Skip to content

Eval statistics

How many test cases for ±10 points at 95% confidence?

93 test cases, if your agent's pass rate could be anywhere. That is the smallest n whose 95% Wilson interval is at most ±10 points wide at a 50% pass rate, the worst case. It suits a quick smoke test that catches large regressions only.

Cases needed by expected pass rate

Expected pass rateWilson (what Kojen reports)Normal approximation
50%9397
70%7881
80%6062
90%3735
95%2519

Margin you get from a given number of cases

Test casesMargin at 50% pass rateMargin at 90% pass rateMeets ±10?
50±13.4 pts±8.5 ptsAt 90% only
100±9.6 pts±6.0 ptsYes
200±6.9 pts±4.2 ptsYes
500±4.4 pts±2.6 ptsYes
1,000±3.1 pts±1.9 ptsYes

Comparing two prompts on 93 cases

Cases where A and B disagreeSmallest difference detected (80% power)
10%9.1 points
20%12.9 points
30%15.8 points

McNemar's test, two-sided at 5% significance (z = 1.960), using Connor's (1987) approximation. Variants of one agent typically disagree on 10–30% of cases; the fewer the disagreements, the smaller the difference you can detect.

Calculator

Pre-filled for ±10 points at 95% confidence.

Cases needed for a margin of error

Test cases needed

93

Worst case (50%)

93

Normal approximation

97

z

1.960

The smallest n whose 95% Wilson interval at a 50% pass rate is at most ±10 points wide.

Interval for a result you already have

Pass rate

80.0%

95% interval

73.9% – 85.0%

Margin

±5.5 pts

Cases

200

Comparing two prompts or models on the same cases

Paired cases needed

296

Difference

10 pts

Disagreement

38.0%

Power at n = 200

63%

McNemar's test, two-sided at 5% significance, via Connor's (1987) approximation. Only the cases where A and B disagree carry information, so variants that mostly agree need far fewer cases.

Assumptions

  • Each test case is an independent pass/fail trial. Near-duplicate cases count as fewer than they look.
  • The margin is half the width of the Wilson score interval, computed by the same function that puts error bars on Kojen's evaluation results.
  • If you sample each case several times, the interval is over cases, not samples: repeated samples of one case are correlated. Kojen uses a bootstrap over cases for that.

Frequently asked questions

How many test cases do I need for a ±10-point margin at 95% confidence?
93 if you don't know the pass rate in advance (the worst case, 50%). At a 90% pass rate you need 37, because rates near 0% or 100% vary less. These use the Wilson score interval, which behaves better than the textbook formula near the extremes.
Is 93 cases enough to tell two prompts apart?
Only for differences of about 12.9 points or more, if the two prompts disagree on 20% of cases (80% power, McNemar's test at 5% significance). Run both on the same cases: the paired test only counts disagreements, so it needs far fewer cases than comparing two independent intervals.
Why not just use the normal approximation, n = z²p(1−p)/m²?
It gives 97 here, close to the Wilson answer at 50%, but it understates uncertainty for small n and high or low pass rates, and its interval can run past 0% or 100%. The Wilson interval is what Kojen reports on every evaluation, so planning with it means the margin you plan is the margin you see.

Other targets

Definitions: Wilson interval, McNemar test, confidence interval.

Sources

  1. [1]Wilson, E. B. (1927), JASA. Probable inference, the law of succession, and statistical inference
  2. [2]Brown, Cai & DasGupta (2001), Statistical Science. Interval Estimation for a Binomial Proportion
  3. [3]McNemar, Q. (1947), Psychometrika. Note on the sampling error of the difference between correlated proportions or percentages
  4. [4]Connor, R. J. (1987), Biometrics. Sample size for testing differences in proportions for the paired-sample design
  5. [5]Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv:2411.00640)

Know when a change is real.

Kojen puts a Wilson interval on every pass rate and runs paired tests when training compares candidates. Free to start.

Start training free