Skip to content

Eval statistics

How many test cases for ±5 points at 99% confidence?

659 test cases, if your agent's pass rate could be anywhere. That is the smallest n whose 99% Wilson interval is at most ±5 points wide at a 50% pass rate, the worst case. It suits safety or compliance checks, where a wrong call is expensive.

Cases needed by expected pass rate

Expected pass rateWilson (what Kojen reports)Normal approximation
50%659664
70%553558
80%422425
90%244239
95%144127

Margin you get from a given number of cases

Test casesMargin at 50% pass rateMargin at 90% pass rateMeets ±5?
50±17.1 pts±11.3 ptsNo
100±12.5 pts±7.9 ptsNo
200±9.0 pts±5.5 ptsNo
500±5.7 pts±3.5 ptsAt 90% only
1,000±4.1 pts±2.5 ptsYes

Comparing two prompts on 659 cases

Cases where A and B disagreeSmallest difference detected (80% power)
10%4.3 points
20%6.0 points
30%7.3 points

McNemar's test, two-sided at 1% significance (z = 2.576), using Connor's (1987) approximation. Variants of one agent typically disagree on 10–30% of cases; the fewer the disagreements, the smaller the difference you can detect.

Calculator

Pre-filled for ±5 points at 99% confidence.

Cases needed for a margin of error

Test cases needed

659

Worst case (50%)

659

Normal approximation

664

z

2.576

The smallest n whose 99% Wilson interval at a 50% pass rate is at most ±5 points wide.

Interval for a result you already have

Pass rate

80.0%

99% interval

71.8% – 86.3%

Margin

±7.2 pts

Cases

200

Comparing two prompts or models on the same cases

Paired cases needed

296

Difference

10 pts

Disagreement

38.0%

Power at n = 200

63%

McNemar's test, two-sided at 5% significance, via Connor's (1987) approximation. Only the cases where A and B disagree carry information, so variants that mostly agree need far fewer cases.

Assumptions

  • Each test case is an independent pass/fail trial. Near-duplicate cases count as fewer than they look.
  • The margin is half the width of the Wilson score interval, computed by the same function that puts error bars on Kojen's evaluation results.
  • If you sample each case several times, the interval is over cases, not samples: repeated samples of one case are correlated. Kojen uses a bootstrap over cases for that.

Frequently asked questions

How many test cases do I need for a ±5-point margin at 99% confidence?
659 if you don't know the pass rate in advance (the worst case, 50%). At a 90% pass rate you need 244, because rates near 0% or 100% vary less. These use the Wilson score interval, which behaves better than the textbook formula near the extremes.
Is 659 cases enough to tell two prompts apart?
Only for differences of about 6.0 points or more, if the two prompts disagree on 20% of cases (80% power, McNemar's test at 1% significance). Run both on the same cases: the paired test only counts disagreements, so it needs far fewer cases than comparing two independent intervals.
Why not just use the normal approximation, n = z²p(1−p)/m²?
It gives 664 here, close to the Wilson answer at 50%, but it understates uncertainty for small n and high or low pass rates, and its interval can run past 0% or 100%. The Wilson interval is what Kojen reports on every evaluation, so planning with it means the margin you plan is the margin you see.

Other targets

Definitions: Wilson interval, McNemar test, confidence interval.

Sources

  1. [1]Wilson, E. B. (1927), JASA. Probable inference, the law of succession, and statistical inference
  2. [2]Brown, Cai & DasGupta (2001), Statistical Science. Interval Estimation for a Binomial Proportion
  3. [3]McNemar, Q. (1947), Psychometrika. Note on the sampling error of the difference between correlated proportions or percentages
  4. [4]Connor, R. J. (1987), Biometrics. Sample size for testing differences in proportions for the paired-sample design
  5. [5]Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv:2411.00640)

Know when a change is real.

Kojen puts a Wilson interval on every pass rate and runs paired tests when training compares candidates. Free to start.

Start training free