How many test cases for ±5 points at 95% confidence?
382 test cases, if your agent's pass rate could be anywhere. That is the smallest n whose 95% Wilson interval is at most ±5 points wide at a 50% pass rate, the worst case. It suits the default for a release gate or a published benchmark number.
Cases needed by expected pass rate
| Expected pass rate | Wilson (what Kojen reports) | Normal approximation |
|---|---|---|
| 50% | 382 | 385 |
| 70% | 320 | 323 |
| 80% | 245 | 246 |
| 90% | 141 | 139 |
| 95% | 83 | 73 |
Margin you get from a given number of cases
| Test cases | Margin at 50% pass rate | Margin at 90% pass rate | Meets ±5? |
|---|---|---|---|
| 50 | ±13.4 pts | ±8.5 pts | No |
| 100 | ±9.6 pts | ±6.0 pts | No |
| 200 | ±6.9 pts | ±4.2 pts | At 90% only |
| 500 | ±4.4 pts | ±2.6 pts | Yes |
| 1,000 | ±3.1 pts | ±1.9 pts | Yes |
Comparing two prompts on 382 cases
| Cases where A and B disagree | Smallest difference detected (80% power) |
|---|---|
| 10% | 4.6 points |
| 20% | 6.4 points |
| 30% | 7.9 points |
McNemar's test, two-sided at 5% significance (z = 1.960), using Connor's (1987) approximation. Variants of one agent typically disagree on 10–30% of cases; the fewer the disagreements, the smaller the difference you can detect.
Calculator
Pre-filled for ±5 points at 95% confidence.
Cases needed for a margin of error
382
382
385
1.960
The smallest n whose 95% Wilson interval at a 50% pass rate is at most ±5 points wide.
Interval for a result you already have
80.0%
73.9% – 85.0%
±5.5 pts
200
Comparing two prompts or models on the same cases
296
10 pts
38.0%
63%
McNemar's test, two-sided at 5% significance, via Connor's (1987) approximation. Only the cases where A and B disagree carry information, so variants that mostly agree need far fewer cases.
Assumptions
- Each test case is an independent pass/fail trial. Near-duplicate cases count as fewer than they look.
- The margin is half the width of the Wilson score interval, computed by the same function that puts error bars on Kojen's evaluation results.
- If you sample each case several times, the interval is over cases, not samples: repeated samples of one case are correlated. Kojen uses a bootstrap over cases for that.
Frequently asked questions
- How many test cases do I need for a ±5-point margin at 95% confidence?
- 382 if you don't know the pass rate in advance (the worst case, 50%). At a 90% pass rate you need 141, because rates near 0% or 100% vary less. These use the Wilson score interval, which behaves better than the textbook formula near the extremes.
- Is 382 cases enough to tell two prompts apart?
- Only for differences of about 6.4 points or more, if the two prompts disagree on 20% of cases (80% power, McNemar's test at 5% significance). Run both on the same cases: the paired test only counts disagreements, so it needs far fewer cases than comparing two independent intervals.
- Why not just use the normal approximation, n = z²p(1−p)/m²?
- It gives 385 here, close to the Wilson answer at 50%, but it understates uncertainty for small n and high or low pass rates, and its interval can run past 0% or 100%. The Wilson interval is what Kojen reports on every evaluation, so planning with it means the margin you plan is the margin you see.
Other targets
- ±10 points at 95%93 cases
- ±3 points at 95%1,067 cases
- ±2 points at 95%2,410 cases
- ±5 points at 90%269 cases
- ±5 points at 99%659 cases
- pass@k calculatorReliability over repeated runs
Definitions: Wilson interval, McNemar test, confidence interval.
Sources
- Wilson, E. B. (1927), JASA. Probable inference, the law of succession, and statistical inference
- Brown, Cai & DasGupta (2001), Statistical Science. Interval Estimation for a Binomial Proportion
- McNemar, Q. (1947), Psychometrika. Note on the sampling error of the difference between correlated proportions or percentages
- Connor, R. J. (1987), Biometrics. Sample size for testing differences in proportions for the paired-sample design
- Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv:2411.00640)
Know when a change is real.
Kojen puts a Wilson interval on every pass rate and runs paired tests when training compares candidates. Free to start.
Start training free