How many test cases for ±2 points at 95% confidence?
2,410 test cases, if your agent's pass rate could be anywhere. That is the smallest n whose 95% Wilson interval is at most ±2 points wide at a 50% pass rate, the worst case. It suits leaderboard-grade precision, where 2-point gaps decide rankings.
Cases needed by expected pass rate
| Expected pass rate | Wilson (what Kojen reports) | Normal approximation |
|---|---|---|
| 50% | 2,410 | 2,401 |
| 70% | 2,020 | 2,017 |
| 80% | 1,536 | 1,537 |
| 90% | 869 | 865 |
| 95% | 469 | 457 |
Margin you get from a given number of cases
| Test cases | Margin at 50% pass rate | Margin at 90% pass rate | Meets ±2? |
|---|---|---|---|
| 50 | ±13.4 pts | ±8.5 pts | No |
| 100 | ±9.6 pts | ±6.0 pts | No |
| 200 | ±6.9 pts | ±4.2 pts | No |
| 500 | ±4.4 pts | ±2.6 pts | No |
| 1,000 | ±3.1 pts | ±1.9 pts | At 90% only |
Comparing two prompts on 2,410 cases
| Cases where A and B disagree | Smallest difference detected (80% power) |
|---|---|
| 10% | 1.9 points |
| 20% | 2.6 points |
| 30% | 3.2 points |
McNemar's test, two-sided at 5% significance (z = 1.960), using Connor's (1987) approximation. Variants of one agent typically disagree on 10–30% of cases; the fewer the disagreements, the smaller the difference you can detect.
Calculator
Pre-filled for ±2 points at 95% confidence.
Cases needed for a margin of error
2,410
2,410
2,401
1.960
The smallest n whose 95% Wilson interval at a 50% pass rate is at most ±2 points wide.
Interval for a result you already have
80.0%
73.9% – 85.0%
±5.5 pts
200
Comparing two prompts or models on the same cases
296
10 pts
38.0%
63%
McNemar's test, two-sided at 5% significance, via Connor's (1987) approximation. Only the cases where A and B disagree carry information, so variants that mostly agree need far fewer cases.
Assumptions
- Each test case is an independent pass/fail trial. Near-duplicate cases count as fewer than they look.
- The margin is half the width of the Wilson score interval, computed by the same function that puts error bars on Kojen's evaluation results.
- If you sample each case several times, the interval is over cases, not samples: repeated samples of one case are correlated. Kojen uses a bootstrap over cases for that.
Frequently asked questions
- How many test cases do I need for a ±2-point margin at 95% confidence?
- 2,410 if you don't know the pass rate in advance (the worst case, 50%). At a 90% pass rate you need 869, because rates near 0% or 100% vary less. These use the Wilson score interval, which behaves better than the textbook formula near the extremes.
- Is 2,410 cases enough to tell two prompts apart?
- Only for differences of about 2.6 points or more, if the two prompts disagree on 20% of cases (80% power, McNemar's test at 5% significance). Run both on the same cases: the paired test only counts disagreements, so it needs far fewer cases than comparing two independent intervals.
- Why not just use the normal approximation, n = z²p(1−p)/m²?
- It gives 2,401 here, close to the Wilson answer at 50%, but it understates uncertainty for small n and high or low pass rates, and its interval can run past 0% or 100%. The Wilson interval is what Kojen reports on every evaluation, so planning with it means the margin you plan is the margin you see.
Other targets
- ±5 points at 95%382 cases
- ±10 points at 95%93 cases
- ±3 points at 95%1,067 cases
- ±5 points at 90%269 cases
- ±5 points at 99%659 cases
- pass@k calculatorReliability over repeated runs
Definitions: Wilson interval, McNemar test, confidence interval.
Sources
- Wilson, E. B. (1927), JASA. Probable inference, the law of succession, and statistical inference
- Brown, Cai & DasGupta (2001), Statistical Science. Interval Estimation for a Binomial Proportion
- McNemar, Q. (1947), Psychometrika. Note on the sampling error of the difference between correlated proportions or percentages
- Connor, R. J. (1987), Biometrics. Sample size for testing differences in proportions for the paired-sample design
- Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv:2411.00640)
Know when a change is real.
Kojen puts a Wilson interval on every pass rate and runs paired tests when training compares candidates. Free to start.
Start training free