How many test cases for ±5 points at 99% confidence?
659 test cases, if your agent's pass rate could be anywhere. That is the smallest n whose 99% Wilson interval is at most ±5 points wide at a 50% pass rate, the worst case. It suits safety or compliance checks, where a wrong call is expensive.
Cases needed by expected pass rate
| Expected pass rate | Wilson (what Kojen reports) | Normal approximation |
|---|---|---|
| 50% | 659 | 664 |
| 70% | 553 | 558 |
| 80% | 422 | 425 |
| 90% | 244 | 239 |
| 95% | 144 | 127 |
Margin you get from a given number of cases
| Test cases | Margin at 50% pass rate | Margin at 90% pass rate | Meets ±5? |
|---|---|---|---|
| 50 | ±17.1 pts | ±11.3 pts | No |
| 100 | ±12.5 pts | ±7.9 pts | No |
| 200 | ±9.0 pts | ±5.5 pts | No |
| 500 | ±5.7 pts | ±3.5 pts | At 90% only |
| 1,000 | ±4.1 pts | ±2.5 pts | Yes |
Comparing two prompts on 659 cases
| Cases where A and B disagree | Smallest difference detected (80% power) |
|---|---|
| 10% | 4.3 points |
| 20% | 6.0 points |
| 30% | 7.3 points |
McNemar's test, two-sided at 1% significance (z = 2.576), using Connor's (1987) approximation. Variants of one agent typically disagree on 10–30% of cases; the fewer the disagreements, the smaller the difference you can detect.
Calculator
Pre-filled for ±5 points at 99% confidence.
Cases needed for a margin of error
659
659
664
2.576
The smallest n whose 99% Wilson interval at a 50% pass rate is at most ±5 points wide.
Interval for a result you already have
80.0%
71.8% – 86.3%
±7.2 pts
200
Comparing two prompts or models on the same cases
296
10 pts
38.0%
63%
McNemar's test, two-sided at 5% significance, via Connor's (1987) approximation. Only the cases where A and B disagree carry information, so variants that mostly agree need far fewer cases.
Assumptions
- Each test case is an independent pass/fail trial. Near-duplicate cases count as fewer than they look.
- The margin is half the width of the Wilson score interval, computed by the same function that puts error bars on Kojen's evaluation results.
- If you sample each case several times, the interval is over cases, not samples: repeated samples of one case are correlated. Kojen uses a bootstrap over cases for that.
Frequently asked questions
- How many test cases do I need for a ±5-point margin at 99% confidence?
- 659 if you don't know the pass rate in advance (the worst case, 50%). At a 90% pass rate you need 244, because rates near 0% or 100% vary less. These use the Wilson score interval, which behaves better than the textbook formula near the extremes.
- Is 659 cases enough to tell two prompts apart?
- Only for differences of about 6.0 points or more, if the two prompts disagree on 20% of cases (80% power, McNemar's test at 1% significance). Run both on the same cases: the paired test only counts disagreements, so it needs far fewer cases than comparing two independent intervals.
- Why not just use the normal approximation, n = z²p(1−p)/m²?
- It gives 664 here, close to the Wilson answer at 50%, but it understates uncertainty for small n and high or low pass rates, and its interval can run past 0% or 100%. The Wilson interval is what Kojen reports on every evaluation, so planning with it means the margin you plan is the margin you see.
Other targets
- ±5 points at 95%382 cases
- ±10 points at 95%93 cases
- ±3 points at 95%1,067 cases
- ±2 points at 95%2,410 cases
- ±5 points at 90%269 cases
- pass@k calculatorReliability over repeated runs
Definitions: Wilson interval, McNemar test, confidence interval.
Sources
- Wilson, E. B. (1927), JASA. Probable inference, the law of succession, and statistical inference
- Brown, Cai & DasGupta (2001), Statistical Science. Interval Estimation for a Binomial Proportion
- McNemar, Q. (1947), Psychometrika. Note on the sampling error of the difference between correlated proportions or percentages
- Connor, R. J. (1987), Biometrics. Sample size for testing differences in proportions for the paired-sample design
- Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv:2411.00640)
Know when a change is real.
Kojen puts a Wilson interval on every pass rate and runs paired tests when training compares candidates. Free to start.
Start training free