Eval sample size calculator
A ±5-point margin at 95% confidence takes about 382 test cases; ±10 points takes about 93. Work out what your eval needs, how wide the interval on a result you already have really is, and how many paired cases it takes to show one prompt beats another.
Cases needed for a margin of error
382
382
385
1.960
The smallest n whose 95% Wilson interval at a 50% pass rate is at most ±5 points wide.
Interval for a result you already have
80.0%
73.9% – 85.0%
±5.5 pts
200
Comparing two prompts or models on the same cases
296
10 pts
38.0%
63%
McNemar's test, two-sided at 5% significance, via Connor's (1987) approximation. Only the cases where A and B disagree carry information, so variants that mostly agree need far fewer cases.
Common targets
| Target | Cases (worst case) | Use it for |
|---|---|---|
| ±5 points at 95% | 382 | the default for a release gate or a published benchmark number |
| ±10 points at 95% | 93 | a quick smoke test that catches large regressions only |
| ±3 points at 95% | 1,067 | tracking small week-to-week changes in a production agent |
| ±2 points at 95% | 2,410 | leaderboard-grade precision, where 2-point gaps decide rankings |
| ±5 points at 90% | 269 | internal iteration, where a slightly higher false-alarm rate is acceptable |
| ±5 points at 99% | 659 | safety or compliance checks, where a wrong call is expensive |
How it works
The margin is half the width of the Wilson score interval, the same interval Kojen reports on every evaluation, computed by the same code. The sample size is the smallest n whose interval meets your margin. The paired comparison uses Connor's (1987) approximation for McNemar's test: with δ the difference in pass rates and ψ the share of cases where the two variants disagree, n = (zα/2√ψ + zβ√(ψ − δ²))² / δ².
For pass@k and pass^k, use the pass@k calculator; for building the test set itself, read Your agent is only as good as its test set and the Academy lesson Test sets that tell the truth.
Frequently asked questions
- How many test cases does an LLM evaluation need?
- For a ±5-point margin at 95% confidence, about 382 cases if the pass rate could be anywhere; for ±10 points, about 93. High or low pass rates need fewer. Comparing two variants on the same cases needs far fewer than two independent margins suggest, because only the disagreements count.
- What is the Wilson interval and why use it?
- The Wilson score interval is a confidence interval for a pass rate that stays inside 0–100% and keeps close to its stated coverage at small n and extreme rates, where the textbook normal interval fails. Brown, Cai and DasGupta (2001) recommend it over the normal interval.
- How do I tell whether prompt B really beats prompt A?
- Run both on the same test cases and apply McNemar's test to the cases where exactly one passes. If B wins most of those disagreements, the improvement is unlikely to be noise. The calculator gives the number of paired cases you need for a given power.
Glossary
Sources
- Wilson, E. B. (1927), JASA. Probable inference, the law of succession, and statistical inference
- Brown, Cai & DasGupta (2001), Statistical Science. Interval Estimation for a Binomial Proportion
- McNemar, Q. (1947), Psychometrika. Note on the sampling error of the difference between correlated proportions or percentages
- Connor, R. J. (1987), Biometrics. Sample size for testing differences in proportions for the paired-sample design
- Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv:2411.00640)
Every Kojen evaluation comes with error bars.
Kojen reports Wilson intervals on every pass rate, McNemar tests when training compares candidates, and pass@k / pass^k for repeated runs, so you know when a change is real.
Start training free