Skip to content

Free tool

Eval sample size calculator

A ±5-point margin at 95% confidence takes about 382 test cases; ±10 points takes about 93. Work out what your eval needs, how wide the interval on a result you already have really is, and how many paired cases it takes to show one prompt beats another.

Cases needed for a margin of error

Test cases needed

382

Worst case (50%)

382

Normal approximation

385

z

1.960

The smallest n whose 95% Wilson interval at a 50% pass rate is at most ±5 points wide.

Interval for a result you already have

Pass rate

80.0%

95% interval

73.9% – 85.0%

Margin

±5.5 pts

Cases

200

Comparing two prompts or models on the same cases

Paired cases needed

296

Difference

10 pts

Disagreement

38.0%

Power at n = 200

63%

McNemar's test, two-sided at 5% significance, via Connor's (1987) approximation. Only the cases where A and B disagree carry information, so variants that mostly agree need far fewer cases.

Common targets

TargetCases (worst case)Use it for
±5 points at 95%382the default for a release gate or a published benchmark number
±10 points at 95%93a quick smoke test that catches large regressions only
±3 points at 95%1,067tracking small week-to-week changes in a production agent
±2 points at 95%2,410leaderboard-grade precision, where 2-point gaps decide rankings
±5 points at 90%269internal iteration, where a slightly higher false-alarm rate is acceptable
±5 points at 99%659safety or compliance checks, where a wrong call is expensive

How it works

The margin is half the width of the Wilson score interval, the same interval Kojen reports on every evaluation, computed by the same code. The sample size is the smallest n whose interval meets your margin. The paired comparison uses Connor's (1987) approximation for McNemar's test: with δ the difference in pass rates and ψ the share of cases where the two variants disagree, n = (zα/2√ψ + zβ√(ψ − δ²))² / δ².

For pass@k and pass^k, use the pass@k calculator; for building the test set itself, read Your agent is only as good as its test set and the Academy lesson Test sets that tell the truth.

Frequently asked questions

How many test cases does an LLM evaluation need?
For a ±5-point margin at 95% confidence, about 382 cases if the pass rate could be anywhere; for ±10 points, about 93. High or low pass rates need fewer. Comparing two variants on the same cases needs far fewer than two independent margins suggest, because only the disagreements count.
What is the Wilson interval and why use it?
The Wilson score interval is a confidence interval for a pass rate that stays inside 0–100% and keeps close to its stated coverage at small n and extreme rates, where the textbook normal interval fails. Brown, Cai and DasGupta (2001) recommend it over the normal interval.
How do I tell whether prompt B really beats prompt A?
Run both on the same test cases and apply McNemar's test to the cases where exactly one passes. If B wins most of those disagreements, the improvement is unlikely to be noise. The calculator gives the number of paired cases you need for a given power.

Sources

  1. [1]Wilson, E. B. (1927), JASA. Probable inference, the law of succession, and statistical inference
  2. [2]Brown, Cai & DasGupta (2001), Statistical Science. Interval Estimation for a Binomial Proportion
  3. [3]McNemar, Q. (1947), Psychometrika. Note on the sampling error of the difference between correlated proportions or percentages
  4. [4]Connor, R. J. (1987), Biometrics. Sample size for testing differences in proportions for the paired-sample design
  5. [5]Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv:2411.00640)

Every Kojen evaluation comes with error bars.

Kojen reports Wilson intervals on every pass rate, McNemar tests when training compares candidates, and pass@k / pass^k for repeated runs, so you know when a change is real.

Start training free