Skip to content

Evaluation

How many test cases do you need to evaluate an LLM?

Wilson confidence intervals for eval scores, a table of interval widths from 20 to 1,000 cases, sample sizes for ±5 and ±10 points, and McNemar's test for telling two prompts apart.

By Will Schulz9 min read
A glass funnel pouring tiny glass beads onto a dark metal tray, where they form a bell-shaped pile with a red band across its middle

You changed a prompt and accuracy went from 76% to 82% on 50 test cases. Is the new prompt better? On 50 cases, probably not provably: the 95% interval around 82% runs from about 69% to 90%. Most eval dashboards would show the six-point gain as a win anyway.

This guide gives you the arithmetic to answer two questions before you spend money on model calls: how many test cases you need to measure one model to a given precision, and how many you need to tell two versions apart. Every number here can be checked with a calculator.

An eval score is a sample, not a fact

Treat each test case as a draw from the population of requests your model will see. If the model's true pass rate is p and you test n cases, the number that pass follows a binomial distribution, and your measured accuracy will wander around p by roughly √(p(1 − p)/n). That spread is the noise every comparison has to beat. Evan Miller's paper on error bars for evals makes the case for reporting it on every score, the way other experimental sciences do.[1]

Use the Wilson interval, not ± 1.96 × SE

The textbook interval, p̂ ± 1.96√(p̂(1 − p̂)/n), behaves badly exactly where LLM evals live: small n and pass rates near 100%. Brown, Cai and DasGupta showed its coverage can fall well below the nominal 95%, erratically, even for fairly large samples, and recommended the Wilson interval for small n.[2] Wilson's 1927 interval inverts the score test, so it stays inside [0, 1] and doesn't collapse to zero width at 0% or 100%.[3]

p̂ = successes / n,  z = 1.96 for 95%

centre     = (p̂ + z²/2n) / (1 + z²/n)
half-width = z × √( p̂(1 − p̂)/n + z²/4n² ) / (1 + z²/n)

27 of 30 correct:
normal approximation  90% ± 10.7  →  79.3% to 100.7%   (impossible upper bound)
Wilson                             →  74.4% to 96.5%

The edge cases are where the difference shows. Thirty out of thirty is not proof of perfection: the Wilson interval is 88.6% to 100%. Zero failures in n cases bounds the true failure rate at about 3/n with 95% confidence, the “rule of three”[4], so 100 clean cases still allow a 3% failure rate.

How wide is the interval at your sample size?

These are 95% Wilson intervals, in percentage points, at three true pass rates. We calculated them with the formula above.

Test casesScore 50%Score 80%Score 90%
2029.9–70.158.4–91.969.9–97.2
3033.2–66.862.7–90.574.4–96.5
5036.6–63.467.0–88.878.6–95.7
10040.4–59.671.1–86.782.6–94.5
20043.1–56.973.9–85.085.1–93.4
40045.1–54.975.8–83.686.7–92.6
1,00046.9–53.177.4–82.488.0–91.7

Precision improves with the square root of n. Halving the interval costs four times the cases. Scores near 50% are the noisiest; scores near the ceiling are tighter, but the interval becomes lopsided.

Cases needed for a target margin

Turn the formula around to get the sample size for a given margin m:

n ≈ z² × p(1 − p) / m²      (m = margin, as a fraction)

p = 0.5, m = 0.05:  1.96² × 0.25 / 0.0025 = 384
p = 0.9, m = 0.05:  1.96² × 0.09 / 0.0025 = 138
p = 0.9, m = 0.10:  1.96² × 0.09 / 0.01   =  35
Expected score±10 points±5 points±3 points
50%973851,068
80%62246683
90%35139385

If you don't know the expected score, plan for 50%: it's the worst case. For a quick check during development, 30–50 cases tell you whether something is badly broken; Anthropic's guide to agent evals suggests starting with 20–50 tasks drawn from real failures rather than waiting for hundreds.[5] For a number you'll put in a release note, aim for a few hundred.

Comparing two prompts: pair the cases and use McNemar

Two separate intervals overlapping doesn't mean there's no difference, and the overlap rule throws away the most useful fact you have: both versions answered the same cases. Most cases are easy or hard for both, so they cancel out. Only the cases where the two disagree tell you which is better.

McNemar's test uses exactly those discordant pairs.[6] Under the null hypothesis that A and B are equally accurate, a disagreement is equally likely to favour either, so the count of “A right, B wrong” cases is binomial with probability ½. Dietterich's comparison of tests for classifiers found McNemar's test had low Type I error, which is why it became the standard for comparing two models on one test set.[7]

200 shared test cases
                     prompt B right   prompt B wrong
prompt A right            142                8
prompt A wrong             20               30

A: 150/200 = 75%     B: 162/200 = 81%
Only the 28 discordant cases carry information:
χ² = (|8 − 20| − 1)² / (8 + 20) = 121 / 28 = 4.32  →  p = 0.038
exact binomial (28 trials, 8 or fewer, two-sided)      →  p = 0.036

Unpaired two-proportion test on the same totals      →  p = 0.15

In this made-up but typical example, the six-point gain is significant when paired (p ≈ 0.04) and not significant when the two runs are treated as independent samples (p ≈ 0.15). Use the exact binomial version when there are fewer than about 25 discordant pairs.

Cases needed to detect a difference

For paired designs, the sample size depends on how often the two versions disagree. Connor's formula gives it:[8]

Paired design (Connor, 1987), α = 0.05 two-sided, 80% power
d  = expected difference in accuracy
ψ  = expected share of cases where the two disagree

n ≈ ( 1.96 √ψ + 0.84 √(ψ − d²) )² / d²

d = 0.05, ψ = 0.15  →  n ≈ 469 cases
d = 0.05, ψ = 0.10  →  n ≈ 312 cases
Unpaired, 75% vs 80% →  n ≈ 1,094 cases per arm

Detecting a five-point improvement reliably needs a few hundred shared cases. If your test set has 50, you can reliably detect only large gains, around 20 points if the versions disagree on a quarter of cases, and should say so. The cheap way to stretch a small set is not to run it more times at temperature 0, which just repeats the same answers, but to add cases.

Practical rules

  • Report an interval with every score. “82% (69–90%, n = 50)” tells the reader what “82%” alone hides.
  • Compare versions on the same cases and test the difference with McNemar (pass/fail) or a paired bootstrap (partial-credit scores).
  • Decide the comparison before you look. Testing ten prompts against a baseline and keeping the best one inflates false positives; confirm the winner on a held-out split.
  • Count errors as failures and report the error rate. Dropping timeouts flatters the model and shrinks n.
  • Multiple samples per case measure consistency, not precision. They tighten nothing about how well the cases represent your traffic; see pass@k vs pass^k for what they do measure.

For how to build the cases themselves, see Your agent is only as good as its test set, and the Test sets that tell the truth lesson.

Frequently asked questions

How many test cases do I need to evaluate an LLM?
For a ±10-point 95% interval you need about 97 cases at a 50% score, 62 at 80% and 35 at 90%. For ±5 points you need about 385, 246 and 139. Use n ≈ 1.96² × p(1 − p) / margin².
Which confidence interval should I use for LLM accuracy?
The Wilson score interval. The normal-approximation interval can extend past 100%, collapses to zero width at 0% or 100%, and under-covers at small sample sizes; Wilson doesn't.
How do I know if one prompt is really better than another?
Run both on the same test cases and apply McNemar's test to the cases where they disagree. With 8 cases only A got right and 20 only B got right, p ≈ 0.04. Treating the runs as independent wastes that pairing.
How many cases do I need to detect a 5-point improvement?
With paired cases, 80% power and α = 0.05, roughly 300–470 cases if the two versions disagree on 10–15% of them. Unpaired, about 1,100 per version for 75% versus 80%.
If my model passes all 30 test cases, is it perfect?
No. The 95% Wilson interval for 30 out of 30 is 88.6% to 100%. By the rule of three, zero failures in n cases still allows a failure rate up to about 3/n.

Go deeper in the Academy

Free lessons with quizzes, citations and a hands-on exercise.

Sources

  1. [1]Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. Miller, arXiv (2024).
  2. [2]Interval Estimation for a Binomial Proportion. Brown, Cai and DasGupta, Statistical Science 16(2) (2001).
  3. [3]Probable Inference, the Law of Succession, and Statistical Inference. Wilson, Journal of the American Statistical Association 22 (1927).
  4. [4]If nothing goes wrong, is everything all right? Interpreting zero numerators. Hanley and Lippman-Hand, JAMA 249(13) (1983).
  5. [5]Demystifying evals for AI agents. Anthropic Engineering (9 January 2026).
  6. [6]Note on the sampling error of the difference between correlated proportions or percentages. McNemar, Psychometrika 12 (1947).
  7. [7]Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Dietterich, Neural Computation 10(7) (1998).
  8. [8]Sample size for testing differences in proportions for the paired-sample design. Connor, Biometrics 43(1) (1987).

Put it into practice.

Kojen gives you the Forge, LLM-judged evaluations, red-team simulations and live endpoints in one place. Free to start, no card needed, then pay as you go.

Start training free

More reading

All articles →