You changed a prompt and accuracy went from 76% to 82% on 50 test cases. Is the new prompt better? On 50 cases, probably not provably: the 95% interval around 82% runs from about 69% to 90%. Most eval dashboards would show the six-point gain as a win anyway.
This guide gives you the arithmetic to answer two questions before you spend money on model calls: how many test cases you need to measure one model to a given precision, and how many you need to tell two versions apart. Every number here can be checked with a calculator.
An eval score is a sample, not a fact
Treat each test case as a draw from the population of requests your model will see. If the model's true pass rate is p and you test n cases, the number that pass follows a binomial distribution, and your measured accuracy will wander around p by roughly √(p(1 − p)/n). That spread is the noise every comparison has to beat. Evan Miller's paper on error bars for evals makes the case for reporting it on every score, the way other experimental sciences do.[1]
Use the Wilson interval, not ± 1.96 × SE
The textbook interval, p̂ ± 1.96√(p̂(1 − p̂)/n), behaves badly exactly where LLM evals live: small n and pass rates near 100%. Brown, Cai and DasGupta showed its coverage can fall well below the nominal 95%, erratically, even for fairly large samples, and recommended the Wilson interval for small n.[2] Wilson's 1927 interval inverts the score test, so it stays inside [0, 1] and doesn't collapse to zero width at 0% or 100%.[3]
p̂ = successes / n, z = 1.96 for 95% centre = (p̂ + z²/2n) / (1 + z²/n) half-width = z × √( p̂(1 − p̂)/n + z²/4n² ) / (1 + z²/n) 27 of 30 correct: normal approximation 90% ± 10.7 → 79.3% to 100.7% (impossible upper bound) Wilson → 74.4% to 96.5%
The edge cases are where the difference shows. Thirty out of thirty is not proof of perfection: the Wilson interval is 88.6% to 100%. Zero failures in n cases bounds the true failure rate at about 3/n with 95% confidence, the “rule of three”[4], so 100 clean cases still allow a 3% failure rate.
How wide is the interval at your sample size?
These are 95% Wilson intervals, in percentage points, at three true pass rates. We calculated them with the formula above.
| Test cases | Score 50% | Score 80% | Score 90% |
|---|---|---|---|
| 20 | 29.9–70.1 | 58.4–91.9 | 69.9–97.2 |
| 30 | 33.2–66.8 | 62.7–90.5 | 74.4–96.5 |
| 50 | 36.6–63.4 | 67.0–88.8 | 78.6–95.7 |
| 100 | 40.4–59.6 | 71.1–86.7 | 82.6–94.5 |
| 200 | 43.1–56.9 | 73.9–85.0 | 85.1–93.4 |
| 400 | 45.1–54.9 | 75.8–83.6 | 86.7–92.6 |
| 1,000 | 46.9–53.1 | 77.4–82.4 | 88.0–91.7 |
Precision improves with the square root of n. Halving the interval costs four times the cases. Scores near 50% are the noisiest; scores near the ceiling are tighter, but the interval becomes lopsided.
Cases needed for a target margin
Turn the formula around to get the sample size for a given margin m:
n ≈ z² × p(1 − p) / m² (m = margin, as a fraction) p = 0.5, m = 0.05: 1.96² × 0.25 / 0.0025 = 384 p = 0.9, m = 0.05: 1.96² × 0.09 / 0.0025 = 138 p = 0.9, m = 0.10: 1.96² × 0.09 / 0.01 = 35
| Expected score | ±10 points | ±5 points | ±3 points |
|---|---|---|---|
| 50% | 97 | 385 | 1,068 |
| 80% | 62 | 246 | 683 |
| 90% | 35 | 139 | 385 |
If you don't know the expected score, plan for 50%: it's the worst case. For a quick check during development, 30–50 cases tell you whether something is badly broken; Anthropic's guide to agent evals suggests starting with 20–50 tasks drawn from real failures rather than waiting for hundreds.[5] For a number you'll put in a release note, aim for a few hundred.
Comparing two prompts: pair the cases and use McNemar
Two separate intervals overlapping doesn't mean there's no difference, and the overlap rule throws away the most useful fact you have: both versions answered the same cases. Most cases are easy or hard for both, so they cancel out. Only the cases where the two disagree tell you which is better.
McNemar's test uses exactly those discordant pairs.[6] Under the null hypothesis that A and B are equally accurate, a disagreement is equally likely to favour either, so the count of “A right, B wrong” cases is binomial with probability ½. Dietterich's comparison of tests for classifiers found McNemar's test had low Type I error, which is why it became the standard for comparing two models on one test set.[7]
200 shared test cases
prompt B right prompt B wrong
prompt A right 142 8
prompt A wrong 20 30
A: 150/200 = 75% B: 162/200 = 81%
Only the 28 discordant cases carry information:
χ² = (|8 − 20| − 1)² / (8 + 20) = 121 / 28 = 4.32 → p = 0.038
exact binomial (28 trials, 8 or fewer, two-sided) → p = 0.036
Unpaired two-proportion test on the same totals → p = 0.15In this made-up but typical example, the six-point gain is significant when paired (p ≈ 0.04) and not significant when the two runs are treated as independent samples (p ≈ 0.15). Use the exact binomial version when there are fewer than about 25 discordant pairs.
Cases needed to detect a difference
For paired designs, the sample size depends on how often the two versions disagree. Connor's formula gives it:[8]
Paired design (Connor, 1987), α = 0.05 two-sided, 80% power d = expected difference in accuracy ψ = expected share of cases where the two disagree n ≈ ( 1.96 √ψ + 0.84 √(ψ − d²) )² / d² d = 0.05, ψ = 0.15 → n ≈ 469 cases d = 0.05, ψ = 0.10 → n ≈ 312 cases Unpaired, 75% vs 80% → n ≈ 1,094 cases per arm
Detecting a five-point improvement reliably needs a few hundred shared cases. If your test set has 50, you can reliably detect only large gains, around 20 points if the versions disagree on a quarter of cases, and should say so. The cheap way to stretch a small set is not to run it more times at temperature 0, which just repeats the same answers, but to add cases.
Practical rules
- Report an interval with every score. “82% (69–90%, n = 50)” tells the reader what “82%” alone hides.
- Compare versions on the same cases and test the difference with McNemar (pass/fail) or a paired bootstrap (partial-credit scores).
- Decide the comparison before you look. Testing ten prompts against a baseline and keeping the best one inflates false positives; confirm the winner on a held-out split.
- Count errors as failures and report the error rate. Dropping timeouts flatters the model and shrinks n.
- Multiple samples per case measure consistency, not precision. They tighten nothing about how well the cases represent your traffic; see pass@k vs pass^k for what they do measure.
For how to build the cases themselves, see Your agent is only as good as its test set, and the Test sets that tell the truth lesson.



