Wilson interval
The Wilson score interval is a confidence interval for a proportion, such as a pass rate, that stays within 0–100% and keeps close to its stated coverage even for small samples and rates near 0 or 100%.
Proposed by E. B. Wilson in 1927, it is recommended over the textbook normal (Wald) interval by Brown, Cai and DasGupta, whose study showed the Wald interval's coverage can be badly off even at large n.
Kojen reports a 95% Wilson interval on every pass rate it measures.
Formula
(p̂ + z²/2n ± z·√(p̂(1−p̂)/n + z²/4n²)) / (1 + z²/n)
Learn more on Kojen
Related terms
- Confidence intervalA confidence interval is a range computed from a sample that, across repeated samples, would contain the true value a stated share of the time, such as 95%.
- McNemar testMcNemar's test checks whether two classifiers, prompts or models differ in accuracy when both are run on the same cases, using only the cases where exactly one of them is correct..
- Holdout setA holdout set is a portion of labelled examples kept out of training and tuning, used only to measure how well a model or prompt generalises to data it hasn't seen..
Sources
- Wilson, E. B. (1927), JASA. Probable inference, the law of succession, and statistical inference
- Brown, Cai & DasGupta (2001), Statistical Science. Interval Estimation for a Binomial Proportion
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free