Confidence interval
A confidence interval is a range computed from a sample that, across repeated samples, would contain the true value a stated share of the time, such as 95%. It is the error bar on an eval score.
An agent that passes 80 of 100 cases has a 95% Wilson interval of roughly 71–87%, so a rival at 84% isn't clearly better. Miller argues every eval result should carry such an interval.
Intervals narrow with the square root of the number of cases: halving the margin takes four times the cases.
Learn more on Kojen
Related terms
- Wilson intervalThe Wilson score interval is a confidence interval for a proportion, such as a pass rate, that stays within 0–100% and keeps close to its stated coverage even for small samples and rates near 0 or 100%..
- McNemar testMcNemar's test checks whether two classifiers, prompts or models differ in accuracy when both are run on the same cases, using only the cases where exactly one of them is correct..
- Holdout setA holdout set is a portion of labelled examples kept out of training and tuning, used only to measure how well a model or prompt generalises to data it hasn't seen..
Sources
- Miller, 2024. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv:2411.00640)
- Brown, Cai & DasGupta (2001), Statistical Science. Interval Estimation for a Binomial Proportion
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free