Skip to content

Glossary · Evaluation

Confidence interval

A confidence interval is a range computed from a sample that, across repeated samples, would contain the true value a stated share of the time, such as 95%. It is the error bar on an eval score.

Reviewed 15 November 2026

An agent that passes 80 of 100 cases has a 95% Wilson interval of roughly 71–87%, so a rival at 84% isn't clearly better. Miller argues every eval result should carry such an interval.

Intervals narrow with the square root of the number of cases: halving the margin takes four times the cases.

Learn more on Kojen

Sources

  1. [1]Miller, 2024. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv:2411.00640)
  2. [2]Brown, Cai & DasGupta (2001), Statistical Science. Interval Estimation for a Binomial Proportion

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free