Skip to content

Glossary · Evaluation

Holdout set

A holdout set is a portion of labelled examples kept out of training and tuning, used only to measure how well a model or prompt generalises to data it hasn't seen.

Also called held-out set, test set · Reviewed 15 November 2026

If you tune prompts or hyperparameters against the same cases you report on, the score overstates real performance. Keep a separate validation split for tuning and touch the final test split rarely.

Size the holdout set for the precision you need: a ±5-point margin at 95% confidence takes about 382 independent cases in the worst case (Wilson interval).

Learn more on Kojen

Sources

  1. [1]Miller, 2024. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv:2411.00640)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free