Holdout set
A holdout set is a portion of labelled examples kept out of training and tuning, used only to measure how well a model or prompt generalises to data it hasn't seen.
If you tune prompts or hyperparameters against the same cases you report on, the score overstates real performance. Keep a separate validation split for tuning and touch the final test split rarely.
Size the holdout set for the precision you need: a ±5-point margin at 95% confidence takes about 382 independent cases in the worst case (Wilson interval).
Learn more on Kojen
- Academy: Test sets that tell the truth
- Blog: Your agent is only as good as its test set
- Tool: eval sample size calculator
Related terms
- Confidence intervalA confidence interval is a range computed from a sample that, across repeated samples, would contain the true value a stated share of the time, such as 95%.
- Wilson intervalThe Wilson score interval is a confidence interval for a proportion, such as a pass rate, that stays within 0–100% and keeps close to its stated coverage even for small samples and rates near 0 or 100%..
- Agent evalAn agent eval measures an AI agent on multi-step tasks, scoring both the final outcome and how it got there, including tool calls, turns, cost and reliability over repeated runs..
Sources
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free