pass@k
pass@k is the probability that at least one of k sampled attempts at a task is correct. It measures whether a model can solve a task given several tries.
Chen et al. introduced the unbiased estimator with the Codex paper: draw n ≥ k samples per task, count the c that are correct, and compute 1 − C(n−c, k)/C(n, k). Averaging that over tasks gives the benchmark's pass@k. Estimating it as 1 − (1 − c/n)^k is biased for small n.
pass@k rises with k, so it flatters agents: a user who runs a task once gets pass@1, not pass@10. For reliability, report pass^k alongside it.
Formula
pass@k = 1 − C(n − c, k) / C(n, k)
Learn more on Kojen
Related terms
- pass^kpass^k is the probability that all k attempts at the same task succeed.
- Agent evalAn agent eval measures an AI agent on multi-step tasks, scoring both the final outcome and how it got there, including tool calls, turns, cost and reliability over repeated runs..
- Confidence intervalA confidence interval is a range computed from a sample that, across repeated samples, would contain the true value a stated share of the time, such as 95%.
Sources
- Chen et al., 2021. Evaluating Large Language Models Trained on Code (arXiv:2107.03374)
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free