Skip to content

Glossary · Evaluation

pass@k

pass@k is the probability that at least one of k sampled attempts at a task is correct. It measures whether a model can solve a task given several tries.

Also called pass at k · Reviewed 15 November 2026

Chen et al. introduced the unbiased estimator with the Codex paper: draw n ≥ k samples per task, count the c that are correct, and compute 1 − C(n−c, k)/C(n, k). Averaging that over tasks gives the benchmark's pass@k. Estimating it as 1 − (1 − c/n)^k is biased for small n.

pass@k rises with k, so it flatters agents: a user who runs a task once gets pass@1, not pass@10. For reliability, report pass^k alongside it.

Formula

pass@k = 1 − C(n − c, k) / C(n, k)

Learn more on Kojen

Sources

  1. [1]Chen et al., 2021. Evaluating Large Language Models Trained on Code (arXiv:2107.03374)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free