Skip to content

Glossary · Evaluation

pass^k

pass^k is the probability that all k attempts at the same task succeed. It measures how reliably an agent succeeds every time, and falls as k grows.

Also called pass hat k, pass^k reliability · Reviewed 15 November 2026

τ-bench introduced pass^k for agents that face the same kind of request again and again, where one failure in k runs matters. Its unbiased estimator from n samples with c correct is C(c, k)/C(n, k), the share of k-sized subsets of your samples that are all correct.

If attempts were independent with success rate p, pass^k would be p^k: a 90% agent passes all of 8 runs only 43% of the time. τ-bench found GPT-4o's function-calling agent below 25% at pass^8 in its retail domain.

Formula

pass^k = C(c, k) / C(n, k)  ≈ p^k for independent attempts

Learn more on Kojen

Sources

  1. [1]Yao et al., 2024. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv:2406.12045)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free