pass^k
pass^k is the probability that all k attempts at the same task succeed. It measures how reliably an agent succeeds every time, and falls as k grows.
τ-bench introduced pass^k for agents that face the same kind of request again and again, where one failure in k runs matters. Its unbiased estimator from n samples with c correct is C(c, k)/C(n, k), the share of k-sized subsets of your samples that are all correct.
If attempts were independent with success rate p, pass^k would be p^k: a 90% agent passes all of 8 runs only 43% of the time. τ-bench found GPT-4o's function-calling agent below 25% at pass^8 in its retail domain.
Formula
pass^k = C(c, k) / C(n, k) ≈ p^k for independent attempts
Learn more on Kojen
Related terms
- pass@kpass@k is the probability that at least one of k sampled attempts at a task is correct.
- Agent evalAn agent eval measures an AI agent on multi-step tasks, scoring both the final outcome and how it got there, including tool calls, turns, cost and reliability over repeated runs..
- Trust score (Kojen)Kojen's trust score is a 0–100 roll-up of an evaluation; for comprehensive evaluations it weights accuracy 50%, hallucination 30% and bias 20%..
Sources
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free