Two agents can both score 75% on your eval and be completely different products. One fails the same quarter of tasks every time, so you know exactly what it can't do. The other fails a random quarter of runs on every task, so no customer can rely on it for anything. A single accuracy number hides that difference. pass@k and pass^k expose it.
This article explains both metrics, the unbiased way to estimate them from a handful of samples, the mistakes that make them misleading, and how Kojen reports them.
Two questions, two metrics
Run the same task k times. pass@k asks whether at least one attempt succeeded. It comes from code generation, where you can run unit tests and keep any passing sample: Kulal et al. used it for program synthesis,[1] and OpenAI's Codex paper made it the standard for HumanEval.[2]
pass^k (read “pass hat k”) asks whether every attempt succeeded. Yao et al. introduced it with τ-bench, a benchmark where an agent serves a simulated customer through real tools, because a support agent that solves a refund only sometimes is not one you can deploy. They found GPT-4o succeeded on fewer than half the tasks, and that pass^8 fell below 25% in the retail domain.[3]
If a task succeeds with probability p on each independent attempt, the two metrics move in opposite directions as k grows:
pass@k = P(at least one of k attempts succeeds) = 1 − (1 − p)^k pass^k = P(all k attempts succeed) = p^k p = 0.75, k = 3: pass@3 = 1 − 0.25³ = 0.984 pass^3 = 0.75³ = 0.422 p = 0.75, k = 8: pass@8 = 0.99998 pass^8 = 0.75⁸ = 0.100
That matches the example in Anthropic's January 2026 guide to agent evals: a 75% per-trial success rate gives roughly 42% over three trials.[4] At eight trials it's 10%. The same agent looks nearly perfect under pass@8 and nearly useless under pass^8.
Which one should you report?
| Situation | Metric | Why |
|---|---|---|
| A verifier picks the best of several attempts (unit tests, a checker, a human) | pass@k | One success is enough because you can tell which one it was |
| Each user gets one attempt, and the same request comes in many times | pass^k | Every run has to work; failures reach users |
| You're deciding whether more samples are worth paying for | Both | The gap between them is the variance you'd be buying your way out of |
Most production agents are in the second row. A customer can't ask for eight answers and keep the right one, and an agent that writes to a database or sends an email doesn't get to retry silently.
Estimating them without bias
You never know p. You have n samples per task, c of which passed. The obvious move is to plug in c/n and raise it to the power k. That estimate is biased: for pass^k it is too optimistic, and for pass@k too pessimistic, because a power of an average isn't the average of a power.
The fix is to treat the n samples as a pool and ask how many ways you could draw k of them. Chen et al. gave the estimator for pass@k: one minus the chance that all k drawn samples come from the n − c failures.[2] pass^k is the mirror image: the chance that all k drawn samples come from the c successes.
n samples per task, c of them correct, k ≤ n pass@k (Chen et al.) = 1 − C(n − c, k) / C(n, k) pass^k = C(c, k) / C(n, k) Example: n = 10, c = 7, k = 3 C(10, 3) = 120 C(3, 3) = 1 C(7, 3) = 35 pass@3 = 1 − 1/120 = 0.992 pass^3 = 35/120 = 0.292 naive (7/10)^3 = 0.343
The naive plug-in says 0.343. How wrong is it on average? If the true p is 0.7 and you take 10 samples, the expected value of (c/10)³ is 0.386, while the true p³ is 0.343. That's a 4.3-point overstatement of reliability built into the arithmetic. The combinatorial estimator’s expected value is exactly 0.343. Those two figures are our own calculations over the binomial distribution, and you can reproduce them in a few lines.
Computing binomial coefficients directly overflows quickly, so implementations use a running product. Here's the same calculation Kojen uses, in Python:
def pass_at_k(n, c, k):
"""Unbiased pass@k: 1 - C(n-c, k) / C(n, k), computed as a stable product."""
if n - c < k:
return 1.0
prod = 1.0
for i in range(n - c + 1, n + 1):
prod *= 1 - k / i
return 1 - prod
def pass_hat_k(n, c, k):
"""Unbiased pass^k: C(c, k) / C(n, k)."""
if c < k:
return 0.0
prod = 1.0
for i in range(k):
prod *= (c - i) / (n - i)
return prod
# Average per task, never pool the samples across tasks first.
tasks = [(10, 10), (10, 5)] # (n, c) for each task
print(sum(pass_hat_k(n, c, 3) for n, c in tasks) / len(tasks)) # 0.542
# Pooling first, (15/20)^3, would say 0.422.Two rules follow from the estimator:
- You need n ≥ k. To report pass^8, sample at least eight times per task. Sampling more than k (say n = 10 for k = 5) lowers the estimator's variance.
- Average per task, then across tasks. Never pool all samples into one success rate first.
Why averaging per task matters
Suppose your eval has two tasks. The agent passes the first every time and the second half the time. Its mean success rate is 75%. If you computed pass^3 from that pooled rate you'd get 0.75³ = 42%. The correct figure averages each task's own pass^3: (1 + 0.5³) / 2 = 56%.
The difference is the point of the metric. Failures that concentrate on a few hard tasks give a higher pass^k than failures spread thinly across every task, because the spread-out kind means nothing is safe. When pass^k falls much faster than pass@1 as k grows, your problem is inconsistency, not capability, and that's a different fix:
| Pattern | What it means | What usually helps |
|---|---|---|
| pass@1 low, pass^k close to pass@1 | Consistent failures on specific tasks | Better instructions, examples or tools for those tasks; a stronger model |
| pass@1 decent, pass^k collapses | The agent can do it, but not reliably | Lower temperature, tighter output formats, clearer stopping rules, prompt tuning against repeated runs |
| pass@k high, pass@1 low | The right answer is often among the samples | A verifier or best-of-n selection, if you can afford the extra calls |
Error bars, because k samples aren't many
A pass^k figure from 30 tasks is an estimate with real uncertainty. Tasks are the unit that varies, so resample tasks, not individual attempts: a bootstrap over tasks gives an honest interval. Evan Miller's paper on error bars for evals makes the same case for treating questions as draws from a larger population and reporting intervals, not just means.[5]
Temperature matters too. pass^k at temperature 0 mostly measures whether your serving stack is deterministic. Measure at the temperature you'll ship with, and say which one you used.
Finally, k multiplies cost. Thirty tasks at k = 8 is 240 agent runs, plus a grader call for each. Start with k = 3 or 5 to find inconsistent tasks, then spend more samples on the ones that matter.
To learn how test cases, judges and agent evals fit together, see the Evaluating agents lesson, or read how to evaluate AI agents for the full method. If you're unsure how many tasks you need, our sample-size guide works through the arithmetic.



