Skip to content

Evaluation

pass@k vs pass^k: measuring whether your agent works every time

pass@k rewards one lucky success; pass^k demands every run succeed. The definitions, the unbiased estimators with worked examples, the averaging mistake to avoid, and how Kojen reports both.

By Will Schulz8 min read
A row of eight glass keys on obsidian stands above a thin red line of light, one key glowing red

Two agents can both score 75% on your eval and be completely different products. One fails the same quarter of tasks every time, so you know exactly what it can't do. The other fails a random quarter of runs on every task, so no customer can rely on it for anything. A single accuracy number hides that difference. pass@k and pass^k expose it.

This article explains both metrics, the unbiased way to estimate them from a handful of samples, the mistakes that make them misleading, and how Kojen reports them.

Two questions, two metrics

Run the same task k times. pass@k asks whether at least one attempt succeeded. It comes from code generation, where you can run unit tests and keep any passing sample: Kulal et al. used it for program synthesis,[1] and OpenAI's Codex paper made it the standard for HumanEval.[2]

pass^k (read “pass hat k”) asks whether every attempt succeeded. Yao et al. introduced it with τ-bench, a benchmark where an agent serves a simulated customer through real tools, because a support agent that solves a refund only sometimes is not one you can deploy. They found GPT-4o succeeded on fewer than half the tasks, and that pass^8 fell below 25% in the retail domain.[3]

If a task succeeds with probability p on each independent attempt, the two metrics move in opposite directions as k grows:

pass@k = P(at least one of k attempts succeeds) = 1 − (1 − p)^k
pass^k = P(all k attempts succeed)                = p^k

p = 0.75, k = 3:  pass@3 = 1 − 0.25³ = 0.984    pass^3 = 0.75³ = 0.422
p = 0.75, k = 8:  pass@8 = 0.99998             pass^8 = 0.75⁸ = 0.100

That matches the example in Anthropic's January 2026 guide to agent evals: a 75% per-trial success rate gives roughly 42% over three trials.[4] At eight trials it's 10%. The same agent looks nearly perfect under pass@8 and nearly useless under pass^8.

Which one should you report?

SituationMetricWhy
A verifier picks the best of several attempts (unit tests, a checker, a human)pass@kOne success is enough because you can tell which one it was
Each user gets one attempt, and the same request comes in many timespass^kEvery run has to work; failures reach users
You're deciding whether more samples are worth paying forBothThe gap between them is the variance you'd be buying your way out of

Most production agents are in the second row. A customer can't ask for eight answers and keep the right one, and an agent that writes to a database or sends an email doesn't get to retry silently.

Estimating them without bias

You never know p. You have n samples per task, c of which passed. The obvious move is to plug in c/n and raise it to the power k. That estimate is biased: for pass^k it is too optimistic, and for pass@k too pessimistic, because a power of an average isn't the average of a power.

The fix is to treat the n samples as a pool and ask how many ways you could draw k of them. Chen et al. gave the estimator for pass@k: one minus the chance that all k drawn samples come from the n − c failures.[2] pass^k is the mirror image: the chance that all k drawn samples come from the c successes.

n samples per task, c of them correct, k ≤ n

pass@k (Chen et al.)  = 1 − C(n − c, k) / C(n, k)
pass^k                = C(c, k) / C(n, k)

Example: n = 10, c = 7, k = 3
C(10, 3) = 120   C(3, 3) = 1   C(7, 3) = 35
pass@3 = 1 − 1/120 = 0.992
pass^3 = 35/120   = 0.292
naive (7/10)^3    = 0.343

The naive plug-in says 0.343. How wrong is it on average? If the true p is 0.7 and you take 10 samples, the expected value of (c/10)³ is 0.386, while the true p³ is 0.343. That's a 4.3-point overstatement of reliability built into the arithmetic. The combinatorial estimator’s expected value is exactly 0.343. Those two figures are our own calculations over the binomial distribution, and you can reproduce them in a few lines.

Computing binomial coefficients directly overflows quickly, so implementations use a running product. Here's the same calculation Kojen uses, in Python:

def pass_at_k(n, c, k):
    """Unbiased pass@k: 1 - C(n-c, k) / C(n, k), computed as a stable product."""
    if n - c < k:
        return 1.0
    prod = 1.0
    for i in range(n - c + 1, n + 1):
        prod *= 1 - k / i
    return 1 - prod

def pass_hat_k(n, c, k):
    """Unbiased pass^k: C(c, k) / C(n, k)."""
    if c < k:
        return 0.0
    prod = 1.0
    for i in range(k):
        prod *= (c - i) / (n - i)
    return prod

# Average per task, never pool the samples across tasks first.
tasks = [(10, 10), (10, 5)]          # (n, c) for each task
print(sum(pass_hat_k(n, c, 3) for n, c in tasks) / len(tasks))   # 0.542
# Pooling first, (15/20)^3, would say 0.422.

Two rules follow from the estimator:

  • You need n ≥ k. To report pass^8, sample at least eight times per task. Sampling more than k (say n = 10 for k = 5) lowers the estimator's variance.
  • Average per task, then across tasks. Never pool all samples into one success rate first.

Why averaging per task matters

Suppose your eval has two tasks. The agent passes the first every time and the second half the time. Its mean success rate is 75%. If you computed pass^3 from that pooled rate you'd get 0.75³ = 42%. The correct figure averages each task's own pass^3: (1 + 0.5³) / 2 = 56%.

The difference is the point of the metric. Failures that concentrate on a few hard tasks give a higher pass^k than failures spread thinly across every task, because the spread-out kind means nothing is safe. When pass^k falls much faster than pass@1 as k grows, your problem is inconsistency, not capability, and that's a different fix:

PatternWhat it meansWhat usually helps
pass@1 low, pass^k close to pass@1Consistent failures on specific tasksBetter instructions, examples or tools for those tasks; a stronger model
pass@1 decent, pass^k collapsesThe agent can do it, but not reliablyLower temperature, tighter output formats, clearer stopping rules, prompt tuning against repeated runs
pass@k high, pass@1 lowThe right answer is often among the samplesA verifier or best-of-n selection, if you can afford the extra calls

Error bars, because k samples aren't many

A pass^k figure from 30 tasks is an estimate with real uncertainty. Tasks are the unit that varies, so resample tasks, not individual attempts: a bootstrap over tasks gives an honest interval. Evan Miller's paper on error bars for evals makes the same case for treating questions as draws from a larger population and reporting intervals, not just means.[5]

Temperature matters too. pass^k at temperature 0 mostly measures whether your serving stack is deterministic. Measure at the temperature you'll ship with, and say which one you used.

Finally, k multiplies cost. Thirty tasks at k = 8 is 240 agent runs, plus a grader call for each. Start with k = 3 or 5 to find inconsistent tasks, then spend more samples on the ones that matter.

To learn how test cases, judges and agent evals fit together, see the Evaluating agents lesson, or read how to evaluate AI agents for the full method. If you're unsure how many tasks you need, our sample-size guide works through the arithmetic.

Frequently asked questions

What is the difference between pass@k and pass^k?
pass@k is the probability that at least one of k attempts at a task succeeds; pass^k is the probability that all k attempts succeed. With a 75% per-attempt success rate, pass@3 is 98% but pass^3 is 42%.
How do you calculate pass@k without bias?
Generate n ≥ k samples per task, count the c correct ones, and use 1 − C(n − c, k) / C(n, k) from the Codex paper (Chen et al., 2021). Average that over tasks. Raising c/n to a power is biased.
How do you calculate pass^k?
With n samples and c correct, the unbiased estimate is C(c, k) / C(n, k): the chance that k samples drawn from the n are all correct. Compute it per task and average across tasks.
When should I use pass^k instead of pass@k?
Use pass^k when each user gets one attempt and the same kind of request recurs, as with customer-facing agents. Use pass@k when a verifier such as unit tests can pick a correct attempt out of several.
Who introduced pass^k?
Yao et al. introduced pass^k in τ-bench (2024), a benchmark of tool-using agents talking to simulated users. They reported GPT-4o succeeded on fewer than half the tasks and that pass^8 fell below 25% in the retail domain.

Go deeper in the Academy

Free lessons with quizzes, citations and a hands-on exercise.

Sources

  1. [1]SPoC: Search-based Pseudocode to Code. Kulal et al., NeurIPS, arXiv (2019).
  2. [2]Evaluating Large Language Models Trained on Code. Chen et al., OpenAI, arXiv (2021).
  3. [3]τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Yao et al., Sierra, arXiv (2024).
  4. [4]Demystifying evals for AI agents. Anthropic Engineering (9 January 2026).
  5. [5]Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. Miller, arXiv (2024).

Put it into practice.

Kojen gives you the Forge, LLM-judged evaluations, red-team simulations and live endpoints in one place. Free to start, no card needed, then pay as you go.

Start training free

More reading

All articles →