Skip to content

Free tool

pass@k and pass^k calculator

pass@k asks whether an agent succeeds at least once in k tries; pass^k asks whether it succeeds every time. With 7 of 10 samples correct, pass@3 is 99.2% but pass^3 is only 29.2%. Enter your own samples to see both, and how fast reliability falls.

pass@3

99.2%

pass^3 (unbiased)

29.2%

p^3

34.3%

Single-try rate (c/n)

70.0%

  • pass@k (at least one of k passes)
  • pass^k (all k pass, unbiased)
  • p^k (independent attempts)
0%25%50%75%100%k = 1k = 3k = 5k = 8k = 10
Show as a table
kpass@kpass^kp^k
170.0%70.0%70.0%
293.3%46.7%49.0%
399.2%29.2%34.3%
4100.0%16.7%24.0%
5100.0%8.3%16.8%
6100.0%3.3%11.8%
7100.0%0.8%8.2%
8100.0%0.0%5.8%
9100.0%0.0%4.0%
10100.0%0.0%2.8%

For one task. Over a test set, average each task's value; Kojen's evaluations report that mean with a bootstrap interval over tasks. The chart shows k up to 100.

The estimators

Given n samples for a task, c of them correct, the unbiased estimators (Chen et al., 2021, for pass@k; Yao et al., 2024, for pass^k) are:

pass@k = 1 − C(n − c, k) / C(n, k)
pass^k = C(c, k) / C(n, k)
p^k    = (c / n)^k        (assumes independent attempts)

Both need n ≥ k. They are computed with running products rather than factorials, so they stay accurate for large n, by the same functions Kojen's evaluation engine uses.

Samplespass@1pass@5pass^5p^5
9 of 10 correct90.0%100.0%50.0%59.0%
7 of 10 correct70.0%100.0%8.3%16.8%
5 of 10 correct50.0%99.6%0.4%3.1%
10 of 20 correct50.0%98.4%1.6%3.1%

To plan how many test cases you need around these numbers, use the eval sample size calculator. Why reliability matters more than best-of-k for agents is the subject of Practice beats prompting and the Academy lesson Evaluating agents.

Frequently asked questions

What is the difference between pass@k and pass^k?
pass@k is the chance that at least one of k attempts succeeds, which rewards trying several times. pass^k is the chance that all k attempts succeed, which measures whether an agent is reliable every time. An agent with a 70% single-try rate has pass@3 near 97% but pass^3 near 34%.
Why use the unbiased estimator instead of p^k?
p^k treats attempts as independent coin flips at the observed rate. The unbiased estimator C(c,k)/C(n,k) counts how many k-subsets of your n samples are all correct, so it is exact for your data and doesn't overstate reliability when n is small. With 10 samples, 7 correct, p^3 is 34.3% but the unbiased pass^3 is 29.2%.
How many samples per task should I draw?
At least k, and ideally several times k: the estimators need n ≥ k, and their variance falls as n grows. Chen et al. drew 200 samples per problem to estimate pass@k for k up to 100.
Which should I report for an agent?
Both. pass@1 is the expected single-run success rate; pass^k for the number of times a user will run the same task tells you how often they'll hit a failure. τ-bench found a GPT-4o function-calling agent succeeded on under half its tasks, with pass^8 below 25% in the retail domain.

Sources

  1. [1]Chen et al., 2021. Evaluating Large Language Models Trained on Code (arXiv:2107.03374)
  2. [2]Yao et al., 2024. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv:2406.12045)

Measure reliability, not one lucky run.

Kojen evaluations sample each case several times and report pass@k and pass^k with bootstrap intervals, so you see how often your agent fails, not just whether it can succeed.

Start training free