pass@k and pass^k calculator
pass@k asks whether an agent succeeds at least once in k tries; pass^k asks whether it succeeds every time. With 7 of 10 samples correct, pass@3 is 99.2% but pass^3 is only 29.2%. Enter your own samples to see both, and how fast reliability falls.
99.2%
29.2%
34.3%
70.0%
- pass@k (at least one of k passes)
- pass^k (all k pass, unbiased)
- p^k (independent attempts)
Show as a table
| k | pass@k | pass^k | p^k |
|---|---|---|---|
| 1 | 70.0% | 70.0% | 70.0% |
| 2 | 93.3% | 46.7% | 49.0% |
| 3 | 99.2% | 29.2% | 34.3% |
| 4 | 100.0% | 16.7% | 24.0% |
| 5 | 100.0% | 8.3% | 16.8% |
| 6 | 100.0% | 3.3% | 11.8% |
| 7 | 100.0% | 0.8% | 8.2% |
| 8 | 100.0% | 0.0% | 5.8% |
| 9 | 100.0% | 0.0% | 4.0% |
| 10 | 100.0% | 0.0% | 2.8% |
For one task. Over a test set, average each task's value; Kojen's evaluations report that mean with a bootstrap interval over tasks. The chart shows k up to 100.
The estimators
Given n samples for a task, c of them correct, the unbiased estimators (Chen et al., 2021, for pass@k; Yao et al., 2024, for pass^k) are:
pass@k = 1 − C(n − c, k) / C(n, k) pass^k = C(c, k) / C(n, k) p^k = (c / n)^k (assumes independent attempts)
Both need n ≥ k. They are computed with running products rather than factorials, so they stay accurate for large n, by the same functions Kojen's evaluation engine uses.
| Samples | pass@1 | pass@5 | pass^5 | p^5 |
|---|---|---|---|---|
| 9 of 10 correct | 90.0% | 100.0% | 50.0% | 59.0% |
| 7 of 10 correct | 70.0% | 100.0% | 8.3% | 16.8% |
| 5 of 10 correct | 50.0% | 99.6% | 0.4% | 3.1% |
| 10 of 20 correct | 50.0% | 98.4% | 1.6% | 3.1% |
To plan how many test cases you need around these numbers, use the eval sample size calculator. Why reliability matters more than best-of-k for agents is the subject of Practice beats prompting and the Academy lesson Evaluating agents.
Frequently asked questions
- What is the difference between pass@k and pass^k?
- pass@k is the chance that at least one of k attempts succeeds, which rewards trying several times. pass^k is the chance that all k attempts succeed, which measures whether an agent is reliable every time. An agent with a 70% single-try rate has pass@3 near 97% but pass^3 near 34%.
- Why use the unbiased estimator instead of p^k?
- p^k treats attempts as independent coin flips at the observed rate. The unbiased estimator C(c,k)/C(n,k) counts how many k-subsets of your n samples are all correct, so it is exact for your data and doesn't overstate reliability when n is small. With 10 samples, 7 correct, p^3 is 34.3% but the unbiased pass^3 is 29.2%.
- How many samples per task should I draw?
- At least k, and ideally several times k: the estimators need n ≥ k, and their variance falls as n grows. Chen et al. drew 200 samples per problem to estimate pass@k for k up to 100.
- Which should I report for an agent?
- Both. pass@1 is the expected single-run success rate; pass^k for the number of times a user will run the same task tells you how often they'll hit a failure. τ-bench found a GPT-4o function-calling agent succeeded on under half its tasks, with pass^8 below 25% in the retail domain.
Glossary
Sources
Measure reliability, not one lucky run.
Kojen evaluations sample each case several times and report pass@k and pass^k with bootstrap intervals, so you see how often your agent fails, not just whether it can succeed.
Start training free