Why reliability, not accuracy
Leaderboards usually report how often a model gets a task right once. An agent in production runs the same kind of task again and again, and one bad run in eight is a support ticket. The question that matters is how often it gets the task right every time. That is pass^k: the probability that k independent attempts at a task all pass. A model at 90% pass@1 can sit far lower at pass^8 if its failures are spread across many tasks rather than concentrated in a few.
Method
- Every model answers every task 8 times through the Vercel AI Gateway with the same short system prompt, temperature 0.7 and a cap of 400 output tokens. No tools, no retrieval, no retries on wrong answers. Calls that error after four network retries count as failures.
- pass@1 is the mean, over tasks, of the share of samples that passed, with a 95% Wilson interval over all samples.
- pass^k uses the unbiased estimator C(c, k) / C(n, k) for a task with c passes in n samples (Yao et al., 2024, τ-bench), averaged over tasks, with a 95% bootstrap interval over tasks. We also publish pass@k (Chen et al., 2021) in the raw data.
- Grading is deterministic: exact match, final number, regular expression, JSON Schema, or format checks such as word and line counts. There is no LLM judge, so anyone can re-grade the raw replies and get the same score.
- Cost is what the gateway reports per call; where it reports none we say so rather than estimate. Latency is wall-clock time per call from our runner, so it includes network time.
- The task file is versioned: each edition records the SHA-256 of the exact suite it ran. Tasks are original to Kojen (not drawn from public benchmarks), released under CC BY 4.0.
The tasks
40 short tasks in four categories. They are deliberately small: each has one verifiable right answer, so a failure is the model's, not the grader's.
Reasoning · 10 tasks
Tool-free arithmetic and logic word problems with a single numeric or one-word answer.
A shop sells pencils at 3 for $0.75. How many cents do 14 pencils cost? Reply with only the final number, no units or working.
Extraction · 10 tasks
Pull named fields out of a short invoice, log line, address or message into JSON.
Extract fields from this text as a JSON object with keys invoice_id (string), customer (string), total (number) and currency (ISO 4217 code). Text: "Invoice INV-20931 was issued on 3 March 2026 to Halvorsen Bakery for a total of €1,284.50, due in 30 days." Reply with only the JSON, no code fence or commentary.
Instruction following · 10 tasks
Follow format constraints exactly: counts of bullets, lines or words, banned words, fixed openings and endings.
Write exactly three bullet points about why sleep matters. Start each bullet with "- ". Write nothing else: no heading, intro or closing line.
JSON formatting · 10 tasks
Produce JSON that validates against a schema, sometimes with exact expected values.
Return a JSON object for a fictional person with exactly these keys: name (string), age (integer) and tags (array of exactly 2 strings). Reply with only the JSON, no code fence or commentary.
Run it on your own agent
The same statistics (pass@1, unbiased pass@k and pass^k, Wilson intervals, McNemar comparisons) are in passk, our open-source CLI and GitHub Action: point it at any OpenAI-compatible endpoint and your own test set. For why repeated runs matter, read Practice beats prompting or the Academy lesson on evaluation and trust.
Data license: CC BY 4.0. Cite as “Kojen Agent Reliability Index, November 2026, kojen.ai/reliability-index”.