Skip to content

Open data · monthly

Agent Reliability Index

Not “can the model do it?” but “does it do it every time?”. Each month we run 40 verifiable agent tasks against hosted LLMs, 8 times each, and publish pass@1 and pass^k with confidence intervals, cost and latency. Tasks, raw results and method are open.

November 2026 edition

Leaderboard

10 models, 40 tasks, 8 samples each at temperature 0.7. Ranked by pass^8: the chance a model gets a task right on all 8 tries. The most reliable model, google/gemma-4-31b-it, passed every try on 95.0% of tasks, against 96.3% on a single try.

#Modelpass@1pass^8Flaky tasksLatency p50 / p95Cost (320 calls)
1google/gemma-4-31b-it96.3%95% CI 94–98%95.0%95% CI 88–100%1 of 401.2 s / 3.2 s$0.006
2anthropic/claude-sonnet-5.595.6%95% CI 93–97%92.5%95% CI 85–100%2 of 400.9 s / 1.6 s$0.206
3anthropic/claude-haiku-5.593.4%95% CI 90–96%92.5%95% CI 85–100%1 of 400.8 s / 1.4 s$0.010
4meta/llama-3.3-70b90.0%95% CI 86–93%90.0%95% CI 80–98%0 of 400.4 s / 0.8 s$0.026
5deepseek/deepseek-v3.294.4%95% CI 91–96%87.5%95% CI 78–98%4 of 400.9 s / 3.1 s$0.024
6openai/gpt-4.1-mini88.1%95% CI 84–91%87.5%95% CI 75–98%1 of 400.7 s / 1.2 s$0.017
7google/gemini-2.5-flash94.7%95% CI 92–97%80.0%95% CI 68–93%8 of 401.4 s / 2.4 s$0.146
8mistral/mistral-small86.3%95% CI 82–90%80.0%95% CI 68–93%3 of 400.6 s / 1.3 s$0.005
9alibaba/qwen-3-32b79.1%95% CI 74–83%67.5%95% CI 53–83%9 of 400.4 s / 0.7 s$0.008
10meta/llama-3.1-8b69.4%95% CI 64–74%60.0%95% CI 45–75%7 of 400.4 s / 0.5 s$0.007

Reliability falls with k

pass^k for k = 1 to 8. At k = 1 it equals pass@1; the steeper a model's curve, the more its successes depend on luck.

Up to five models at a time.

By task category

ModelReasoningExtractionInstruction followingJSON formatting
google/gemma-4-31b-it100.0%100.0%85.0%100.0%
anthropic/claude-sonnet-5.583.8%100.0%98.8%100.0%
anthropic/claude-haiku-5.5100.0%100.0%73.8%100.0%
meta/llama-3.3-70b70.0%100.0%90.0%100.0%
deepseek/deepseek-v3.295.0%100.0%83.8%98.8%
openai/gpt-4.1-mini80.0%100.0%72.5%100.0%
google/gemini-2.5-flash98.8%98.8%83.8%97.5%
mistral/mistral-small76.3%100.0%68.8%100.0%
alibaba/qwen-3-32b51.2%100.0%68.8%96.3%
meta/llama-3.1-8b13.8%100.0%83.8%80.0%

Share of all samples passed in each category (pass@1).

Why reliability, not accuracy

Leaderboards usually report how often a model gets a task right once. An agent in production runs the same kind of task again and again, and one bad run in eight is a support ticket. The question that matters is how often it gets the task right every time. That is pass^k: the probability that k independent attempts at a task all pass. A model at 90% pass@1 can sit far lower at pass^8 if its failures are spread across many tasks rather than concentrated in a few.

Method

  • Every model answers every task 8 times through the Vercel AI Gateway with the same short system prompt, temperature 0.7 and a cap of 400 output tokens. No tools, no retrieval, no retries on wrong answers. Calls that error after four network retries count as failures.
  • pass@1 is the mean, over tasks, of the share of samples that passed, with a 95% Wilson interval over all samples.
  • pass^k uses the unbiased estimator C(c, k) / C(n, k) for a task with c passes in n samples (Yao et al., 2024, τ-bench), averaged over tasks, with a 95% bootstrap interval over tasks. We also publish pass@k (Chen et al., 2021) in the raw data.
  • Grading is deterministic: exact match, final number, regular expression, JSON Schema, or format checks such as word and line counts. There is no LLM judge, so anyone can re-grade the raw replies and get the same score.
  • Cost is what the gateway reports per call; where it reports none we say so rather than estimate. Latency is wall-clock time per call from our runner, so it includes network time.
  • The task file is versioned: each edition records the SHA-256 of the exact suite it ran. Tasks are original to Kojen (not drawn from public benchmarks), released under CC BY 4.0.

The tasks

40 short tasks in four categories. They are deliberately small: each has one verifiable right answer, so a failure is the model's, not the grader's.

Reasoning · 10 tasks

Tool-free arithmetic and logic word problems with a single numeric or one-word answer.

A shop sells pencils at 3 for $0.75. How many cents do 14 pencils cost? Reply with only the final number, no units or working.

Extraction · 10 tasks

Pull named fields out of a short invoice, log line, address or message into JSON.

Extract fields from this text as a JSON object with keys invoice_id (string), customer (string), total (number) and currency (ISO 4217 code).

Text: "Invoice INV-20931 was issued on 3 March 2026 to Halvorsen Bakery for a total of €1,284.50, due in 30 days."

Reply with only the JSON, no code fence or commentary.

Instruction following · 10 tasks

Follow format constraints exactly: counts of bullets, lines or words, banned words, fixed openings and endings.

Write exactly three bullet points about why sleep matters. Start each bullet with "- ". Write nothing else: no heading, intro or closing line.

JSON formatting · 10 tasks

Produce JSON that validates against a schema, sometimes with exact expected values.

Return a JSON object for a fictional person with exactly these keys: name (string), age (integer) and tags (array of exactly 2 strings). Reply with only the JSON, no code fence or commentary.

Run it on your own agent

The same statistics (pass@1, unbiased pass@k and pass^k, Wilson intervals, McNemar comparisons) are in passk, our open-source CLI and GitHub Action: point it at any OpenAI-compatible endpoint and your own test set. For why repeated runs matter, read Practice beats prompting or the Academy lesson on evaluation and trust.

Data license: CC BY 4.0. Cite as “Kojen Agent Reliability Index, November 2026, kojen.ai/reliability-index”.

Measure your agent's pass^k, then raise it.

Kojen runs your own test cases many times, reports pass^k with intervals, and tunes prompts and models until the agent is reliable, not just lucky.

Start free