Skip to content

Open source · MIT · v0.1.0

passk: is your agent reliable, or did it get lucky?

passk runs every case in your test set several times against any OpenAI-compatible model and reports pass@1, pass@k and pass^k with confidence intervals, the cases that flip between runs, and whether a change beat your baseline or just got noise. One command locally, one step in CI. No dependencies.

Quickstart
# OpenAI (any OpenAI-compatible endpoint works)
export PASSK_API_KEY=sk-...
npx passk tests.jsonl --model gpt-4o-mini --samples 8

# No API key? Try the pipeline with a mock model
npx passk tests.jsonl --mock --mock-fail-rate 0.2
Output (illustrative)
  Model              gpt-4o-mini
  Cases × samples    40 × 8
  pass@1             81.3% (95% CI 77.2%–84.7%)
  pass@8             95.0%
  pass^8             52.5%
  Flaky cases        17

In that example a single try passes 81% of the time, yet only about half of the cases pass all eight tries. If your agent runs the same task every day, pass^k is the number your users feel.

Write a test set

JSONL with one case per line, or CSV with a header row. Each case has an input (a prompt or a message list), usually an expected answer, and optionally its own grader. Give cases stable ids so runs can be compared.

tests.jsonl
{"id": "capital", "input": "Capital of Australia? City only.", "expected": "Canberra", "grader": {"type": "exact", "ignoreCase": true}}
{"id": "math", "input": "17 * 23? Number only.", "expected": "391"}
{"id": "iso", "input": "4 July 2026 in ISO 8601?", "grader": {"type": "regex", "pattern": "2026-07-04"}}
{"id": "contact", "input": "Extract name and email as JSON: 'Ada, ada@example.com'", "grader": {"type": "json-schema", "schema": {"type": "object", "required": ["name", "email"]}}}
{"id": "refund", "input": "Customer wants a refund on day 40; policy is 30 days.", "grader": {"type": "llm-judge", "rubric": "Declines politely, cites the policy, offers an alternative."}}

Graders

exact
Equals the expected answer after trimming whitespace (optionally ignoring case).
contains
Contains the expected string, any of a list, or all of a list.
regex
Matches a regular expression.
json-schema
Holds JSON (bare, fenced or inside prose) that validates against a schema: type, properties, required, enum, items, ranges, lengths, pattern.
llm-judge
A judge model you choose grades the reply against the reference and a rubric, through any OpenAI-compatible endpoint.

A sample whose call fails after retries counts as a failure, so an outage can never raise a score.

Any OpenAI-compatible endpoint

Point --base-url at anything that serves /chat/completions. The key comes from PASSK_API_KEY (or OPENAI_API_KEY). Requests run in parallel with retries on 429 and 5xx (honouring Retry-After) and a per-request timeout.

# A Kojen deployment (model comes from the deployment)
PASSK_API_KEY=kojen_dk_... npx passk tests.jsonl \
  --base-url https://kojen.ai/api/v1/endpoints/<deployment-id>

# vLLM
npx passk tests.jsonl --base-url http://localhost:8000/v1 -m Qwen/Qwen3-8B

# Ollama
npx passk tests.jsonl --base-url http://localhost:11434/v1 -m llama3.1:8b

# Judge with a different model and endpoint
PASSK_JUDGE_API_KEY=sk-... npx passk tests.jsonl -m llama3.1:8b \
  --base-url http://localhost:11434/v1 \
  --judge-base-url https://api.openai.com/v1 --judge-model gpt-4o

What the numbers mean

  • pass@1: how often one try passes, with a 95% Wilson interval, which behaves at 0%, 100% and small sample sizes.
  • pass@k: the chance at least one of k tries passes, using the unbiased estimator 1 − C(n−c, k) / C(n, k) from Chen et al. (2021).
  • pass^k: the chance all k tries pass, C(c, k) / C(n, k), the reliability metric from Yao et al. (2024, τ-bench).
  • Flaky cases: cases that passed some samples and failed others. Prompt and model changes show up there first.
  • McNemar's test: with a baseline, a paired test on the cases that changed, so you know whether a gain is real.

The Academy lesson on evaluation and trust covers these in depth, and Practice beats prompting explains why pass^k matters for agents.

GitHub Action

The action runs passk, writes the table to the job summary, and posts a pull request comment that it updates in place on every push. Fail the check when pass@1 drops under a threshold or when the comparison finds a significant regression.

.github/workflows/passk.yml
# .github/workflows/passk.yml
name: passk
on: pull_request

permissions:
  contents: read
  pull-requests: write

jobs:
  reliability:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aw3-technology/passk@v0
        with:
          test-set: evals/tests.jsonl
          model: gpt-4o-mini
          base-url: https://api.openai.com/v1
          samples: 5
          fail-under: 0.8
          baseline: evals/baseline.json
        env:
          PASSK_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Compare against a baseline

npx passk tests.jsonl -m gpt-4o-mini -n 8 --out baseline.json
# ...change the prompt or the model...
npx passk tests.jsonl -m gpt-4o-mini -n 8 --compare baseline.json --fail-on-regression

  vs baseline pass@1  78.1% → 84.4% (+6.3 pts)
  McNemar             5 improved, 1 regressed, p = 0.2188 → no significant difference

A six-point gain on 40 cases can easily be noise. passk says so instead of letting a lucky run through.

Track runs in Kojen (optional)

With --upload, the run summary goes to your Kojen account and appears under Evaluations, so you can follow reliability across commits. Only aggregate numbers and per-case pass counts are sent, never prompts or outputs. Create an API key with the write scope in Settings.

KOJEN_API_KEY=kojen_... npx passk tests.jsonl -m gpt-4o-mini \
  --upload --name "main@$(git rev-parse --short HEAD)"

Frequently asked questions

What is the difference between pass@k and pass^k?
pass@k is the chance that at least one of k tries passes, which suits tasks you can retry or pick the best answer for. pass^k is the chance that all k tries pass: a reliability measure. A model with 80% pass@1 has roughly 33% pass^5 if its failures are spread evenly across cases.
How are pass@k and pass^k computed?
For each case passk draws n samples and counts c correct, then uses the unbiased estimators 1 − C(n−c, k)/C(n, k) for pass@k (Chen et al., 2021) and C(c, k)/C(n, k) for pass^k (Yao et al., 2024), averaged over cases. Pick k no larger than n.
Which models and endpoints does passk work with?
Anything that serves OpenAI-style POST /chat/completions: OpenAI, Kojen deployment endpoints, vLLM, Ollama, LiteLLM, OpenRouter and most hosted inference providers. Set the base URL and key with --base-url and PASSK_API_KEY.
How does passk decide if a change is a real improvement?
With --compare baseline.json it pairs cases by id, counts a case as passing when most of its samples pass, and runs McNemar's test on the cases that changed. Small gains on small test sets often come out as “no significant difference”, which is the honest answer.
Does passk send my prompts anywhere?
Only to the model endpoint you configure (and the judge, if you use one). The optional --upload to Kojen sends aggregate numbers and per-case pass counts, never prompts or outputs.

passk measures it. Kojen fixes it.

Run prompt tuning, model search and red-teaming against the same test set, deploy the winner behind an OpenAI-compatible endpoint, and point passk at it.

Start free