Skip to content

Open source & open data

Measure reliability, not luck

An agent that passes a test once can still fail it on the next run. These free tools and datasets measure what happens over repeated tries: pass@k, pass^k, flaky cases and confidence intervals. Use them with or without a Kojen account.

Tool · MIT

passk

A zero-dependency CLI and GitHub Action that runs your test set N times against any OpenAI-compatible model and reports pass@1, pass@k and pass^k with confidence intervals, flaky cases and a paired significance test against a baseline.

Open

Open data · CC BY 4.0

Agent Reliability Index

A monthly, fixed public task suite run against popular models with repeated samples, published as a downloadable dataset: pass@1, pass^k curves, intervals, cost and latency.

Open

Game · Free to play

Break the Sensei

A red-team game: talk seven increasingly guarded agents into leaking their secret password. Opted-in attempts become an open dataset of prompts that broke the guards.

Open

Why these exist

Most evaluation numbers come from a single sample per case. That hides the thing that breaks agents in production: inconsistency. pass^k, the chance that all k tries succeed, falls fast when a model is right only most of the time, and it is the number a user feels when the same task runs every day.

passk brings that measurement to any CI pipeline. The Agent Reliability Index publishes it for popular models each month, on tasks written for the index rather than copied from public benchmarks. Break the Sensei is the adversarial side: how easily a guarded agent leaks what it was told to keep secret.

The statistics match what Kojen uses for its own evaluations. Source for passk: github.com/aw3-technology/passk.

Then make it more reliable

Kojen tunes prompts, searches models, red-teams and retrains your agent against the same test set, and shows you the pass^k after every change.

Start free