Why these exist
Most evaluation numbers come from a single sample per case. That hides the thing that breaks agents in production: inconsistency. pass^k, the chance that all k tries succeed, falls fast when a model is right only most of the time, and it is the number a user feels when the same task runs every day.
passk brings that measurement to any CI pipeline. The Agent Reliability Index publishes it for popular models each month, on tasks written for the index rather than copied from public benchmarks. Break the Sensei is the adversarial side: how easily a guarded agent leaks what it was told to keep secret.
The statistics match what Kojen uses for its own evaluations. Source for passk: github.com/aw3-technology/passk.