Agent eval
An agent eval measures an AI agent on multi-step tasks, scoring both the final outcome and how it got there, including tool calls, turns, cost and reliability over repeated runs.
Anthropic's guidance separates tasks, trials, graders and transcripts: grade outcomes with code where possible, use model judges for open-ended parts, and run each task several times because agents are non-deterministic.
Benchmarks such as τ-bench simulate users and tools and report pass^k, since an agent that succeeds once in a while isn't one you can ship.
Learn more on Kojen
Related terms
- pass^kpass^k is the probability that all k attempts at the same task succeed.
- pass@kpass@k is the probability that at least one of k sampled attempts at a task is correct.
- LLM-as-a-judgeLLM-as-a-judge is the practice of using a language model to grade another model's outputs against a rubric or a reference answer, instead of a human grader..
Sources
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free