Skip to content

Glossary · Evaluation

Agent eval

An agent eval measures an AI agent on multi-step tasks, scoring both the final outcome and how it got there, including tool calls, turns, cost and reliability over repeated runs.

Also called agent evaluation · Reviewed 15 November 2026

Anthropic's guidance separates tasks, trials, graders and transcripts: grade outcomes with code where possible, use model judges for open-ended parts, and run each task several times because agents are non-deterministic.

Benchmarks such as τ-bench simulate users and tools and report pass^k, since an agent that succeeds once in a while isn't one you can ship.

Learn more on Kojen

Sources

  1. [1]Anthropic. Demystifying evals for AI agents
  2. [2]Yao et al., 2024. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv:2406.12045)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free