Evaluation & Trust
Lesson 4 of 5Intermediate 10 min

Evaluating agents: trials, transcripts and pass^k

Agents act over many turns and change the world around them. Grade the outcome, read the transcript, and run every task more than once.

In this lesson you'll

  • Use the vocabulary of agent evals: task, trial, grader, transcript, outcome
  • Choose between pass@k and pass^k
  • Grade outcomes rather than claims

Single-turn evals send a prompt and grade a reply. Agent evals give the model tools and an environment and let it work for many turns, so mistakes compound and creative solutions can surprise the grader [1]. Benchmarks such as SWE-bench, built from real GitHub issues graded by the repository's own tests [2], and τ-bench, where an agent serves a simulated user under a domain policy [3], set the pattern most teams follow in 2026.

Vocabulary

  • Task: one test with inputs and success criteria.
  • Trial: one attempt at a task. Outputs vary, so run several.
  • Transcript: the full record of a trial, including tool calls and reasoning.
  • Outcome: the final state of the environment. A booking agent saying "booked" isn't the outcome; a reservation existing in the database is [1].

pass@k or pass^k?

pass@k is the chance at least one of k trials succeeds; it rises with k. pass^k, introduced with τ-bench, is the chance all k trials succeed; it falls with k [3]. An agent that succeeds 75% of the time per trial passes all three of three trials only about 42% of the time [1]. Use pass@k where one success is enough (generating options), and pass^k for customer-facing agents that must be reliable every time.

Read the transcripts

Scores tell you something changed; transcripts tell you why. A failure should look fair when you read it. If it doesn't, the grader is wrong, not the agent [1].

In Kojen

Scenario simulations run given or generated scenarios and grade each pass/fail, and every Agents run keeps a live log you can read like a transcript.

Key takeaways

  • Grade the outcome in the environment, not what the agent says it did.
  • Run several trials; report pass^k for anything customer-facing.
  • Read transcripts to check the grader as well as the agent.

Practise in the dojo

Run a Scenario simulation against your model three times and compare which scenarios pass every time.

Run a scenario simulation

Check your understanding

Score 70 or more to complete the lesson. Progress is saved in this browser.

1. An agent succeeds 75% of the time per trial. Roughly what is pass^3 (all three trials succeed)?
2. What should a booking agent's eval check?

Further reading

Sources

Checked against the original sources in November 2026.

  1. [1]Anthropic (2026). Demystifying evals for AI agents. Anthropic Engineering, January 2026.
  2. [2]Jimenez, C. E. et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. ICLR 2024 · arXiv:2310.06770.
  3. [3]Yao, S. et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045.