Evaluating a chatbot means grading a reply. Evaluating an agent means grading a sequence: it reads a request, picks tools, calls them with arguments, reads the results, decides what to do next and eventually stops. Any step can go wrong, the same request can take a different path every time, and a wrong step can still end in a plausible-looking answer.
This guide is the method we'd use to evaluate an agent from scratch, in the order we'd do it. It draws on Anthropic's January 2026 guide to agent evals,[1] the τ-bench and SWE-bench benchmarks, and the statistics we've written about separately.
1. Write down what success means, per task
Start with tasks, not metrics. A task is one request with a defined starting state and explicit success criteria. Anthropic's guide uses the same vocabulary: a task is one test, a trial is one attempt at it, a transcript is the full record of what the agent did, and the outcome is the final state of the environment.[1]
{
"id": "refund-damaged-item-07",
"input": "My order 4471 arrived broken. I want my money back.",
"environment": { "orders": [{ "id": 4471, "status": "delivered", "total": 89.00 }] },
"success_criteria": [
"outcome: a refund of 89.00 exists for order 4471",
"outcome: no other order was modified",
"transcript: agent confirmed the order number before refunding",
"reply: tells the customer when the money will arrive"
],
"should_not": "issue store credit instead of a refund"
}Notice that most criteria are about the outcome, not the words. The refund either exists in the database or it doesn't. τ-bench grades this way: it compares the database state at the end of a conversation with the annotated goal state.[2] SWE-bench does the same for coding agents by running the repository's tests after the agent's patch.[3]
Include tasks where the agent should not act: a request it should refuse, an ambiguous one where it should ask, an order that isn't eligible. An eval made only of “do the thing” tasks rewards an agent that always does the thing.
2. Start with 20–50 tasks from real failures
You don't need hundreds of tasks to begin. Anthropic suggests 20–50 simple tasks drawn from real failures, user reports and the checks you were already doing by hand.[1] Spread them across:
- The common path, weighted by how often each request type actually arrives.
- Edge cases: missing information, conflicting instructions, tool errors, empty search results.
- Policy boundaries: things the agent must refuse or escalate.
- Adversarial inputs: prompt injection in retrieved content or tool output (see red-teaming your agent).
Twenty tasks are enough to find broken behaviour, not to measure small improvements. Our sample-size guide shows why: at 30 tasks, an 80% score has a 95% interval of roughly 63–91%. Grow the set as production surfaces new failures, and keep a held-out portion you never tune against.
3. Pick the cheapest grader that's reliable
| Grader | Good for | Watch out for |
|---|---|---|
| Code (state checks, tests, regex, schema validation) | Outcomes, tool arguments, output format | Brittle when many answers are valid |
| LLM judge with a rubric | Tone, helpfulness, whether a reply explains the next step | Bias and drift; must be checked against human labels |
| Human review | Calibrating the judge, ambiguous or high-stakes tasks | Slow and expensive; use it to audit, not to grade everything |
Use code wherever the outcome is checkable, a model where it isn't, and people to keep the model honest. Anthropic's guide makes the same ordering and recommends grading each rubric dimension separately and giving the judge an explicit “unknown” option.[1] If you use an LLM judge, measure its agreement with humans before trusting it; we explain how in Can you trust your LLM judge?
Grade outcomes, not paths
It's tempting to assert an exact tool sequence: search, then fetch, then refund. Agents often find another valid route, and a path check fails them for it. Grade the outcome, and add transcript checks only for things that must happen regardless of route, such as confirming identity before a refund or never calling a destructive tool without approval. The guide puts it plainly: it's often better to grade what the agent produced, not the path it took.[1]
4. Run every task more than once
One trial per task measures whether the agent can do the task. Several trials measure whether it will. τ-bench's authors found GPT-4o succeeded on fewer than half their tasks and that pass^8, the chance of succeeding in all eight trials, fell below 25% in retail.[2] Report pass@1 and pass^k together; if pass^k falls much faster than pass@1, the problem is consistency. The full arithmetic is in pass@k vs pass^k.
Isolate trials. Each one should start from a clean environment, or shared state from a previous run will make failures correlated and scores misleading.[1] Budget for the multiplication:
40 tasks × 5 trials = 200 agent runs × ~8 model calls per run = 1,600 calls + 200 judge calls (one per trial) = 1,800 calls per eval pass
5. Measure cost and latency next to quality
An agent that succeeds in 40 tool calls and 90 seconds may be worse than one that succeeds slightly less often in four calls. Record, per trial: tool calls, tokens, wall-clock time and dollars. Report latency as percentiles (p50 and p95), not a mean, because a few long loops drag the mean around. Anthropic's advice on building agents is to add complexity only when it measurably improves outcomes,[4] and you can only apply that rule if cost is on the same dashboard as accuracy.
6. Read the transcripts
Scores tell you that something failed. Transcripts tell you why. Read a sample of failures every time the score moves, and a few passes too: graders get fooled, and an agent that passes by luck or by gaming a weak check looks the same as one that passes on merit. Hamel Husain's widely shared advice on evals makes looking at your data the first and most neglected step,[5] and Anthropic's guide ends on the same instruction.[1]
A 0% pass rate across many trials usually means a broken task (an impossible criterion, a missing tool, a bad environment) rather than an incapable agent. Fix the task before you fix the agent.
7. Split capability evals from regression evals
- Capability evals are hard on purpose. They start with low pass rates and show whether a change helps.
- Regression evals should sit near 100%. They run on every change and catch backsliding.
When a capability task passes reliably, graduate it into the regression suite.[1] Compare versions on the same tasks and test the difference with a paired test such as McNemar's, rather than eyeballing two percentages.
8. Keep evaluating after launch
Production traffic drifts away from any test set. Log real requests, sample them for review, turn new failures into tasks, and let user feedback flag what to look at. The eval set is a living document; the agent's real job keeps changing.
To go deeper, take the free Evaluation & Trust course in the Kojen Academy.



