Exact-match grading works when the answer is a label: billing, bug, how-to. Agents rarely give you labels. They write summaries, draft replies, explain a refund policy in their own words. Grading that by hand doesn't scale, and string metrics like BLEU and ROUGE correlate poorly with human judgement on open-ended text.[1]
So most teams now use a strong model as the grader, an approach known as LLM-as-a-judge. It works, with caveats. Here's what the research says about both.
The case for an AI judge
The foundational study is Judging LLM-as-a-Judge from the LMSYS team behind Chatbot Arena. They found that strong judges like GPT-4 matched both controlled and crowdsourced human preferences with over 80% agreement, about the same rate at which humans agree with each other.[2] An AI judge isn't perfect, but neither is a single human reviewer, and the AI judge can grade ten thousand answers overnight.
Structure helps. G-Eval had GPT-4 reason through an evaluation step by step and fill in a scoring form, rather than giving a bare number. On summarisation it reached a Spearman correlation of 0.514 with human ratings, well ahead of earlier automatic metrics.[1]
The known failure modes
The same research that validates AI judges also documents how they go wrong:
- Position bias. When comparing two answers, judges favour one slot. Wang et al. showed that simply swapping the order let Vicuna-13B “beat” ChatGPT on 66 of 80 queries with ChatGPT as the judge.[3]
- Verbosity bias. Longer answers look more thorough and get rated higher, even when they aren't better.[2]
- Self-enhancement. Judges tend to prefer answers that read like their own output.[2][1]
- Limited reasoning. On maths and logic, a judge can be talked into accepting a confident wrong answer.[2]
How to make a judge trustworthy
- Grade against a reference. Give the judge the expected answer, not just the question. It turns an open-ended opinion into a comparison.
- Ask for the reasoning before the verdict. Wang et al.'s “multiple evidence calibration” has the judge produce its evidence before assigning a score.[3]
- Grade one answer at a time. If you must compare, run both orders and aggregate, which Wang et al. call balanced position calibration.[3]
- Write an explicit rubric. “Is this good?” invites the judge's taste. “Does it cite the policy section and avoid promising a refund?” doesn't.
- Spot-check the judge. Read a sample of verdicts every run. When you disagree with the judge, that's a rubric bug.
Bias needs its own test
A model can be accurate on average and still treat people differently. The standard way to detect that is with paired prompts: two questions identical except for one demographic attribute, where any change in the answer is a signal. The BBQ benchmark (Parrish et al.) built question sets around attested social biases across nine social dimensions for exactly this purpose.[4]
What Kojen's evaluations do
Every Kojen evaluation runs your model on real test cases, measures p50/p95/p99 latency on those real calls, and has an LLM judge give each answer a verdict. You pick the lens:
- Accuracy: correct versus the expected answer.
- Hallucination: fabricated or unsupported claims.
- Bias: paired prompts that differ only in a demographic attribute.
- Custom: your own rubric. Criteria are required, for the reasons above.
- Comprehensive: all three core checks together.
The trust score
Each evaluation rolls up into a trust score from 0 to 100. For comprehensive evaluations, accuracy is weighted 50%, hallucination 30% and bias 20%. It's a single number for dashboards and gates, but the per-answer verdicts are kept, so you can always open one up and check the judge's reasoning.
