An LLM judge turns an eval into a number you can track, but only if the judge agrees with the people whose standard it stands in for. Most teams check that by eyeballing a few verdicts, or by quoting percent agreement. Both can be badly misleading. This article shows how to measure a judge properly with a small labelled sample, using Cohen's kappa, sensitivity and specificity, and how to correct a score for a judge you know is imperfect.
Why percent agreement isn't enough
Suppose 95% of your agent's answers are good. A judge that says “pass” to everything agrees with people 95% of the time while detecting nothing. Thakur et al., who compared many judge models against human annotators, found that judges with high percent agreement can still assign very different scores, and argued for metrics beyond simple agreement.[1] The standard fix is to correct for the agreement you'd get by chance.
Cohen's kappa, worked
Cohen's kappa compares observed agreement with the agreement expected if the two raters labelled independently at their own base rates.[2] A kappa of 1 is perfect agreement, 0 is no better than chance.
200 answers, each labelled pass/fail by a person and by the judge
human pass human fail total
judge pass 130 20 150
judge fail 10 40 50
total 140 60 200
observed agreement pₒ = (130 + 40) / 200 = 0.85
chance agreement pₑ = (150/200)(140/200) + (50/200)(60/200)
= 0.525 + 0.075 = 0.60
Cohen's kappa κ = (pₒ − pₑ) / (1 − pₑ) = 0.25 / 0.40 = 0.625Eighty-five percent agreement sounds strong; a kappa of 0.63 is a more honest “substantial, not excellent”. And the lenient judge from above:
A lenient judge that passes everything, on a set where people pass 95%: pₒ = 0.95 pₑ = (1.00)(0.95) + (0)(0.05) = 0.95 κ = 0 95% agreement, and the judge has told you nothing.
That's the well-known kappa paradox: when one class dominates, kappa can be low even when agreement is high.[3] Report kappa together with the full 2×2 table, not instead of it.
How good is good enough?
| Kappa | Landis and Koch label |
|---|---|
| below 0.00 | Poor |
| 0.00–0.20 | Slight |
| 0.21–0.40 | Fair |
| 0.41–0.60 | Moderate |
| 0.61–0.80 | Substantial |
| 0.81–1.00 | Almost perfect |
These bands come from Landis and Koch,[4] who called them arbitrary themselves; they're conventions, not thresholds of truth. A better benchmark is your own human-to-human agreement: have two people label the same 50 answers. If they reach kappa 0.7, a judge at 0.65 is nearly as good as a second person. If they only reach 0.4, the rubric is ambiguous and no judge will fix that.
For context, the MT-Bench study found strong LLM judges matched human preferences about as well as humans matched each other, at over 80% agreement on pairwise comparisons.[5] That was for general chat quality. Your task, rubric and judge will differ, which is the reason to measure.
Look at both error types
Kappa is one number for two different mistakes. Split them:
- Sensitivity: of the answers people pass, how many the judge passes. Low sensitivity means a harsh judge, which makes a good model look bad.
- Specificity: of the answers people fail, how many the judge fails. Low specificity means a lenient judge, which lets regressions through. Thakur et al. found leniency was a common judge failure.[1]
For a release gate you usually care most about specificity: you want the judge to catch bad answers. In the example, it misses a third of them.
Correct the score when the judge is imperfect
If you know a judge's sensitivity and specificity, you can estimate the true pass rate from the judge's observed rate. Epidemiologists use the same correction for imperfect screening tests:[6]
From the table above:
sensitivity (judge passes what people pass) = 130 / 140 = 0.929
specificity (judge fails what people fail) = 40 / 60 = 0.667
On a new run the judge passes 80% of answers. Corrected estimate
(Rogan–Gladen):
true rate = (observed + specificity − 1) / (sensitivity + specificity − 1)
= (0.80 + 0.667 − 1) / (0.929 + 0.667 − 1)
= 0.467 / 0.596 ≈ 0.78The judge said 80%; the corrected estimate is about 78%. The correction is only as good as the sensitivity and specificity estimates, which carry their own uncertainty with 140 and 60 labelled cases, so treat it as a sanity check rather than a replacement number.
A calibration procedure
- Sample 100–200 real outputs, including clear passes, clear fails and borderline cases. Don't label only failures; you need both classes.
- Have people label them blind to the judge's verdict, using the same rubric. Have a second person label 50 of them to measure human agreement.
- Run the judge on the same outputs and compute kappa, sensitivity and specificity.
- Read every disagreement. They usually trace to a vague rubric line, a missing reference answer, or a bias such as preferring longer answers.
- Fix the rubric, not just the judge prompt, and re-run on a fresh sample so you don't overfit the judge to the calibration set.
- Recalibrate whenever you change the judge model, the rubric, or the model under test. Shankar et al. found that people's grading criteria themselves shift as they see more outputs, so calibration is a loop, not a one-off.[7]
from sklearn.metrics import cohen_kappa_score, confusion_matrix
human = [1, 1, 0, 1, 0, ...] # your labels
judge = [1, 1, 1, 1, 0, ...] # the judge's verdicts, same order
print(cohen_kappa_score(human, judge))
tn, fp, fn, tp = confusion_matrix(human, judge).ravel()
print("sensitivity", tp / (tp + fn), "specificity", tn / (tn + fp))Practical tips
- Grade one criterion per judge call; multi-criterion verdicts blur together. Anthropic's agent-eval guide recommends grading each dimension separately and giving the judge an “unknown” option.[8]
- Give the judge a reference answer whenever one exists.
- Don't use a judge from the same model family as the model under test; it may share its blind spots.
- Ask for the reason before the verdict, and keep the reasons: they're how you debug disagreements.
For the biases that make judges disagree with people in the first place, see LLM-as-a-judge, honestly and the LLM-as-a-judge lesson.



