Skip to content

Evaluation

Can you trust your LLM judge? Measuring agreement with humans

Why percent agreement misleads, Cohen's kappa worked by hand, sensitivity and specificity of a judge, correcting a pass rate for an imperfect judge, and a six-step calibration procedure.

By Will Schulz9 min read
Two glass tuning forks on a dark metal base with red waves of light between them, almost perfectly overlapping

An LLM judge turns an eval into a number you can track, but only if the judge agrees with the people whose standard it stands in for. Most teams check that by eyeballing a few verdicts, or by quoting percent agreement. Both can be badly misleading. This article shows how to measure a judge properly with a small labelled sample, using Cohen's kappa, sensitivity and specificity, and how to correct a score for a judge you know is imperfect.

Why percent agreement isn't enough

Suppose 95% of your agent's answers are good. A judge that says “pass” to everything agrees with people 95% of the time while detecting nothing. Thakur et al., who compared many judge models against human annotators, found that judges with high percent agreement can still assign very different scores, and argued for metrics beyond simple agreement.[1] The standard fix is to correct for the agreement you'd get by chance.

Cohen's kappa, worked

Cohen's kappa compares observed agreement with the agreement expected if the two raters labelled independently at their own base rates.[2] A kappa of 1 is perfect agreement, 0 is no better than chance.

200 answers, each labelled pass/fail by a person and by the judge

                   human pass   human fail   total
judge pass             130           20        150
judge fail              10           40         50
total                  140           60        200

observed agreement  pₒ = (130 + 40) / 200                    = 0.85
chance agreement    pₑ = (150/200)(140/200) + (50/200)(60/200)
                       = 0.525 + 0.075                       = 0.60
Cohen's kappa       κ  = (pₒ − pₑ) / (1 − pₑ) = 0.25 / 0.40  = 0.625

Eighty-five percent agreement sounds strong; a kappa of 0.63 is a more honest “substantial, not excellent”. And the lenient judge from above:

A lenient judge that passes everything, on a set where people pass 95%:

pₒ = 0.95    pₑ = (1.00)(0.95) + (0)(0.05) = 0.95    κ = 0

95% agreement, and the judge has told you nothing.

That's the well-known kappa paradox: when one class dominates, kappa can be low even when agreement is high.[3] Report kappa together with the full 2×2 table, not instead of it.

How good is good enough?

KappaLandis and Koch label
below 0.00Poor
0.00–0.20Slight
0.21–0.40Fair
0.41–0.60Moderate
0.61–0.80Substantial
0.81–1.00Almost perfect

These bands come from Landis and Koch,[4] who called them arbitrary themselves; they're conventions, not thresholds of truth. A better benchmark is your own human-to-human agreement: have two people label the same 50 answers. If they reach kappa 0.7, a judge at 0.65 is nearly as good as a second person. If they only reach 0.4, the rubric is ambiguous and no judge will fix that.

For context, the MT-Bench study found strong LLM judges matched human preferences about as well as humans matched each other, at over 80% agreement on pairwise comparisons.[5] That was for general chat quality. Your task, rubric and judge will differ, which is the reason to measure.

Look at both error types

Kappa is one number for two different mistakes. Split them:

  • Sensitivity: of the answers people pass, how many the judge passes. Low sensitivity means a harsh judge, which makes a good model look bad.
  • Specificity: of the answers people fail, how many the judge fails. Low specificity means a lenient judge, which lets regressions through. Thakur et al. found leniency was a common judge failure.[1]

For a release gate you usually care most about specificity: you want the judge to catch bad answers. In the example, it misses a third of them.

Correct the score when the judge is imperfect

If you know a judge's sensitivity and specificity, you can estimate the true pass rate from the judge's observed rate. Epidemiologists use the same correction for imperfect screening tests:[6]

From the table above:
sensitivity (judge passes what people pass)  = 130 / 140 = 0.929
specificity (judge fails what people fail)   =  40 / 60  = 0.667

On a new run the judge passes 80% of answers. Corrected estimate
(Rogan–Gladen):
true rate = (observed + specificity − 1) / (sensitivity + specificity − 1)
          = (0.80 + 0.667 − 1) / (0.929 + 0.667 − 1)
          = 0.467 / 0.596 ≈ 0.78

The judge said 80%; the corrected estimate is about 78%. The correction is only as good as the sensitivity and specificity estimates, which carry their own uncertainty with 140 and 60 labelled cases, so treat it as a sanity check rather than a replacement number.

A calibration procedure

  1. Sample 100–200 real outputs, including clear passes, clear fails and borderline cases. Don't label only failures; you need both classes.
  2. Have people label them blind to the judge's verdict, using the same rubric. Have a second person label 50 of them to measure human agreement.
  3. Run the judge on the same outputs and compute kappa, sensitivity and specificity.
  4. Read every disagreement. They usually trace to a vague rubric line, a missing reference answer, or a bias such as preferring longer answers.
  5. Fix the rubric, not just the judge prompt, and re-run on a fresh sample so you don't overfit the judge to the calibration set.
  6. Recalibrate whenever you change the judge model, the rubric, or the model under test. Shankar et al. found that people's grading criteria themselves shift as they see more outputs, so calibration is a loop, not a one-off.[7]
from sklearn.metrics import cohen_kappa_score, confusion_matrix

human = [1, 1, 0, 1, 0, ...]   # your labels
judge = [1, 1, 1, 1, 0, ...]   # the judge's verdicts, same order

print(cohen_kappa_score(human, judge))
tn, fp, fn, tp = confusion_matrix(human, judge).ravel()
print("sensitivity", tp / (tp + fn), "specificity", tn / (tn + fp))

Practical tips

  • Grade one criterion per judge call; multi-criterion verdicts blur together. Anthropic's agent-eval guide recommends grading each dimension separately and giving the judge an “unknown” option.[8]
  • Give the judge a reference answer whenever one exists.
  • Don't use a judge from the same model family as the model under test; it may share its blind spots.
  • Ask for the reason before the verdict, and keep the reasons: they're how you debug disagreements.

For the biases that make judges disagree with people in the first place, see LLM-as-a-judge, honestly and the LLM-as-a-judge lesson.

Frequently asked questions

How do you calibrate an LLM-as-a-judge?
Have people label 100–200 real outputs blind with the same rubric, run the judge on the same outputs, and compute Cohen's kappa, sensitivity and specificity. Read every disagreement, fix the rubric, and re-check on a fresh sample.
What is Cohen's kappa?
A measure of agreement between two raters corrected for chance: κ = (pₒ − pₑ) / (1 − pₑ), where pₒ is observed agreement and pₑ is the agreement expected from each rater's base rates. 1 is perfect, 0 is chance level.
What is a good Cohen's kappa for an LLM judge?
Landis and Koch call 0.61–0.80 substantial and above 0.80 almost perfect, but the better benchmark is agreement between two of your own human labellers on the same rubric. A judge close to that level is about as good as a second person.
Why is percent agreement misleading for LLM judges?
When one class dominates, a judge can agree often by chance. If 95% of answers are good, a judge that passes everything agrees 95% of the time and has a kappa of 0.
Can I correct an eval score for an imperfect judge?
Yes, approximately. With the judge's sensitivity and specificity from a labelled sample, the Rogan–Gladen estimate is (observed + specificity − 1) / (sensitivity + specificity − 1). Treat it as a sanity check, since those inputs are estimated too.

Go deeper in the Academy

Free lessons with quizzes, citations and a hands-on exercise.

Sources

  1. [1]Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges. Thakur et al., arXiv (2024; GEM² 2025).
  2. [2]A Coefficient of Agreement for Nominal Scales. Cohen, Educational and Psychological Measurement 20(1) (1960).
  3. [3]High agreement but low kappa: I. The problems of two paradoxes. Feinstein and Cicchetti, Journal of Clinical Epidemiology 43(6) (1990).
  4. [4]The Measurement of Observer Agreement for Categorical Data. Landis and Koch, Biometrics 33(1) (1977).
  5. [5]Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Zheng et al., arXiv (2023).
  6. [6]Estimating prevalence from the results of a screening test. Rogan and Gladen, American Journal of Epidemiology 107(1) (1978).
  7. [7]Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. Shankar et al., arXiv (2024).
  8. [8]Demystifying evals for AI agents. Anthropic Engineering (9 January 2026).

Put it into practice.

Kojen gives you the Forge, LLM-judged evaluations, red-team simulations and live endpoints in one place. Free to start, no card needed, then pay as you go.

Start training free

More reading

All articles →