Skip to content

Glossary · Evaluation

LLM-as-a-judge

LLM-as-a-judge is the practice of using a language model to grade another model's outputs against a rubric or a reference answer, instead of a human grader.

Also called LLM judge, AI grader · Reviewed 15 November 2026

Zheng et al. found strong judges such as GPT-4 agreed with human preferences over 80% of the time, about as often as humans agree with each other. Judges also show biases: they favour the first answer shown (position bias), longer answers (verbosity bias) and their own outputs (self-enhancement).

Mitigations include swapping answer order, specific rubrics, reference answers, and checking agreement against a human-labelled sample, for example with Cohen's kappa.

Learn more on Kojen

Sources

  1. [1]Zheng et al., 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv:2306.05685)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free