Cohen's kappa
Cohen's kappa measures agreement between two raters on categorical labels, corrected for the agreement expected by chance: 1 is perfect, 0 is chance level.
Raw agreement can look high when one label dominates. Kappa subtracts expected agreement: κ = (p_o − p_e)/(1 − p_e). It is a standard way to check whether an LLM judge agrees with human graders beyond chance.
There is no universal threshold; report kappa with raw agreement and the label balance.
Formula
κ = (p_o − p_e) / (1 − p_e)
Learn more on Kojen
Related terms
- LLM-as-a-judgeLLM-as-a-judge is the practice of using a language model to grade another model's outputs against a rubric or a reference answer, instead of a human grader..
- Confidence intervalA confidence interval is a range computed from a sample that, across repeated samples, would contain the true value a stated share of the time, such as 95%.
Sources
- Cohen, J. (1960), Educational and Psychological Measurement. A Coefficient of Agreement for Nominal Scales
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free