Skip to content

Glossary · Evaluation

Cohen's kappa

Cohen's kappa measures agreement between two raters on categorical labels, corrected for the agreement expected by chance: 1 is perfect, 0 is chance level.

Also called kappa, κ · Reviewed 15 November 2026

Raw agreement can look high when one label dominates. Kappa subtracts expected agreement: κ = (p_o − p_e)/(1 − p_e). It is a standard way to check whether an LLM judge agrees with human graders beyond chance.

There is no universal threshold; report kappa with raw agreement and the label balance.

Formula

κ = (p_o − p_e) / (1 − p_e)

Learn more on Kojen

Sources

  1. [1]Cohen, J. (1960), Educational and Psychological Measurement. A Coefficient of Agreement for Nominal Scales

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free