Skip to content

Glossary · Evaluation

McNemar test

McNemar's test checks whether two classifiers, prompts or models differ in accuracy when both are run on the same cases, using only the cases where exactly one of them is correct.

Also called McNemar's test · Reviewed 15 November 2026

Cases both get right or both get wrong say nothing about which is better, so the test compares the two kinds of disagreement: A-only correct versus B-only correct. Below about 25 disagreements an exact binomial test is used; above it, a chi-square approximation.

Paired testing is far more sensitive than comparing two separate confidence intervals, which is why it is the right test for prompt A/B comparisons.

Formula

χ² = (|b − c| − 1)² / (b + c)

Learn more on Kojen

Sources

  1. [1]McNemar, Q. (1947), Psychometrika. Note on the sampling error of the difference between correlated proportions or percentages
  2. [2]Miller, 2024. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv:2411.00640)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free