McNemar test
McNemar's test checks whether two classifiers, prompts or models differ in accuracy when both are run on the same cases, using only the cases where exactly one of them is correct.
Cases both get right or both get wrong say nothing about which is better, so the test compares the two kinds of disagreement: A-only correct versus B-only correct. Below about 25 disagreements an exact binomial test is used; above it, a chi-square approximation.
Paired testing is far more sensitive than comparing two separate confidence intervals, which is why it is the right test for prompt A/B comparisons.
Formula
χ² = (|b − c| − 1)² / (b + c)
Learn more on Kojen
Related terms
- Confidence intervalA confidence interval is a range computed from a sample that, across repeated samples, would contain the true value a stated share of the time, such as 95%.
- Wilson intervalThe Wilson score interval is a confidence interval for a proportion, such as a pass rate, that stays within 0–100% and keeps close to its stated coverage even for small samples and rates near 0 or 100%..
- LLM-as-a-judgeLLM-as-a-judge is the practice of using a language model to grade another model's outputs against a rubric or a reference answer, instead of a human grader..
Sources
- McNemar, Q. (1947), Psychometrika. Note on the sampling error of the difference between correlated proportions or percentages
- Miller, 2024. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv:2411.00640)
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free