McNemar's Test for Comparing ML Models: Proper Usage Under Random Seeds

The Problem

In machine learning research, one commonly overlooked step is: how do you scientifically prove that "your proposed method genuinely outperforms the baseline"? Many papers simply present an accuracy comparison table — say, "our method achieves 94.2% versus the baseline's 92.8%" — and declare the improvement valid. But this approach ignores a critical question: is that 1.4-percentage-point difference a real performance gain, or just a result of random fluctuation?

A Reddit user raised a very specific question: when a paper uses random seeds across multiple experimental runs, how exactly do researchers apply McNemar's test to compare a baseline against a proposed method? This seemingly minor detail cuts to the heart of statistical significance testing in ML experimental design.

What McNemar's Test Actually Tests

Core Principle and Historical Background

McNemar's test was introduced by American psychologist Quinn McNemar in 1947, originally for pre-post comparisons in social science research — for example, assessing whether an intervention changed participants' attitudes. Its core idea is rooted in the statistical philosophy of matched-pair design: by eliminating noise from individual differences, you isolate the net effect between two treatments. This logic translates naturally to machine learning, where each sample in the test set serves as a natural "paired unit" without requiring additional matching.

McNemar's test is a statistical test for paired nominal data, particularly suited for comparing two classifiers on the same test set. Its core idea is not to compare the overall accuracy of two models, but to focus on where the two models disagree on individual samples.

Specifically, for each sample in the test set, there are four possible outcomes, forming a 2×2 contingency table:

	Model B Correct	Model B Wrong
Model A Correct	a	b
Model A Wrong	c	d

The informative cells are b and c — the number of samples where the two models disagree. Here, b represents samples where "A is correct but B is wrong," and c represents samples where "A is wrong but B is correct." Cells a and d (where both agree) are treated as non-informative, since they cannot distinguish the two models. This focus on "discordant pairs" makes McNemar's test highly efficient in its use of sample data: even when overall accuracy is similar, asymmetry in the discordant distribution can reveal significant differences.

The Test Statistic

The McNemar test statistic is built on the difference between b and c. With sufficient sample size, it approximately follows a chi-squared distribution with 1 degree of freedom:

χ² = (|b - c| - 1)² / (b + c)

The -1 is a continuity correction (Yates' correction), which improves the approximation when mapping discrete data to the continuous chi-squared distribution. When b and c are small (typically b+c < 25), an exact binomial test should be used instead (treating b as an observation from a binomial distribution with p=0.5). If the p-value falls below the chosen significance level (e.g., 0.05), there is statistical evidence that the two models perform significantly differently.

The Complexity Introduced by Random Seeds

Multiple Sources of Randomness in Deep Learning

Random seeds in deep learning control multiple sources of randomness: weight initialization (e.g., the random components of He or Xavier initialization), data shuffling order during loading, random neuron dropout in Dropout layers, and stochastic gradient estimation in some optimizers (such as Adam). Together, these determine which local optimum the model converges to on the loss surface. Research shows that even with identical architectures and hyperparameters, models trained with different random seeds can exhibit natural accuracy variation of 0.5% to 2% — a range that sometimes overlaps with the "improvement" claimed in papers. This is precisely why statistical testing is indispensable.

The Core Tension

This is exactly where the original question gets tricky. The standard McNemar's test assumes we are comparing two deterministic models on a single fixed test set. But in modern deep learning experiments, model training typically involves random seeds — different seeds lead to different weight initializations, data shuffling, dropout patterns, and so on, ultimately producing multiple slightly different model instances.

This creates a fundamental tension: McNemar's test handles the divergence in model predictions, while random seeds introduce variability in the training process itself. These two sources of uncertainty operate at conceptually different levels.

Common Practices in Research

In practice, researchers typically adopt one of the following strategies to reconcile this tension:

Approach 1: Fixed-Seed Single-Run Paired Test

The simplest approach is to fix a seed for each model, train two deterministic models, and run McNemar's test on the same test set. This strictly satisfies the original assumptions of McNemar's test, but the drawback is clear — it only reflects results under one specific seed and cannot demonstrate whether the improvement holds across different random initializations.

Approach 2: Multi-Seed Repetition + Separate Statistical Framework

More rigorous papers train both models under multiple random seeds (e.g., 5 or 10), obtaining multiple accuracy measurements, then use a paired t-test or Wilcoxon signed-rank test to compare the distributions of those accuracy scores.

Notably, the paired t-test assumes that pairwise differences are normally distributed and works best with larger sample sizes (typically n > 30) or when normality can be assumed. The Wilcoxon signed-rank test is the non-parametric alternative, making no assumptions about the shape of the distribution — only that it is symmetric. In multi-seed ML experiments where repetition counts are often just 5 to 10, normality is hard to verify, making the Wilcoxon test a more conservative and robust choice. Some researchers also use permutation tests as a distribution-free alternative, directly estimating the null distribution of the test statistic from the data.

Importantly, in this scenario, McNemar's test is typically no longer the right tool, because the test object shifts from "prediction disagreements on individual samples" to "a sequence of accuracy scores across multiple runs."

Approach 3: Hybrid Reporting

Some papers report both: McNemar's test to illustrate sample-level prediction discrepancies under a representative seed, alongside mean ± standard deviation across multiple seeds to demonstrate stability. This approach balances statistical rigor with practical transparency.

How to Choose the Right Test

Clarify Your Research Question

The key is to be clear about what question you're trying to answer:

• If the question is "do these two specifically trained models show significantly different prediction behavior," then McNemar's test is appropriate — fix the seed and use the same test set.
• If the question is "is my method generally better than the baseline, accounting for training randomness," then multi-seed experiments paired with a paired t-test or non-parametric test are the right approach.

A Practical Recommendation

For most research scenarios, a combined strategy is recommended:

1. Train both the baseline and proposed method under multiple random seeds (at least 5 recommended)
2. Report mean and standard deviation of accuracy for each model; use a paired test to evaluate the significance of the overall difference
3. For one representative seed (e.g., the run with median performance), additionally run McNemar's test to illustrate the sample-level prediction disagreement pattern
4. Clearly state in the paper the testing method used, the significance level, and any multiple comparison corrections applied (if applicable)

This approach satisfies reviewers' expectations for statistical rigor while clearly explaining the source and stability of the performance improvement.

Easily Overlooked Pitfalls

There are several common mistakes to watch out for when applying statistical tests.

Multiple comparisons problem: When comparing multiple models or multiple datasets simultaneously, multiple comparison corrections are necessary — otherwise you will systematically overestimate significance. Bonferroni correction is the most conservative option, dividing the significance threshold by the number of comparisons (e.g., dropping from 0.05 to 0.01 for 5 comparisons). Benjamini-Hochberg FDR (false discovery rate) correction is more lenient, controlling the proportion of false positives among all significant results rather than the error rate per individual test — it is more commonly used in exploratory research. Unfortunately, multiple comparison corrections remain underused in machine learning papers, representing a systematic weakness in the field's statistical practice.

Test set independence: McNemar's test requires that both models be evaluated on exactly the same test set. Any inconsistency invalidates the test.

Statistical significance ≠ practical significance: Even with a very small p-value, if the actual improvement is negligible (e.g., 0.1%), its engineering value may still be limited. Statistical conclusions must be interpreted in the context of the actual application. In statistical terms, this distinction corresponds to statistical power versus effect size — the former measures the ability to detect a true difference, the latter measures the practical importance of that difference.

Summary

The question of how random seeds interact with McNemar's test fundamentally reflects a deeper issue in ML experimental design: how to strike a balance between algorithmic randomness and the deterministic assumptions of statistical inference. Understanding the preconditions for each testing method matters far more than mechanically applying formulas.

For researchers, rather than asking "what do papers typically do," it's better to start with "what am I trying to prove" — then choose the statistical tool that truly matches the research question, and transparently report methodological details in the paper.

Key Takeaways