McNemar's Test
A statistical test proposed by psychologist Quinn McNemar in 1947 for paired nominal data. It is particularly suited for comparing the performance of two classifiers on the same test set, assessing the significance of differences by analyzing samples on which the two models disagree.
Timeline (last 90 days)
The exact McNemar test compares binary outcomes of two related samples using only the discordant pairs (the number of gains b and the number of losses c); under the null hypothesis b and c follow a binomial distribution with success probability 0.5, and the exact version directly computes binomial tail probabilities rather than the large-sample chi-square approximation, still giving reliable p-values when b+c is small.
The McNemar test is applicable to paired binary data, and the p=0.00006 (15 fixes, 0 breaks across the three packages combined) and p=0.03 (6 fixes, 0 breaks on the check packages only) reported in this paper both come from that test.
AUC is a metric measuring the ranking ability of a binary classification model, ranging from 0.5 to 1, and does not depend on a specific threshold; the McNemar test is a statistical test for paired binary classification data, with p<0.05 generally considered significant.
All Facts (6)
该项目采用配对McNemar检验作为统计显著性验证方法
90%UnverifiedMcNemar's test is a non-parametric test method specifically designed for paired binary classification data
60%UnverifiedThe exact McNemar test compares binary outcomes of two related samples using only the discordant pairs (the number of gains b and the number of losses c); under the null hypothesis b and c follow a binomial distribution with success probability 0.5, and the exact version directly computes binomial tail probabilities rather than the large-sample chi-square approximation, still giving reliable p-values when b+c is small.
50%UnverifiedThe McNemar test is applicable to paired binary data, and the p=0.00006 (15 fixes, 0 breaks across the three packages combined) and p=0.03 (6 fixes, 0 breaks on the check packages only) reported in this paper both come from that test.
50%UnverifiedAUC is a metric measuring the ranking ability of a binary classification model, ranging from 0.5 to 1, and does not depend on a specific threshold; the McNemar test is a statistical test for paired binary classification data, with p<0.05 generally considered significant.
50%UnverifiedMcNemar 检验要求两个模型在完全相同的测试集上评估，任何测试集的不一致都会使检验失效
50%