McNemar's Test
A statistical test method for paired binary classification data, suitable for comparing whether the performance difference between two models on the same batch of samples is significant.
Timeline (last 90 days)
The comparison used Wilson confidence intervals and a paired McNemar test with Holm-Bonferroni correction.
After decontamination, the model achieved 22.6% on HumanEval, 21.3% on HumanEval+, and 35.0% on MBPP; compared with the published version, HumanEval dropped by 4.3 points, McNemar test p = 0.039.
Determining whether a model truly outperforms the baseline usually requires larger samples (hundreds to thousands), paired tests (such as the McNemar test), and repeated experiments under multiple random seeds.
All Facts (3)
The comparison used Wilson confidence intervals and a paired McNemar test with Holm-Bonferroni correction.
50%UnverifiedAfter decontamination, the model achieved 22.6% on HumanEval, 21.3% on HumanEval+, and 35.0% on MBPP; compared with the published version, HumanEval dropped by 4.3 points, McNemar test p = 0.039.
50%UnverifiedDetermining whether a model truly outperforms the baseline usually requires larger samples (hundreds to thousands), paired tests (such as the McNemar test), and repeated experiments under multiple random seeds.
50%