Paired Design
A statistical experimental design method that pairs subjects to eliminate noise from individual differences, focusing on the net effect of two treatments. In machine learning, this corresponds to sample-level comparisons on the same test set, enabling direct model-to-model evaluation.
Timeline (last 90 days)
The paired design feeds the same case to two models separately and compares the pass/fail status on the same case, eliminating the variance caused by differences in case difficulty.
Under a paired design, any routing jitter, temperature sampling or re-running of unstable endpoints degrades paired data into unpaired data, invalidating the premises of the McNemar test.
Under a paired design, each case is judged only once per fixed backend (or by majority vote after k repetitions), and both legs are guaranteed to use exactly the same prompt.
All Facts (3)
The paired design feeds the same case to two models separately and compares the pass/fail status on the same case, eliminating the variance caused by differences in case difficulty.
50%UnverifiedUnder a paired design, any routing jitter, temperature sampling or re-running of unstable endpoints degrades paired data into unpaired data, invalidating the premises of the McNemar test.
50%UnverifiedUnder a paired design, each case is judged only once per fixed backend (or by majority vote after k repetitions), and both legs are guaranteed to use exactly the same prompt.
50%