Paired Testing
An evaluation method that compares the outputs of two models case by case on the same set of test cases to determine agreement. By analyzing discordant pairs, it detects statistically significant performance differences and prevents average scores from masking regressions.
Timeline (last 90 days)
The paired test revealed a systematic bias: some models claim that a better alternative exists regardless of whether the optimal plan has already been provided.
The paired test is an experimental design: two variants are constructed for the same scenario, one containing the optimal plan and the other removing it, and the model is asked to judge 'whether a better solution exists'.
Both backends score 71% on the same frozen test set, but a case-by-case comparison may reveal that nine cases got worse and nine got better, with the average masking these flips.
Paired testing focuses only on discordant pairs, so even when two runs have the same average score, it can still detect a statistically significant regression.
All Facts (4)
The paired test revealed a systematic bias: some models claim that a better alternative exists regardless of whether the optimal plan has already been provided.
50%UnverifiedThe paired test is an experimental design: two variants are constructed for the same scenario, one containing the optimal plan and the other removing it, and the model is asked to judge 'whether a better solution exists'.
50%UnverifiedBoth backends score 71% on the same frozen test set, but a case-by-case comparison may reveal that nine cases got worse and nine got better, with the average masking these flips.
50%UnverifiedPaired testing focuses only on discordant pairs, so even when two runs have the same average score, it can still detect a statistically significant regression.
50%