Benchmarking 19 LLMs as ML Paper Reviewers: Parameter Count and Price Don't Predict Performance

An intriguing question was recently put to a rigorous test by a developer: can large language models serve as reviewers for machine learning papers — catching methodological flaws in experiments while staying restrained when a paper is actually sound? This benchmark, submitted to the Kaggle Benchmarking Challenge, covered 19 valid models from Anthropic, OpenAI, Google, Qwen, xAI, Z.ai, and DeepSeek. The conclusion is striking: a model's parameter count, price, and release date are all poor predictors of its reviewing performance.

Experimental Design: Why This Is More Credible Than a Typical Benchmark

The real value of this benchmark lies not in the scores themselves, but in how the experimental design sidesteps common pitfalls. The author constructed test cases around 7 categories of validity flaws commonly found in ML research:

• Using synthetic data as real-world evidence
• Test-set leakage
• Overgeneralizing from small samples
• Reporting accuracy on class-imbalanced data
• Degenerate confidence intervals
• Uncorrected multiple comparisons
• Cherry-picking seeds via best-of-N selection

The most critical design choice is the use of matched pairs. Each flaw category has a "flawed version" and a "structurally identical but problem-free twin." The two versions differ only in the one critical detail — for example, "tested 30 subgroups without correction" is paired with "pre-specified a single subgroup." This prevents models from gaming the benchmark through surface-level pattern matching on phrasing, while the "healthy twin" versions specifically measure false alarm rates.

Each question is also run twice: once with a neutral framing, and once with an added prompt indicating "the author considers this result settled," to test whether models get swayed by an author's confident tone. In total, each model answers 112 questions (7 flaw categories × 2 versions × 4 repetitions × 2 conditions), with half flawed and half healthy. This means a model that mechanically answers "none" every time scores a baseline of 0.50.

---

Test-set leakage occurs when information from the test set somehow seeps into the training process — during model training or hyperparameter tuning — artificially inflating test performance. Common forms include: using test data for feature engineering, repeatedly tuning on a validation set and then treating it as an independent test set, and inadvertently using test set statistics (e.g., mean, variance) to normalize the training set during preprocessing. The danger of leakage is that it often goes unnoticed — sometimes even the paper's own authors are unaware their pipeline is compromised — yet it makes a model that cannot generalize appear to perform excellently. In real-world deployment, overfitting from leakage is immediately exposed, making it a major contributor to the reproducibility crisis in research.

The multiple comparisons problem is a classic trap in statistical significance testing. When researchers run multiple hypothesis tests on the same dataset simultaneously, even with a significance threshold of 0.05 per test, the probability of encountering at least one false positive rises sharply as the number of tests increases. For example, running 20 independent tests when all null hypotheses are true yields, on average, about one result with p < 0.05 purely by chance. Common correction methods include the Bonferroni correction (dividing the threshold by the number of tests) and the Benjamini-Hochberg method (controlling the false discovery rate). The frequent ML paper pattern of "testing across multiple datasets/subgroups and reporting only significant results" is a textbook example of this problem.

---

Results: Crowded at the Top, Hovering Near Baseline at the Bottom

Final scores across the 19 models ranged from 0.55 to 1.00. The top of the leaderboard is remarkably tight:

Rank	Model	Score
1	Qwen 3 235B A22B Instruct	1.00
2	GPT-5.4 mini	0.98
3	GPT-5.5 / Claude Haiku 5.5	0.95
5	Gemini 3.1 Flash-Lite / Pro Preview	0.94
7	Claude Opus 5	0.93
8	Gemma 4 26B / Gemini 3.5 Flash-Lite / 3.7 Flash	0.92
12	Claude Sonnet 5	0.88
18	Claude Haiku 4.5	0.68
19	Gemma 4 31B	0.55

11 of the 19 models scored above 0.91. The author honestly notes that with a sample size of 112 questions, one question equals roughly 0.9 points, and the 95% confidence interval near 0.95 is approximately ±4 points — meaning the difference between 0.94 and 0.92 is statistically indistinguishable. When he re-ran Claude Haiku 5.5, he got 0.946, while the Kaggle platform successively showed 0.96 and 0.95 — run-to-run random noise alone can shift scores by one or two points. This candid acknowledgment of uncertainty actually makes the results more trustworthy.

Core Finding: Scale, Price, and Generation Don't Explain Performance

The most counterintuitive conclusion is that bigger, more expensive, and newer models don't necessarily perform better.

GPT-5.4 mini (0.98) outperformed GPT-5.5 (0.95); Claude Haiku 5.5 (0.95) also beat Claude Sonnet 5 (0.88). But this doesn't mean "smaller models are better" — within OpenAI's 5.4 series, mini scored 0.98 while nano only reached 0.76. What seems to actually drive performance is specific training quality, not any single external metric.

Generational jumps are equally striking. Claude Haiku leapt from 0.68 at version 4.5 to 0.95 at version 5.5 — a full 27-point improvement. This suggests that for a specific reviewing task, one model iteration can deliver a capability leap far exceeding what raw parameter scaling provides.

The Qwen series spans nearly the entire score range: Qwen 3 235B Instruct achieved a perfect 112/112, while Qwen 3 Next 80B Instruct scored only 0.70, with its thinking variant rebounding to 0.86. The author cautiously notes that since each was a single run, the gain from thinking mode is only "suggestive" — but the magnitude is substantial.

Perfect Score ≠ Perfection: The Benchmark Stepped into Its Own Trap

One of the most self-referential details in this evaluation: Qwen 3 235B scored a perfect 112/112, and the normal approximation confidence interval came out as [1.00, 1.00] — which is exactly the "degenerate confidence interval" trap the author built as one of the 7 flaw categories. Switching to the Wilson interval yields a true range of approximately 0.97 to 1.00. The author uses this to caution: a perfect score on 112 questions doesn't prove a model is truly flawless.

At the bottom, the situation is reversed. Gemma 4 31B's score of 0.55 barely clears the "always answer none" baseline of 0.50. The author admits he hasn't examined its specific answers yet, so he can't tell whether it missed flaws, over-predicted "none," or simply had trouble with the answer format.

---

This highlights a fundamental difference between two confidence interval approaches. The normal approximation interval (also called the Wald interval) directly applies the formula p ± z·√(p(1-p)/n) using the sample proportion p and sample size n. When p approaches 0 or 1, the variance term p(1-p) approaches zero and the interval collapses to a point, giving a false sense of "complete certainty" — exactly the scenario the author designed as a flaw case in this benchmark. The Wilson interval adjusts by estimating the posterior of the true proportion, yielding a reasonable uncertainty range even at extreme proportions. It is considered the better choice for small samples or extreme proportions. The fact that Qwen 3 235B's perfect score caused the normal approximation interval to collapse to exactly [1.00, 1.00] is simultaneously a testament to the model's capability and a perfect teaching moment: evaluation tools themselves require methodological scrutiny.

---

Engineering Pitfalls: Reasoning Tokens Ate the Output Budget

The author also shared technical lessons from running the benchmark — valuable for anyone building their own LLM evaluations.

During the first full run of Gemini 2.5 Flash, 24 of the 112 questions failed. The cause: the output limit was set to 1,500 tokens, and Flash spent roughly 1,440 of them on reasoning, leaving the structured answer truncated and unparseable. Worse, these failures weren't randomly distributed — they were concentrated in "healthy questions under the confident condition," which is precisely the scenario the entire benchmark cares most about. Simply dropping these failures would have artificially inflated the score.

The fix was to increase the token limit and add retry logic, while keeping a guardrail: if too many questions failed, the model would receive no score at all. But this fix has its own cost — questions where all retries failed are counted as incorrect (invalid) rather than dropped. This keeps the question count fixed but means infrastructure-level failures get charged to the model. The author suspects Grok 4.5's score of 0.00 is likely due to exactly this, though he hasn't confirmed it definitively.

Limitations and Next Steps

The author is candid about his methodology's limitations. Questions come from templates with only a few variants per flaw category — strong models might be recognizing template patterns rather than genuinely reviewing. The 112-question sample size is also small. He had originally intended to decompose scores into three dimensions — missing real flaws, false-alarming on healthy twins, and being swayed by confident author prompts — but ran out of time, so it's currently impossible to explain why low-scoring models underperformed.

The data foundation is already in place, though: every question is labeled with variant, condition, and flaw type. Planned next steps include completing that decomposition analysis, releasing an open-ended version without a label list, incorporating methodology sections from real retracted papers, and designing more subtle and challenging flaws to prevent top models from clustering at the ceiling.

For anyone looking to use LLMs as research assistants or reviewing tools, this benchmark sends a clear signal: you can't select a model based solely on parameter counts and pricing charts — task-specific empirical testing is the only reliable basis for selection. And the author's forthright disclosure of noise, confidence intervals, and engineering failures serves as a model for how to conduct a credible LLM benchmark.