AI Peer Review Benchmark: 100 Planted Errors, Best System Catches Only 71

100 planted errors test AI review: best single system catches 71%, ensemble reaches 93%, omission errors remain a blind spot.
A new study planted 100 known errors into 10 open-access psychology papers to benchmark AI review systems. The best single system caught 71 errors, the worst only 30; ensembling all systems raised the catch count to 93, showing different models' error detection is only partially correlated and combination improves recall. However, 7 errors were missed by every system, all "omission-type" (missing information), revealing AI is better at finding commission errors than judging missing information. Commercial tool Refine.ink contributed the most unique catches but had a high cost per error. False positives were not measured, and the difference between planted and real error distributions is unknown, suggesting AI review should currently serve as an auxiliary tool to human review.
Introduction: Can AI Peer Review Replace Humans? A Benchmark Offers a Quantitative Answer
As AI begins to participate in academic peer review, a key question emerges: how reliable is it really? A new study built a unique benchmark by planting 100 known errors into 10 open-access psychology papers, systematically evaluating cutting-edge AI models and two commercial AI review tools. The results are both surprising and cautionary.

Test Design: A Real-World Trial with 100 "Artificial Errors"
Rather than relying on conventional static Q&A or abstract comparison, the research team chose a method closer to real review scenarios: they manually planted 100 errors into 10 published open-access psychology papers, then had multiple frontier AI models and two commercial AI review tools (one of which was Refine.ink) review the modified papers.
This "planted error" approach directly tests whether a system can spot substantive errors in long-form academic text, rather than merely assessing language fluency or formatting compliance. All papers, error annotations, model outputs, and complete experimental logs have been made public, providing a reproducible foundation for future research.
Key Findings: Best Single System Catches 71%, Ensemble Reaches 93%
The results show a clear performance gradient. The best-performing single system successfully caught 71 of the 100 errors, while the worst caught only 30. This means that even top-tier AI review tools still miss nearly 30% of planted errors.
However, a more revealing finding is that when all systems' outputs were aggregated, the total catch count jumped to 93. This indicates that different models' error detection is only partially correlated—the errors each one misses do not fully overlap. As a result, ensembling multiple AI systems becomes an effective lever for improving error detection rates. For journals or research institutions, rather than relying on a single tool, it is better to combine multiple systems for cross-review.
Ensemble learning is a machine learning strategy that improves overall performance by combining predictions from multiple models. Its core premise is that different models (especially those based on different architectures or training data) make different errors, so aggregating their results can compensate for individual weaknesses. In the AI review context, ensembling typically means running multiple models on the same paper and then merging (or voting on) their lists of suspected errors. The key to ensemble gains in recall is error diversity—if all models miss the same error, the ensemble cannot help. The 7 errors missed by all systems in this study are precisely a manifestation of insufficient diversity.
Blind Spots: 7 Errors No System Could Catch
The most alarming finding is that 7 errors were not caught by any system, and all 7 were "omission-type"—meaning information that should have been present was deleted from the paper, rather than incorrect content being inserted.
This phenomenon reveals a structural weakness in current AI review systems: they are better at detecting "what errors are present" (such as data contradictions or logical fallacies) but struggle to judge "what is missing." For academic papers, omitting key methodological details, statistical information, or ethical disclosures can be equally fatal, yet AI is nearly powerless to address this. This suggests that AI review tools will still need supplementation by human experts for the foreseeable future, especially in assessing paper completeness and transparency.
Omission errors and commission errors are a fundamental pair of categories in error analysis. The former refers to information that should be present being deleted or unreported (e.g., missing sample sizes, undeclared conflicts of interest), while the latter refers to added incorrect or misleading content (e.g., miscalculated data, mismatched citations). From a cognitive and computational perspective, detecting omission errors is far more difficult: the system must first build a complete expectation model of "what elements this paper should contain" and then compare item by item; whereas detecting commission errors only requires local verification of whether the given content is contradictory. Current large language models excel at the latter's pattern matching but lack a systematic expectation of academic paper structural completeness, making omission errors a structural blind spot.
Commercial Tool Comparison: Refine.ink Has Most Unique Catches but High Cost
Among the two commercial AI review tools, Refine.ink contributed the most "unique catches" (i.e., errors not found by other systems). This means it has a unique advantage in certain error types, possibly stemming from its specialized optimization for academic text.
But the study also notes that Refine.ink's cost per error is relatively high—calculated per error discovered, its cost is significantly higher than using frontier general-purpose models. For budget-limited teams, combining multiple general-purpose models may be more cost-effective; for scenarios pursuing maximum recall, Refine.ink is still worth considering.
Limitations and Unknowns: False Positives Not Measured, True Error Distribution Unknown
This benchmark is not without limitations. The study explicitly states that false positives were not measured—that is, how many of the "errors" reported by AI systems were actually false alarms. In real review scenarios, false positives also consume editors' and authors' time and may even trigger unnecessary disputes. Without this dimension, we cannot fully assess system precision.
Another unresolved question is whether the distribution of these 100 planted errors matches the distribution of errors naturally occurring in real papers. The research team acknowledges that it is currently unclear how planted errors (mostly clearly verifiable factual errors) differ from real errors (which may be more subtle and more dependent on domain knowledge). This means the benchmark results may overestimate or underestimate AI performance in real review.
False positives in AI review refer to cases where the system flags something as an "error" but it is not actually an error. The counterpart is false negatives, i.e., real errors the system misses. The catch rate in this study essentially measures recall—the proportion of true positives among all planted errors; whereas precision—the proportion of true positives among all flagged items—cannot be calculated because false positives were not measured. In practical review, a high false positive rate would cause editors and authors to waste considerable time verifying false alarms and could even damage trust in AI tools. Therefore, even if a system has high recall, low precision would greatly reduce its practical usability. A complete evaluation needs to report both recall and precision, or use composite metrics such as the F1 score.
Significance and Outlook: Toward a Cross-Disciplinary Review Benchmark
Despite its limitations, the value of this work lies in providing a publicly reproducible evaluation framework. All materials—papers, errors, model outputs, experimental logs—are open, and other researchers can build on this to expand error types, add disciplines, and gradually construct a comprehensive cross-disciplinary AI review benchmark.
For the academic publishing community, this study conveys a pragmatic message: AI review tools are currently best suited as a "second pair of eyes" to assist human reviewers in spotting obvious errors, rather than replacing human judgment. Ensembling multiple systems can significantly improve recall, but handling "omission-type" errors and false positives still requires deep involvement from human experts.
As AI models continue to evolve, we have reason to expect further improvements in catch rates. But until then, transparent benchmarking and cautious deployment strategies are key to making AI truly serve academic quality.
One-line Summary
100 planted errors test AI review: best single system catches 71%, ensemble reaches 93%, omission errors remain a blind spot.
Paragraph Summary
A new study built a benchmark by planting 100 known errors into 10 open-access psychology papers to evaluate AI review systems. Results show the best single system caught 71 errors, the worst only 30; ensembling all systems raised the catch count to 93, indicating different models' error detection is only partially correlated and combination can improve recall. However, 7 errors were missed by every system, all of them "omission-type" (missing information in the paper), revealing that AI is better at finding commission errors than judging missing information. The commercial tool Refine.ink contributed the most unique catches but had a high cost per error. The study did not measure false positives, and the difference between planted and real error distributions is unknown, suggesting AI review should currently serve as an auxiliary tool to human review.
Related articles

Daybreak Red Deployment Guide: A Reference Architecture for Supervised Security Investigation
A breakdown of the Daybreak Red reference architecture: security control design, deployment configuration process, and runtime monitoring for supervised security investigation.

Gemini 4 Neon Tops the Models Engineering Index
Gemini 4 Neon tops the Models Engineering Index, sparking Hacker News debate over ranking credibility and evaluation methods. Here's what it means for developers.

Apple's Self-Developed AI Agent: An Agent That Can Operate Your Mac Is Coming
Apple is reportedly developing an AI agent that can operate a Mac, targeting Meta Muse, Claude, OpenAI Dots, and others. This article analyzes Apple's system-level advantages, the competitive landscape, and likely rollout paths in the agent race.