Peerify: A Benchmark Study on AI-Powered Peer Review Claim Verification

The Verification Problem in Peer Review

Peer review is the cornerstone of academic publishing, yet a persistent pain point remains: whether the claims made by reviewers are actually grounded in the manuscript's content. This verification process still relies heavily on manual effort — time-consuming and difficult to standardize. When a reviewer writes something like "the method lacks comparison with baselines" or "the experimental setup is biased," editors and authors must trace back through the paper word by word to determine whether those statements hold up.

A new paper on arXiv introduces Peerify, a system designed precisely for this task. It builds an automated pipeline for "manuscript fact-checking" — attempting to determine whether each reviewer claim can be supported by evidence found in the paper.

How Peerify Works

Peerify's core approach breaks down into three steps. Given a manuscript and a review comment, the system first decomposes the comment into atomic claims — splitting a complex comment into the smallest independently verifiable factual units. It then retrieves relevant evidence from the manuscript, locating passages in the paper that correspond to each claim. Finally, it determines whether each claim is supported by the paper.

This "decompose–retrieve–verify" design is deliberate. Review comments are often structurally complex and referentially ambiguous; directly judging the truth of an entire comment yields poor accuracy. Breaking it down into atomic claims and matching each one to evidence significantly reduces ambiguity. The paper's experimental results confirm this: retrieval-centric verification strategies and claim decomposition are the key drivers of verification quality.

Atomic claim decomposition is a common preprocessing technique in fact-checking and retrieval-augmented generation (RAG). The core insight is that complex natural language sentences often contain multiple propositions that can independently be true or false. Treating them as a single verification unit means that any one failing part can taint the whole. Take the review comment "the authors did not compare against SOTA baselines, and the ablation study is insufficient" — this actually contains two independent claims. If a paper includes SOTA comparisons but has weak ablations, the truth value of the whole sentence can't be cleanly determined. Once decomposed into atomic claims, each one maps to a specific passage or table in the manuscript, enabling finer-grained retrieval and judgment, and making it easier to report to editors or authors exactly which specific allegations are evidence-backed and which are not.

An 800-Claim Benchmark Dataset

To train and evaluate the pipeline, the research team constructed a benchmark dataset containing 800 claims drawn from real review interactions at two top machine learning venues: NeurIPS 2024 and ICLR 2024. These are representative and genuinely challenging examples.

The dataset also includes a 300-claim human-annotated subset, specifically designed to audit the reliability of automated labeling. This design choice matters: if the automatically generated labels are themselves unreliable, all subsequent evaluations built on them become meaningless. The audit found that automatic labels agreed with human consensus on 90.3% of claims, with a Cohen's Kappa of 0.87 — a level considered high agreement, confirming that the automated supervision signal is sufficiently trustworthy.

Cohen's Kappa is a statistical measure of agreement between two annotators (or between a human and an automated system), ranging from -1 to 1. Unlike a simple percentage agreement, Kappa adjusts for chance agreement, making it a better reflection of true annotation quality. A Kappa > 0.80 is generally considered "high agreement," and the 0.87 achieved here means the disagreement between the automated system and human judgment is essentially within the range of inherent subjectivity in the annotation task. This figure is especially important for a semantically complex domain like academic peer review — it indirectly validates that the "gold standard" labels used to train and evaluate models are trustworthy, rather than noisy proxy signals.

The Gap in Off-the-Shelf Model Performance

The study benchmarked multiple state-of-the-art language models and retrieval strategies within the Peerify pipeline, comparing them against entailment baselines. The results reveal a clear capability gap.

Using off-the-shelf entailment models directly to judge claim truthfulness yields a macro-averaged F1 below 0.24 — quite limited. This shows that traditional textual entailment approaches struggle with the specialized, long-document, semantically nuanced demands of academic review fact-checking. By contrast, the full pipeline with retrieval and claim decomposition dramatically outperforms this baseline.

The takeaway is clear: academic claim verification is not a simple "do these two sentences contradict each other" problem. It's a compound task requiring precise evidence localization within a long manuscript followed by fine-grained reasoning.

Textual Entailment, sometimes called Natural Language Inference (NLI), is the task of determining whether a "premise" text logically supports a "hypothesis" text. Traditional entailment models (such as classifiers trained on SNLI or MultiNLI) typically operate on short sentence pairs — a one- or two-sentence premise and hypothesis — with labels of "entailment," "contradiction," or "neutral." Directly transferring these models to academic peer review poses two fundamental challenges: first, manuscripts are thousands-of-words-long documents, not single sentences; second, the semantics of review claims often require joint reasoning across multiple sections, not resolvable within a single paragraph window. This explains why the entailment baseline F1 drops below 0.24 — there is a fundamental out-of-distribution shift between the task's input space and what the model was trained on.

Remaining Challenges

The paper also honestly acknowledges the limitations of the current approach. The biggest difficulty comes from vague, interpretive review claims. Some review comments are inherently subjective — evaluations of novelty, significance of contribution, and similar judgments — and it's hard to find clear "supporting" or "opposing" evidence for these in the manuscript, let alone define an objective verification standard.

This is precisely where automated verification systems are most fragile. They handle factual, verifiable claims like "did the paper run a specific experiment" well, but struggle with interpretive judgments like "is this improvement significant enough."

Implications for the Academic Ecosystem

Peerify represents a pragmatic direction for integrating large language models into academic workflows. Rather than attempting to replace reviewer judgment, it focuses on a specific, measurable assistive task: helping to quickly cross-check the correspondence between reviewer claims and manuscript evidence.

As submission volumes continue to grow and the review burden intensifies, tools like this — if they mature enough for deployment — could help alleviate inconsistent review quality and the pressure of author rebuttal verification. Of course, there's still a significant gap between a benchmark study and a genuinely usable editorial assistance system, especially when it comes to handling subjective claims and ensuring fairness. This work's dataset and evaluation framework provide a reproducible foundation for future research.