Peerify
An automated pipeline system for fact-checking academic peer reviews. It verifies whether reviewer claims are supported by manuscript content through a three-step process of atomic claim decomposition, evidence retrieval, and adjudication, and is accompanied by a standardized benchmark dataset.
Timeline (last 90 days)
The Peerify system constructs an automated pipeline for manuscript fact-checking, used to determine whether each claim made in a review can be supported by evidence in the paper.
Peerify's core pipeline consists of three steps: decomposing reviews into atomic claims, retrieving relevant manuscript evidence, and determining whether each claim is supported by the paper.
The research team constructed a benchmark dataset containing 800 claims for training and evaluating the pipeline.
The claims in the benchmark dataset come from real review interactions at two machine learning conferences, NeurIPS 2024 and ICLR 2024.
The dataset contains a manually annotated subset of 300 entries used to audit the reliability of automated annotation.
The audit results show that automatic labels agree with human consensus on 90.3% of claims, with a Cohen's Kappa coefficient of 0.87.
The greatest challenge for Peerify's current approach comes from vague, interpretive reviewer claims for which clear supporting or refuting evidence is difficult to find.
All Facts (7)
The Peerify system constructs an automated pipeline for manuscript fact-checking, used to determine whether each claim made in a review can be supported by evidence in the paper.
50%UnverifiedPeerify's core pipeline consists of three steps: decomposing reviews into atomic claims, retrieving relevant manuscript evidence, and determining whether each claim is supported by the paper.
50%UnverifiedThe research team constructed a benchmark dataset containing 800 claims for training and evaluating the pipeline.
50%UnverifiedThe claims in the benchmark dataset come from real review interactions at two machine learning conferences, NeurIPS 2024 and ICLR 2024.
50%UnverifiedThe dataset contains a manually annotated subset of 300 entries used to audit the reliability of automated annotation.
50%UnverifiedThe audit results show that automatic labels agree with human consensus on 90.3% of claims, with a Cohen's Kappa coefficient of 0.87.
50%UnverifiedThe greatest challenge for Peerify's current approach comes from vague, interpretive reviewer claims for which clear supporting or refuting evidence is difficult to find.
50%