R2VC: A Modular Fact-Checking Architecture for More Trustworthy LLM Verification
R2VC's four-stage modular architecture improves fact-checking accuracy by ~14% on FEVER while making model confidence genuinely trustworthy.
R2VC is a modular solution to the "black-box" problem in LLM-based fact-checking. It decomposes the end-to-end pipeline into four independent stages — hybrid retrieval, SFT+DPO-trained multi-candidate generation, NLI cross-encoder verification, and lightweight confidence calibration with abstention support. On FEVER, an 8B backbone with R2VC achieves a 13.74% accuracy gain over the baseline, and ablation studies show that removing the calibration module nearly doubles the Brier score. Manual analysis of 250 error cases reveals that retrieval failures — especially wrong-entity evidence — remain the pipeline's biggest bottleneck, pointing to entity disambiguation as the most valuable direction for future work.
The Problem with End-to-End Fact-Checking
Large language models are increasingly being used to automate fact-checking, but the dominant end-to-end prompting approach has a fundamental flaw: it entangles evidence retrieval, reasoning, and uncertainty estimation into a single undifferentiated process. When the system produces a wrong verdict, it's nearly impossible to pinpoint whether the failure came from poor retrieval, flawed reasoning logic, or miscalibrated confidence. Worse, this black-box output makes the model's expressed "certainty" untrustworthy — it may appear completely confident in a conclusion that is entirely wrong.
A new paper on arXiv titled R2VC: Modular Fact-Checking with Retrieval, Verification, and Confidence Calibration addresses exactly this problem. The authors argue for replacing monolithic end-to-end pipelines with a modular architecture that breaks fact-checking into independently diagnosable and optimizable components.
R2VC's Four-Stage Modular Design
The name R2VC reflects its four core stages: Retrieve, Reason, Verify, and Calibrate. The architecture targets evidence-backed fact-checking with citations and abstention support, with each module handling a distinct responsibility.
Hybrid Retrieval: Sparse + Dense
The first stage performs hybrid retrieval over a Wikipedia corpus, combining both sparse and dense retrieval methods. Sparse retrieval excels at precise keyword matching, while dense retrieval captures semantic-level relevance. Together, they enable more comprehensive evidence recall in open-domain settings.
Generating Diverse Structured Verdict Candidates
The generator in the second stage is trained via supervised fine-tuning (SFT) and DPO (Direct Preference Optimization) alignment. Rather than committing to a single answer, it produces multiple diverse structured verdict candidates. This "multi-candidate" strategy creates room for downstream selection, avoiding premature lock-in on a potentially incorrect conclusion.
External Verification and Confidence Calibration
The third stage introduces an external NLI cross-encoder that filters the candidate verdicts against retrieved evidence, selecting the one best supported by the evidence. The fourth stage is a lightweight sequence-level calibrator responsible for confidence estimation and selective abstention — when the model lacks sufficient certainty about a judgment, it can choose to withhold a verdict rather than force out a low-confidence answer.
Results: Higher Accuracy and More Trustworthy Confidence
On FEVER, the standard benchmark for fact-checking, an 8B backbone model equipped with the R2VC architecture achieved a 13.74% accuracy improvement over the baseline. This is a substantial gain, demonstrating that modular design isn't just engineering cleanliness — it delivers real performance benefits.
Ablation studies further reveal the contribution of each component. The researchers found that verifier-based candidate selection and confidence calibration are the two primary drivers of improvement:
- Removing the candidate selection stage drops FEVER accuracy to 76.24%;
- Removing the calibration module nearly doubles the Brier score (a metric measuring probabilistic prediction reliability) to 0.161 — meaning the model's expressed confidence becomes far less trustworthy.
These figures make it clear that verification and calibration are critical to making a model's confidence worth trusting — something end-to-end methods struggle to guarantee.
Retrieval Remains the Biggest Bottleneck
Worth highlighting is the team's manual analysis of 250 error cases. Despite the overall strong performance of the modular architecture, retrieval failures remain the dominant bottleneck, with "wrong-entity evidence" being the most prominent issue — that is, the system retrieves evidence that appears relevant but actually pertains to the wrong entity.
This finding is instructive: it suggests that future research may benefit more from improving retrieval entity disambiguation than from building stronger reasoning capabilities. No matter how well the reasoning and calibration stages perform, if the evidence fed into the pipeline points to the wrong entity, the entire fact-checking chain will still fail.
Implications for Automated Fact-Checking
The core message of R2VC is that a modular fact-checking pipeline can simultaneously improve both prediction accuracy and confidence reliability in open-domain verification. By decoupling retrieval, reasoning, verification, and calibration, the system not only performs better — failures are also easier to diagnose, since each stage can be evaluated and replaced independently.
At a time when LLMs are widely deployed for information verification, this "interpretable, abstention-capable, and calibrated" design philosophy offers more engineering value than simply scaling up end-to-end models. It acknowledges that models make mistakes and provides a clear fallback mechanism (abstention) for those mistakes — a quality that is especially important for high-stakes, real-world applications that demand trustworthy outputs.
Related articles
CCPS Sampling: Preserving Reasoning Diversity to Boost LLM Performance Without Fine-Tuning
A new arXiv paper proposes CCPS, which boosts LLM reasoning accuracy without any training by preserving reasoning diversity via Chopthin resampling and semantic majority selection — achieving up to 10.6pp absolute gains.
MIT Spinout Transforms Plastic Waste into High-Resilience Building Materials
MIT spinout Atlas Building Composites converts plastic waste into resilient structural components for buildings and infrastructure, pioneering a high-value recycling pathway.
Repair Before Reinforce: Context-Augmented Knowledge Graph Reasoning Tackles Multi-Hop QA
New arXiv paper proposes a context-augmented KG reasoning framework using a "Repair Before Reinforce" strategy to improve LLM multi-hop QA. Validated on Gastroparesis and Diabetes KGs with Qwen3-14B, achieving 100% single-hop accuracy after repair.