R2VC: A Modular Fact-Checking Architecture for More Trustworthy LLM Verification
R2VC's four-stage modular architecture improves fact-checking accuracy by ~14% on FEVER while making model confidence genuinely trustworthy.
R2VC is a modular solution to the "black-box" problem in LLM-based fact-checking. It decomposes the end-to-end pipeline into four independent stages — hybrid retrieval, SFT+DPO-trained multi-candidate generation, NLI cross-encoder verification, and lightweight confidence calibration with abstention support. On FEVER, an 8B backbone with R2VC achieves a 13.74% accuracy gain over the baseline, and ablation studies show that removing the calibration module nearly doubles the Brier score. Manual analysis of 250 error cases reveals that retrieval failures — especially wrong-entity evidence — remain the pipeline's biggest bottleneck, pointing to entity disambiguation as the most valuable direction for future work.
The Problem with End-to-End Fact-Checking
Large language models are increasingly being used to automate fact-checking, but the dominant end-to-end prompting approach has a fundamental flaw: it entangles evidence retrieval, reasoning, and uncertainty estimation into a single undifferentiated process. When the system produces a wrong verdict, it's nearly impossible to pinpoint whether the failure came from poor retrieval, flawed reasoning logic, or miscalibrated confidence. Worse, this black-box output makes the model's expressed "certainty" untrustworthy — it may appear completely confident in a conclusion that is entirely wrong.
A new paper on arXiv titled R2VC: Modular Fact-Checking with Retrieval, Verification, and Confidence Calibration addresses exactly this problem. The authors argue for replacing monolithic end-to-end pipelines with a modular architecture that breaks fact-checking into independently diagnosable and optimizable components.
R2VC's Four-Stage Modular Design
The name R2VC reflects its four core stages: Retrieve, Reason, Verify, and Calibrate. The architecture targets evidence-backed fact-checking with citations and abstention support, with each module handling a distinct responsibility.
Hybrid Retrieval: Sparse + Dense
The first stage performs hybrid retrieval over a Wikipedia corpus, combining both sparse and dense retrieval methods. Sparse retrieval excels at precise keyword matching, while dense retrieval captures semantic-level relevance. Together, they enable more comprehensive evidence recall in open-domain settings.
Generating Diverse Structured Verdict Candidates
The generator in the second stage is trained via supervised fine-tuning (SFT) and DPO (Direct Preference Optimization) alignment. Rather than committing to a single answer, it produces multiple diverse structured verdict candidates. This "multi-candidate" strategy creates room for downstream selection, avoiding premature lock-in on a potentially incorrect conclusion.
External Verification and Confidence Calibration
The third stage introduces an external NLI cross-encoder that filters the candidate verdicts against retrieved evidence, selecting the one best supported by the evidence. The fourth stage is a lightweight sequence-level calibrator responsible for confidence estimation and selective abstention — when the model lacks sufficient certainty about a judgment, it can choose to withhold a verdict rather than force out a low-confidence answer.
Results: Higher Accuracy and More Trustworthy Confidence
On FEVER, the standard benchmark for fact-checking, an 8B backbone model equipped with the R2VC architecture achieved a 13.74% accuracy improvement over the baseline. This is a substantial gain, demonstrating that modular design isn't just engineering cleanliness — it delivers real performance benefits.
Ablation studies further reveal the contribution of each component. The researchers found that verifier-based candidate selection and confidence calibration are the two primary drivers of improvement:
- Removing the candidate selection stage drops FEVER accuracy to 76.24%;
- Removing the calibration module nearly doubles the Brier score (a metric measuring probabilistic prediction reliability) to 0.161 — meaning the model's expressed confidence becomes far less trustworthy.
These figures make it clear that verification and calibration are critical to making a model's confidence worth trusting — something end-to-end methods struggle to guarantee.
Retrieval Remains the Biggest Bottleneck
Worth highlighting is the team's manual analysis of 250 error cases. Despite the overall strong performance of the modular architecture, retrieval failures remain the dominant bottleneck, with "wrong-entity evidence" being the most prominent issue — that is, the system retrieves evidence that appears relevant but actually pertains to the wrong entity.
This finding is instructive: it suggests that future research may benefit more from improving retrieval entity disambiguation than from building stronger reasoning capabilities. No matter how well the reasoning and calibration stages perform, if the evidence fed into the pipeline points to the wrong entity, the entire fact-checking chain will still fail.
Implications for Automated Fact-Checking
The core message of R2VC is that a modular fact-checking pipeline can simultaneously improve both prediction accuracy and confidence reliability in open-domain verification. By decoupling retrieval, reasoning, verification, and calibration, the system not only performs better — failures are also easier to diagnose, since each stage can be evaluated and replaced independently.
At a time when LLMs are widely deployed for information verification, this "interpretable, abstention-capable, and calibrated" design philosophy offers more engineering value than simply scaling up end-to-end models. It acknowledges that models make mistakes and provides a clear fallback mechanism (abstention) for those mistakes — a quality that is especially important for high-stakes, real-world applications that demand trustworthy outputs.
Related articles
A New Paradigm for Text Editing in Compose: TextFieldState Replaces the Old API
Jetpack Compose introduces the TextFieldState API to fix dropped characters and cursor jumps from async input. The old value/onValueChange callbacks will be deprecated in 1.13. This article covers InputTransformation, OutputTransformation, four state hoisting patterns, and new formatted text editing capabilities.
Remote Compose: A New Document-Driven Paradigm for Cross-Platform UI on Android
Remote Compose transmits compact binary documents instead of screen pixels, enabling efficient UI rendering on remote surfaces like Wear OS and widgets. This article covers its document creation API, expression animations, size optimization techniques, and Wear OS progress.
How Crashlytics Eliminates Android Memory and Responsiveness Issues
Through the Chroma Clip case study, this article explains how Firebase Crashlytics uses Application Exit Info, Profiling Manager, GWP ASAN, and ARM MTE to pinpoint ANRs, OOMs, memory limit terminations, and native memory corruption—solving the challenge of stack traces that don't point to the real culprit.