The Truth Behind Rising Fact-Check Scores: Evidence Matters More Than Answers

When Fact-Check Scores Go Up, What's Actually Changing?

Automated fact-verification systems are typically evaluated using a joint score that assesses both whether the model's answer is correct and whether the supporting evidence it submits is complete. The problem is that when this combined score rises, it's difficult to tell whether the improvement comes from better answer quality or better evidence retrieval. This arXiv paper (arXiv:2609.27064v1) tackles exactly this question, using rigorous experimental design to disentangle the contributions of "answers" and "evidence" to overall performance.

The study centers on the FEVEROUS dataset. In this setting, the strict score is defined as the percentage of claims for which the system both provides a correct answer and includes all annotated evidence groups in its submitted evidence. In other words, a claim only scores a point when both the answer and the evidence are right. This dual threshold makes decomposition analysis not just useful, but essential.

FEVEROUS (Fact Extraction and VERification Over Unstructured and Structured information) is a benchmark dataset designed for complex fact-checking, requiring systems to handle both unstructured text and structured table data from Wikipedia. Compared to the earlier FEVER dataset, the core challenge in FEVEROUS is that evidence is often scattered across multiple heterogeneous sources — a single claim may require joint reasoning across paragraphs and tables to verify. Each claim in the dataset comes with a human-annotated "gold evidence set" — the minimal combination of evidence needed to correctly verify the claim. The strict score's dual-threshold design stems directly from this: if a system's submitted evidence doesn't fully cover the gold evidence set, the claim doesn't count toward the score even if the answer label is correct. The design intent is to encourage systems to produce "explainable" verification conclusions rather than just guessing the right label.

Evidence Substitution Yields Far Greater Gains Than Answer Accuracy

The research team ran controlled experiments across four trained DeBERTa checkpoints on 7,890 claims. The core finding is striking: replacing DCUF evidence with UnifEE evidence improved the strict score by 9.61 percentage points, while the same operation improved answer accuracy by only 1.96 percentage points. This nearly fivefold gap shows that most of the improvement in the joint score comes from evidence quality, not from the model getting more answers right.

The paired 95% confidence interval of [8.77, 10.43] (conditioned on these checkpoints) confirms that this gain is statistically robust. The study goes further with a more granular attribution analysis: when only the evidence passed to the scorer was swapped — while keeping answers generated from DCUF or UnifEE evidence respectively — the strict score improvements were 7.92 and 9.08 percentage points. This directly quantifies the independent contribution of "just swapping the evidence" to the final score, isolating it from any changes in answers.

The implication for the field is significant: many methods that claim to improve fact-checking performance may be deriving most of their gains from better evidence retrieval pipelines, not from advances in core reasoning ability. Without decomposition, it's easy to mistakenly credit evidence-side gains to the model's answer prediction capabilities.

The DCUF and UnifEE systems involved in the experiments are two evidence retrieval systems for FEVEROUS, representing different retrieval pipeline designs. DCUF (Decompose, Classify, Unite, Filter) uses a decomposition strategy, breaking claims into sub-questions for individual retrieval before merging results. UnifEE (Unified Evidence Extraction) processes text and tables simultaneously through a unified extraction framework, achieving better completeness in evidence recall. DeBERTa (Decoding-Enhanced BERT with Disentangled Attention) is a pre-trained language model from Microsoft Research that excels at natural language inference and textual entailment tasks, commonly used as the verdict prediction module in fact-checking pipelines. Combining DeBERTa checkpoints with different evidence sources makes it possible to manipulate evidence quality independently while holding model capability fixed — enabling causal attribution of contributions.

How Context Budget Affects Evidence Gains

To check whether these evidence gains depend on specific evaluation choices, the research team also generated 470,400 responses across FEVER, FEVEROUS, and SciFact using two 8B-scale large language models (Qwen and Llama), covering two answer formats and two context budgets.

The results show that expanding context from 256 tokens to 2,048 tokens improved "evidence gains conditioned on fixed answers" on FEVEROUS by 3.84 percentage points (Qwen) and 3.10 percentage points (Llama). This means giving models a larger context window does improve evidence-side performance, though the magnitude is modest.

The study also set a pre-specified criterion for determining whether effects were significant across datasets. The actual effects did not meet this threshold, though some confidence intervals did exceed the "two percentage points" boundary for small effects. This cautious framing reminds readers that while there is some evidence for the existence of evidence gains, their magnitude and cross-dataset consistency should be interpreted with reservations.

Why Aggregate Metrics Obscure Real Changes

One of the paper's important methodological contributions is pointing out that commonly used aggregate metrics — such as overall accuracy and evidence coverage — miss claim-level patterns. Post-hoc analysis quantifies how answers and submitted evidence each change, and reveals under what conditions macro-level accuracy and coverage numbers diverge from what's actually happening claim by claim.

The study proposes an analytical framework built around four combinations of answer and evidence outcomes: correct/incorrect answers crossed with complete/incomplete evidence, forming four quadrants. This fine-grained accounting reveals gaps left by endpoint metrics and aggregate metrics alike. For example, a method might appear unchanged in overall accuracy, yet have large numbers of claims shifting from "correct answer but missing evidence" to "correct answer with complete evidence" — a shift that's only visible at the claim level.

For researchers working on fact-checking, retrieval-augmented generation (RAG), and trustworthy AI evaluation, this accounting framework offers a more honest evaluation perspective: don't just look at how much a composite score rose — ask where the gains came from and how different claims are flowing between the four quadrants.

This "answer × evidence" four-quadrant analysis shares conceptual DNA with heterogeneous treatment effect analysis in causal inference: system-level average improvements can mask large numbers of individual changes in opposite directions. In RAG system evaluation practice, similar problems are widespread — common end-to-end metrics like RAGAS scores or answer faithfulness conflate retrieval quality and generation quality, making it difficult to guide targeted optimization. The value of the four-quadrant framework lies in distinguishing between "correct but evidence-poor" and "evidence-rich but incorrect" failure modes — the former points to retrieval pipeline improvements, while the latter indicates deficiencies in the reasoning module, leading to entirely different engineering interventions.

Implications for Trustworthy AI Evaluation

While this work focuses on a fairly specialized evaluation problem, its underlying logic has broad applicability. As large models are increasingly used for tasks that require citing sources and providing evidence, distinguishing between "getting the answer right" and "providing the right basis for the answer" becomes critically important. A system that reaches correct conclusions through incorrect or incomplete evidence is equally untrustworthy in high-stakes scenarios.

The core message the research delivers is this: when evaluating fact-checking systems, evidence quality and answer correctness should be tracked separately, with changes examined at the claim level. A single joint score is convenient but easily obscures true capability boundaries. For practitioners looking to make trade-offs between retrieval, evidence aggregation, and generation, this decomposition approach can help direct resources toward the components that actually drive meaningful gains.