Hybrid Retrieval + NLI + Open-Source LLMs for Scientific Claim Verification: Why Does Fusion by Rules Hurt Performance?

Scientific Claim Verification involves determining whether a source text supports or refutes a claim, or whether there is insufficient evidence. This task seems like a natural fit for Natural Language Inference (NLI), but a new experimental study offers a more nuanced answer: NLI performs adequately on single-sentence evidence but struggles with cross-sentence evidence; open-source large models can reason across sentences but may draw conclusions not supported by the text. The researchers attempted to combine the strengths of both using a conservative fusion rule, only to find that fusion actually degraded performance.

Task Setup: Three Types of Judgments and an Easily Overlooked Category

Claim verification is not a simple binary classification. The study categorizes judgments into three types: Support, Contradict, and Not Enough Evidence, with the last category comprising a high proportion in real scientific literature scenarios. The authors constructed a benchmark of 2,035 claims from the SciFact and CliniFact datasets, distributed as 858 Support, 152 Contradict, and 1,025 Not Enough Evidence.

This distribution itself illustrates the issue: contradiction samples are scarce, and insufficient evidence samples account for nearly half. Any model that tends to make black-and-white judgments will suffer on this benchmark. This is also key background for understanding the subsequent experimental results.

Macro-F1 is a commonly used metric for evaluating multi-class tasks. It first calculates the F1 score (harmonic mean of precision and recall) for each category separately, then takes the arithmetic mean across all categories. Unlike micro-averaged F1, macro-averaging treats each category equally, without giving higher weight to categories with more samples. In this benchmark, the "Not Enough Evidence" class accounts for nearly half, while the "Contradict" class has only 152 samples. If accuracy or micro-averaged F1 were used, a model could achieve seemingly good scores simply by favoring the majority class; however, Macro-F1 amplifies errors on minority classes, thus better reflecting the model's balanced capability across the three types of judgments. This also explains why the NLI baseline has a Macro-F1 of only 0.3157—it nearly fails on the scarce "Contradict" and "Not Enough Evidence" classes, and macro-averaging fully exposes this flaw.

Retrieval Stage: Hybrid Semantic and Lexical Retrieval Slightly Better

Each claim first needs to find relevant evidence sentences. The study employs a hybrid approach combining semantic retrieval and lexical retrieval (BM25), retrieving the five most relevant sentences for each claim and providing these sentences to all verification models.

The results show that hybrid retrieval performs slightly better than using semantic retrieval or BM25 alone. This conclusion is not surprising—semantic retrieval excels at capturing paraphrases, while BM25 excels at exact term matching, and scientific texts require both. As a detail, retrieval quality is a common variable here: all verification models receive the same set of retrieved sentences, so subsequent performance differences mainly stem from the models' own judgment capabilities.

BM25 is a classic lexical retrieval algorithm belonging to the sparse retrieval family. It scores based on term matching: the more frequently and rarely a query term appears in a document, the higher the score, while normalizing for document length to avoid favoring long documents. BM25 does not rely on vector representations, so it is highly sensitive to exact terms, abbreviations, and numbers, but cannot handle synonym substitution or paraphrasing. Semantic retrieval maps queries and documents into a dense vector space and recalls via cosine similarity, excelling at capturing semantically similar but differently worded sentences. Scientific texts contain both a large number of standardized terms (favoring BM25) and definitional paraphrases and cross-sentence expressions (favoring semantic retrieval); the complementarity of the two is precisely why hybrid retrieval performs slightly better. Common practices include normalizing and weighted fusion of the two retrieval scores, or using Reciprocal Rank Fusion (RRF) to merge rankings.

Model Comparison: Qwen3 Leads, NLI Baseline Far Behind

The study compares an NLI baseline model (DeBERTa-v3-base-mnli-fever-anli) with three open-source large models: Llama 3.1 8B, Qwen3 8B, and Mistral 7B.

The performance gap is significant. Qwen3 achieves the highest Macro-F1 (0.7937), Llama 3.1 scores 0.7204, Mistral 0.6388, while the NLI baseline is only 0.3157. The gap between NLI models and LLMs is not a minor lag but an order-of-magnitude capability difference—consistent with its design positioning: NLI models excel at judging single-sentence entailment relationships, but the five retrieved sentences are scattered evidence, and cross-sentence integration exceeds its training objective.

There is also a gradient among the three LLMs. At roughly the same 8B scale, Qwen3 significantly outperforms Llama 3.1 and Mistral, indicating that in fine-grained judgments on scientific texts, differences between models cannot be simply explained by parameter count.

DeBERTa-v3-base-mnli-fever-anli is a discriminative model fine-tuned on natural language inference data. DeBERTa-v3 improves encoder performance through disentangled attention mechanisms and ELECTRA-style pre-training; this checkpoint was sequentially trained on MNLI, FEVER, and ANLI, thus possessing the ability to judge entailment, contradiction, or neutral relationships between "premise-hypothesis" pairs. Its input format is typically "premise sentence [SEP] hypothesis sentence," outputting three-class probabilities, essentially a sentence-pair-level classifier. This architecture is naturally suited for entailment judgments on single-sentence evidence but is not designed for aggregating multi-sentence evidence—when the input is five scattered retrieval results, the model lacks an explicit cross-sentence reasoning mechanism, which is the structural reason for its significant lag on this benchmark.

Fusion Rule Failure: Only Speaking When Consistent, Yet Dragging Down Overall Performance

The researchers' hypothesis was intuitive: since NLI and LLMs each have strengths, adopt a conservative fusion rule—only label Support or Contradict when both models give consistent judgments, otherwise default to Not Enough Evidence. Theoretically, this could filter out overconfidence from individual models.

The experimental results contradicted the hypothesis. After fusion, Macro-F1 dropped to 0.7392 (Qwen3), 0.6730 (Llama 3.1), and 0.5991 (Mistral), all lower than each model's standalone performance.

Why does a conservative rule harm performance? The key lies in the low judgment quality of the NLI baseline. When NLI frequently gives judgments inconsistent with the LLM, the fusion rule reclassifies many originally correct LLM predictions as insufficient evidence. The premise of a conservative strategy is that both parties have some reliability, but in this setting, the noise from the weaker party cancels out the correct judgments of the stronger party.

Error Analysis: Disagreement Often Means the LLM Is Right

To explain the performance drop, the study analyzed agreement, disagreement, and common error patterns between the two verifiers. The core finding is that a considerable portion of disagreements between NLI and LLM involve correct LLM answers being overturned by incorrect NLI judgments.

In other words, the fusion rule did not serve as "double verification" but rather added a layer of noise on top of correct outputs. This also suggests a more general engineering lesson: model fusion is not an unconditional gain; when the capability gap between components is too large, simple voting or consistency rules may backfire.

Model ensembling classically assumes "wisdom of the crowd": when errors from multiple models are independent, voting or averaging can cancel individual errors and improve overall robustness. However, this assumption requires two prerequisites—members have roughly comparable capability levels, and error patterns are uncorrelated. If one member is significantly weaker, its errors are no longer random noise but systematic bias; in this case, consistency rules treat the weaker member's errors as veto votes, suppressing the stronger member's correct judgments. The conservative fusion in this paper falls into exactly this trap: the NLI baseline's systematic failure on cross-sentence evidence causes the rule of "only speaking when both agree" to frequently reclassify correct LLM answers as insufficient evidence. A more robust engineering approach is to weight members based on validation set performance, or set confidence thresholds rather than hard consistency.

Practical Implications

The value of this experiment lies not in negating the fusion approach but in delineating its applicable boundaries. To introduce fusion in claim verification tasks, at least three points should be considered: the capabilities of models participating in fusion should be roughly comparable; fusion rules should weight the reliability of different models rather than equal-weight consistency; and the determination of the insufficient evidence category needs separate design, as it has a high proportion in the data and is most vulnerable to conservative rules.

For scenarios directly implementing scientific claim verification, the signal from the study is clear: at the current level of open-source models, choosing a more capable LLM for standalone use is more reliable than building a fusion pipeline with mismatched capabilities. On the retrieval side, the hybrid approach is a low-cost positive improvement and worth prioritizing.

One-line Summary

Experiments show NLI lacks cross-sentence capability, LLMs tend to over-infer, and conservative fusion actually degrades performance.

Paragraph Summary

This study compares an NLI baseline with three open-source large models on a benchmark of 2,035 scientific claims from SciFact and CliniFact, evaluating performance on Support, Contradict, and Not Enough Evidence judgments. Retrieval uses a hybrid semantic and BM25 approach to fetch Top-5 evidence sentences. Results show Qwen3 8B achieves the highest Macro-F1 of 0.7937, Llama 3.1 scores 0.7204, Mistral 0.6388, while the NLI baseline is only 0.3157. The researchers attempted a conservative fusion rule of "only speak when consistent, otherwise judge insufficient evidence" to combine NLI and LLM, but all three LLMs' performance dropped. Error analysis shows that a considerable portion of disagreements between NLI and LLM involve the LLM being correct but overturned by NLI, meaning fusion introduces noise rather than verification.