Sieve and Sage: Teaching Retrieval-Augmented Models to Know What They Don't Know

When AI Needs to Learn to Say "I Don't Know"

Socrates was famously known for his wisdom in acknowledging his own ignorance. For retrieval-augmented language models (RALMs), this same capacity for self-reflection is equally critical — when retrieved evidence fails to support a reliable answer, a model should choose to abstain rather than fabricate a response. A recent paper published on arXiv proposes a framework called Sieve and Sage, designed specifically to address this challenge.

Retrieval-Augmented Generation (RAG) has become the dominant approach for reducing hallucinations in large language models, but it's far from foolproof. The retrieval step can return missing, contradictory, or even adversarial information. If a model blindly trusts this noisy input, the generated answer can actually be more dangerous than if no retrieval had occurred at all. Teaching models to recognize "when to stay silent" has therefore become a core challenge in building reliable AI systems.

Decomposing Retrieval Failure into Two Distinct States

Most existing approaches rely on a single monolithic LLM to handle all types of retrieval failure in one step. The researchers argue that this one-size-fits-all approach both limits abstention performance and incurs unnecessarily high computational costs.

The central insight behind Sieve and Sage is decomposing retrieval failure into two fundamentally different states:

Unanswerable State

This occurs when the required evidence simply does not exist in the retrieved results. No matter how capable the model is, it cannot provide a well-grounded answer — the correct response is to abstain immediately.

Distracted State

This is the more subtle case: relevant evidence does exist, but it is mixed with conflicting, negated, or adversarial information. The model must distinguish signal from noise — and this is precisely where errors are most likely to occur.

The value of this decomposition lies in the fact that the two failure modes require different handling strategies. Lumping them together under a single module inevitably compromises both effectiveness and efficiency.

Retrieval-Augmented Generation (RAG) works by fetching relevant passages from an external knowledge base (such as a document corpus or search engine) before the language model generates an answer, feeding those passages as "context" into the model. This mechanism effectively reduces hallucinations caused by relying solely on parametric memory, but it also introduces new vulnerabilities. Retrieval systems are fundamentally based on semantic similarity matching — they can judge whether a document is relevant, but not whether it is correct. When the knowledge base lacks the answer, retrieval returns similar but irrelevant noise. When the knowledge base contains contradictory statements, the model may pick whichever supports a "fluent" answer rather than the most accurate one. Both scenarios are extremely common in real-world deployments, yet have long lacked a systematic framework for handling them.

A Two-Stage Pipeline: Lightweight Filtering + Heavy-Duty Reasoning

Building on this decomposition, the framework introduces two components with clearly defined responsibilities:

• Sieve: A lightweight module that screens the retrieved document set for distracting evidence before invoking the expensive large model.
• Sage: Handles grounded generation based on the filtered evidence and makes abstention decisions when necessary.

This "filter first, then reason" design philosophy is fundamentally about allocating computational resources where they're needed most. The lightweight Sieve handles the bulk of coarse filtering, preventing the costly Sage from wasting effort on obviously noisy inputs. This is both an accuracy optimization and a cost optimization.

The two-stage pipeline is a classic paradigm in ML system design, built on the logic of "coarse filter + fine judgment": the first stage uses a low-cost model to quickly eliminate clearly unsuitable candidates, while the second stage applies high-cost reasoning only to the remaining candidates. This design has mature applications in information retrieval (e.g., the recall-ranking architecture in search engines) and computer vision (e.g., region proposal generation in object detection). Sieve and Sage brings this thinking to the abstention decision problem in RAG: the Sieve module doesn't need to understand the semantic details of the question — it only needs to detect whether conflicts or adversarial signals exist within the document set, a relatively simple classification task well-suited to a lightweight model. Only when the document set passes screening and evidence quality is trustworthy does the full reasoning task get handed off to the more capable Sage model. This precise matching of task difficulty to model capability is the structural source of its efficiency gains.

Experimental Results: Gains in Both Accuracy and Speed

The research team evaluated the framework across two settings: general-domain and high-stakes expert-domain scenarios. The results are striking:

• Accuracy improved by up to 69.4 percentage points compared to single-stage baselines;
• Macro-F1 improved by up to 55.2 percentage points;
• Inference speed achieved up to 1.99× acceleration.

The substantial gains in accuracy and F1 confirm that pre-screening for distracting noise genuinely addresses a core weakness of single-stage approaches. The near-2× speedup validates the efficiency advantages of the two-stage pipeline — delegating simple judgments to a lightweight module and reserving the heavy model only for cases that truly require it.

The "high-stakes expert domain" test setting is particularly noteworthy. In fields like medicine and law where errors carry serious consequences, a model's ability to "rather say nothing than say something wrong" is far more valuable than answering more questions. This is precisely the real-world motivation behind abstention research.

Macro-F1 (macro-averaged F1 score) is a commonly used aggregate metric for multi-class classification tasks. It is computed by calculating the F1 score for each class separately and then taking the arithmetic mean — rather than weighting by class sample size. The key property of this metric is that it treats minority classes equally. In abstention evaluation, "correctly abstaining" samples are typically far fewer than "normally answered" samples; weighted averaging would easily obscure performance on the abstention class. The large improvement in Macro-F1 (55.2 percentage points) is therefore especially meaningful: it indicates the model performs well not only on common answerable questions, but also accurately identifies the rarer "should abstain" samples — rather than simply riding the majority class to inflate overall scores. This also reflects that Sieve's detection of distracting evidence remains effective even in real-world scenarios with imbalanced class distributions.

Implications for RAG System Design

The contribution of Sieve and Sage extends beyond its specific performance numbers — it also offers a broader architectural philosophy. At a time when the industry tends to throw larger models at every problem, this research reminds us: task decomposition and staged processing can often deliver better results with less compute.

For teams building enterprise-grade RAG applications, this approach offers direct, practical inspiration. Rather than hoping a single large model can digest all retrieval noise on its own, consider adding a dedicated "gatekeeper" earlier in the pipeline to filter out unreliable evidence. Reliability and efficiency don't always have to be a zero-sum trade-off.

It's worth noting that this article is based on the paper's abstract. Specific implementation details of the Sieve module, training methodology, and full experimental setup should be sought in the original paper.