Refusal Capability in RAG Applications: Teaching AI to Say "I Don't Know"

Getting an AI application to answer questions is exciting, but getting it to recognize when the documents don't contain the answer—that's the more interesting challenge.

On Day 1 of a 90-day AI challenge, one developer chose a seemingly simple yet critical question as a starting point: How can a RAG application avoid answering questions that its source documents cannot support?

The essence of this problem is giving the AI system the judgment to know what it knows and what it doesn't.

The Problem Scenario: When AI Confidently Fabricates Answers

Imagine you upload a company vacation policy document, then ask: "How many days of annual leave do employees get per year?"

If the policy document contains this information, the application should provide an answer and show the supporting passage. But if you ask a different question: "Does the company reimburse international travel expenses?"—and the document says nothing about it—the application should clearly state that it cannot find the answer in the provided materials.

This sounds obvious, but it's easily overlooked in practice. A confident guess could easily be mistaken by users for company policy. In document-based AI applications, the risk of unsupported answers far exceeds that of refusing to answer. Users rely on these systems for accurate information, not plausible-sounding fabrications.

Why RAG Alone Cannot Solve This Problem

Retrieval-Augmented Generation (RAG) works by retrieving relevant content and passing it to a language model to assist with answering. The process seems complete, but there's a fundamental flaw.

The best-matching passage is not necessarily the passage that can answer the question.

Search engines can still return results when information is missing—this is the nature of retrieval mechanisms. Retrieval is a useful starting point, but it doesn't guarantee that answers are grounded. Passages with high semantic similarity may only be topically related, without actually containing the answer the question requires.

This is the common pathway through which RAG applications hallucinate: the retriever finds content that "looks relevant," the language model "reasonably infers" an answer based on that content, and that answer has no support in the original documents.

What is the inherent limitation of semantic similarity retrieval?

Most RAG systems use vector embeddings for retrieval—converting text into high-dimensional vectors and finding the passages "closest" to the question by computing metrics like cosine similarity. The fundamental assumption of this approach is that semantically proximate text contains the answer to the question. But in reality, semantic similarity measures topical relevance, not answer entailment. A passage might discuss the same topic and use similar terminology while lacking the specific facts needed to answer the question. For example, if the question asks about "number of annual leave days," the retrieved passage might discuss the "annual leave application process"—highly relevant in topic but containing no answer. This "relevant but insufficient" retrieval result is precisely what directly triggers the language model to make unfounded inferences.

Day 1 Implementation: Small-Scale Validation

The developer's Day 1 plan was deliberately kept minimal, with a core workflow of five steps:

1. Load a short document and split it into passages
2. Retrieve relevant passages for the question
3. Instruct the model to answer using only these passages
4. Require answers to include source citations
5. Return an explicit fallback response when passage evidence is insufficient

The fallback response can be simple: "I cannot find this information in the provided documents."

The key is—making this kind of response a legitimate and expected output, rather than a sign of system failure. The pitfall of many RAG applications is treating "refusal" as an exceptional case, which leads the model to still lean toward giving an answer when evidence is insufficient.

On the prompt engineering principle of "answer using only these passages"

Instructing the model to "answer using only the provided passages" is a form of grounded prompting. Its core mechanism is: explicitly limiting the model's knowledge boundary in the system prompt or user prompt, transforming the model's role from "general assistant" to "document reader." Combined with source citation requirements, this strategy effectively builds a verifiable closed loop—the model must not only provide an answer but also indicate which passage the answer comes from, allowing users to independently verify. It's worth noting that instruction-following capability varies by model: some models will still "supplement with common knowledge" when passages are insufficient, so the fallback response format for insufficient evidence must be explicitly defined in the prompt, redefining it from "failure" to "correct behavior."

Three Types of Test Questions: Exposing the System's True Capabilities

To validate the approach's effectiveness, the developer designed three categories of test questions:

• Answerable: The document directly contains the answer
• Relevant but unsupported: The question is related to the document's topic, but the answer is missing
• Irrelevant: The question falls outside the document's scope

The second category is the most challenging. A passage can appear relevant on the surface yet still fail to support a specific answer. These cases best expose the "overconfidence" tendency of RAG systems.

Each answer needs to be checked across four dimensions:

• Is the answer correct
• Do the cited passages truly support the answer
• Does the application refuse to answer when evidence is missing
• Does the application unnecessarily refuse questions that could be answered

The last point is equally important—an overly cautious RAG application diminishes its usefulness. In some scenarios, users would rather receive a qualified answer than face blanket refusals.

Two Pitfalls to Watch Out For

The developer specifically highlighted two easily overlooked issues.

A valid citation ID does not equal a correct answer. A citation ID only proves that the cited passage exists; it does not prove that the passage supports the claim in the answer. Verifying citation quality requires checking whether the content of the cited passage truly supports the corresponding claim.

Retrieval score thresholds cannot be assumed to be universally effective. Choosing a score cutoff and assuming it applies to all scenarios is a dangerous oversimplification. Any threshold needs to be tested and validated with actual documents and retrieval methods. Different document structures, terminology distributions, and retriever characteristics all affect a threshold's effectiveness.

The nature and risks of retrieval score thresholds

Vector retrieval typically returns a similarity score (such as cosine similarity), and developers often try to set a threshold to filter out low-quality results. But the absolute value of a threshold is highly dependent on specific conditions: the type of embedding model (score distributions differ significantly across embedding models from different vendors), document length and chunking granularity, query phrasing, and more. A threshold that performs well on a short Q&A dataset may completely fail when transferred to long technical documents. A more robust approach combines relative ranking (e.g., keeping only passages in the top K results whose scores are significantly higher than the rest) with verification-based judgment (having the model determine whether a passage truly supports the answer), rather than relying on a single static threshold.

Why This Foundational Capability Deserves Priority

Day 1 focused on just one thing: building a small baseline, observing where it fails, and recording results before adding complexity.

Starting here was chosen because "knowing when to stop answering" is a foundational capability for document-based AI applications. If a system cannot judge what it doesn't know, then all the answers it provides are suspect.

If you're also building a RAG application, try asking a question that "sounds relevant but whose answer is deliberately missing"—did your application notice?

The value of this test lies in this: it directly examines the system's honesty. And honesty is precisely the cornerstone of user trust.

One-line Summary

RAG applications must learn to explicitly refuse when documents don't support an answer—this is a foundational capability for building user trust.

Paragraph Summary

This article documents a developer's practice on Day 1 of a 90-day AI challenge, building "refusal capability" for a RAG application. The core question: when documents don't contain an answer, how do you get the AI to clearly say "I don't know" rather than confidently fabricating one? The article points out that RAG retrievers always return "seemingly relevant" passages, but semantic similarity doesn't equal grounded answers—this is the common pathway to hallucination. The solution includes grounded prompting, source citations, and fallback responses when evidence is insufficient, defining refusal as a legitimate output. The author designed three types of test questions—answerable, relevant but unsupported, and irrelevant—evaluated across four dimensions: answer correctness, citation quality, refusal accuracy, and excessive refusal. The article also warns of two pitfalls: a valid citation ID doesn't equal a correct answer, and retrieval score thresholds can't be assumed to be universally effective. Ultimately, it emphasizes that judging "what you don't know" is a foundational capability for document-based AI.