RAG Systems: The LLM Isn't the Hard Part—Retrieval Quality Is

Core thesis: Retrieval-augmented generation (RAG) failures are often not because the model is weak, but because the retrieval stage fails to supply sufficient evidence. This article examines retrieval relevance, chunking strategy, hybrid search, evaluation methods, and more to dissect what's truly hard in RAG engineering.

RAG system architecture diagram

Why Do RAG Systems "Look Fine" but Answer Wrong?

A typical scenario: a user asks, "After the contract renewal, can the customer still request a refund?" The knowledge base contains a general refund policy, the contract renewal process, and an old policy that has since been replaced. The system retrieves related content, the model generates a fluent answer, and the application returns a response successfully—but the answer is unsupported.

This is the hidden failure mode of RAG systems: the search returns results, the model produces output, but the key evidence was never retrieved. Retrieval-augmented generation (RAG) is conceptually simple: index documents, retrieve relevant passages, and hand them to a language model to generate an answer. But the difficulty of building reliable systems is this—finding text about a topic is not the same as finding the evidence needed to answer a question.

The truly hard work happens before generation.

Retrieval Relevance Is Not Answer Relevance

The goal of traditional search is to return documents that match a query. RAG's responsibility is more specific: retrieve the evidence that supports the answer.

Vector search may find a general refund policy but miss the renewal process that contains the decisive condition. The retrieved text is semantically relevant but insufficient to answer the question. The model cannot reliably recover a rule it has never seen—it may guess, combine unrelated conditions, or give an incomplete answer with excessive confidence.

This distinction changes how you debug: if the failure point is retrieval, tweaking prompts or upgrading the model only adds cost without solving the problem.

Vector search (also called semantic search) maps queries and documents into dense vectors in a high-dimensional space and measures semantic closeness via cosine similarity or dot product. Its strength is finding conceptually related content across literal differences—for example, a query for "refund" can match a passage about "returning payment." However, vector representations are compressed semantic summaries that lose details like exact identifiers, negations, and numeric boundaries, and may rank semantically similar but opposite-conclusion texts higher. This is precisely the technical root of "semantically relevant but insufficient to answer" mentioned in the article: similarity measures topic closeness, not evidence sufficiency.

Every Stage of the RAG Pipeline Can Introduce Different Failures

A complete RAG pipeline includes: document sources → ingestion → chunking → indexing → retrieval → filtering → reranking → LLM generation → answer.

Each stage has its own distinct failure modes:

Document sources: documents missing, duplicated, or outdated.
Ingestion: content extraction errors.
Chunking: rules and exceptions separated.
Indexing: representations missing or stale.
Retrieval: relevant evidence not found.
Filtering: useful or authorized evidence removed.
Reranking: relevant passages ranked too low.
Generation: evidence misinterpreted or unsupported claims added.

Answer quality depends on every stage, not just the language model.

Chunking Strategy Is Fundamentally a Data Modeling Decision

Chunking is often treated as a configuration option: pick a token limit, add some overlap, index the results. But the more useful question is: in this application, what is the smallest meaningful unit of knowledge?

Suppose the refund policy explains the general rule in one section and the renewal exception in another. Fixed-size splitting may separate the exception from the rule it modifies. Both chunks are valid text, but neither provides sufficient context when retrieved alone.

Chunk boundaries should reflect the structure of the source material:

Policies need to preserve conditions, exceptions, and section headings.
Contracts may need clause boundaries and cross-references.
Product documentation should keep version details together with feature descriptions.
Tables need captions, units, and values to stay associated.

Metadata (such as document title, section, version, effective date, and source identifier) can help preserve context without duplicating the entire document in every chunk.

For a product limits table, if the indexing pipeline extracts the values but loses the column headers, the retrieved text may contain the correct numbers without stating what they mean. The model has to infer relationships the original document already defined.

Chunking is part of knowledge modeling, not just text preprocessing. The goal is to create units that are focused enough to retrieve and complete enough to interpret.

Chunking refers to splitting long documents into text units suitable for indexing and fitting into the context window. Common approaches include fixed token windows with overlap, splitting by paragraph or heading, and dynamically merging by semantic similarity. Chunk size involves trade-offs: too small and context is missing and references can't be resolved; too large and retrieval precision drops, and mixing multiple topics in one chunk dilutes the relevance signal. Overlapping windows can mitigate boundary cut-off issues but increase index size and duplicate recall. In practice, teams often combine metadata tagging (section paths, document titles) with parent-child chunk structures—retrieving small chunks and returning the larger chunk they belong to—to balance localization precision with interpretive completeness.

Vector Search Should Not Solve All Search Problems

Vector search can find conceptually related material when query wording differs from source text. But enterprise questions often contain exact terms, product codes, error messages, or version numbers.

"Explain our refund policy" is different from "Does policy RF-204 apply after renewal?" The former may benefit from semantic matching; the latter may depend on exact identifiers and specific document versions.

Hybrid retrieval combines keyword-based and vector search, then merges the results. Keyword search can capture critical terms like "renewal" and exact identifiers, while vector search tolerates wording differences. The value of hybrid search is in combining complementary strengths, not assuming one retrieval method works equally well for all queries.

Microsoft's hybrid search overview explains how text and vector search work together. The right retrieval strategy depends on the data and the questions users ask—a technical support system may need exact error code matching, while a policy assistant may rely more on semantic similarity but still need exact policy identifiers and version constraints.

Hybrid retrieval typically runs sparse keyword retrieval such as BM25 in parallel with dense vector retrieval, then merges the two result sets using reciprocal rank fusion (RRF) or weighted score normalization. BM25 is based on term frequency and inverse document frequency, is highly sensitive to exact terms, product numbers, and error codes, and naturally carries term importance weights; vector retrieval compensates for paraphrases and long-tail expressions. The key difficulty in fusion is that scores from the two paths are not directly comparable, requiring rank-level fusion or unified normalization. Platforms like Azure AI Search and Elasticsearch have built-in capabilities for this. Whether to enable hybrid retrieval should be validated by comparison on a fixed evaluation set, not enabled by default.

Retrieval Needs More Than a Similarity Score

A similarity score estimates how well a retrieved item matches the query, but it cannot prove that the item contains the answer. A passage may be highly similar to the question yet omit the condition that determines the outcome; another passage with a lower score may contain the decisive exception.

This raises a series of engineering questions: how many candidates should retrieval return? Should keyword and vector results be merged? Which documents are authoritative? How should document versions and effective dates affect ranking? When should a reranker be used? What to do when evidence is incomplete or contradictory?

A typical approach is to retrieve a broader candidate set and then rerank using more detailed relevance assessment. This can improve result ordering but cannot recover evidence that the initial retrieval stage never returned. Filtering also requires care: removing outdated or unauthorized documents is necessary, but overly broad filtering may eliminate the only passage that can answer the question.

For systems handling private or tenant-specific information, authorization must be enforced during retrieval, not left to the language model to choose to ignore unauthorized passages.

What About Questions That Require Multiple Documents?

Some questions cannot be answered from a single passage. For example: "Under current policy, is this customer eligible for a refund?" A reliable answer may require three pieces of evidence: the current refund policy, the customer's contract renewal date, and the applicable exception for that product or contract type.

Retrieving only the general policy is insufficient; retrieving only the contract without the exception is also insufficient. This calls for multi-step retrieval: the system may need to identify missing facts, retrieve additional documents, and combine evidence before generating an answer.

But additional retrieval steps introduce their own risks: an early error can steer subsequent searches in the wrong direction; the system may accumulate irrelevant passages, or treat two documents as consistent when they are not.

A useful design principle is to make evidence requirements explicit. For questions that depend on multiple conditions, define which facts must be established before an answer is considered complete. The system can then distinguish between questions with sufficient evidence and those that still need more information.

With Larger Context Windows, Why Not Just Give the Model Everything?

Larger context windows make it possible to pass more information, but they do not eliminate the need for retrieval. Stuffing an entire knowledge base into the prompt can increase latency, cost, and the amount of irrelevant information the model must process. Large inputs can also make it harder for the model to consistently use the most important details.

More context is not necessarily better context. A good retrieval system narrows evidence to the material that matters for the current question, preserving enough surrounding information to interpret the evidence correctly while avoiding irrelevant passages. The goal is not to minimize context at all costs, but to provide the minimal sufficient set of trustworthy evidence.

This balance depends on the task: a short troubleshooting question may need only one document section; a complex contract question may need multiple clauses and their cross-references. The context window is a resource, and retrieval is how the system decides what deserves to use that resource.

Measure Retrieval First, Then Blame Generation

A fluent answer does not prove that retrieval worked. A convincing evaluation must examine the evidence that reaches the model.

A useful starting point is Recall@K: does the expected relevant source appear among the top K retrieval results? If a question's expected source appears in the top five results, its source recall@5 is 1; if it does not appear, it is 0. Averaged over a set of questions, this indicates how often retrieval finds the expected evidence. For questions requiring multiple documents, evaluation should also check whether all required sources were retrieved.

The following Python function computes source recall@K:

def retrieval_metrics_at_k(expected_sources, retrieved_sources, k=5):
    """
    expected_sources: iterable of relevant source IDs
    retrieved_sources: source IDs ordered by retrieval rank
    """
    expected = set(expected_sources)
    retrieved = retrieved_sources[:k]
    if not expected:
        return {
            "recall_at_k": None,
            "all_sources_found": None,
        }
    found = expected.intersection(retrieved)
    return {
        "recall_at_k": len(found) / len(expected),
        "all_sources_found": found == expected,
    }

For single-source questions, Recall@K is 0 or 1; for questions requiring multiple sources, it measures the proportion of expected sources retrieved. The all_sources_found field indicates whether all expected sources appear in the top K.

This is only a starting point. Source-level evaluation may miss problems within documents: the correct file was retrieved, but the relevant section is not in the returned chunk. Stronger evaluation should track relevant passages, ranking quality, and whether the final answer is supported by the retrieved evidence.

The next step is to evaluate generation separately: given the correct evidence, does the model produce the correct answer? Does it preserve conditions and exceptions? Does it introduce claims not supported by the sources? Separating retrieval and generation metrics makes failures easier to diagnose. If retrieval recall is poor, investigate ingestion, chunking, indexing, query construction, and ranking; if retrieval is strong but answers are still incorrect, investigate context assembly, instructions, model behavior, and answer validation.

Reranking is the process of using a heavier model, such as a cross-encoder, to jointly score the query and each candidate passage after initial recall, producing a more fine-grained relevance judgment than independent vector encoding. A typical flow is to first recall dozens to hundreds of candidates, then have the reranker select the top few to send to generation. It can significantly improve ranking quality, but is limited by the recall set: if decisive evidence never enters the candidate set, reranking cannot remedy it. Therefore reranking optimizes ranking metrics (such as MRR, NDCG), while Recall@K reflects the recall ceiling—the two must be measured separately.

Retrieval Is Also a Lifecycle Problem

Documents change. Policies are revised, products evolve, contracts expire, old processes become obsolete. A retrieval system that worked well at launch can gradually become unreliable if the index no longer reflects current source material.

This is not just an indexing problem—it's a lifecycle problem. Production RAG systems need clear answers to: how are changed documents detected? How quickly are updated documents reindexed? How are deleted or superseded documents removed? How is the authoritative version identified? How are generated answers traced back to sources? How is retrieval degradation detected after index or ranking changes?

Document versions and effective dates are especially important in enterprise systems. The latest document is not always the correct document for every question—historical questions may require the policy in effect on a specific date. Similarly, a document may be technically up to date but not applicable to a specific customer, product, or region.

Retrieval therefore needs to consider authority, time, scope, and access permissions, not just semantic similarity. Evaluation sets should also evolve with the knowledge base. When important documents or business rules change, teams should update test questions and verify that previously correct retrieval behavior still holds.

A Practical Improvement Order

When a RAG system produces unreliable answers, switching models should not automatically be the first reaction. A more effective order is:

Validate source material: check that the required information exists, is current, and can be extracted correctly.
Check chunking: confirm that headings, conditions, exceptions, table relationships, and relevant metadata survive preprocessing.
Inspect retrieval results: capture the actual returned passages for representative questions; don't rely only on final answers.
Improve candidate retrieval: test keyword search, vector search, hybrid retrieval, and appropriate filtering.
Evaluate ranking: check whether relevant passages appear near the top and whether reranking improves their position.
Verify evidence completeness: for multi-document questions, confirm that all necessary facts were retrieved.
Evaluate generation independently: give the model known-good evidence and assess whether its answer is accurate and supported.
Test changes on a fixed evaluation set: compare results before and after each meaningful change rather than relying on anecdotal improvements.
Monitor the lifecycle: track document freshness, indexing failures, retrieval degradation, latency, and cost.

This order helps teams spend effort where it produces measurable differences, and it prevents a common mistake: trying to fix retrieval failures with a more expensive model.

Not Every Question Should Produce an Answer

Even a well-designed retrieval pipeline will sometimes fail to find sufficient evidence. The system should be able to recognize this limitation.

If the retrieved documents cannot determine whether a refund is allowed, the correct response may be to state that the available evidence is insufficient. If two authoritative sources conflict, the system may need to present the conflict rather than silently choosing one. If required documents are inaccessible, the system should not fabricate their contents.

This behavior requires more than a prompt that says "don't hallucinate." It depends on how the application defines evidence sufficiency, handles conflicting sources, and validates answers. For some questions, the application can require specific facts to be present before generating; for others, it can return a source-supported answer with explicit limitations, or ask the user for the missing information.

Abstention is not a substitute for good retrieval—it is a necessary part of handling situations where retrieval cannot provide sufficient evidence. A reliable system must know how to answer, and it must also know when there isn't enough information to answer responsibly.

The Real Work Happens Before the Answer

RAG is often presented as a simple connection between a search index and a language model. That description hides a great deal of engineering work. The quality of the final answer depends on the documents entering the system, how content is represented, which passages are retrieved, how evidence is ranked and assembled, and whether the resulting context is sufficient to answer the question.

The model matters, but it is only one component in a longer decision chain. When an answer is wrong, first check what the model actually received: was the correct document available? Was the relevant passage preserved? Did retrieval find it? Was it ranked high enough? Is the evidence complete, current, and authorized?

These questions are usually more useful than asking "is the model smart enough?" The core question for a RAG system is not whether it can generate an answer, but whether it has retrieved the right evidence, from the right sources, for the right question. That is the starting point for reliable RAG engineering.