The Fatal Blind Spot in RAG Retrieval: The Paragraph Was Found, but the Heading Was Lost

Title

The Fatal Blind Spot in RAG Retrieval: The Paragraph Was Found, but the Heading Was Lost

Meta Description

Why does a RAG system answer a basic-plan refund question with enterprise-plan terms? The problem isn't the model — it's that chunking drops headings and hierarchy paths. This article breaks down this common engineering trap and offers four fixes: structure-aware splitting, heading-path injection, metadata filtering, and citation verification.

Article Content

A user asks about the refund window, and the AI assistant confidently answers "thirty days," complete with a citation to the policy text. But this user is on the basic plan — which only allows fourteen days. The cited content is real; it just comes from the enterprise account's terms.

The root cause isn't the model — it's document chunking. This is a technical short read worth careful attention from anyone building RAG (Retrieval-Augmented Generation) systems. Author Serguey Asael Shinder lays out a widely overlooked engineering trap in poetic form.

How the Problem Happens

The policy document has a clear structure: each plan has a heading, and under each heading is a paragraph that opens with the same sentence pattern — "Refunds may be requested within..." The basic-plan and enterprise-plan paragraphs are worded almost identically; the only distinction is buried in the heading.

Before the document is indexed, a script splits it top-down by token count — a few hundred tokens per chunk — with no regard for where sections begin. The result: the heading lands at the end of one chunk, and the paragraph lands at the start of the next.

The index ends up with two passages about refunds, nearly identical in wording, but neither states which plan it belongs to. The retriever ranks by semantic similarity, both passages score the same against the question, and the model receives two unlabeled "confident answers" and picks one at random.

The Model Isn't Wrong — What It's Fed Is

The author emphasizes a point that's easy to misattribute emotionally: the model is entirely innocent. It faithfully answers based on the context it receives, with no hallucination whatsoever. The problem is that once text is stripped of its context, its meaning collapses.

The author uses three analogies to describe this information loss:

• A row of data losing its column name
• A step losing the process it belongs to
• A constraint losing the product it constrains

This is the essential risk of chunking: mechanically splitting by token count amounts to deliberately discarding the semantic hierarchy the author already built into the document structure. Headings, sections, list items, table headers — these structural elements aren't decoration; they're prerequisites for disambiguation.

Semantic collapse borrows the metaphor of "gravitational collapse" from physics: the full meaning of a passage is supported by two dimensions — what it says and where it says it. Remove the latter, and its coordinates in semantic space cave in toward the nearest common attractor, with multiple previously distinguishable points contracting into one. In embedding space, this manifests as two passages that should be far apart receiving nearly identical vectors — because the embedding model only sees phrases like "Refunds may be requested within...," unable to perceive the heading above them. It's worth stressing that this is not a flaw of the embedding model: any method that compresses text into a fixed-length vector must discard some information. The issue is that we actively throw away structural cues the author already provided for free. Understanding this explains why "swap in a stronger embedding model" often can't fix this class of bug — the collapse happens before encoding, and no model, however powerful, can undo it.

How to Fix It: Give Every Chunk an "Address"

The author offers concrete solutions that can be distilled into four operational principles.

1. Split along structure, not by count

Chunk according to the structure the author already built — headings, sections, list items, data rows that keep their headers. Document structure itself is the most natural chunking boundary.

2. Inject the hierarchy path before embedding

Before embedding a chunk, prepend the document title and the path formed by all headings above it. For example, "Pricing Policy > Basic Plan > Refunds." This way every chunk carries its own contextual coordinates.

Hierarchy-path injection (sometimes called a contextual chunk header or breadcrumb prefix) works because it puts disambiguating information into the same vector as the body text, rather than leaving it in external metadata to be passively used at retrieval time. In practice you typically prepend a path line to the chunk body, like "Pricing Policy > Basic Plan > Refunds\nRefunds may be requested within fourteen days...," then embed the whole thing. The cost is that the prefix dilutes the weight of the body and slightly lengthens the sequence, so the path shouldn't go too deep — two or three levels up is usually enough. An equivalent variant is "contextual retrieval": first use an LLM call to generate a sentence describing each chunk's "situated context" before embedding. The effect is similar but the cost is higher. Either way, the core idea is to let each chunk carry a retrievable contextual coordinate rather than relying on the retriever to guess.

3. Store metadata as filterable fields

Store key dimensions like "plan type" as standalone fields that support filtering. When you know who's asking, filter out irrelevant branches directly instead of gambling on vector similarity.

Metadata filtering is part of hybrid retrieval: first narrow the candidate set with structured conditions, then rank by vector similarity within that reduced scope — commonly known as pre-filtering. It complements the previous principle — hierarchy paths solve "how to distinguish within a document," while metadata filtering solves "how to distinguish across documents and across users." Two practical caveats: first, filter fields must be extracted at index time; adding them later usually requires rebuilding the entire index. Second, over-filtering reduces recall — when filter conditions are too narrow and the candidate set is empty, the system needs a graceful fallback (e.g., relaxing conditions or indicating it can't answer) rather than forcing out an irrelevant snippet. Making high-discrimination dimensions like "plan type" into filterable fields is essentially using structured knowledge to do the elimination the retriever is bad at.

4. Show the model the heading and require citations

Give the model both the heading and the body, and have it cite both. If the model must output "Source: Pricing Policy > Basic Plan > Refunds," then the problem of lost headings will surface during citation.

Test Case: Same Sentence, Different Section, Different Answer

The author adds one crucial suggestion: write a test that catches this class of bug.

The test design: the answer varies by section, but the wording is the same — same phrasing, different plan, different number. If the retrieval system answers these cases incorrectly, it means structural information was already lost during chunking.

This kind of test is far more useful than "asking a question that can be answered correctly," because it specifically attacks the weakest seam in the chunking logic.

In engineering, this kind of test is often called a differential test or contrast test: construct a pair of inputs that differ only along one dimension (here, the section they belong to) while everything else is identical, then assert the system must produce different answers. It's the opposite mindset from an ordinary "Q&A accuracy test" — the latter aims for "getting it right," while the former specifically targets "not getting it wrong at highly confusable points," making it better at exposing systematic retrieval-layer defects. When designing tests, cover three directions: positive cases (asking about the basic plan should yield fourteen days), distractor cases (asking about the enterprise plan should yield thirty days, verifying the system doesn't conflate the two passages), and citability (requiring the model to output the source path, verifying the heading actually entered the context). Bake these cases into regression tests, and any change to the chunking strategy will immediately surface in CI — far cheaper than discovering the problem through user complaints after launch.

An Overlooked Engineering Common Sense

The closing line of the article deserves to be pinned to every RAG project's README:

A paragraph is not a fact. It is a fact plus where it was written.

A passage of text isn't a fact — it's a fact plus where it was written in the original.

A large share of the "hallucinations," "irrelevant answers," and "citation errors" in RAG systems, traced to their source, aren't model capability problems at all — they're information loss during index construction. Chunking strategy, metadata design, hierarchy injection — these seemingly boring engineering details are precisely what determine a system's trustworthiness on edge cases.

The model handles reasoning, but only if you haven't already cut away the answer's context before it ever sees the question.

One-line Summary

Many RAG "hallucinations" stem from splitting documents by token count, which separates headings from body text and causes semantic collapse.

Paragraph Summary

The article opens with a real case: an AI assistant gives a basic-plan user the thirty-day enterprise-plan refund window, when that user actually has only fourteen days. The author points out the problem isn't the model but document chunking — mechanically splitting by token count separates headings from body text across different chunks, stripping two nearly identically worded refund passages of their disambiguating context, leaving the retriever to pick at random. This is framed as "semantic collapse": text loses information when detached from its context. The author offers four fix principles: split along document structure, inject hierarchy paths before embedding, store key dimensions as filterable metadata, and show the model the heading while requiring source citations; and recommends writing test cases where "same wording, different section, different answer" to expose this seam problem. The core conclusion: a passage of text isn't a fact — it's a fact plus where it was written.