How to Choose a RAG Chunking Strategy: A Practical Guide from Fixed Splitting to Late Chunking

You uploaded a document, asked your RAG application a question, and got an incomplete answer. You go back to the source text, and the information is clearly there—so why didn't the system find it? The problem often lies in how the document was split. This article is based on a Day 2 note from a 90-day AI challenge, combined with Chroma's research report and an ACL 2025 paper, to lay out the logic behind choosing RAG chunking strategies and the common pitfalls.

Why chunking determines whether RAG answers correctly

The original text gives a typical example. A company policy states: "Employees may carry over up to five days of unused annual leave. This option is only available after completing one year of employment." The user asks: "Can I carry over unused vacation days?" If the system only retrieves the first sentence, it will answer "Yes" while missing the "one year of employment" precondition. The model got the relevant passage, but not all the information needed to explain the policy.

This is the core problem that chunking strategies must solve: the retrieved text must both locate the answer and support a complete explanation.

RAG (Retrieval-Augmented Generation) is a system architecture that combines information retrieval with text generation: after a user asks a question, the system first retrieves text fragments relevant to the question from a document library, then passes these fragments as context to a large language model to generate an answer. Retrieval quality directly determines what evidence the model can see, so chunking—splitting the original document into the smallest retrievable units—is one of the earliest and most critical decisions in the RAG pipeline. If chunks are poorly cut, subsequent embedding, retrieval, and generation are hard-pressed to compensate.

The dilemma between small and large chunks

A RAG system splits documents into chunks, and these chunks are the basic unit of retrieval. Small chunks are more likely to hit the most specific facts, but once a fact is separated from its heading, definition, or exception clause, its meaning may change. Large chunks preserve more context but also tend to bring irrelevant information into the results.

There is also an easily overlooked embedding problem: when a chunk contains multiple topics, the single vector generated for it must represent all topics at once, making questions about a specific detail harder to match. The original text proposes a criterion: how much text does the system need to "find" the answer, and how much text does it need to "explain" the answer? These two amounts need not be the same.

Embedding is the process of mapping a piece of text into a high-dimensional vector; semantically similar texts are closer together in vector space, and retrieval matches queries with document chunks by computing vector similarity. Because the standard approach generates one vector per chunk, when a chunk mixes multiple topics, this vector can only be some kind of "average" of the semantics of each topic, making queries about a single detail hard to hit precisely. This is one of the fundamental reasons why chunk granularity requires trade-offs: the larger the chunk, the more mixed the semantics the vector carries, and the lower the retrieval discriminability.

Structure-aware chunking: use headings when they exist

Policies, contracts, and technical manuals usually come with their own clues about information attribution. For example, the headings "Full-time employees," "Contract employees," and "Probationary employees" change how the text beneath them is interpreted. If the headings are stripped away during processing, a passage looks like it applies to everyone.

Structure-aware chunking uses headings and sections to preserve this layer of context; chunks can carry a heading path like "Employment Policy → Full-time Employees → Annual Leave." One detail is worth noting here: storing the path as metadata does not automatically make it enter the embedding. Writing the path into the embedded text helps retrieval matching; putting the path into the context sent to the answering model helps the model understand the semantics. These two happen at different stages of the pipeline, and both are useful, but they cannot replace each other. This also explains why document parsing matters—headings and table relationships lost during parsing cannot be recovered by the chunker out of thin air.

Recursive splitting: a reliable starting point when structure is messy

Not all documents have clean sections. For text with inconsistent formatting, recursive splitting is a practical starting point: it first tries larger boundaries like paragraphs, and if that doesn't work, falls back to smaller units. It is simple to run and does not require an additional model to judge each split point.

Its limitations are also straightforward: paragraph breaks do not guarantee that the explanation is complete. A rule may be in one paragraph, and its exception in the next. According to Chroma's research report, a well-configured recursive splitting performs competitively in their evaluation—this reminds us that before introducing complex solutions, we should give simple baselines a fair chance to compare.

Semantic chunking: when topics change but there are no headings

Some documents switch between topics without markers. Semantic chunking uses signals such as sentence similarity to determine which paragraphs should be grouped together, avoiding stuffing unrelated discussions into the same chunk. But there are pitfalls: similar wording does not mean the information belongs to the same group. The descriptions of two products may be almost identical while their constraints differ; a policy exception may also sound nothing like the rule it modifies.

The ACL 2025 paper "Document Segmentation Matters for Retrieval-Augmented Generation" starts from this problem and proposes the PIC method, which uses document summaries to guide sentence grouping. The paper reports improvements on its benchmark—this gives us a reason to consider the method, but it does not mean it dominates on all documents.

What overlap can and cannot save

Overlap between chunks repeats some text in adjacent chunks. If useful context happens to fall near a boundary, overlap lets both sides retain it. But exception clauses pages apart cannot be recovered by repeating a few sentences.

Too much overlap brings another problem: duplicate search results. If retrieval returns five chunks and three of them are nearly identical, the model actually gets far less new evidence than the number of results suggests. The most valuable use of overlap is to protect adjacent context while keeping results from being filled with duplicate content.

Overlap refers to keeping some repeated text between adjacent chunks, a common practice being to let the next chunk look back several tokens or sentences from the end of the previous one. Its purpose is to prevent key information from being cut exactly at a boundary and left incomplete on both sides. But overlap increases index size and storage costs, and may also lead to highly duplicated retrieval results. In practice, an overlap ratio of 10%–20% is usually set, combined with deduplication or diversity re-ranking to suppress duplicate results. Overlap is not a panacea; it can only protect adjacent context and cannot recover related information scattered across long distances.

Parent document retrieval: search with small passages, answer with large sections

This is where RAG design gets interesting: search and answering need not use the same amount of text. The parent–child retrieval process is: first use small chunks for precise matching, then locate the larger section to which the matched chunk belongs, and finally send the relevant section to the answering model.

Returning to the annual leave example, a small chunk can help the system locate "may carry over five days," while the surrounding section brought back can also recover the "one year of employment" eligibility condition. This separates two things: finding the correct location, and providing enough context to explain it. Of course, limits are needed—returning an entire section for every match wastes the context budget, and overlapping sections should be merged to avoid the model receiving the same content repeatedly.

Late chunking: what if a chunk loses its meaning before retrieval?

Look at this sentence: "It applies only to employees based in London." What does "it" refer to? A person reading the previous paragraph knows immediately, but an embedding model processing this sentence in isolation lacks the information.

Late chunking targets exactly this problem. The conventional approach is to split first and then embed each chunk separately; late chunking instead lets a compatible embedding model first process a longer piece of text, then generate vectors for each chunk from the token representations that have already "seen" the surrounding context. This way each chunk's vector carries more context.

But note two things: the answering model still needs the actual supporting text, and better embeddings will not automatically add the previous paragraph to the retrieval results; late chunking also requires a compatible embedding pipeline—simply throwing a long document at a standard embedding API does not amount to implementing it.

Late chunking relies on long-context embedding models (such as models supporting thousands of input tokens). Its approach is to first perform a forward encoding of the entire long text to obtain context-dependent representations for each token, then pool these token representations according to chunk boundaries (such as average pooling) to generate vectors for each chunk. This is the reverse of the conventional "split first, then encode separately" process, so the chunk vectors naturally carry surrounding context. Implementation requires models and inference frameworks that support token-level output and custom pooling, rather than calling a standard embedding interface that returns a single sentence vector.

How to judge whether chunking actually helps

Checking whether the retrieval results contain the answer is only the starting point. In the policy example, finding "five days of unused annual leave" is only partial success—the eligibility condition is missing. A more rigorous evaluation should ask: did retrieval find all the required evidence? Does the answer preserve the conditions and exceptions? How much irrelevant text entered the model? How much of the retrieved text is duplicated? Do the cited passages support the complete answer?

Chroma's report evaluates retrieval at the token level, measuring both relevant text and redundant, duplicated content. This also explains why a large chunk containing the answer may still be an inefficient retrieval result. In addition, comparisons between different configurations must be fair: five short chunks and five long chunks give the model completely different amounts of information, so recording the actual amount of retrieved text is what makes comparisons meaningful.

What to consider when choosing a chunking strategy

Start from the type of information the document contains and the questions users will ask. If headings are clear, structure-aware chunking is worth using; if formatting is messy, recursive splitting is a reasonable baseline; if topics switch without markers, consider semantic chunking. Overlap handles adjacent boundaries, parent–child retrieval recovers surrounding explanations, and late chunking solves context loss during embedding. These methods solve different problems and can be combined.

There is only one goal: preserve enough meaning so that a relevant retrieval result is also useful evidence. When your RAG application misses conditions and gives incomplete answers, look at the retrieved passages—did the model receive a complete explanation, or a sentence that looks like one?

One-line Summary

The core of RAG chunking is balancing the amount of text needed to "find the answer" and to "explain the answer," and choosing a strategy based on document structure.

Paragraph Summary

Starting from a retrieval failure case involving annual leave policy, the article points out the core contradiction of RAG chunking: retrieving the text where the answer is located does not equal retrieving all the context needed to explain the answer. Around this contradiction, the article compares strategies such as structure-aware chunking, recursive splitting, semantic chunking, inter-chunk overlap, parent–child retrieval, and late chunking, explaining their applicable scenarios and limitations. The article emphasizes easily overlooked details such as metadata not automatically entering the embedding, overlap being unable to recover exception clauses across long distances, and late chunking relying on a compatible embedding pipeline, and advocates combining these methods based on document structure and user questions, evaluating chunking effectiveness by whether retrieval provides complete evidence rather than whether it hits keywords.