RAG Chunking Strategies in Practice: 5 Runnable Python Solutions

Chunking is the most often glossed-over step in RAG tutorials—most just drop the line "split the document into 500-token chunks," and poor retrieval performance usually starts right here. The right chunking strategy depends on the structure of the source document and the shape of the queries, not on some blog's default number. This article walks through 5 chunking strategies that cover roughly 95% of production RAG pipelines, each with runnable Python code.

Prerequisite reading: if you haven't built a full pipeline yet, start with "Building a RAG Pipeline from Scratch." This article corresponds to step 1 of that guide.

Why chunking is so critical

A chunk is the basic unit of retrieval. After the user asks a question, the retriever returns the top-K chunks whose embeddings are closest to the question. Three failure modes arise directly from the shape of the chunks:

• Chunks too large: embeddings average out specificity. The chunk gets retrieved for too many weakly related queries, pushing genuinely relevant chunks out of the top-K.
• Chunks too small: chunks lose context. A retrieved passage reads fine on its own, but the LLM can't answer because the answer's anchor lies in another chunk that wasn't retrieved.
• Chunks split across semantic boundaries: the definition lives in one chunk and the example in another; only one is retrieved, and the LLM gets half an answer.

The five strategies below address these three failure modes in different ways.

Embedding is the process of mapping a piece of text into a high-dimensional vector—texts with similar semantics have vectors pointing in similar directions in that space. At retrieval time, the user's question is also encoded into a vector, and the system uses a distance metric such as cosine similarity to find the top-K chunks closest to the question vector. This explains a counterintuitive phenomenon: the larger the chunk, the more topics its vector covers, and the more "centered" its direction becomes, so it isn't sufficiently similar to any specific question; the smaller the chunk, the more focused the vector, but the less semantic information it carries. Chunking is therefore fundamentally a trade-off between "retrieval precision" and "context completeness," and that trade-off has no universal optimum—it depends on whether your queries are detail-oriented or summary-oriented.

Top-K retrieval refers to the retrieval stage returning the top K chunks by similarity, where K is usually 3 to 10. Its key constraint is: once K is fixed, the more chunks there are, the lower the probability that any single chunk gets selected; the larger the chunks, the higher the risk that they hog the slots. This is the mechanism behind the article's point that "chunks too large push genuinely relevant chunks out of the top-K."

Common setup

All strategies share the same scaffolding. Install the dependencies first:

pip install langchain-text-splitters sentence-transformers numpy

Define a helper data class for displaying chunks:

from dataclasses import dataclass
from typing import Callable

@dataclass
class Chunk:
    text: str
    index: int
    strategy: str
    metadata: dict

def show(chunks: list[Chunk]) -> None:
    for c in chunks:
        preview = c.text[:80].replace("\n", " ")
        print(f"[{c.strategy} #{c.index:02d}] ({len(c.text)} chars) {preview}...")

Strategy 1: Fixed-size chunking

Split by N characters (or tokens), keeping M characters of overlap between adjacent chunks. It's structure-, sentence-, and semantics-agnostic. Fast, reproducible, and often a reasonable default when the corpus is uniform.

def fixed_size_chunk(text: str, size: int = 400, overlap: int = 50) -> list[Chunk]:
    chunks = []
    start = 0
    i = 0
    while start < len(text):
        end = min(start + size, len(text))
        chunks.append(Chunk(
            text=text[start:end],
            index=i,
            strategy="fixed",
            metadata={"start": start, "end": end},
        ))
        start = end - overlap
        i += 1
    return chunks

When to use it: uniform prose, logs, chat logs, source code where line boundaries don't carry semantics.

When not to use it: structured documents where an arbitrary cut separates a heading from its content, or splits a code block in the middle of a function.

Strategy 2: Recursive character chunking

Walk a priority-ordered list of separators—paragraphs, newlines, sentences, words, characters—and split at the highest-priority separator while keeping chunks under the limit. This is LangChain's RecursiveCharacterTextSplitter, and it's a reasonable default for mixed-structure content.

from langchain_text_splitters import RecursiveCharacterTextSplitter

def recursive_chunk(text: str, size: int = 400, overlap: int = 50) -> list[Chunk]:
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=size,
        chunk_overlap=overlap,
        separators=["\n\n", "\n", ". ", " ", ""],
    )
    raw = splitter.split_text(text)
    return [
        Chunk(text=c, index=i, strategy="recursive", metadata={})
        for i, c in enumerate(raw)
    ]

When to use it: prose of varying structure—documents, articles, Notion pages, Markdown with lists and code blocks. It's usually the default for team docs too.

When not to use it: extremely long paragraphs with no internal line breaks (it degenerates to character-level splitting); when the separator list doesn't match the document format (e.g., source code's hierarchy is class → method → statement).

Strategy 3: Sentence-boundary chunking

First split the document into sentences, then greedily group consecutive sentences into chunks until the size budget is reached. Chunks never cut mid-sentence, and embeddings come out cleaner.

import re

def sentence_chunk(text: str, size: int = 400) -> list[Chunk]:
    # Simplified sentence splitting. In production, prefer spaCy or NLTK to handle abbreviations, etc.
    sentences = re.split(r"(?<=[.!?])\s+", text.strip())

    chunks: list[Chunk] = []
    current: list[str] = []
    current_len = 0
    i = 0

    for s in sentences:
        if current_len + len(s) > size and current:
            chunks.append(Chunk(
                text=" ".join(current),
                index=i,
                strategy="sentence",
                metadata={"n_sentences": len(current)},
            ))
            i += 1
            current = [s]
            current_len = len(s)
        else:
            current.append(s)
            current_len += len(s) + 1

    if current:
        chunks.append(Chunk(
            text=" ".join(current),
            index=i,
            strategy="sentence",
            metadata={"n_sentences": len(current)},
        ))
    return chunks

When to use it: QA corpora, FAQ documents, interview transcripts, conversations. Any corpus where the natural semantic unit is one or two sentences.

When not to use it: code (semicolons end statements, but functions are the meaningful unit); text with lots of abbreviations that confuse sentence tokenizers; non-English languages without clear end-of-sentence punctuation.

Strategy 4: Document-structure chunking

Use the document's own structural markers—Markdown headings, HTML <h1>/<h2>, docstring sections—to define chunk boundaries. Each chunk is a self-contained section. Chunk sizes will be uneven, which is fine as long as a single section doesn't exceed the embedding model's context window.

def markdown_section_chunk(text: str, min_size: int = 100) -> list[Chunk]:
    lines = text.split("\n")
    chunks: list[Chunk] = []
    buffer: list[str] = []
    current_heading = ""
    i = 0

    def flush():
        nonlocal i
        if not buffer:
            return
        section = "\n".join(buffer).strip()
        if len(section) >= min_size:
            chunks.append(Chunk(
                text=section,
                index=i,
                strategy="structure",
                metadata={"heading": current_heading},
            ))
            i += 1

    for line in lines:
        if re.match(r"^#{1,6}\s", line):
            flush()
            buffer = [line]
            current_heading = line.lstrip("# ").strip()
        else:
            buffer.append(line)
    flush()
    return chunks

When to use it: technical documentation, Markdown knowledge bases, structured reports, policy documents, textbook chapters. Corpora where the author deliberately used headings to organize ideas.

When not to use it: unstructured prose (novels, emails, chats); documents whose single section is too large to embed as one chunk; when section headings are too generic ("Overview," "Introduction") to help embeddings carry context.

Strategy 5: Semantic chunking

Embed every sentence, then look at the cosine similarity between adjacent sentence embeddings. Where similarity drops below a threshold is a topic boundary—split there. Chunks are grouped by semantic coherence rather than character count.

The cost is higher than the other four (one embedding call per sentence at index time), but when the corpus is dense and topics drift smoothly within sections, it produces the cleanest chunks.

import numpy as np
from sentence_transformers import SentenceTransformer

_model = None

def _embedder():
    global _model
    if _model is None:
        _model = SentenceTransformer("all-MiniLM-L6-v2")
    return _model

def semantic_chunk(text: str, similarity_threshold: float = 0.5) -> list[Chunk]:
    sentences = re.split(r"(?<=[.!?])\s+", text.strip())
    if len(sentences) < 2:
        return [Chunk(text=text, index=0, strategy="semantic", metadata={"n_sentences": len(sentences)})]

    embeddings = _embedder().encode(sentences)
    sims = []
    for i in range(len(sentences) - 1):
        a, b = embeddings[i], embeddings[i + 1]
        sims.append(float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))))

    chunks: list[Chunk] = []
    buffer = [sentences[0]]
    i = 0
    for idx, sim in enumerate(sims):
        if sim < similarity_threshold:
            chunks.append(Chunk(
                text=" ".join(buffer),
                index=i,
                strategy="semantic",
                metadata={"split_similarity": round(sim, 3)},
            ))
            i += 1
            buffer = [sentences[idx + 1]]
        else:
            buffer.append(sentences[idx + 1])

    if buffer:
        chunks.append(Chunk(
            text=" ".join(buffer),
            index=i,
            strategy="semantic",
            metadata={},
        ))
    return chunks

The similarity threshold is a tunable parameter: lower values yield larger, fewer chunks (more tolerant of topic drift); higher values yield smaller, more numerous chunks (stricter about topic coherence). Start at 0.5 and fine-tune on an evaluation set.

When to use it: long papers, academic articles, dense technical writing where topics drift gradually but there are no clear structural markers. News archives, podcast transcripts, meeting notes.

When not to use it: short documents (too few sentences to cluster); content that already has good structure (document-structure chunking is cheaper and more faithful to reality); heterogeneous corpora where no single threshold generalizes across documents.

Cosine similarity measures how close two vectors' directions are, ranging from -1 to 1, independent of vector length, which makes it suitable for comparing embeddings of texts of different lengths. It's computed as the dot product of the two vectors divided by the product of their magnitudes. The np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)) in the semantic_chunk function is a direct implementation of this formula; for already-normalized sentence-transformers outputs, the dot product itself approximates cosine similarity.

all-MiniLM-L6-v2 is a lightweight model of about 22MB in the sentence-transformers library that outputs 384-dimensional vectors, striking a good balance between speed and semantic quality, and is commonly used for local prototyping. Its context window is limited (about 256 tokens), which is why semantic chunking typically encodes sentence by sentence rather than paragraph by paragraph. Note that this model was trained on English corpora; for multilingual scenarios, switch to a model like paraphrase-multilingual-MiniLM.

How to choose a strategy: a rough rule

Match the strategy to the source document's dominant shape:

• Uniform prose, logs, chat logs → fixed-size
• Mixed Markdown / HTML / Notion-style documents → recursive
• QA, FAQ, conversations, interviews → sentence boundary
• Technical docs with headings, textbooks, policies, specs → document structure
• Long papers, academic articles, meeting notes without structural markers → semantic

Before making a final decision, benchmark two or three candidates on a held-out evaluation set. In practice, recursive chunking with Markdown separators works as a default for most production corpora, and the other four fit specific scenarios better.

What this article doesn't cover

The long tail of chunking in production goes well beyond these five strategies. Not covered: a 30-second decision table from corpus shape to strategy; edge cases such as multilingual mixed scripts, PDFs with tables mangled by crude extractors, and legal contracts with nested clause structures; chunk metadata schemas; chunker versioning; and incremental reindexing strategies—detecting which chunks actually changed after a source document is edited.

Related reading: Building a RAG Pipeline from Scratch, vector database comparison (pgvector, Qdrant, Pinecone, Chroma), when not to use RAG, and a RAG production FAQ.

One-line Summary

A walkthrough of 5 production-grade RAG chunking strategies and how to choose between them, with runnable Python code.

Paragraph Summary

The article points out that chunking is the most overlooked step in RAG tutorials yet directly determines retrieval quality, with three main failure modes: chunks too large, too small, and split across semantic boundaries. The author presents 5 strategies covering roughly 95% of production pipelines: fixed-size, recursive character, sentence boundary, document structure, and semantic chunking, each with runnable Python code and notes on when to use and not use it. The core conclusion is that the chunking strategy should depend on source document structure and query shape, not on some default token count; recursive chunking with Markdown separators can serve as the default for most production corpora, while the other four fit specific scenarios—and before deciding, benchmark two or three candidates on a held-out evaluation set.