chunking

Chunk boundaries should reflect the structure of the source material; for example, policies need to preserve conditions, exceptions and section headings, while contracts may need clause boundaries and cross-references.

The goal of an efficient chunking strategy is to create units that are focused enough to be retrieved and complete enough to be interpreted.

Chunking refers to splitting long documents into text units suitable for indexing and fitting into the context window; common approaches include fixed token windows with overlap, splitting by paragraph or heading, and dynamic merging by semantic similarity.

Chunking is a key step in the RAG pipeline; common splitting methods include splitting by fixed length, by paragraph boundaries, and by semantic similarity, and chunk size requires trade-offs.

RAG tutorials rarely cover a key issue: chunk boundaries are almost always wrong; when documents are chunked and embedded, assumptions are actually being made about which context belongs together, and these assumptions often do not hold.

If a RAG pipeline returns results that are technically correct but contextually useless, the problem is almost certainly in the chunking or metadata, not the embedding model.

The choice of chunking granularity is essentially a trade-off between precision and contextual completeness: chunks that are too small allow precise location of code but lose context, while chunks that are too large retain context but degrade retrieval accuracy and vector representation quality.