Chunkr: A Rust-Powered Text Chunking Library That's 20x Faster

When building RAG (Retrieval-Augmented Generation) systems, text chunking is often an overlooked step that can have a real impact on performance. As document volumes grow, mainstream chunking tools in the Python ecosystem start to show speed bottlenecks. One developer set out to solve this problem in his own system by writing a chunking library from scratch in Rust — called Chunkr — and published a set of impressive benchmark numbers to back it up.

Why Build Another Chunking Library

The author's motivation was straightforward: he needed a faster chunking library without sacrificing chunking accuracy, and no existing solution adequately satisfied both requirements. So he built Chunkr in Rust, covering all the major chunking strategies in use today: Character chunking, Recursive chunking, Markdown heading-based chunking, Late chunking, and Hierarchical chunking.

Beyond chunking strategies, Chunkr also includes a native PDF loader with support for several other file types. This means it's not just a string-splitting utility — it integrates the entire "document loading + chunking" pipeline into a single tool, which happens to be the most time-consuming part of RAG preprocessing.

Late Chunking is a relatively novel strategy that inverts the traditional "chunk first, then embed" workflow. Instead, it performs a full contextual embedding of the entire document first, then splits within the embedding space. This ensures each chunk's vector representation carries global context, effectively reducing the semantic truncation that traditional chunking causes — especially useful for long texts with cross-paragraph coreferences. Hierarchical Chunking splits documents into two levels: coarse-grained "parent chunks" and fine-grained "child chunks." Retrieval uses child chunks for precision, while responses include parent chunks for context. This is a widely adopted architecture in modern RAG systems for balancing precise matching with semantic completeness. Both strategies are far more complex to implement than simple character splitting, and combining them with high throughput is one of Chunkr's key differentiators over basic string-splitting tools.

Chunking Speed: Benchmarks Against Popular Python Tools

The author ran benchmarks on a MacBook (M4 chip, 16GB RAM), comparing Chunkr against LangChain, LlamaIndex, Chonkie, semchunk, and text-splitter. With aligned parameters (e.g., chunk size 1000 / overlap 200), the performance gaps were striking:

• Recursive chunking (1MB, 1000/200): Chunkr hits 2,264 MB/s, versus LangChain at 769 MB/s, LlamaIndex at just 10 MB/s, and Chonkie at 225 MB/s.
• Fixed character chunking (1MB): Chunkr at 750 MB/s versus LangChain at just 1.7 MB/s — a gap of hundreds of times.
• Python code chunking (200KB, 1500/200): Chunkr reaches 3,232 MB/s versus text-splitter at 5.7 MB/s.
• Parallel batch processing (100 documents × 50KB): Chunkr sustains 3,224 MB/s, demonstrating its parallel processing capability.

One notable exception is the BPE tokenization (cl100k_base, 512/50) scenario: Chunkr comes in at 38 MB/s, while Chonkie leads with 151 MB/s. This shows that when a real tokenizer is involved, the performance landscape shifts — Chunkr is not the fastest in every dimension.

BPE (Byte Pair Encoding) is the subword tokenization algorithm used by GPT-series models, and cl100k_base is the specific vocabulary used by models like GPT-4 and text-embedding-ada-002. BPE-based chunking converts text into a token sequence first, then splits by token count — ensuring each chunk stays within a strictly controlled token budget, which is critical when you need to precisely match a model's context window limit. Unlike character-level chunking, BPE tokenization involves calling a real tokenizer model (typically via libraries like tiktoken), and that step has a fixed computational overhead that masks the speed advantage of the underlying string processing. This explains why Chonkie outperforms Chunkr in this scenario — Chonkie has been specifically optimized for tiktoken integration, while Chunkr's tokenizer-layer integration is not yet at the same level.

PDF Parsing and End-to-End Pipeline

Beyond chunking, PDF parsing itself is a common bottleneck. The author benchmarked Chunkr's PDFLoader against PyMuPDF (fitz) and pypdf (pure Python), using pypdf as the baseline (1.0×):

• Chunkr PDFLoader (full document): 747.9 ms, 2,762 pages/sec — 15.9× faster than pypdf.
• Chunkr PDFLoader (per-page document): 721.0 ms, 2,865 pages/sec — 16.5× faster.
• PyMuPDF (fitz): 2,616.8 ms, 789.5 pages/sec — 4.5× faster.
• pypdf (pure Python): 11,900.5 ms, 173.6 pages/sec — baseline.

More representative of real-world usage is the end-to-end test. Chunkr's full "PDF + recursive chunking" pipeline takes 798.1 ms (2,589 pages/sec), compared to pypdf + LangChain RecursiveTextSplitter at 12,054.5 ms (171.4 pages/sec) — an overall speedup of roughly 14.9×. Even against the PyMuPDF + LangChain combination (2,659.3 ms), Chunkr shows a clear advantage.

The value of these numbers lies in what they measure: the complete "file read to finished chunks" pipeline, not a single isolated step — which is far more reflective of real-world data pipeline gains.

PyMuPDF (fitz) is a Python binding for the MuPDF rendering engine, which is itself implemented in C/C++, which is why it stands out among Python-based PDF tools (~4.5× faster than pypdf). Even so, Chunkr's native Rust PDF loader still outpaces PyMuPDF by roughly 3.5×, demonstrating that Rust's advantages in memory management and zero-copy string handling remain significant for mixed I/O and parsing tasks like PDF text extraction. End-to-end pipeline testing is more practically meaningful: real RAG preprocessing never benchmarks PDF parsing and chunking in isolation — they run sequentially, and any bottleneck in either step drags down the whole pipeline. Chunkr merges both steps into a single Rust call, eliminating the Python-layer data transformation and memory copy overhead between them. This is also why the end-to-end multiplier (14.9×) is close to the PDF parsing multiplier (15.9×) rather than being the product of the two.

How to Interpret These Numbers

The results show that Rust delivers real language-level performance advantages for CPU-intensive, string-and-I/O-mixed tasks like these — especially for pure character chunking and code chunking, where it pulls ahead of pure Python implementations by orders of magnitude. For retrieval systems that need to process large volumes of documents in bulk, this kind of throughput improvement can directly translate to shorter index build times and lower preprocessing costs.

That said, a few things are worth keeping in mind. First, all data comes from the author's own single-machine (M4) testing, with no third-party reproduction. Second, throughput (MB/s) is not the same as chunking quality — the author claims accuracy is unaffected, but the benchmarks include no comparison of accuracy or recall metrics. Third, Chunkr falling behind Chonkie in the BPE tokenizer scenario is a reminder that tool selection still needs to be evaluated against your specific chunking strategy and downstream embedding approach.

For teams building RAG pipelines where document parsing and chunking have become bottlenecks, Chunkr is worth adding to the candidate list for a real-world test. The author has open-sourced the project on GitHub and is actively seeking feedback — interested developers can go verify these numbers firsthand.