Chunkr
A high-performance text chunking library implemented in Rust, supporting multiple chunking strategies including character-based, recursive, Markdown, lazy, and hierarchical chunking. It includes a built-in PDF loader and achieves significantly higher throughput compared to mainstream Python-based chunking tools.
Timeline (last 90 days)
Chunkr is a text chunking library implemented in Rust that supports multiple chunking strategies including character chunking, recursive chunking, Markdown heading chunking, late chunking, and hierarchical chunking, and includes a native PDF loader and support for other file types.
In the recursive chunking benchmark (1MB, 1000/200), Chunkr achieved 2,264 MB/s, LangChain achieved 769 MB/s, LlamaIndex only 10 MB/s, and Chonkie 225 MB/s.
In the BPE tokenization (cl100k_base, 512/50) scenario, Chunkr achieved 38 MB/s, while Chonkie led at 151 MB/s.
In the full PDF parsing test, Chunkr PDFLoader took 747.9 ms (2,762 pages/second), which is 15.9× faster than the baseline pypdf and 4.5× faster than PyMuPDF.
All benchmarks were conducted on the author's MacBook (M4 chip, 16GB RAM) as a single machine, with no third-party reproduction.
Chunkr's benchmarks only measure throughput (MB/s) and do not provide comparative metrics for chunking accuracy or recall.
Chonkie has been specifically optimized for tiktoken integration, while Chunkr has not yet reached the same level of adaptation at the tokenizer call layer, which is the reason Chunkr lags behind in the BPE scenario.
All Facts (7)
Chunkr is a text chunking library implemented in Rust that supports multiple chunking strategies including character chunking, recursive chunking, Markdown heading chunking, late chunking, and hierarchical chunking, and includes a native PDF loader and support for other file types.
50%UnverifiedIn the recursive chunking benchmark (1MB, 1000/200), Chunkr achieved 2,264 MB/s, LangChain achieved 769 MB/s, LlamaIndex only 10 MB/s, and Chonkie 225 MB/s.
50%UnverifiedIn the BPE tokenization (cl100k_base, 512/50) scenario, Chunkr achieved 38 MB/s, while Chonkie led at 151 MB/s.
50%UnverifiedIn the full PDF parsing test, Chunkr PDFLoader took 747.9 ms (2,762 pages/second), which is 15.9× faster than the baseline pypdf and 4.5× faster than PyMuPDF.
50%UnverifiedAll benchmarks were conducted on the author's MacBook (M4 chip, 16GB RAM) as a single machine, with no third-party reproduction.
50%UnverifiedChunkr's benchmarks only measure throughput (MB/s) and do not provide comparative metrics for chunking accuracy or recall.
50%UnverifiedChonkie has been specifically optimized for tiktoken integration, while Chunkr has not yet reached the same level of adaptation at the tokenizer call layer, which is the reason Chunkr lags behind in the BPE scenario.
50%