Chonkie
A text chunking library within the Python ecosystem, specifically optimized for tiktoken integration. It delivers strong performance in chunking scenarios that rely on BPE (Byte Pair Encoding) tokenization, making it well-suited for workflows centered on OpenAI-compatible tokenizers.
Timeline (last 90 days)
In the recursive chunking benchmark (1MB, 1000/200), Chunkr achieved 2,264 MB/s, LangChain achieved 769 MB/s, LlamaIndex only 10 MB/s, and Chonkie 225 MB/s.
In the BPE tokenization (cl100k_base, 512/50) scenario, Chunkr achieved 38 MB/s, while Chonkie led at 151 MB/s.
Chonkie has been specifically optimized for tiktoken integration, while Chunkr has not yet reached the same level of adaptation at the tokenizer call layer, which is the reason Chunkr lags behind in the BPE scenario.
All Facts (3)
In the recursive chunking benchmark (1MB, 1000/200), Chunkr achieved 2,264 MB/s, LangChain achieved 769 MB/s, LlamaIndex only 10 MB/s, and Chonkie 225 MB/s.
50%UnverifiedIn the BPE tokenization (cl100k_base, 512/50) scenario, Chunkr achieved 38 MB/s, while Chonkie led at 151 MB/s.
50%UnverifiedChonkie has been specifically optimized for tiktoken integration, while Chunkr has not yet reached the same level of adaptation at the tokenizer call layer, which is the reason Chunkr lags behind in the BPE scenario.
50%