Tokka-Bench: A New Tokenizer Evaluation Framework Spanning 100 Languages

The capabilities of large language models depend not only on parameter scale and training data, but also on the underlying tokenizer — which is itself a decisive factor. An open-source research project called Tokka-Bench argues that tokenizer quality varies significantly across languages, yet the field has long lacked a standardized, multi-metric framework for comparing tokenizers side by side. This arXiv paper aims to fill exactly that gap.

Why Tokenizer Evaluation Matters

Subword tokenizers are the first gateway through which large language models process text. They split raw text into tokens the model can understand, and the efficiency and quality of that splitting directly affects model performance on specific languages, inference costs, and how effectively the context window is used.

The problem is that a tokenizer that performs well on English may become far less efficient when confronted with Chinese, Arabic, Hindi, or other languages that use different writing systems. When a single word gets split into too many tokens, it not only wastes context length but also drives up computational costs. Yet before Tokka-Bench, researchers lacked a unified, multilingual, multi-dimensional evaluation toolkit to quantify these differences.

The dominant algorithms for subword tokenization include BPE (Byte-Pair Encoding), WordPiece, and SentencePiece. BPE, for example, starts from individual characters and iteratively merges the most frequently co-occurring adjacent symbol pairs in the training corpus, ultimately building a vocabulary that balances high-frequency complete words with low-frequency subword units. This approach is highly efficient for Latin-script languages like English, where frequent words get encoded as single tokens. But for morphologically rich languages (like Finnish or Turkish) or non-Latin writing systems (such as Chinese characters, Arabic script, or Devanagari), if the training corpus underrepresents these languages, the algorithm struggles to learn enough high-frequency merge rules — causing ordinary words to be broken into many fine-grained tokens. This phenomenon is known in the literature as "over-tokenization," and its direct consequence is that non-English text may consume several times as many tokens as equivalent English content, multiplying both inference costs and context window usage.

Tokka-Bench's Five Evaluation Metrics

The core contribution of Tokka-Bench is its proposal of five complementary evaluation dimensions, rather than relying on any single metric:

• Bytes per token: measures tokenization compression efficiency.
• Unique token coverage: reflects how well the vocabulary covers a given language.
• Subword fertility: the average number of tokens a word is split into.
• Word-split rate: the proportion of words that are broken apart during tokenization.
• Vocabulary composition: analyzes the distribution of token types within the vocabulary.

This framework covers 100 natural languages (spanning more than 30 writing systems) and 20 programming languages, and applies language-aware segmentation tailored to each writing system. This means evaluation results more faithfully reflect how different language families are actually processed, rather than imposing an English-centric standard across all languages.

"Subword fertility" is one of the most frequently cited metrics in NLP for assessing a tokenizer's multilingual friendliness, first systematized by work such as Acs (2019). It is calculated by dividing the total number of tokens produced by the tokenizer for all words in a target-language corpus by the original word count. A higher ratio means the language is being split more finely. A fertility rate between 1.0 and 1.5 is generally considered efficient; exceeding 2.0 suggests the model faces a significant structural disadvantage when processing that language. The "word-split rate" offers a complementary binary perspective: any word split into two or more tokens counts as "split," and this rate reflects how well the tokenizer preserves complete word boundaries. Used together, these two metrics capture both the average granularity of tokenization and how frequently over-splitting occurs — each complementing the other.

Head-to-Head Comparison of Seven Major Tokenizers

The research team compared seven BPE-based tokenizers within individual languages, covering the major players in today's industry: GPT-2, GPT-4, gpt-oss, Llama 3.1, Gemma 3, Qwen3, and Kimi K2.

This comparison yielded two particularly insightful findings:

Vocabulary Allocation Strategy Matters More Than Vocabulary Size

A common intuition is that a larger vocabulary means better multilingual capability. But Tokka-Bench's findings challenge this assumption — vocabulary allocation strategy matters more than raw vocabulary size. In other words, how the limited vocabulary budget is distributed across different languages and scripts often matters more for real-world efficiency than simply expanding vocabulary capacity. This is an important directional signal for tokenizer designers.

Vocabulary allocation strategy is typically implemented during tokenizer training by adjusting the sampling ratios for each language. For instance, boosting a language's sampling weight in the training corpus from 1% to 5% allows the BPE algorithm to learn more high-frequency merge rules for that language, effectively reserving more vocabulary slots for its complete words or common subword units. In recent years, several multilingual models (such as mT5 and Llama 3) have explicitly designed "language budgets" when building their tokenizers, allocating vocabulary slots by writing system or language family rather than letting high-resource languages (especially English) dominate the merging process. Tokka-Bench's findings provide further quantitative evidence for the importance of this design decision: even when two tokenizers have similar vocabulary sizes, differences in low-resource language sampling ratios during training can lead to dramatically different multilingual processing efficiency. This gives teams designing tokenizers from scratch a clear optimization direction.

Programming Language Tokenization Efficiency Has Converged

Another interesting finding is that while these tokenizers show notably divergent natural-language profiles, their efficiency in handling programming languages has largely converged. This likely reflects the industry-wide emphasis on code capabilities in recent years, making code tokenization optimization a common baseline. For applications centered on coding tasks, the choice of tokenizer may be far less consequential than in natural-language scenarios.

The Value of Open-Source and Reproducibility

Tokka-Bench's framework, datasets, and an interactive dashboard are all publicly available. This openness carries real value for the community: researchers and engineers can use it directly to evaluate how their model's tokenizer performs on target languages — especially when building applications for specific languages or multilingual settings, where quantifying tokenization costs upfront enables more informed technical decisions.

For researchers focused on low-resource languages and multilingual model fairness, a tool covering more than 30 writing systems also provides a quantifiable handle for revealing and addressing cross-linguistic inequality.

Summary

Tokka-Bench fills the gap in multilingual, multi-metric tokenizer evaluation. With five complementary metrics and coverage spanning 120 languages, it delivers conclusions that go beyond the intuition that "bigger vocabulary is better": allocation strategy beats scale, code tokenization is converging, and natural language remains the primary battleground for tokenizer differentiation. For any team deploying large language models in multilingual environments, this is a benchmark worth consulting.