Tokeniser Coverage: The Invisible Ceiling of Small Language Models in Multilingual Scenarios

Introduction: The Overlooked Efficiency Metric

When we talk about the efficiency of small language models (SLMs), almost always one number is cited: memory footprint. Techniques like quantization and parameter-efficient fine-tuning are widely promoted as the path to multilingual AI under compute-constrained conditions. However, a study covering 11 Indian script systems reveals an overlooked fact: efficiency metrics are defined in English and cannot be directly transferred to other languages.

The study is based on a corpus of 131,000 sentences covering 11 Indian scripts, comparing the tokenizer performance of models such as TinyLlama-1.1B, mBERT, and MuRIL. The results point directly to a core issue: vocabulary coverage, not parameter count, is the fundamental constraint for small language models in multilingual scenarios.

The Truth About Tokenizer Efficiency: English-Centric Metrics

Huge Differences in Subword Token Counts

The study shows that TinyLlama-1.1B's tokenizer produces an average of 1.34 subword tokens per word when processing Latin script; when processing Kannada, it produces an average of 22.60 tokens per word. This is not simply inefficiency but a manifestation of missing vocabulary.

This difference directly affects the model's effective context length. Within a 2048-token context window, English users get 1528 words of usable context, while Kannada users get only 91 words. This means that the same model offers nearly 17 times difference in actual usable capacity for users of different languages.

Fertility is a common metric for measuring tokenizer efficiency, defined as the average number of subword tokens a word is split into. Higher fertility means more tokens are consumed for the same text, and the model's effective context length is compressed more severely. Ideally, a tokenizer trained for a language should have fertility close to 1 for that language (one token per word). In cross-lingual comparisons, differences in fertility directly reflect how well the vocabulary covers the language—if the vocabulary lacks common subword units for a language, the tokenizer can only fall back to splitting words more finely, even down to the byte level.

Alarming Vocabulary Coverage

In TinyLlama's vocabulary, the codepoint coverage for Odia is 0%, for Telugu 1%, and for Kannada 2%. Among the 11 scripts, 5 have fertility (tokens produced per word) hitting the UTF-8 byte fallback limit, with Odia reaching 99.6% of the limit, where 93.6% of tokens are raw bytes.

More insidious is that because SentencePiece's byte fallback mechanism never produces unknown token markers (UNK), this failure is completely silent. TinyLlama's UNK rate is 0.0% across all scripts, including those read as bytes. The model appears to "know" all scripts, but in reality, it is just piecing together bytes.

SentencePiece's byte fallback mechanism is key to understanding "silent failure." When the tokenizer encounters characters completely absent from the vocabulary, SentencePiece by default does not output an UNK (unknown) marker but instead decomposes the character into a UTF-8 raw byte sequence, mapping each byte to a token. This design ensures any input can be encoded, but at the cost that a character may expand into 2-4 byte tokens, and these tokens carry no semantic information; the model must learn from scratch how to assemble characters from bytes. More seriously, because the UNK rate is zero, conventional efficiency reports cannot detect this degradation—the tokenizer appears "normal" but actually operates at extremely low semantic density.

The Trap of Low Fertility: mBERT's False Efficiency

The study further reveals that low fertility may mean the opposite of efficiency. mBERT achieves its best score on Odia (1.10 tokens/word) but simultaneously discards 87.9% of the text as UNK. This "efficiency" comes at the cost of information—the model largely ignores input content but appears to perform excellently because the UNK rate is not included in efficiency reports.

This reminds us that single metrics like memory or token count can be misleading. True efficiency should comprehensively consider vocabulary coverage, UNK rate, and byte fallback usage.

UNK rate (unknown token ratio) is another core metric for evaluating tokenizer coverage quality, referring to the proportion of tokens in the input text mapped to UNK markers. Unlike byte fallback, UNK means that part of the input is completely discarded, and the model cannot obtain any information from it. In efficiency reports, fertility and UNK rate must be interpreted together: if low fertility is accompanied by a high UNK rate, it indicates that the tokenizer reduces token counts by "ignoring" rather than "understanding" the input—this is information loss, not efficiency gain. Reporting only fertility or only memory footprint may mask severe tokenizer deficiencies in specific languages.

Vocabulary, Not Parameter Count: The MuRIL Lesson

MuRIL is an encoder with Indian language vocabulary. Its parameter count is only one-fifth of TinyLlama's, yet it achieves reading efficiency of 1.23-2.26 tokens/word across all scripts, with zero UNK and zero byte fallback. This contrast clearly shows that the binding constraint is vocabulary, not parameter count.

For multilingual AI deployment in resource-constrained environments, blindly pursuing smaller parameter counts or higher quantization precision may yield little benefit. Prioritizing the construction or selection of tokenizers that cover the target language vocabulary is the key to improving actual efficiency.

MuRIL (Multilingual Representations for Indian Languages) is an encoder model pre-trained by Google specifically for Indian languages based on the BERT architecture. Its vocabulary incorporates a large number of subword units from Indian scripts and uses transliteration and translation-based training objectives to enhance cross-script alignment. Unlike general multilingual models, MuRIL's design assumption is that vocabulary coverage of the target language is a prerequisite for model practicality. The comparison in this study is powerful precisely because MuRIL achieves efficient reading of 11 Indian scripts with a very small parameter count, demonstrating that in low-resource multilingual scenarios, upfront investment in vocabulary design is more cost-effective than stacking model size.

Conclusions and Implications

This study issues an important warning to practitioners of multilingual AI:

• Efficiency reports need to distinguish languages; English benchmarks cannot be directly extrapolated.
• Tokenizer vocabulary coverage should be a core evaluation metric, as important as memory footprint.
• Although byte fallback avoids UNK, it may mask severe vocabulary deficiencies.
• In low-resource scenarios such as Indian languages, choosing models with targeted vocabularies (like MuRIL) is more effective than simply reducing model size.

The "efficiency" myth of small language models may be an illusion in languages other than English. True multilingual fairness requires rethinking from the tokenizer level.

One-line Summary

The efficiency myth of small language models collapses in non-English languages; vocabulary coverage, not parameter count, is the real bottleneck in multilingual scenarios.

Paragraph Summary

A study of 131,000 sentences across 11 Indian scripts reveals that efficiency metrics for small language models are English-centric and cannot be directly transferred. TinyLlama-1.1B produces 22.60 tokens per word in Kannada, 17 times that of Latin script, leaving only 91 words usable within a 2048 context window. Among the 11 scripts, 5 hit the UTF-8 byte fallback limit for fertility, with Odia codepoint coverage at 0%, yet UNK rate shows 0.0%, meaning the failure is completely silent. mBERT achieves low fertility on Odia while discarding 87.9% of text as UNK, indicating that low fertility may mask information loss. MuRIL, with only one-fifth of TinyLlama's parameters, achieves 1.23-2.26 tokens/word with zero UNK, proving that vocabulary, not parameter count, is the binding constraint.