When STEM Data Slips into Dravidian Language Models: Hidden Issues Exposed by the BPB Metric

When training multilingual models, we're used to judging model quality by watching the loss curve. But a practice involving a language model for the Dravidian language family shows that this approach can unknowingly mask severe performance degradation.

Background: The Dravidian Language Challenge in Multilingual Models

A developer shared on Reddit their experience training a 253M-parameter language model. The model covers 4 major Dravidian languages (Kannada, Tamil, Telugu, Malayalam), 4 low-resource Dravidian languages (Tulu, Koya, Kuvi, Kolami), and English.

During this training process, the team discovered two issues at the tokenization and multilingual training levels, which are valuable reference points for researchers working with Indic scripts.

Tokenizer Problem: How Standard BPE "Dismembers" South Indian Scripts

The Tokenization Efficiency Dilemma of Standard BPE

Standard byte-level BPE tokenizers chop South Indian scripts into short UTF-8 fragments, covering only about 3.67 bytes per token on a typical 48k-vocabulary GPT-style baseline. This means a large number of tokens are consumed encoding byte fragments of individual characters, resulting in very low effective information density.

Byte-level BPE is the tokenization scheme adopted by mainstream models such as the GPT series. It first encodes all text into UTF-8 byte sequences, then learns merges at the byte level. For Latin letters, a character typically corresponds to 1-2 bytes, so a single token can often cover multiple characters. However, South Indian scripts (such as Kannada and Tamil) have code points located outside the Unicode Basic Multilingual Plane or use multi-byte encodings, requiring 3-4 UTF-8 bytes per character. Byte-level BPE cannot allocate sufficient merge rules for these characters under a limited vocabulary, causing byte fragments of individual characters to each become separate tokens, with a large vocabulary budget consumed on low-information-density byte units. This explains the low average coverage of only 3.67 bytes per token in the baseline—the tokenizer is effectively "reading" South Indian scripts byte by byte rather than understanding them as writing units.

Improvement from the Akshara-Preserving Strategy

The team adopted Unicode NFC normalization along with a pre-tokenization regex that groups [\p{L}\p{M}\u200c\u200d]+, and set the vocabulary size to 32768. After adjustment, the token-weighted average UTF-8 byte coverage per language improved significantly: Kannada reached 9.717, Tamil 9.654, Telugu 8.910, Malayalam 10.531, with the average of the four major languages at 9.680 bytes/token.

The average across all 8 Dravidian languages is 8.693 bytes/token, while English is only 3.159. This difference reflects the complex agglutinative morphology and rich character combinations—with a reasonable pre-tokenization strategy, the same vocabulary size can carry more information.

Akshara is the basic writing unit in Indic scripts, roughly equivalent to a consonant letter together with its vowel sign (matra) and possibly nasalization or pause marks. In Unicode, an akshara often consists of multiple code points: a base letter (\p{L} category) followed by one or more combining marks (\p{M} category, such as vowel signs, virama, etc.), and may also include zero-width non-joiner (\u200c) and zero-width joiner (\u200d) to control ligature rendering. Standard pre-tokenization regexes typically only split on whitespace and punctuation, making it difficult to ensure that code points within an akshara are not separated. The [\p{L}\p{M}\u200c\u200d]+ regex used in the article merges all letters, combining marks, and join control characters into the same pre-tokenization unit, ensuring that BPE starts from complete aksharas during merging and avoids splitting out combining marks separately. Combined with NFC normalization to pre-compose composable code point sequences, this strategy significantly increases the number of UTF-8 bytes carried per token, bringing tokenization efficiency close to the information density expected for agglutinative languages.

A More Insidious Problem: BPB Degradation Behind Declining Training Loss

The Divergence Between Loss and BPB

When pretraining with mixed STEM, math, and Python data, training loss dropped from about 2.9 to about 2.0, which looks like clear progress. But bits per byte (BPB) on the held-out set deteriorated simultaneously across all 9 languages—Kuvi BPB rose by 7%, and Kannada went from 0.489 to 0.500.

This means the model fit the training distribution better, but its true compression ability for each language was declining. BPB measures the model's prediction efficiency on raw byte sequences, and compared to loss, it better reflects generalization across languages and character sets. If you only look at loss, this degradation is almost "silent."

Bits per byte (BPB) is a language modeling evaluation metric that is independent of tokenization, defined as the average negative log-likelihood (base 2) of the model over raw UTF-8 byte sequences, i.e., the cross-entropy loss divided by the number of bytes corresponding to each token and appropriately normalized. Unlike training loss, BPB directly measures the model's ability to compress raw text and is unaffected by vocabulary size and tokenization granularity—regardless of how the tokenizer segments, the same text byte sequence will yield comparable BPB values. This makes BPB particularly suitable for cross-lingual comparison: when tokenization efficiency differs greatly across languages (e.g., English at 3.159 bytes/token vs. Malayalam at 10.531 bytes/token in this article), loss values can be misleading due to different average information content per token, whereas BPB can reveal simultaneous changes in the model's true predictive ability for each language. In the article, BPB rises while loss falls, indicating that the model is fitting high-frequency patterns in the training distribution (such as the regular structure of code and STEM data), but its prediction efficiency on raw bytes of the held-out set is degrading—this is precisely the value of BPB as a decoupled metric.

Fixing with a Multilingual Replay Strategy

The team switched the next stage to a 50/50 multilingual replay ratio, and BPB recovered on all 11 validation sets. This shows that when introducing English code or STEM data, maintaining sufficient replay of the target languages is necessary—otherwise, high-resource, structurally regular data will crowd out the model's modeling capacity for low-resource languages.

Evaluation Performance of the Final Model

In a held-out check of 300 prompts, the final model achieved 100% script purity (80/80) on Kannada, Tamil, Telugu, and Malayalam, with a format compliance rate of 96.25% (77/80, with all 3 failures in Tamil). On the Belebele Dravidian evaluation, the average score was 17.18%, with a best score of 25.00%.

Tulu, Koya, Kuvi, and Kolami remain in the experimental stage, with the main constraint being the scarcity of clean, commercially usable text resources with validation sets.

Open Questions to Be Resolved

The developer posed two questions to the community: Does akshara-level pre-tokenization, compared to finer-grained subwords, harm morphological generalization? When mixing English code or STEM data, what replay ratio and data sources work better?

From this practice, at least two points can be confirmed: for agglutinative and low-resource Indic languages, tokenization strategies need to be customized to the writing system; and monitoring metrics should not rely solely on training loss—decoupled metrics like BPB are better at revealing hidden degradation in multilingual mixed training.

One-line Summary

Dravidian language family multilingual model practice reveals: declining training loss may mask BPB degradation, and tokenization strategy and replay ratio are crucial.

Paragraph Summary

This article shares training experience with a 253M-parameter multilingual language model covering 8 Dravidian languages and English. The team found that a standard byte-level BPE tokenizer chopped South Indian scripts into short UTF-8 fragments, covering only about 3.67 bytes per token on average; through Unicode NFC normalization, akshara-grouping pre-tokenization regex, and a 32k vocabulary, the average bytes/token for the four major Dravidian languages rose to 9.680. A more insidious problem was that after mixing STEM/math/Python data, training loss dropped from about 2.9 to about 2.0, but held-out BPB deteriorated simultaneously across all 9 languages, with Kuvi BPB rising 7%. After switching to a 50/50 multilingual replay ratio, BPB recovered on all 11 validation sets. The final model achieved 100% script purity and 96.25% format compliance on the four major languages, with an average Belebele score of 17.18%. Low-resource languages remain constrained by the scarcity of clean, commercially usable text.