Latent Core Tokenizer: Compression Rate Doesn't Equal Representation Quality — A New Approach to Multilingual Tokenization

Title

Latent Core Tokenizer: Compression Rate Doesn't Equal Representation Quality — A New Approach to Multilingual Tokenization

Meta Description

Latent Core Tokenizer (LCT) proposes a new approach to multilingual tokenization: using minimum description length, entropy boundary signals, and morphotactic constraints to discover language structure before building the vocabulary. Across 104 languages, it outperforms BPE and Unigram on fertility and MorphScore, with up to a 2.00-point gain on downstream benchmarks.

Article Content

Tokenizers are the first step in virtually every language model, yet their design has long been dominated by a single metric: compression rate. The smaller the vocabulary and the shorter the sequence, the more "efficient" it appears. However, a study on the Latent Core Tokenizer (LCT) presents a counterintuitive conclusion — compressing better does not mean learning better.

The Hidden Bias of Tokenizers: Uneven Capacity Allocation

The optimization objectives of mainstream tokenization algorithms (BPE, Unigram) are essentially all about compression: minimizing the encoding length of a corpus under a fixed vocabulary size. The problem is that a compact vocabulary does not equal a balanced allocation of capacity across languages.

If a language accounts for a high proportion of the training corpus, its high-frequency segments will be prioritized into the vocabulary; the morphological structure of low-resource languages, on the other hand, may be fragmented into a large number of short tokens. The result: an English sentence may occupy only a dozen or so tokens, while a morphologically complex language (such as Turkish or Finnish) requires two to three times as many tokens for the same semantics. This not only slows down inference but also affects the model's depth of understanding of low-resource languages.

Prior work has attempted to mitigate this inequality with methods such as "parity-aware BPE," but the compression objective itself remains unchanged — it merely compensates during allocation. LCT's starting point is: change the objective function.

BPE (Byte Pair Encoding) is one of the most mainstream tokenization algorithms: it starts from characters (or bytes), repeatedly counts the highest-frequency adjacent symbol pairs in the corpus, and merges them into new symbols until the vocabulary reaches a preset size. Unigram is the opposite: it first builds a massive set of candidate subwords, then progressively prunes it via language model likelihood, retaining the set of subwords that maximizes corpus probability. Although the two take different paths, both objective functions point to the same thing — minimizing the encoding length of the corpus under a given vocabulary size, i.e., maximizing compression rate. For this reason, they are highly sensitive to the distribution of the training corpus: whichever appears more frequently in the corpus is more likely to have its subwords incorporated into the vocabulary. This is precisely the algorithmic root of imbalanced capacity allocation in multilingual settings.

Core Method: Discover Structure First, Then Build the Vocabulary

LCT's design splits tokenizer construction into two independent stages: structure discovery and vocabulary construction. This is completely different from BPE's greedy strategy of merging while compressing.

The structure discovery stage introduces three signals:

• Minimum Description Length (MDL): measures whether a candidate language unit is worth retaining, weighing "the benefit brought by reuse" against "the cost of storing that unit."
• Entropy-based boundary signals: uses information-theoretic methods to identify morpheme boundaries. Character combinations at boundaries have higher uncertainty, and entropy values can serve as a basis for segmentation.
• Morphotactic constraints: restricts which morpheme sequences are legal, avoiding linguistically unreasonable segmentations.

This pipeline first identifies reusable language units (such as roots and affixes), then builds a shared vocabulary on that basis, rather than directly performing statistical merges on character sequences. In other words, LCT attempts to make the tokenizer "understand a bit of morphology" rather than purely perform byte-level compression.

Experimental Results: A Comprehensive Comparison Across 104 Languages

Under the setting of 104 languages and a 200K vocabulary, the paper compares LCT with BPE, Unigram, and parity-aware BPE. The main conclusions are as follows:

• Lower fertility: that is, each word is on average split into fewer tokens, indicating that segmentation is closer to the natural units of language;
• Higher MorphScore: this is a metric measuring the consistency between tokenization results and true morphological structure, and LCT clearly leads;
• Cross-lingual tokenization cost disparity remains similar: that is, while improving morphological alignment, it does not sacrifice fairness across languages.

On four multilingual downstream benchmarks, LCT improves aggregate scores by 1.48, 1.83, and 2.00 points compared with BPE, Unigram, and parity-aware BPE, respectively.

This magnitude of improvement is not small in tokenizer research — the tokenizer is usually the most "static" component in model training, and changes often bring limited gains.

Fertility is a commonly used metric for measuring tokenizer efficiency, referring to the average number of tokens into which each word is split. The lower the fertility, the more likely a word is to be represented as a whole or by only a few tokens; if a language's fertility is noticeably high (such as in morphologically complex agglutinative languages), it means its words are split very finely, and the model needs longer sequences to express the same semantics, which both increases inference cost and dilutes the modeling effect of long-range dependencies. MorphScore evaluates tokenization quality from another dimension: it compares tokenization boundaries with manually annotated true morpheme boundaries, measuring the degree of consistency between segmentation results and linguistic morphological structure. A tokenizer can very well have low fertility (short sequences) but poor MorphScore (segmentation points misaligned with morpheme boundaries). LCT's value lies precisely in improving both metrics simultaneously.

Key Insight: How Frequency Should Be Used

Perhaps the most valuable point of this research is not the specific algorithm, but the repositioning of the role of "frequency."

The implicit assumption of BPE-style methods is: high-frequency segments are more worth including in the vocabulary. This largely holds in monolingual settings, but in multilingual settings, high frequency often equates to "high corpus share," so vocabulary capacity naturally tilts toward high-resource languages.

LCT's approach is to postpone the role of frequency: the structure discovery stage does not rely on frequency, only on MDL, entropy, and morphotactic constraints; frequency only comes into play in the final vocabulary allocation stage. In this way, whether a language's morphological units can be identified depends on whether they are "structurally reusable," not on how often they appear.

The paper's summary is very direct: compression alone does not predict representation quality — compression rate itself cannot predict representation quality. This constitutes a methodological challenge to the many current tokenizer works that market themselves on "lower compression rate."

Minimum Description Length (MDL) originates from information theory and the idea of Occam's razor, with the core being to strike a balance between "model complexity" and "data fit." When used for tokenization, its intuition is: if a candidate language unit (such as a root or affix) is reused more and saves more encoding length, it is more worth retaining; but storing the unit itself, as well as the segmentation ambiguity it introduces, also constitutes a cost. MDL requires minimizing the total description length (storage cost plus encoding cost), so it evaluates a unit's "structural reusability" rather than mere frequency. This forms a sharp contrast with BPE, which merges only by frequency — MDL favors units that, although not always high-frequency, genuinely carry semantics structurally, which precisely preserves space for the morphological structure of low-resource languages.

Implications for Engineering Practice

For teams building multilingual models, LCT suggests an actionable direction: if your vocabulary has abnormally high fertility on low-resource languages, the problem may not lie in vocabulary size, but in the construction objective. Simply enlarging the vocabulary or adding more corpus may not solve morphological segmentation errors.

Of course, LCT is still a research-stage method. The paper does not mention details of inference overhead and training cost, and the construction of morphotactic constraints also requires linguistic knowledge or annotated resources. Whether this is scalable for truly low-resource languages still needs verification. But at least it puts a long-obscured problem on the table: the goal of a tokenizer should be to let every language be "meaningfully" represented, not merely to make the overall sequence shorter.

Additional Background

It is worth distinguishing that the "postponement of frequency" in LCT does not mean "frequency is useless." In the final vocabulary allocation stage, frequency still determines how limited vocabulary slots are traded off among languages and units; otherwise the vocabulary would be filled with a large number of extremely low-frequency morphological units, which would instead harm the performance of high-resource languages. The key of LCT is to separate frequency from the prior judgment of "discovering which structures exist," retaining only its weight in "how to allocate resources after structures are determined." This two-stage decoupling allows structure identification and resource allocation to each use the most appropriate signal, avoiding the systematic erosion of multilingual fairness by the implicit assumption in BPE that "high frequency means important," and also leaving room for further tuning of vocabulary allocation strategies in the future.

One-line Summary

Latent Core Tokenizer proposes a two-stage tokenization framework that discovers structure first and builds the vocabulary second, proving that compression rate cannot predict representation quality.

Paragraph Summary

To address the flaw of mainstream tokenizers (BPE, Unigram) that use compression rate as the sole objective, LCT proposes splitting tokenizer construction into two stages: structure discovery and vocabulary construction. The structure discovery stage does not rely on frequency, but instead combines minimum description length, entropy boundary signals, and morphotactic constraints to identify reusable language units; frequency only comes into play in the final vocabulary allocation. Under the setting of 104 languages and a 200K vocabulary, LCT achieves lower fertility and higher MorphScore than BPE, Unigram, and parity-aware BPE, while cross-lingual cost disparity remains on par with the baselines, and it improves four multilingual downstream benchmarks by 1.48, 1.83, and 2.00 points, respectively. Its core insight is: compression rate itself cannot predict representation quality.