Latent Core Tokenizer
A method for constructing multilingual tokenizers that splits tokenizer construction into two independent stages, structure discovery and vocabulary construction, combining MDL, entropy boundary signals and morphosyntactic constraints to improve morphological alignment and downstream task performance without sacrificing language fairness.
Timeline (last 90 days)
Latent Core Tokenizer (LCT) splits tokenizer construction into two independent stages: structure discovery and vocabulary construction.
In the structure discovery stage, LCT introduces three signals: Minimum Description Length (MDL), entropy-based boundary signals, and morphosyntactic constraints.
The paper compares LCT with BPE, Unigram, and parity-aware BPE under the setting of 104 languages and a 200K vocabulary.
Compared with BPE, Unigram, and parity-aware BPE, LCT improves the aggregate score on four multilingual downstream benchmarks by 1.48, 1.83, and 2.00 points, respectively.
LCT achieves lower fertility and higher MorphScore on 104 languages, while the cross-language tokenization cost differences remain similar.
The paper's core conclusion is: compression alone does not predict representation quality.
LCT defers the role of frequency: the structure discovery stage does not depend on frequency and relies only on MDL, entropy, and morphological constraints; frequency only acts in the final vocabulary allocation stage.
In LCT's final vocabulary allocation stage, frequency still determines how limited vocabulary slots are traded off among languages and units.
LCT is currently still a research-stage method, and the paper does not mention details of inference overhead and training cost.
If a multilingual model's vocabulary has abnormally high fertility on low-resource languages, the problem may not lie in vocabulary size but in the construction objective.
All Facts (10)
Latent Core Tokenizer (LCT) splits tokenizer construction into two independent stages: structure discovery and vocabulary construction.
50%UnverifiedIn the structure discovery stage, LCT introduces three signals: Minimum Description Length (MDL), entropy-based boundary signals, and morphosyntactic constraints.
50%UnverifiedThe paper compares LCT with BPE, Unigram, and parity-aware BPE under the setting of 104 languages and a 200K vocabulary.
50%UnverifiedCompared with BPE, Unigram, and parity-aware BPE, LCT improves the aggregate score on four multilingual downstream benchmarks by 1.48, 1.83, and 2.00 points, respectively.
50%UnverifiedLCT achieves lower fertility and higher MorphScore on 104 languages, while the cross-language tokenization cost differences remain similar.
50%UnverifiedThe paper's core conclusion is: compression alone does not predict representation quality.
50%UnverifiedLCT defers the role of frequency: the structure discovery stage does not depend on frequency and relies only on MDL, entropy, and morphological constraints; frequency only acts in the final vocabulary allocation stage.
50%UnverifiedIn LCT's final vocabulary allocation stage, frequency still determines how limited vocabulary slots are traded off among languages and units.
50%UnverifiedLCT is currently still a research-stage method, and the paper does not mention details of inference overhead and training cost.
50%UnverifiedIf a multilingual model's vocabulary has abnormally high fertility on low-resource languages, the problem may not lie in vocabulary size but in the construction objective.
50%