parity-aware BPE
An improved BPE tokenization method that introduces a fairness compensation mechanism during vocabulary allocation to mitigate unequal capacity allocation between high-resource and low-resource languages in multilingual settings, while leaving the compression objective itself unchanged.
Timeline (last 90 days)
The paper compares LCT with BPE, Unigram, and parity-aware BPE under the setting of 104 languages and a 200K vocabulary.
Compared with BPE, Unigram, and parity-aware BPE, LCT improves the aggregate score on four multilingual downstream benchmarks by 1.48, 1.83, and 2.00 points, respectively.
All Facts (2)
The paper compares LCT with BPE, Unigram, and parity-aware BPE under the setting of 104 languages and a 200K vocabulary.
50%UnverifiedCompared with BPE, Unigram, and parity-aware BPE, LCT improves the aggregate score on four multilingual downstream benchmarks by 1.48, 1.83, and 2.00 points, respectively.
50%