Unigram
A subword tokenization algorithm that assumes a vocabulary composed of independent subword units and selects an optimal tokenization scheme through a probabilistic model. It is commonly used in the SentencePiece tool.
Timeline (last 90 days)
The optimization objectives of the mainstream tokenization algorithms BPE and Unigram are essentially both compression: minimizing the corpus encoding length under a limited vocabulary.
The paper compares LCT with BPE, Unigram, and parity-aware BPE under the setting of 104 languages and a 200K vocabulary.
Compared with BPE, Unigram, and parity-aware BPE, LCT improves the aggregate score on four multilingual downstream benchmarks by 1.48, 1.83, and 2.00 points, respectively.
The Unigram language model tokenization method was proposed by Kudo in 2018, starting from a large vocabulary and progressively pruning it to retain the set of subwords that minimizes the likelihood loss on the corpus.
The SentencePiece toolkit includes a complete implementation of Unigram language model tokenization.
The choice among the three tokenization algorithms involves different trade-offs among vocabulary size, rare-word coverage, and cross-lingual generalization ability.
All Facts (6)
The optimization objectives of the mainstream tokenization algorithms BPE and Unigram are essentially both compression: minimizing the corpus encoding length under a limited vocabulary.
50%UnverifiedThe paper compares LCT with BPE, Unigram, and parity-aware BPE under the setting of 104 languages and a 200K vocabulary.
50%UnverifiedCompared with BPE, Unigram, and parity-aware BPE, LCT improves the aggregate score on four multilingual downstream benchmarks by 1.48, 1.83, and 2.00 points, respectively.
50%UnverifiedThe Unigram language model tokenization method was proposed by Kudo in 2018, starting from a large vocabulary and progressively pruning it to retain the set of subwords that minimizes the likelihood loss on the corpus.
50%UnverifiedThe SentencePiece toolkit includes a complete implementation of Unigram language model tokenization.
50%UnverifiedThe choice among the three tokenization algorithms involves different trade-offs among vocabulary size, rare-word coverage, and cross-lingual generalization ability.
50%