SentencePiece Unigram
One of the tokenization algorithms supported by Google's SentencePiece toolkit. It works top-down from a large vocabulary, pruning low-contribution subwords via the EM algorithm, and supports probabilistic tokenization and sampling over multiple segmentations.
Timeline (last 90 days)
A paper on a Sindhi tokenizer trained a SentencePiece Unigram-based tokenizer and systematically compared it with 14 existing tokenizers on four evaluation sets.
The paper trained SentencePiece Unigram tokenizers at four vocabulary sizes (32k to 150k), all with byte fallback enabled.
At the 64k vocabulary setting, 88.6% of words were retained as a single token.
On the three metrics of fertility, single-token retention rate, and UNK rate, configurations of 64k and above are the best or tied for best across all four evaluation sets.
All Facts (4)
A paper on a Sindhi tokenizer trained a SentencePiece Unigram-based tokenizer and systematically compared it with 14 existing tokenizers on four evaluation sets.
50%UnverifiedThe paper trained SentencePiece Unigram tokenizers at four vocabulary sizes (32k to 150k), all with byte fallback enabled.
50%UnverifiedAt the 64k vocabulary setting, 88.6% of words were retained as a single token.
50%UnverifiedOn the three metrics of fertility, single-token retention rate, and UNK rate, configurations of 64k and above are the best or tied for best across all four evaluation sets.
50%