[KongchangAI]
ConceptUnigram / Unigram语言模型 / SentencePiece Unigram模型

SentencePiece Unigram

One of the tokenization algorithms supported by Google's SentencePiece toolkit. It works top-down from a large vocabulary, pruning low-contribution subwords via the EM algorithm, and supports probabilistic tokenization and sampling over multiple segmentations.

Timeline (last 90 days)

Nov 17

A paper on a Sindhi tokenizer trained a SentencePiece Unigram-based tokenizer and systematically compared it with 14 existing tokenizers on four evaluation sets.

Unverified50%
Nov 17

The paper trained SentencePiece Unigram tokenizers at four vocabulary sizes (32k to 150k), all with byte fallback enabled.

Unverified50%
Nov 17

At the 64k vocabulary setting, 88.6% of words were retained as a single token.

Unverified50%
Nov 17

On the three metrics of fertility, single-token retention rate, and UNK rate, configurations of 64k and above are the best or tied for best across all four evaluation sets.

Unverified50%

All Facts (4)

Source Articles