WordPiece
A subword tokenization algorithm that determines symbol merges by maximizing language model likelihood. It serves as the core tokenization method adopted by models such as BERT, producing a vocabulary of subword units that balances coverage and compactness.
Core Facts
Timeline (last 90 days)
In a warm English encoding with bert-base-uncased, Normalization consumed 88% of the encoding time, while WordPiece accounted for only 8%.
BAAI/bge-small-en-v1.5 uses a WordPiece tokenizer, which splits snake_case and camelCase into meaningless fragments.
The WordPiece tokenizer's ## prefix marks the latter part of a subword; for example, gatito is split into ga, ##ti, ##to, and gato also contains ##to, causing shared tokens between the query and the document and producing false similarity.
A Tokenizer is responsible for splitting raw text into token sequences that the model can process; mainstream approaches include BPE, WordPiece, and SentencePiece.
The choice among the three tokenization algorithms involves different trade-offs among vocabulary size, rare-word coverage, and cross-lingual generalization ability.
WordPiece was proposed by Google and adopted by BERT; its merge criterion maximizes the likelihood of a language model rather than purely counting frequencies.
Mainstream subword tokenizers (such as BPE and WordPiece) treat different written forms of the same word (e.g., hello, Hello, HELLO, Héllo) as independent, mutually unrelated vocabulary entries.
Fixed vocabulary sizes typically range from 30,000 to 100,000 entries.
WordPiece determines its merging strategy with the objective of maximizing language model likelihood.
All Facts (10)
Modern LLMs generally use BPE (Byte Pair Encoding) or WordPiece algorithms for tokenization.
80%UnverifiedIn a warm English encoding with bert-base-uncased, Normalization consumed 88% of the encoding time, while WordPiece accounted for only 8%.
50%UnverifiedBAAI/bge-small-en-v1.5 uses a WordPiece tokenizer, which splits snake_case and camelCase into meaningless fragments.
50%UnverifiedThe WordPiece tokenizer's ## prefix marks the latter part of a subword; for example, gatito is split into ga, ##ti, ##to, and gato also contains ##to, causing shared tokens between the query and the document and producing false similarity.
50%UnverifiedA Tokenizer is responsible for splitting raw text into token sequences that the model can process; mainstream approaches include BPE, WordPiece, and SentencePiece.
50%UnverifiedWordPiece was proposed by Google and adopted by BERT; its merge criterion maximizes the likelihood of a language model rather than purely counting frequencies.
50%UnverifiedThe choice among the three tokenization algorithms involves different trade-offs among vocabulary size, rare-word coverage, and cross-lingual generalization ability.
50%UnverifiedMainstream subword tokenizers (such as BPE and WordPiece) treat different written forms of the same word (e.g., hello, Hello, HELLO, Héllo) as independent, mutually unrelated vocabulary entries.
50%UnverifiedFixed vocabulary sizes typically range from 30,000 to 100,000 entries.
50%UnverifiedWordPiece determines its merging strategy with the objective of maximizing language model likelihood.
50%