Subword Tokenization
A tokenization method that compromises between whole words and single characters, splitting text into subword units (tokens), collapsing common words into a single token and breaking rare words into several subword fragments. It both controls vocabulary size and produces almost no unknown words, and is the mainstream approach to tokenization in modern LLMs.
Timeline (last 90 days)
Subword tokenization cuts words into smaller segments, common algorithms include BPE, WordPiece and Unigram, and SentencePiece is one language-agnostic engineering implementation among them.
Subword schemes can cover almost any text while keeping the vocabulary size controllable, at the cost that the same word may be cut into different Token sequences, with slightly lower interpretability.
All Facts (2)
Subword tokenization cuts words into smaller segments, common algorithms include BPE, WordPiece and Unigram, and SentencePiece is one language-agnostic engineering implementation among them.
50%UnverifiedSubword schemes can cover almost any text while keeping the vocabulary size controllable, at the cost that the same word may be cut into different Token sequences, with slightly lower interpretability.
50%