BPE
A mainstream LLM tokenization algorithm originating from data compression. It iteratively merges high-frequency adjacent character pairs to build a subword vocabulary, naturally handling out-of-vocabulary words.
Core Facts
The BPE algorithm originates from the field of data compression and builds a vocabulary by iteratively merging high-frequency character pairs.
Mainstream tokenizer approaches include BPE (Byte Pair Encoding), WordPiece, and SentencePiece.
Modern LLMs generally use BPE (Byte Pair Encoding) or WordPiece algorithms for tokenization.
Timeline (last 90 days)
BPE was originally a data compression algorithm proposed by Philip Gage in 1994: repeatedly find the most common adjacent byte pair in the data and replace it with a newly added unused symbol.
Rico Sennrich et al. quantified in their 2016 paper "Neural Machine Translation of Rare Words with Subword Units": on a 100-million-word German corpus, keeping whole words requires a vocabulary of 1.75 million entries, and the test set still leaves 1,079 unknown words; splitting into characters requires about 3,000 symbols, but the corpus balloons to 550 million positions; the subword method BPE uses about 63,000 symbols, 112 million positions, and zero unknown words.
BPE operates at the byte level, so any Unicode text (including emoji, Chinese, and code symbols) can be encoded losslessly.
The same word may have a different number of tokens in different models, because each model is trained on a different corpus and vocabulary size, with different merge rules.
The optimization objectives of the mainstream tokenization algorithms BPE and Unigram are essentially both compression: minimizing the corpus encoding length under a limited vocabulary.
The implicit assumption of BPE-type methods is that high-frequency segments are more worthy of entering the vocabulary, which causes vocabulary capacity in multilingual scenarios to naturally tilt toward high-resource languages.
Compared with BPE, Unigram, and parity-aware BPE, LCT improves the aggregate score on four multilingual downstream benchmarks by 1.48, 1.83, and 2.00 points, respectively.
The paper compares LCT with BPE, Unigram, and parity-aware BPE under the setting of 104 languages and a 200K vocabulary.
A Tokenizer is responsible for splitting raw text into token sequences that the model can process; mainstream approaches include BPE, WordPiece, and SentencePiece.
Vocabulary allocation strategies are typically implemented during the tokenizer training phase by adjusting the sampling ratio for each language.
36 more timeline events
All Facts (20)
The BPE algorithm originates from the field of data compression and builds a vocabulary by iteratively merging high-frequency character pairs.
85%VerifiedMainstream tokenizer approaches include BPE (Byte Pair Encoding), WordPiece, and SentencePiece.
80%VerifiedModern LLMs generally use BPE (Byte Pair Encoding) or WordPiece algorithms for tokenization.
80%VerifiedOpenAI uses the BPE (Byte Pair Encoding) algorithm for token encoding.
80%VerifiedBPE（Byte Pair Encoding）是大语言模型使用的Token化算法，通过统计训练语料中字符对的共现频率迭代合并最常见的字符组合
80%VerifiedThe vocabulary of a standard autoregressive language model typically contains 32,000 to 128,000 tokens.
80%VerifiedGPT-series models use the Byte Pair Encoding (BPE) algorithm to segment text into tokens.
80%VerifiedIn GPT-series models, one Chinese character typically corresponds to 1 to 2 tokens, and one English word corresponds to approximately 1 to 1.5 tokens.
80%VerifiedA token is the smallest semantic unit segmented by subword tokenization algorithms such as BPE, and is not simply equivalent to a single character or word.
75%VerifiedModern large models use tokenization algorithms such as BPE or SentencePiece to build vocabularies, with mainstream models having vocabulary sizes ranging from 30,000 to 150,000 tokens.
75%VerifiedThe BPE (Byte Pair Encoding) algorithm commonly used in the GPT series builds a vocabulary by counting the frequency of adjacent character pairs and merging the highest-frequency character pairs.
75%VerifiedGPT series models use byte-level BPE (Byte-level BPE), which takes raw bytes as the basic unit to eliminate the out-of-vocabulary character problem.
75%VerifiedBoth the GPT series and the Claude series employ a tokenization mechanism similar to Byte-Pair Encoding (BPE).
75%VerifiedByte Pair Encoding (BPE) was proposed by Philip Gage in 1994 and was later introduced into the field of neural machine translation by Sennrich et al. in 2016.
70%VerifiedThe BPE (Byte Pair Encoding) algorithm was originally introduced into NLP by Sennrich et al. in 2016, and is essentially a lossless data compression technique.
70%VerifiedBPE (Byte Pair Encoding) was introduced into the NLP field by Sennrich et al. in 2016.
70%VerifiedThe Byte Pair Encoding (BPE) tokenization algorithm was originally introduced by OpenAI in GPT-2, splitting text into subword units by iteratively merging high-frequency character pairs.
65%VerifiedMainstream large models generally use Byte Pair Encoding (BPE) or its variants (such as SentencePiece and tiktoken) for tokenization.
65%UnverifiedBPE starts at the character level and progressively merges the most frequently co-occurring adjacent character pairs into longer subword units
90%Partially VerifiedToken是大语言模型处理文本的基本计量单位，由BPE（字节对编码）等分词算法生成
80%