SentencePiece
A language-agnostic subword tokenization tool supporting multiple algorithms including BPE and Unigram. It is widely used as a tokenizer for large language models and can learn tokenization models directly from raw text.
Timeline (last 90 days)
A Tokenizer is responsible for splitting raw text into token sequences that the model can process; mainstream approaches include BPE, WordPiece, and SentencePiece.
Metaspace is a special pre-tokenization processing technique common in the SentencePiece tokenization system, which replaces spaces between words with a special underscore character (▁).
The SentencePiece toolkit includes a complete implementation of Unigram language model tokenization.
The GLM series uses a SentencePiece-based tokenizer and includes several special control tokens such as [gMASK] and <sop>.
Mainstream large language models generally adopt BPE (Byte Pair Encoding) or its variants (such as SentencePiece) as tokenization algorithms, and GPT-4's vocabulary is approximately 100,000 tokens.
SentencePiece is a language-agnostic tokenization framework developed by Google, particularly well-suited for languages without clear word boundaries such as Chinese, Japanese, and Korean.
The GPT series uses the BPE algorithm with vocabularies such as cl100k_base, the Llama series uses SentencePiece, and Gemini has its own proprietary tokenization scheme.
The project does not rely on heavyweight frameworks such as Hugging Face; the core code is under 400 lines, with only 5 dependencies: PyTorch, SentencePiece, NumPy, TensorBoard, and TQDM.
The core code of the generated TinyGPT project is under 400 lines, with only 5 dependencies: PyTorch, SentencePiece, NumPy, TensorBoard, and TQDM.
SentencePiece is a language-agnostic tokenization framework proposed by Google that models directly on raw text and is suitable for multilingual use and languages without clear word boundaries.
4 more timeline events
All Facts (15)
SentencePiece is a language-agnostic tokenization framework proposed by Google that models directly on raw text and is suitable for multilingual use and languages without clear word boundaries.
60%UnverifiedDifferent models use different tokenization schemes (such as BPE, SentencePiece, etc.), and the same piece of text may be counted as a different number of tokens across different models.
60%UnverifiedOpenAI uses the BPE (Byte Pair Encoding) algorithm for tokenization, while Google's SentencePiece adopts a language-agnostic tokenization method trained directly on raw text.
60%UnverifiedOpenAI uses the tiktoken tokenization algorithm, and Google uses the SentencePiece tokenization algorithm.
60%UnverifiedA Tokenizer is responsible for splitting raw text into token sequences that the model can process; mainstream approaches include BPE, WordPiece, and SentencePiece.
50%UnverifiedMetaspace is a special pre-tokenization processing technique common in the SentencePiece tokenization system, which replaces spaces between words with a special underscore character (▁).
50%UnverifiedThe SentencePiece toolkit includes a complete implementation of Unigram language model tokenization.
50%UnverifiedThe GLM series uses a SentencePiece-based tokenizer and includes several special control tokens such as [gMASK] and <sop>.
50%UnverifiedMainstream large language models generally adopt BPE (Byte Pair Encoding) or its variants (such as SentencePiece) as tokenization algorithms, and GPT-4's vocabulary is approximately 100,000 tokens.
50%UnverifiedSentencePiece is a language-agnostic tokenization framework developed by Google, particularly well-suited for languages without clear word boundaries such as Chinese, Japanese, and Korean.
50%UnverifiedThe GPT series uses the BPE algorithm with vocabularies such as cl100k_base, the Llama series uses SentencePiece, and Gemini has its own proprietary tokenization scheme.
50%UnverifiedThe project does not rely on heavyweight frameworks such as Hugging Face; the core code is under 400 lines, with only 5 dependencies: PyTorch, SentencePiece, NumPy, TensorBoard, and TQDM.
50%UnverifiedThe core code of the generated TinyGPT project is under 400 lines, with only 5 dependencies: PyTorch, SentencePiece, NumPy, TensorBoard, and TQDM.
50%Unverified现代语言模型的词汇表规模通常在32,000到100,000个token之间，使用BPE或SentencePiece等分词方法
50%Unverified不同模型的Token化方式存在差异，OpenAI使用BPE算法，Google使用SentencePiece
50%