cl100k_base
A token encoding scheme in the tiktoken library used for the GPT-3.5/4 series, mapping text to token sequences. It differs from the tokenizer used by Claude, so counting Claude tokens with it can only be used for relative ranking.
Core Facts
The GPT-4 series uses the cl100k_base vocabulary (approximately 100,000 tokens), while the GPT-4o series upgrades to o200k_base (approximately 200,000 tokens).
The cl100k_base vocabulary used by GPT-4 contains approximately 100,000 token units, and the number of tokens in a text can be precisely calculated using the tiktoken library.
OpenAI has used a variant of BPE since GPT-2, and GPT-4's cl100k_base vocabulary contains approximately 100,000 tokens.
Timeline (last 90 days)
o200k_base is the tokenizer OpenAI introduced for the GPT-4o series, with a vocabulary of about 200,000, and its compression efficiency for multilingual (especially non-English) text is clearly better than the earlier cl100k_base.
In a test with a nested configuration object, encoded using OpenAI's cl100k_base tokenizer, JSON was 99 tokens and 322 characters, while YAML was 66 tokens and 223 characters, with YAML reducing the token count by 33.3%.
In a test with regular tabular data of 10 rows and 4 fields per row, encoded using the cl100k_base tokenizer, JSON was 323 tokens and 963 characters, while YAML was 221 tokens and 590 characters, with YAML ahead by 31.6%; the TOON format was 97 tokens and 235 characters, 70% fewer than JSON and 56.1% fewer than YAML.
The o200k_base vocabulary is expanded to approximately 200,000 tokens, offering more efficient encoding of common symbols and multi-byte characters in code compared to cl100k_base.
cl100k_base is the specific vocabulary used by models such as GPT-4 and text-embedding-ada-002.
In the BPE tokenization (cl100k_base, 512/50) scenario, Chunkr achieved 38 MB/s, while Chonkie led at 151 MB/s.
The GPT series uses the BPE algorithm with vocabularies such as cl100k_base, the Llama series uses SentencePiece, and Gemini has its own proprietary tokenization scheme.
OpenAI has used a variant of BPE since GPT-2, and GPT-4's cl100k_base vocabulary contains approximately 100,000 tokens.
GPT-4 使用 cl100k_base tokenizer，与 Claude 的 tokenizer 对中文、代码等内容的切分策略不同
All Facts (11)
The GPT-4 series uses the cl100k_base vocabulary (approximately 100,000 tokens), while the GPT-4o series upgrades to o200k_base (approximately 200,000 tokens).
90%VerifiedThe cl100k_base vocabulary used by GPT-4 contains approximately 100,000 token units, and the number of tokens in a text can be precisely calculated using the tiktoken library.
80%VerifiedOpenAI has used a variant of BPE since GPT-2, and GPT-4's cl100k_base vocabulary contains approximately 100,000 tokens.
65%Unverifiedo200k_base is the tokenizer OpenAI introduced for the GPT-4o series, with a vocabulary of about 200,000, and its compression efficiency for multilingual (especially non-English) text is clearly better than the earlier cl100k_base.
50%UnverifiedIn a test with a nested configuration object, encoded using OpenAI's cl100k_base tokenizer, JSON was 99 tokens and 322 characters, while YAML was 66 tokens and 223 characters, with YAML reducing the token count by 33.3%.
50%UnverifiedIn a test with regular tabular data of 10 rows and 4 fields per row, encoded using the cl100k_base tokenizer, JSON was 323 tokens and 963 characters, while YAML was 221 tokens and 590 characters, with YAML ahead by 31.6%; the TOON format was 97 tokens and 235 characters, 70% fewer than JSON and 56.1% fewer than YAML.
50%UnverifiedThe o200k_base vocabulary is expanded to approximately 200,000 tokens, offering more efficient encoding of common symbols and multi-byte characters in code compared to cl100k_base.
50%Unverifiedcl100k_base is the specific vocabulary used by models such as GPT-4 and text-embedding-ada-002.
50%UnverifiedIn the BPE tokenization (cl100k_base, 512/50) scenario, Chunkr achieved 38 MB/s, while Chonkie led at 151 MB/s.
50%UnverifiedThe GPT series uses the BPE algorithm with vocabularies such as cl100k_base, the Llama series uses SentencePiece, and Gemini has its own proprietary tokenization scheme.
50%UnverifiedGPT-4 使用 cl100k_base tokenizer，与 Claude 的 tokenizer 对中文、代码等内容的切分策略不同
50%