Verified65% confidenceFactExact time
OpenAI has used a variant of BPE since GPT-2, and GPT-4's cl100k_base vocabulary contains approximately 100,000 tokens.
3
Sources
65%
Confidence
Long-term
Relevance
8/27/2026
First Seen
Sources
Related Entities
Related Claims
Partially VerifiedOpenAI's tiktoken expanded from the 50,000-token vocabulary of the GPT-2 era to the 100,000-token vocabulary of GPT-4.79% similarVerifiedThe cl100k_base vocabulary used by GPT-4 contains approximately 100,000 token units, and the number of tokens in a text can be precisely calculated using the tiktoken library.78% similarVerifiedThe GPT-4 series uses the cl100k_base vocabulary (approximately 100,000 tokens), while the GPT-4o series upgrades to o200k_base (approximately 200,000 tokens).77% similarUnverifiedMainstream large language models generally adopt BPE (Byte Pair Encoding) or its variants (such as SentencePiece) as tokenization algorithms, and GPT-4's vocabulary is approximately 100,000 tokens.77% similarUnverifiedWhen GPT-4 was released, OpenAI simultaneously published a nearly 100-page technical report covering performance data across multiple standardized exams and benchmark tests.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/809209API
curl https://kongchang.com/api/v1/knowledge/claims/809209MCP
get_claim(id=809209)