Partially Verified65% confidenceFactExact time
OpenAI's tiktoken expanded from the 50,000-token vocabulary of the GPT-2 era to the 100,000-token vocabulary of GPT-4.
3
Sources
65%
Confidence
Long-term
Relevance
7/6/2026
First Seen
Sources
Claude Sonnet 5 Hands-On Review: The Hidden Token Trap That Makes 'Low Cost' an Illusion
bilibili浮生千山路w6/30/2026
Related Claims
UnverifiedOpenAI's GPT series uses the tiktoken library, with vocabulary sizes typically ranging between 50,000 and 100,000 Tokens84% similarVerifiedOpenAI has used a variant of BPE since GPT-2, and GPT-4's cl100k_base vocabulary contains approximately 100,000 tokens.79% similarUnverifiedtiktoken is OpenAI's open-source tokenizer library, cl100k_base is the encoding scheme used for the GPT-3.5/4 series, and its tokenization results differ from Anthropic Claude's proprietary tokenizer, usually by a single-digit to low-teen percentage.77% similarVerifiedThe cl100k_base vocabulary used by GPT-4 contains approximately 100,000 token units, and the number of tokens in a text can be precisely calculated using the tiktoken library.77% similarVerifiedThe GPT-4 series uses the cl100k_base vocabulary (approximately 100,000 tokens), while the GPT-4o series upgrades to o200k_base (approximately 200,000 tokens).76% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/114167API
curl https://kongchang.com/api/v1/knowledge/claims/114167MCP
get_claim(id=114167)