Unverified50% confidenceFactExact time
tiktoken is OpenAI's open-source tokenizer library, cl100k_base is the encoding scheme used for the GPT-3.5/4 series, and its tokenization results differ from Anthropic Claude's proprietary tokenizer, usually by a single-digit to low-teen percentage.
1
Sources
50%
Confidence
Long-term
Relevance
10/12/2026
First Seen
Sources
Related Entities
Related Claims
Partially VerifiedOpenAI's tiktoken expanded from the 50,000-token vocabulary of the GPT-2 era to the 100,000-token vocabulary of GPT-4.77% similarVerifiedThe GPT series uses tiktoken (based on BPE encoding) for tokenization.76% similarUnverifiedOpenAI's GPT series uses the tiktoken library, with vocabulary sizes typically ranging between 50,000 and 100,000 Tokens72% similarVerifiedThe cl100k_base vocabulary used by GPT-4 contains approximately 100,000 token units, and the number of tokens in a text can be precisely calculated using the tiktoken library.72% similarUnverifiedGPT系列使用tiktoken分词器，LLaMA系列使用SentencePiece分词器72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1006884API
curl https://kongchang.com/api/v1/knowledge/claims/1006884MCP
get_claim(id=1006884)