Unverified50% confidenceFactExact time
Mainstream large language models generally adopt BPE (Byte Pair Encoding) or its variants (such as SentencePiece) as tokenization algorithms, and GPT-4's vocabulary is approximately 100,000 tokens.
1
Sources
50%
Confidence
Long-term
Relevance
9/11/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedGPT-4 使用约 10 万 Token 的词表78% similarUnverifiedGPT series models use byte-level BPE (Byte-level BPE), which takes raw bytes as the basic unit to eliminate the out-of-vocabulary character problem.77% similarUnverifiedOpenAI has used a variant of BPE since GPT-2, and GPT-4's cl100k_base vocabulary contains approximately 100,000 tokens.77% similarUnverifiedThe BPE (Byte Pair Encoding) algorithm commonly used in the GPT series builds a vocabulary by counting the frequency of adjacent character pairs and merging the highest-frequency character pairs.75% similarVerifiedThe cl100k_base vocabulary used by GPT-4 contains approximately 100,000 token units, and the number of tokens in a text can be precisely calculated using the tiktoken library.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/893826API
curl https://kongchang.com/api/v1/knowledge/claims/893826MCP
get_claim(id=893826)