Unverified50% confidenceFactExact time
GPT-4的tiktoken分词器中，一个英文单词平均对应1.3个token，每个汉字约占2-3个token
1
Sources
50%
Confidence
Long-term
Relevance
7/7/2026
First Seen
Sources
AI编程注意力瓶颈：小龙虾之父的Agent工作流实战
bilibiliPalantir研究7/5/2026
Related Claims
UnverifiedGPT-4o introduced a new tokenizer optimized for multilingual use, improving Chinese token efficiency by approximately 3 times, with roughly 1–1.5 tokens now corresponding to one Chinese character.74% similarUnverifiedIn the GPT series of models, one token corresponds to approximately 4 English characters or 0.75 English words.73% similarVerifiedThe context window of the early GPT-3 was only 4,096 tokens, approximately 3,000 English words.67% similarPartially VerifiedOpenAI's tiktoken expanded from the 50,000-token vocabulary of the GPT-2 era to the 100,000-token vocabulary of GPT-4.66% similarUnverifiedEvery large language model has a fixed token limit, such as GPT-4's 128K and Claude 3's 200K.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/173351API
curl https://kongchang.com/api/v1/knowledge/claims/173351MCP
get_claim(id=173351)