Unverified50% confidenceFactExact time
tiktoken's README gives an average of about 4 bytes of text per token.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/13/2026
First Seen
Valid until: 1/11/2027
Sources
Related Entities
Related Claims
UnverifiedToken由BPE（字节对编码）算法生成，英文文本平均每个token约对应4个字符（约0.75个单词）70% similarUnverifiedWhen a model reads text, approximately 3/4 of a word counts as one token.68% similarUnverifiedIn English, approximately every 4 characters form 1 Token, and OpenAI uses the BPE (Byte Pair Encoding) algorithm to map high-frequency words to single Tokens.64% similarUnverified现代 LLM 通常使用字节对编码（BPE）或类似算法进行分词，英文中平均约 4 个字符对应 1 个 Token61% similarVerifiedThe context window refers to the maximum amount of text a model can process in a single conversation; early models had only 4K tokens, while modern models have expanded to hundreds of thousands of tokens.61% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1022415API
curl https://kongchang.com/api/v1/knowledge/claims/1022415MCP
get_claim(id=1022415)