Unverified70% confidenceFactTime unknown
现代LLM的分词器通常将文本压缩为约10万词汇量的Token序列
1
Sources
70%
Confidence
Long-term
Relevance
6/1/2026
First Seen
Sources
Related Entities
Related Claims
Unverified上下文窗口决定模型单次能处理的文本量：8K约6000词，32K约24000词，128K可处理整本书内容70% similarUnverified现代 LLM 通常使用字节对编码（BPE）或类似算法进行分词，英文中平均约 4 个字符对应 1 个 Token67% similarUnverified1M 上下文意味着模型可以一次性处理约 75 万个英文单词或数十万行代码67% similarUnverifiedPrompt compression techniques such as LLMLingua can compress prompt length by 30%–50% while preserving semantics.66% similarUnverifiedLarge models process text using Token (word element) as the minimum processing unit, with tokenizers handling the segmentation.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/34360API
curl https://kongchang.com/api/v1/knowledge/claims/34360MCP
get_claim(id=34360)