Unverified50% confidenceFactExact time
不同模型使用不同的分词算法，如 OpenAI 的 tiktoken、Google 的 SentencePiece，导致同一段文字在不同模型中消耗的 token 数量可能相差 10%-30%
1
Sources
50%
Confidence
Long-term
Relevance
7/7/2026
First Seen
Sources
Kontext：一键迁移AI对话上下文，打破多模型切换壁垒
hackernewshackernews7/3/2026
Related Claims
UnverifiedOpenAI uses the tiktoken tokenization algorithm, and Google uses the SentencePiece tokenization algorithm.75% similarUnverified不同 Tokenizer（tiktoken、SentencePiece、Hugging Face 分词器）对同一文本的切分结果可能相差 20%-40%70% similarUnverifiedOpenAI的tiktoken库与Anthropic的分词器在词汇表构建策略上存在差异，导致相同内容在不同模型间的Token消耗量不同69% similarUnverifiedThe context of a large language model is essentially the total amount of information the model can process in a single inference, typically measured in the number of tokens.68% similarUnverifiedA token is the smallest unit of measurement for text processed by a model; OpenAI uses the BPE algorithm to encode high-frequency character combinations as single tokens, with approximately 75 English words equaling roughly 100 tokens.68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/195029API
curl https://kongchang.com/api/v1/knowledge/claims/195029MCP
get_claim(id=195029)