Unverified50% confidenceFactExact time
The Llama series uses SentencePiece BPE tokenization, while the Qwen series uses a custom tokenizer based on byte-level BPE.
1
Sources
50%
Confidence
Long-term
Relevance
9/11/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedLlama系列模型使用SentencePiece的BPE模式76% similarVerifiedModern LLMs generally use BPE (Byte Pair Encoding) or WordPiece algorithms for tokenization.73% similarVerifiedMainstream large models generally use Byte Pair Encoding (BPE) or its variants (such as SentencePiece and tiktoken) for tokenization.69% similarUnverifiedLLM的分词器（如BPE）为自然语言设计，会将数值拆解为多个不相关的token，破坏数字的连续性和大小语义68% similarUnverifiedLLaMA 3的分词器从SentencePiece升级为基于tiktoken的BPE分词器，词表大小扩展至128,25666% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/893139API
curl https://kongchang.com/api/v1/knowledge/claims/893139MCP
get_claim(id=893139)