Unverified50% confidenceFactExact time
BPE was introduced into the field of natural language processing for subword tokenization by Sennrich et al. in 2016.
1
Sources
50%
Confidence
Long-term
Relevance
9/3/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedMainstream tokenizer approaches include BPE (Byte Pair Encoding), WordPiece, and SentencePiece.70% similarUnverified现代语言模型的词汇表规模通常在32,000到100,000个token之间，使用BPE或SentencePiece等分词方法64% similarUnverified研究界逐步转向结合大语言模型的新一代评估框架如COMET和BERTScore以更好捕捉语义层面翻译质量61% similarUnverifiedThe tiktoken tokenizer used by OpenAI is based on a BPE variant, where common English words are typically preserved as whole units while rare words are split into multiple subword units.61% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/849953API
curl https://kongchang.com/api/v1/knowledge/claims/849953MCP
get_claim(id=849953)