Unverified50% confidenceFactExact time
SentencePiece由Google的Kudo和Richardson在2018年提出，将输入文本视为原始字符流，不依赖预分词步骤
1
Sources
50%
Confidence
Long-term
Relevance
8/8/2026
First Seen
Sources
Related Claims
Unverified现代 LLM 使用子词分词算法，最常见的包括 OpenAI 提出的 BPE 和 Google 提出的 SentencePiece62% similarUnverifiedOpenAI uses the tiktoken tokenization algorithm, and Google uses the SentencePiece tokenization algorithm.62% similarUnverified2018年Google发布的BERT引入双向上下文编码，Sentence-BERT专为句子级语义相似度优化61% similarUnverifiedRecursive character chunking is LangChain's default approach, which sequentially attempts to split at hierarchical boundaries such as paragraphs, sentences, and words in order to prioritize semantic integrity.58% similarUnverifiedThe development of Big Words was prompted by the fact that Simon Willison's macOS presentation tool only accepted URLs as input, and he needed a way to add plain text slides.56% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/707211API
curl https://kongchang.com/api/v1/knowledge/claims/707211MCP
get_claim(id=707211)