Unverified60% confidenceFactTime unknown
OpenAI uses the tiktoken tokenization algorithm, and Google uses the SentencePiece tokenization algorithm.
2
Sources
60%
Confidence
Long-term
Relevance
6/1/2026
First Seen
Sources
One API：统一管理30+大模型的开源API网关部署教程与详解
githubsongquanpeng
Related Entities
Related Claims
Unverified现代 LLM 使用子词分词算法，最常见的包括 OpenAI 提出的 BPE 和 Google 提出的 SentencePiece82% similarUnverified不同模型使用不同的分词算法，如 OpenAI 的 tiktoken、Google 的 SentencePiece，导致同一段文字在不同模型中消耗的 token 数量可能相差 10%-30%75% similarUnverifiedCommon semantic search implementation solutions include OpenAI's text-embedding model series or open-source sentence-transformers, paired with vector databases such as Pinecone and Qdrant.74% similarUnverifiedOpenAI uses the BPE (Byte Pair Encoding) algorithm for tokenization, while Google's SentencePiece adopts a language-agnostic tokenization method trained directly on raw text.74% similarUnverifiedModern large language models universally use BPE algorithm variants to build tokenizers, with OpenAI's tiktoken, Google's SentencePiece, and HuggingFace's Tokenizers library being the three mainstream implementations71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/34785API
curl https://kongchang.com/api/v1/knowledge/claims/34785MCP
get_claim(id=34785)