Unverified50% confidenceFactExact time
大模型处理文本以Token（词元）为最小处理单元，由分词器完成切分
1
Sources
50%
Confidence
Long-term
Relevance
7/21/2026
First Seen
Sources
Related Claims
VerifiedIn large language models, a Token is the smallest unit of text that a model processes85% similarUnverifiedAI模型调用的计费单位通常是Token，即文本被分词器切割后的最小语义单元80% similarUnverifiedToken是大模型计费的基本单位，本质上是文本被分割后的最小语义片段76% similarUnverified大语言模型底层的Transformer架构不直接处理原始字符，而是先通过分词器将文本切分为子词单元，再映射为高维向量进行计算75% similarUnverifiedtoken是大模型计费的依据，由分词器将原始文本切分而来，是大模型处理文本的基本单位74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/582537API
curl https://kongchang.com/api/v1/knowledge/claims/582537MCP
get_claim(id=582537)