Unverified50% confidenceFactExact time
The smallest unit most LLMs process is the token, which can be a complete word, part of a word, a punctuation mark or another text fragment, with the specific segmentation depending on the tokenizer.
1
Sources
50%
Confidence
Long-term
Relevance
10/19/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedLarge models process text using Token (word element) as the minimum processing unit, with tokenizers handling the segmentation.74% similarUnverifiedA token is the basic unit of text processed by an LLM, and it can be a complete word, part of a word, punctuation, or a space.70% similarUnverifiedLarge language model API pricing is typically measured in tokens, which are the smallest units a model uses to process text70% similarVerifiedIn large language models, a Token is the smallest unit of text that a model processes68% similarUnverifiedAI模型调用的计费单位通常是Token，即文本被分词器切割后的最小语义单元68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1067372API
curl https://kongchang.com/api/v1/knowledge/claims/1067372MCP
get_claim(id=1067372)