Unverified50% confidenceFactExact time
The input to an LLM is tokens rather than words: text is split into tokens, and each token corresponds to an ID in a fixed vocabulary.
1
Sources
50%
Confidence
Long-term
Relevance
10/18/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedLLM models do not directly read characters or words; the tokenizer splits text into chunks according to a fixed vocabulary, and what the model sees is a string of chunk IDs.80% similarUnverifiedLLMs operate on numerical representations of text rather than reading the original text word by word like humans, so stylistic information (tone, rhetoric, copyright notices) is largely diluted during vectorization.71% similarUnverified现代 LLM 通常使用字节对编码（BPE）或类似算法进行分词，英文中平均约 4 个字符对应 1 个 Token70% similarUnverifiedLLM处理的是token（词元）而非单个字符，导致在字符计数任务上存在天然缺陷70% similarVerifiedModern LLMs generally use BPE (Byte Pair Encoding) or WordPiece algorithms for tokenization.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1058842API
curl https://kongchang.com/api/v1/knowledge/claims/1058842MCP
get_claim(id=1058842)