Token
Token是自然语言处理和大语言模型中文本的基本处理单元，由分词器（Tokenizer）将原始文本切分而成，可对应一个单词、子词、字符或标点符号。模型在处理输入和生成输出时均以Token为单位进行计算，Token数量直接影响模型的上下文窗口限制、推理速度及API调用费用。不同语言和模型的Token划分规则存在差异。
Core Facts
Timeline (last 90 days)
商业API定价模型中，厂商通常按输入token和输出token分别计费
解决Token爆炸的常见策略包括对话历史摘要压缩、滑动窗口截断、按重要性筛选上下文、分层记忆机制
编程对话通常涉及大量代码上下文的输入输出，Token消耗量远高于普通文本对话
在AI大模型时代，Token是一种信息单位（词元），大语言模型通过分词器将输入文本拆分成Token序列
API调用费用按输入token和输出token分别计价，且输出token的单价通常高于输入token，因为输出阶段需要自回归推理
A single English word may be split into one or more Tokens, while Chinese characters typically correspond to one or two Tokens
Output Tokens are typically 2-4x more expensive than input Tokens because generation requires autoregressive decoding Token by Token, with each generated Token requiring a complete forward pass computation
In large language models, a Token is the smallest unit of text that a model processes
Tokens are the basic unit of measurement for how large language models process text, roughly equivalent to 3/4 of a word in English or 1-2 characters in Chinese
Current mainstream model pricing is split into input Tokens and output Tokens, with output Tokens typically costing 2-4x more than input Tokens
12 more timeline events
All Facts (20)
Token是大语言模型处理文本的基本单位，大约对应0.75个英文单词或1.5个汉字
90%Verified大模型API中输出Token的单价通常是输入Token的2-4倍
80%Verified大模型API的定价通常基于Token计费机制
70%Verified大模型本质上是基于概率的文本生成器，通过预测下一个token来生成内容
70%Verified同样的中文文本在不同大模型服务商处消耗的 Token 数量可能相差 30%-50%
65%UnverifiedThe larger the context, the more input Tokens consumed
95%UnverifiedIn large language models, a Token is the smallest unit of text that a model processes
95%UnverifiedA single English word may be split into one or more Tokens, while Chinese characters typically correspond to one or two Tokens
90%UnverifiedTokens are the basic unit of measurement for how large language models process text, roughly equivalent to 3/4 of a word in English or 1-2 characters in Chinese
85%UnverifiedOutput Tokens are typically 2-4x more expensive than input Tokens because generation requires autoregressive decoding Token by Token, with each generated Token requiring a complete forward pass computation
85%UnverifiedCurrent mainstream model pricing is split into input Tokens and output Tokens, with output Tokens typically costing 2-4x more than input Tokens
80%UnverifiedAI Agent工作流的算力消耗可能是单轮对话的50到100倍以上
65%Unverified大模型输出 Token 单价普遍高于输入 Token
60%Unverified国内日均Token调用量飙涨超千倍
60%Unverified商业API定价模型中，厂商通常按输入token和输出token分别计费
50%Unverified解决Token爆炸的常见策略包括对话历史摘要压缩、滑动窗口截断、按重要性筛选上下文、分层记忆机制
50%Unverified编程对话通常涉及大量代码上下文的输入输出，Token消耗量远高于普通文本对话
50%Unverified在AI大模型时代，Token是一种信息单位（词元），大语言模型通过分词器将输入文本拆分成Token序列
50%UnverifiedAPI调用费用按输入token和输出token分别计价，且输出token的单价通常高于输入token，因为输出阶段需要自回归推理
50%UnverifiedAI模型的推理成本主要由Token消耗量与单价共同决定，通常以每百万Token计价
50%