Tokenization
The process of splitting raw text into a sequence of tokens. It is divided into three approaches: character-level, word-level, and subword-level. Tokenization is the first step in how embedding models and language models process text.
Timeline (last 90 days)
The tokenization strategy directly determines the granularity of vocabulary validation: too coarse a split will miss fabricated content, while too fine a split will misjudge fragments with no real meaning as semantic words and cause false positives; synonyms, morphological variations and cross-language mixing affect matching, and non-English scenarios require specially designed tokenization and morphological normalization schemes.
The inference engine reuses shared prefixes via the KV cache, letting new tokens skip repeated prefill computation, but the tokenization step has no similar reuse shortcut, and the CPU frontend of the serving engine must reconvert the entire request into token IDs from the first character every time.
After KV caching, attention optimization, quantization, and other stages have been repeatedly polished, the preprocessing step of tokenization may still be an important contributor to TTFT, especially in long-prompt and Agent multi-turn scenarios.
Research from Georgia Tech found that in agent multi-turn conversation scenarios, the history must be re-tokenized on the CPU every turn, while the GPU's burden actually shrinks because it has cached the content it has already seen.
The research found that, given sufficient CPU resources, the time from sending a request to receiving the first token (TTFT) was shortened by 1.47 to 7.11 times across the various configurations tested.
In large-model inference pipelines, text must first be converted into tokens by the CPU before the GPU can start working, so the CPU cost of tokenization lies at the front of the GPU computation's critical path.
Neither study can determine how much of the idle time is attributable to tokenization rather than output reassembly, parsing, or launching tasks on the chip.
In a PyTorch blog post, an analysis by LightSeek Foundation advocates moving the tokenization work entirely out of the Python process.
When Hugging Face released its new-generation tokenization library, it judged that tokenization has historically never been a bottleneck, but becomes a bottleneck when serving many requests simultaneously or repeatedly processing extremely long inputs.
Tokenization is essentially table lookup and a small amount of string matching, with complexity approximately linear in sequence length, and unit cost far lower than self-attention or matrix multiplication in a Transformer.
All Facts (10)
The tokenization strategy directly determines the granularity of vocabulary validation: too coarse a split will miss fabricated content, while too fine a split will misjudge fragments with no real meaning as semantic words and cause false positives; synonyms, morphological variations and cross-language mixing affect matching, and non-English scenarios require specially designed tokenization and morphological normalization schemes.
50%UnverifiedThe inference engine reuses shared prefixes via the KV cache, letting new tokens skip repeated prefill computation, but the tokenization step has no similar reuse shortcut, and the CPU frontend of the serving engine must reconvert the entire request into token IDs from the first character every time.
50%UnverifiedAfter KV caching, attention optimization, quantization, and other stages have been repeatedly polished, the preprocessing step of tokenization may still be an important contributor to TTFT, especially in long-prompt and Agent multi-turn scenarios.
50%UnverifiedResearch from Georgia Tech found that in agent multi-turn conversation scenarios, the history must be re-tokenized on the CPU every turn, while the GPU's burden actually shrinks because it has cached the content it has already seen.
50%UnverifiedThe research found that, given sufficient CPU resources, the time from sending a request to receiving the first token (TTFT) was shortened by 1.47 to 7.11 times across the various configurations tested.
50%UnverifiedIn large-model inference pipelines, text must first be converted into tokens by the CPU before the GPU can start working, so the CPU cost of tokenization lies at the front of the GPU computation's critical path.
50%UnverifiedNeither study can determine how much of the idle time is attributable to tokenization rather than output reassembly, parsing, or launching tasks on the chip.
50%UnverifiedIn a PyTorch blog post, an analysis by LightSeek Foundation advocates moving the tokenization work entirely out of the Python process.
50%UnverifiedWhen Hugging Face released its new-generation tokenization library, it judged that tokenization has historically never been a bottleneck, but becomes a bottleneck when serving many requests simultaneously or repeatedly processing extremely long inputs.
50%UnverifiedTokenization is essentially table lookup and a small amount of string matching, with complexity approximately linear in sequence length, and unit cost far lower than self-attention or matrix multiplication in a Transformer.
50%