Delta Tokenizer
A CPU-frontend tokenization optimization scheme for LLM inference serving. Through session-incremental reuse and segmented parallel encoding, it substantially reduces time-to-first-token (TTFT) in long-prompt scenarios without altering output semantics.
Timeline (last 90 days)
Delta Tokenizer is a scheme designed for the CPU frontend tokenization bottleneck in LLM inference, with the core idea of tokenizing only the delta between the current request and the previous request in the same session.
Delta Tokenizer retains the previous token sequence, re-encodes only the appended text plus a short tail that may be affected by the appending, and then concatenates at a position where the tokenizer's own segmentation rules cannot move.
In a 16-core environment, Delta Tokenizer encodes a 4-million-character request 8.2 times faster than serial encoding and 1.6 to 2.1 times faster than LoPT.
Delta Tokenizer uses a shared CPU pool to simultaneously manage the chunking of long requests and short incremental updates, so that an incremental update can be executed inserted between two chunks rather than having to queue after the entire long task.
The incremental path shortens the tokenization time of a single-turn session by about 40 times relative to the Hugging Face reference tokenizer.
Inside SGLang, after using Delta Tokenizer, the median TTFT of the chat interface at 100K tokens is reduced by 59%.
Under load, the P99 TTFT of Agent sessions drops by 9.5 times; when the frontend is saturated by long requests, this figure reaches 23 times.
Every checked output of Delta Tokenizer matches the reference implementation, including the 41.8 billion documents processed by its fast encoder.
Introducing incremental reuse and segmented parallelism into the CPU frontend, combined with CPU pool scheduling, can yield considerable latency gains without changing the model and output semantics.
All Facts (9)
Delta Tokenizer is a scheme designed for the CPU frontend tokenization bottleneck in LLM inference, with the core idea of tokenizing only the delta between the current request and the previous request in the same session.
50%UnverifiedDelta Tokenizer retains the previous token sequence, re-encodes only the appended text plus a short tail that may be affected by the appending, and then concatenates at a position where the tokenizer's own segmentation rules cannot move.
50%UnverifiedIn a 16-core environment, Delta Tokenizer encodes a 4-million-character request 8.2 times faster than serial encoding and 1.6 to 2.1 times faster than LoPT.
50%UnverifiedDelta Tokenizer uses a shared CPU pool to simultaneously manage the chunking of long requests and short incremental updates, so that an incremental update can be executed inserted between two chunks rather than having to queue after the entire long task.
50%UnverifiedThe incremental path shortens the tokenization time of a single-turn session by about 40 times relative to the Hugging Face reference tokenizer.
50%UnverifiedInside SGLang, after using Delta Tokenizer, the median TTFT of the chat interface at 100K tokens is reduced by 59%.
50%UnverifiedUnder load, the P99 TTFT of Agent sessions drops by 9.5 times; when the frontend is saturated by long requests, this figure reaches 23 times.
50%UnverifiedEvery checked output of Delta Tokenizer matches the reference implementation, including the 41.8 billion documents processed by its fast encoder.
50%UnverifiedIntroducing incremental reuse and segmented parallelism into the CPU frontend, combined with CPU pool scheduling, can yield considerable latency gains without changing the model and output semantics.
50%