ITL
A key metric for measuring user experience in LLM serving, referring to the time interval between two consecutive output tokens. It is mainly determined by the decode phase and is affected by memory bandwidth, batch size and the volume of KV cache reads.
Timeline (last 90 days)
Inter-token latency (ITL) is the time interval between adjacent tokens during the generation process and determines the smoothness of streaming text output.
Deterministic execution architecture minimizes variance in time-to-first-token (TTFT) and inter-token latency (ITL), enabling SLA-like latency commitments to be offered externally.
TTFT determines how long a user waits from sending a request to seeing the first token, while ITL determines the pace at which subsequent content is generated.
All Facts (3)
Inter-token latency (ITL) is the time interval between adjacent tokens during the generation process and determines the smoothness of streaming text output.
50%UnverifiedDeterministic execution architecture minimizes variance in time-to-first-token (TTFT) and inter-token latency (ITL), enabling SLA-like latency commitments to be offered externally.
50%UnverifiedTTFT determines how long a user waits from sending a request to seeing the first token, while ITL determines the pace at which subsequent content is generated.
50%