TTFT
A key user-experience metric in large language model inference, measuring the elapsed time from request submission to the appearance of the first output token. It is primarily determined by the prefill stage of inference.
Core Facts
Time to First Token (TTFT) refers to the time interval between a user sending a request and the model outputting the first token, and is primarily constrained by the duration of the prefill stage.
TPS measurement is typically divided into two phases: Time to First Token (TTFT) and the generation rate (throughput) of subsequent tokens.
TPS (Tokens Per Second) is the core metric for measuring the inference speed of large language models, specifically referring to the number of tokens the model outputs per second during the generation phase.
Timeline (last 90 days)
Long context significantly raises time to first token, because prefill computation grows with input length; hitting the cache can skip most of the prefill and directly improve first-token latency.
The research found that, given sufficient CPU resources, the time from sending a request to receiving the first token (TTFT) was shortened by 1.47 to 7.11 times across the various configurations tested.
The prefill stage refers to the process in which the model processes the input prompt all at once; the computation is proportional to input length, and in long-document scenarios the time to first token (TTFT) is significantly prolonged.
Most public APIs do not expose two metrics: time to first token and token throughput per second.
Using only completion time as an optimization metric obscures time-to-first-byte (TTFB/TTFT), which is critical to real user experience.
Longer context means higher time-to-first-token (TTFT), and token count is directly linearly correlated with inference costs.
Aggressively batching prefill can improve throughput but inflates TTFT, while guaranteeing TTFT constrains prefill batch sizes, creating a conflict between the two optimization objectives.
Higher memory bandwidth means lower time to first token (TTFT) and higher tokens per second (TPS) generation.
Prefix caching, by reusing KV vector caches for identical context prefixes, can bring 50%–80% computational savings in AI agent scenarios and significantly reduce time-to-first-token (TTFT) latency.
In large model inference, introducing cross-region selection logic on the hot path adds network round-trip time (RTT), easily stacking tens to hundreds of milliseconds, directly degrading the time-to-first-token (TTFT) metric.
35 more timeline events
All Facts (20)
Time to First Token (TTFT) refers to the time interval between a user sending a request and the model outputting the first token, and is primarily constrained by the duration of the prefill stage.
90%VerifiedTPS measurement is typically divided into two phases: Time to First Token (TTFT) and the generation rate (throughput) of subsequent tokens.
85%VerifiedTPS (Tokens Per Second) is the core metric for measuring the inference speed of large language models, specifically referring to the number of tokens the model outputs per second during the generation phase.
75%VerifiedInference latency is typically decomposed into two core metrics: time to first token (TTFT) and subsequent token generation throughput.
75%VerifiedIn digital human scenarios, the WebRTC end-to-end latency typically needs to be kept within 500 milliseconds to ensure a natural and smooth interactive experience.
75%VerifiedTime to First Token (TTFT) refers to the time interval between a user submitting a request and the model outputting the first token, and is primarily constrained by the time cost of the prefill phase in which the model processes the input prompt.
65%VerifiedTTFT is determined by the Prefill phase and is compute-bound, while TPOT corresponds to the Decode phase and is memory-bandwidth-bound.
65%Unverified传统混合部署中Prefill阶段的大批量计算会阻塞Decode阶段的token生成，导致TTFT波动剧烈
85%UnverifiedPD分离架构通过物理隔离消除Prefill对Decode的干扰，显著改善TTFT的P99延迟
85%UnverifiedLarge model inference is divided into two phases, Prefill and Decode; Prefill is a major component of Time to First Token (TTFT), while Decode corresponds to Time Per Output Token (TPOT).
60%UnverifiedTime to First Byte (TTFB) reflects the latency from when a request is sent to when the model returns the first token, encompassing network transmission, request queue waiting time, and model warm-up time.
60%UnverifiedWhen current mainstream large models handle complex code tasks, the time to first token (TTFT) is often between 2 and 8 seconds.
60%UnverifiedLong context significantly raises time to first token, because prefill computation grows with input length; hitting the cache can skip most of the prefill and directly improve first-token latency.
50%UnverifiedThe research found that, given sufficient CPU resources, the time from sending a request to receiving the first token (TTFT) was shortened by 1.47 to 7.11 times across the various configurations tested.
50%UnverifiedThe prefill stage refers to the process in which the model processes the input prompt all at once; the computation is proportional to input length, and in long-document scenarios the time to first token (TTFT) is significantly prolonged.
50%UnverifiedMost public APIs do not expose two metrics: time to first token and token throughput per second.
50%UnverifiedUsing only completion time as an optimization metric obscures time-to-first-byte (TTFB/TTFT), which is critical to real user experience.
50%UnverifiedLonger context means higher time-to-first-token (TTFT), and token count is directly linearly correlated with inference costs.
50%UnverifiedAggressively batching prefill can improve throughput but inflates TTFT, while guaranteeing TTFT constrains prefill batch sizes, creating a conflict between the two optimization objectives.
50%UnverifiedHigher memory bandwidth means lower time to first token (TTFT) and higher tokens per second (TPS) generation.
50%