TPS
A core performance metric for large language model inference, measuring the number of tokens a model generates per second. TPS determines the total time required for long-text generation and reflects the overall throughput capacity of an inference system.
Core Facts
Time to First Token (TTFT) refers to the time interval between a user sending a request and the model outputting the first token, and is primarily constrained by the duration of the prefill stage.
TPS measurement is typically divided into two phases: Time to First Token (TTFT) and the generation rate (throughput) of subsequent tokens.
TPS (Tokens Per Second) is the core metric for measuring the inference speed of large language models, specifically referring to the number of tokens the model outputs per second during the generation phase.
Timeline (last 90 days)
The latency of streaming LLMs consists of two segments of completely different nature: the cold-start segment of queueing plus first-token inference, and the steady-state segment of subsequent token-by-token generation; SLOs should be set separately for each and cannot be lumped into a single average response time.
Higher memory bandwidth means lower time to first token (TTFT) and higher tokens per second (TPS) generation.
The Parker Probe's carbon composite Thermal Protection System (TPS) is approximately 11.4 centimeters thick, consisting of carbon foam sandwiched between two layers of carbon fiber panels.
The officially advertised 7x+ speedup primarily refers to TPS (tokens per second), i.e., throughput improvement in the generation phase, and does not necessarily mean an equivalent reduction in TTFT.
To assess whether this fix is effective for a given workload, one can monitor metrics such as GPU utilization stability, time to first token (TTFT), and tokens per second (TPS).
Fireworks achieves lower time-to-first-token latency and higher token throughput per second through underlying GPU scheduling optimization and inference engine customization.
The larger the batch size, the higher the per-token amortized compute efficiency and the generally higher the TPS, but latency will also increase.
The Decode phase of TPS is the response speed actually perceived by the user, and hardware memory bandwidth is often the true bottleneck of the decoding phase rather than pure floating-point compute.
When a model's generation speed exceeds 15–20 TPS, users no longer perceive noticeable waiting during interaction, because humans read English at approximately 4–5 tokens per second
TPS (Tokens Per Second) is the core metric for measuring the inference speed of large language models, specifically referring to the number of tokens the model outputs per second during the generation phase.
1 more timeline events
All Facts (12)
Time to First Token (TTFT) refers to the time interval between a user sending a request and the model outputting the first token, and is primarily constrained by the duration of the prefill stage.
90%VerifiedTPS measurement is typically divided into two phases: Time to First Token (TTFT) and the generation rate (throughput) of subsequent tokens.
85%VerifiedTPS (Tokens Per Second) is the core metric for measuring the inference speed of large language models, specifically referring to the number of tokens the model outputs per second during the generation phase.
75%UnverifiedWhen a model's generation speed exceeds 15–20 TPS, users no longer perceive noticeable waiting during interaction, because humans read English at approximately 4–5 tokens per second
60%UnverifiedThe latency of streaming LLMs consists of two segments of completely different nature: the cold-start segment of queueing plus first-token inference, and the steady-state segment of subsequent token-by-token generation; SLOs should be set separately for each and cannot be lumped into a single average response time.
50%UnverifiedHigher memory bandwidth means lower time to first token (TTFT) and higher tokens per second (TPS) generation.
50%UnverifiedThe Parker Probe's carbon composite Thermal Protection System (TPS) is approximately 11.4 centimeters thick, consisting of carbon foam sandwiched between two layers of carbon fiber panels.
50%UnverifiedThe officially advertised 7x+ speedup primarily refers to TPS (tokens per second), i.e., throughput improvement in the generation phase, and does not necessarily mean an equivalent reduction in TTFT.
50%UnverifiedTo assess whether this fix is effective for a given workload, one can monitor metrics such as GPU utilization stability, time to first token (TTFT), and tokens per second (TPS).
50%UnverifiedFireworks achieves lower time-to-first-token latency and higher token throughput per second through underlying GPU scheduling optimization and inference engine customization.
50%UnverifiedThe larger the batch size, the higher the per-token amortized compute efficiency and the generally higher the TPS, but latency will also increase.
50%UnverifiedThe Decode phase of TPS is the response speed actually perceived by the user, and hardware memory bandwidth is often the true bottleneck of the decoding phase rather than pure floating-point compute.
50%