Throughput
The number of requests or generated tokens a system can complete per unit of time, typically measured in tokens/s or requests/s. It can be improved through techniques such as batching, KV cache reuse, quantization and speculative decoding.
Timeline (last 90 days)
Single-stream decode speed figures cannot be equated with throughput under multiple concurrent requests.
Achieving deterministic inference may require sacrificing more than 30% of throughput, and this trade-off is not worthwhile in most production settings.
Throughput refers to the number of requests a model can process per unit of time.
When evaluating an inference engine, throughput and latency must be distinguished: throughput usually refers to the total number of tokens generated by the whole system per unit of time, while latency focuses on the response speed of a single request, divided into first-token latency and per-token generation interval; a larger batch can improve throughput but may raise the latency of a single request.
The author recommends replacing subjective future estimates with real delivery data based on historical performance for forecasting.
Throughput is typically measured in tokens generated per second (tokens/s), determining how many concurrent requests a system can serve simultaneously.
Pursuing extreme throughput often comes at the cost of per-request latency, as throughput and latency metrics constrain each other.
When concurrency increases, individual requests may slow down, but overall system throughput may actually be higher.
In chip architecture design, throughput and latency are typically a pair of metrics that require trade-offs.
In actual production environments, service providers are more concerned with overall throughput (number of users served per second), as single-stream speed and overall efficiency under high concurrency are two different evaluation dimensions.
All Facts (10)
Single-stream decode speed figures cannot be equated with throughput under multiple concurrent requests.
50%UnverifiedAchieving deterministic inference may require sacrificing more than 30% of throughput, and this trade-off is not worthwhile in most production settings.
50%UnverifiedThroughput refers to the number of requests a model can process per unit of time.
50%UnverifiedWhen evaluating an inference engine, throughput and latency must be distinguished: throughput usually refers to the total number of tokens generated by the whole system per unit of time, while latency focuses on the response speed of a single request, divided into first-token latency and per-token generation interval; a larger batch can improve throughput but may raise the latency of a single request.
50%UnverifiedThe author recommends replacing subjective future estimates with real delivery data based on historical performance for forecasting.
50%UnverifiedThroughput is typically measured in tokens generated per second (tokens/s), determining how many concurrent requests a system can serve simultaneously.
50%UnverifiedPursuing extreme throughput often comes at the cost of per-request latency, as throughput and latency metrics constrain each other.
50%UnverifiedWhen concurrency increases, individual requests may slow down, but overall system throughput may actually be higher.
50%UnverifiedIn chip architecture design, throughput and latency are typically a pair of metrics that require trade-offs.
50%UnverifiedIn actual production environments, service providers are more concerned with overall throughput (number of users served per second), as single-stream speed and overall efficiency under high concurrency are two different evaluation dimensions.
50%