Latency
A key metric in performance evaluation, referring to the time elapsed from issuing a single request to receiving the complete response. It is a basic measure for assessing the response speed of a model inference service.
Core Facts
Timeline (last 90 days)
When evaluating an inference engine, throughput and latency must be distinguished: throughput usually refers to the total number of tokens generated by the whole system per unit of time, while latency focuses on the response speed of a single request, divided into first-token latency and per-token generation interval; a larger batch can improve throughput but may raise the latency of a single request.
The research presents four typical trade-offs between privacy and performance: latency vs privacy, model accuracy vs differential privacy, scalability vs federated learning, and transparency vs proprietary SDKs.
Latency is one of the most sensitive probes in distributed systems, because connection pool exhaustion, downstream dependency jitter, and cold starts all leave traces in P99 latency first, often before the error rate spikes.
A good P50 median latency does not mean that P99 tail latency is acceptable; for real-time decision scenarios, tail latency is often the true bottleneck.
Pursuing extreme throughput often comes at the cost of per-request latency, as throughput and latency metrics constrain each other.
Pursuing extremely low latency means decreased GPU utilization and rising per-inference costs.
In chip architecture design, throughput and latency are typically a pair of metrics that require trade-offs.
Increasing batch size to improve throughput typically sacrifices per-request latency
Under a linear conversation structure, token consumption is cumulative—every request requires retransmitting the entire history, increasing inference latency and driving up API call costs.
All Facts (10)
Increasing batch size to improve throughput typically sacrifices per-request latency
65%UnverifiedWhen evaluating an inference engine, throughput and latency must be distinguished: throughput usually refers to the total number of tokens generated by the whole system per unit of time, while latency focuses on the response speed of a single request, divided into first-token latency and per-token generation interval; a larger batch can improve throughput but may raise the latency of a single request.
50%UnverifiedThe research presents four typical trade-offs between privacy and performance: latency vs privacy, model accuracy vs differential privacy, scalability vs federated learning, and transparency vs proprietary SDKs.
50%UnverifiedLatency is one of the most sensitive probes in distributed systems, because connection pool exhaustion, downstream dependency jitter, and cold starts all leave traces in P99 latency first, often before the error rate spikes.
50%UnverifiedA good P50 median latency does not mean that P99 tail latency is acceptable; for real-time decision scenarios, tail latency is often the true bottleneck.
50%UnverifiedPursuing extreme throughput often comes at the cost of per-request latency, as throughput and latency metrics constrain each other.
50%UnverifiedPursuing extremely low latency means decreased GPU utilization and rising per-inference costs.
50%UnverifiedIn chip architecture design, throughput and latency are typically a pair of metrics that require trade-offs.
50%UnverifiedUnder a linear conversation structure, token consumption is cumulative—every request requires retransmitting the entire history, increasing inference latency and driving up API call costs.
50%Unverified路由决策若用小模型可能判断不准，若用大模型则路由开销会抵消节省，需要在准确性、延迟、成本之间权衡
50%