Unverified50% confidenceFactExact time
Long context significantly raises time to first token, because prefill computation grows with input length; hitting the cache can skip most of the prefill and directly improve first-token latency.
1
Sources
50%
Confidence
Long-term
Relevance
10/13/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedHigh throughput is achieved through batching, which increases the total number of tokens processed per unit of time but raises time-to-first-byte latency; low latency requires small batches or no batching, which reduces hardware utilization and raises the cost per token.80% similarUnverifiedIf the speedup mainly comes from reducing the number of visual tokens, the improvement in prefill latency is more significant, while the impact on decoding speed for long text generation is limited.75% similarUnverified追求高吞吐与追求低延迟是两个相互制衡的优化方向，扩大批处理规模可提升吞吐但拉长单请求等待时间74% similarUnverifiedCompared to native framework inference, TensorRT typically delivers a 2–5× throughput improvement and significant latency reduction.74% similarUnverifiedTensor parallelism can effectively reduce single-inference latency and is suitable for latency-sensitive scenarios, while pipeline parallelism is better suited for improving throughput.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1019571API
curl https://kongchang.com/api/v1/knowledge/claims/1019571MCP
get_claim(id=1019571)