Unverified50% confidenceFactExact time
Memory capacity determines whether the parameter count plus the KV Cache can fit into the device; memory bandwidth determines the speed of weight reads during each token generation in the decoding phase, and thus directly determines inference latency.
1
Sources
50%
Confidence
Long-term
Relevance
10/22/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedGeneration throughput (tok/s) reflects decoding-phase efficiency and is affected by KV Cache size and memory bandwidth, with far lower sensitivity to input length than prefilling.75% similarUnverifiedThe autoregressive generation process of large language models requires frequent reading of the KV Cache (key-value cache), placing extremely high demands on memory bandwidth and capacity.75% similarUnverified长上下文在首次推理时需完整计算KV Cache，消耗GPU计算资源并占用显存带宽，拉长首token延迟（TTFT）75% similarVerifiedDuring autoregressive decoding, each newly generated token requires reading the full KV Cache, making the decoding stage almost always memory-bandwidth-bound rather than compute-bound.75% similarUnverifiedIn large model API call scenarios, cache hit rate refers to the reuse ratio of the KV Cache (key-value cache), where the model can directly read the attention key-value pairs of already-processed tokens from the cache.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1086112API
curl https://kongchang.com/api/v1/knowledge/claims/1086112MCP
get_claim(id=1086112)