Unverified50% confidenceFactExact time
The autoregressive generation phase is a memory-bandwidth-sensitive workload: generating each token requires reading the model weights in full once (or at least reading the weights involved in the activation path), so the actual speed is approximately equal to the VRAM bandwidth divided by the weight volume that needs to be read per token.
1
Sources
50%
Confidence
Long-term
Relevance
10/17/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedIn large model API call scenarios, cache hit rate refers to the reuse ratio of the KV Cache (key-value cache), where the model can directly read the attention key-value pairs of already-processed tokens from the cache.76% similarUnverifiedMemory bandwidth determines generation speed, because autoregressive decoding moves all weights token by token, and the higher the bandwidth, the more tokens generated per second.76% similarUnverifiedMemory bandwidth, rather than compute (FLOPS), is the core bottleneck determining local LLM inference speed, because generating each token requires reading nearly all model weights from memory into the compute units75% similarUnverifiedEvery time a model generates a token, the entire model must be read completely from memory, yielding the speed upper bound: memory bandwidth ÷ model size.74% similarUnverified自回归生成中每生成一个token需读取全部权重一次，内存带宽而非算力峰值才是真正的性能瓶颈73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1051448API
curl https://kongchang.com/api/v1/knowledge/claims/1051448MCP
get_claim(id=1051448)