Unverified50% confidenceFactExact time
The decode stage generates only one token at a time, and matrix multiplication degenerates into vector-by-matrix multiplication, which is a memory-bound operation whose performance is limited by memory bandwidth.
1
Sources
50%
Confidence
Long-term
Relevance
10/31/2026
First Seen
Sources
LLM Inference Bottlenecks and KV Cache: Core Mechanisms and Mainstream Optimization Approaches
bilibili先进编译实验室10/20/2026
Related Entities
Related Claims
UnverifiedMatrix multiplication is typically compute-bound in high-batch inference scenarios, while autoregressive decoding with a batch size of 1 is almost invariably bandwidth-bound.75% similarUnverifiedThe top 20 decoder layers do not read the prompt; their global key-values are projected from the encoder's final hidden states via per-layer independent weight matrices, reducing prefill complexity to nearly half (n×L/2).73% similarUnverifiedThe prefill stage is highly parallel with large throughput but latency that grows with context length, while the decode stage generates tokens auto-regressively and its speed is limited by memory bandwidth.73% similarUnverifiedTensor parallelism splits a single-layer weight matrix along a certain dimension, and requires one All-Reduce communication at the end of each forward pass step, making communication frequency extremely high.71% similarUnverifiedFlattening a matrix and feeding it to an MLP results in a non-fixed input dimension due to variable row counts, and breaks permutation invariance.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1121454API
curl https://kongchang.com/api/v1/knowledge/claims/1121454MCP
get_claim(id=1121454)