Unverified50% confidenceFactExact time
The arithmetic intensity of the MLP layer is B×T, the arithmetic intensity of the attention layer is S×T/(S+T), and during generation the arithmetic intensity of the attention layer is approximately 1 and is completely independent of batch size.
1
Sources
50%
Confidence
Long-term
Relevance
9/23/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedFor a model with L layers, H attention heads, and per-head dimension d, caching n tokens requires VRAM of 2 × L × H × d × n × (precision bytes).71% similarUnverifiedThe computational complexity of standard self-attention is quadratic with respect to sequence length, and achieving a context of one million tokens requires introducing techniques such as sparse attention, sliding window attention, and KV Cache compression.69% similarVerified自注意力机制的计算复杂度随序列长度呈近似二次方增长，序列长度为n时注意力矩阵计算量正比于n²69% similarUnverified模型中注意力层的QKV投影权重和输出投影通常比MLP层的中间权重更敏感，模型首尾层比中间层更脆弱68% similarUnverified注意力头的维度（head_dim）通常保持在64到128之间以充分利用硬件矩阵乘法加速68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/942434API
curl https://kongchang.com/api/v1/knowledge/claims/942434MCP
get_claim(id=942434)