Unverified50% confidenceFactExact time
Grouped-query attention (GQA) was proposed by Ainslie et al. in 2023 and is a compromise between multi-head attention (MHA) and multi-query attention (MQA); Llama 3.1 8B uses 32 query heads divided into 8 groups, with each group of 4 query heads sharing 1 KV head, shrinking the KV cache by 4x compared to MHA.
1
Sources
50%
Confidence
Long-term
Relevance
10/12/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedKV Cache memory usage is proportional to the number of layers × number of attention heads × context length × precision; GQA reduces cache size by having multiple query heads share the same set of Key-Value heads, and this design has been adopted by the Llama 2/3 and Qwen series.71% similarUnverifiedGQA allows multiple query heads to share the same set of KV heads, compressing the KV cache size to 1/4 to 1/8 of the original.69% similarUnverifiedMiniMax's new architecture reduces prefill compute consumption by approximately 5x and shrinks the KV cache by approximately 4.5x under million-level context by sharing KV cache across layers and using an intelligent filtering attention mechanism.67% similarUnverified使用非GQA架构（KV头数等于查询头数）在相同条件下的KV缓存是GQA的4倍67% similarUnverifiedMLA（Multi-head Latent Attention）通过低秩压缩将 KV 缓存的显存占用降低至传统 MHA 的约 1/864% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1006016API
curl https://kongchang.com/api/v1/knowledge/claims/1006016MCP
get_claim(id=1006016)