GQA
An attention mechanism optimization technique in which multiple query heads share a single set of key and value heads. This approach significantly reduces the GPU memory footprint of the KV Cache while preserving model performance close to that of standard multi-head attention.
Core Facts
GQA (Grouped-Query Attention) and MQA (Multi-Query Attention) were the industry's solutions prior to MLA for alleviating the KV cache problem by sharing Key/Value heads.
GPT-OSS adopts a MoE (Mixture of Experts) architecture combined with Grouped Multi-Query Attention (GQA).
Timeline (last 90 days)
Methods for optimizing the KV cache include grouped-query attention (GQA), paged attention (such as vLLM's PagedAttention), and sliding window attention.
Grouped-query attention (GQA) was proposed by Ainslie et al. in 2023 and is a compromise between multi-head attention (MHA) and multi-query attention (MQA); Llama 3.1 8B uses 32 query heads divided into 8 groups, with each group of 4 query heads sharing 1 KV head, shrinking the KV cache by 4x compared to MHA.
Different models differ in attention mechanisms, activation functions, and normalization methods (e.g., GQA, RoPE, SwiGLU), and each variant requires the runtime framework to provide corresponding operator implementations.
After adopting GQA, Llama 3.3 70B has a KV Cache of approximately 320 KB per token.
Llama 3 has broadly adopted GQA due to inference cost considerations.
The compressed KV sharing strategy is in the same spirit as Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), but with a coarser sharing granularity.
Agnes-3.0-Flash's global attention layers use 6:1 GQA with 24 query heads and 4 KV heads, a head dim of 256, RMS-norm applied to q and k, and sigmoid-gated output
GQA is the standard design in current mainstream open-source models (Llama, Mistral, Qwen, and other series).
On consumer-grade hardware, a small GQA model with a large context window is generally preferable to a larger model locked to a 4K context window.
Compared to MHA, GQA can compress the KV cache size to one-quarter without noticeably degrading model quality.
26 more timeline events
All Facts (20)
GQA (Grouped-Query Attention) and MQA (Multi-Query Attention) were the industry's solutions prior to MLA for alleviating the KV cache problem by sharing Key/Value heads.
75%UnverifiedGrouped Query Attention (GQA) strikes a balance between inference efficiency and model quality.
70%UnverifiedFlash系列模型通常采用更小的参数规模、多查询注意力(MQA)或分组查询注意力(GQA)等高效注意力机制，以及模型蒸馏技术
70%Partially VerifiedGPT-OSS adopts a MoE (Mixture of Experts) architecture combined with Grouped Multi-Query Attention (GQA).
65%UnverifiedMainstream open-source models such as Llama 3 and Qwen2/3 have already adopted GQA.
60%UnverifiedGQA allows multiple query heads to share the same set of KV heads; LLaMA-3 70B uses 8 KV heads corresponding to 64 query heads, making the KV Cache only 1/8 that of standard MHA.
60%UnverifiedGQA (Grouped Query Attention) is an attention mechanism between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), which reduces KV Cache memory usage by grouping multiple Query heads to share Key-Value projections.
60%UnverifiedFlash Attention reduces the number of memory accesses through block-wise computation, while MQA and GQA reduce computational overhead by sharing key-value heads.
60%UnverifiedLlama 3 8B has 32 layers, 8 KV heads (GQA architecture with 32 query heads), a head dimension of 128, and a KV cache of approximately 1.07 GB for a single sequence under FP16 with an 8192 context length.
60%UnverifiedDeepSeek 8B adopts the GQA (Grouped Query Attention) attention mechanism to reduce VRAM usage during inference.
60%UnverifiedMethods for optimizing the KV cache include grouped-query attention (GQA), paged attention (such as vLLM's PagedAttention), and sliding window attention.
50%UnverifiedGrouped-query attention (GQA) was proposed by Ainslie et al. in 2023 and is a compromise between multi-head attention (MHA) and multi-query attention (MQA); Llama 3.1 8B uses 32 query heads divided into 8 groups, with each group of 4 query heads sharing 1 KV head, shrinking the KV cache by 4x compared to MHA.
50%UnverifiedDifferent models differ in attention mechanisms, activation functions, and normalization methods (e.g., GQA, RoPE, SwiGLU), and each variant requires the runtime framework to provide corresponding operator implementations.
50%UnverifiedAfter adopting GQA, Llama 3.3 70B has a KV Cache of approximately 320 KB per token.
50%UnverifiedLlama 3 has broadly adopted GQA due to inference cost considerations.
50%UnverifiedThe compressed KV sharing strategy is in the same spirit as Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), but with a coarser sharing granularity.
50%UnverifiedAgnes-3.0-Flash's global attention layers use 6:1 GQA with 24 query heads and 4 KV heads, a head dim of 256, RMS-norm applied to q and k, and sigmoid-gated output
50%UnverifiedOn consumer-grade hardware, a small GQA model with a large context window is generally preferable to a larger model locked to a 4K context window.
50%UnverifiedCompared to MHA, GQA can compress the KV cache size to one-quarter without noticeably degrading model quality.
50%UnverifiedGQA is the standard design in current mainstream open-source models (Llama, Mistral, Qwen, and other series).
50%