GQA
An attention mechanism optimization technique in which multiple query heads share a single set of key and value heads. This approach significantly reduces the GPU memory footprint of the KV Cache while preserving model performance close to that of standard multi-head attention.
Core Facts
GQA (Grouped-Query Attention) and MQA (Multi-Query Attention) were the industry's solutions prior to MLA for alleviating the KV cache problem by sharing Key/Value heads.
GPT-OSS adopts a MoE (Mixture of Experts) architecture combined with Grouped Multi-Query Attention (GQA).
Timeline (last 90 days)
Llama 3.1 70B uses 8 KV heads instead of 64 attention heads, reducing the KV cache size to about one-eighth of the original.
On-device deployment often comes with mechanisms such as KV cache quantization (e.g., compressing to 4 or 8 bits), grouped-query attention (GQA), and sliding-window or sparse attention.
MQA (multi-query attention) has all heads share a single set of KVs; compared with GQA, GQA achieves a better balance between quality and efficiency.
RoPE, SwiGLU and GQA are structural components widely adopted by current mainstream large models (including the Qwen series).
LFM2.5-24B-A2B uses a 40-layer architecture: 30 double-gated short convolution blocks + 10 GQA attention layers, 64 experts per MoE block, top-4 routing, a 131,072 token context, and a 128K vocabulary.
The architecture of LFM2 (Liquid AI) is 10 gated short convolution blocks plus 6 GQA blocks, discovered through hardware-in-the-loop search rather than human design.
For a 70B model using GQA (n_layers=80, n_kv_heads=8, head_dim=128, FP16 cache), the KV Cache size per token is about 320 KB.
GQA compresses the KV Cache by about 8x relative to MHA and has become the standard choice for mainstream open-source models such as Llama 2/3, Qwen, and Mistral.
Methods for optimizing the KV cache include grouped-query attention (GQA), paged attention (such as vLLM's PagedAttention), and sliding window attention.
Grouped-query attention (GQA) was proposed by Ainslie et al. in 2023 and is a compromise between multi-head attention (MHA) and multi-query attention (MQA); Llama 3.1 8B uses 32 query heads divided into 8 groups, with each group of 4 query heads sharing 1 KV head, shrinking the KV cache by 4x compared to MHA.
34 more timeline events
All Facts (20)
GQA (Grouped-Query Attention) and MQA (Multi-Query Attention) were the industry's solutions prior to MLA for alleviating the KV cache problem by sharing Key/Value heads.
75%UnverifiedGrouped Query Attention (GQA) strikes a balance between inference efficiency and model quality.
70%UnverifiedFlash系列模型通常采用更小的参数规模、多查询注意力(MQA)或分组查询注意力(GQA)等高效注意力机制，以及模型蒸馏技术
70%Partially VerifiedGPT-OSS adopts a MoE (Mixture of Experts) architecture combined with Grouped Multi-Query Attention (GQA).
65%UnverifiedLFM2.5-24B-A2B uses a 40-layer architecture: 30 double-gated short convolution blocks + 10 GQA attention layers, 64 experts per MoE block, top-4 routing, a 131,072 token context, and a 128K vocabulary.
60%UnverifiedMainstream open-source models such as Llama 3 and Qwen2/3 have already adopted GQA.
60%UnverifiedGQA allows multiple query heads to share the same set of KV heads; LLaMA-3 70B uses 8 KV heads corresponding to 64 query heads, making the KV Cache only 1/8 that of standard MHA.
60%UnverifiedGQA (Grouped Query Attention) is an attention mechanism between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), which reduces KV Cache memory usage by grouping multiple Query heads to share Key-Value projections.
60%UnverifiedFlash Attention reduces the number of memory accesses through block-wise computation, while MQA and GQA reduce computational overhead by sharing key-value heads.
60%UnverifiedLlama 3 8B has 32 layers, 8 KV heads (GQA architecture with 32 query heads), a head dimension of 128, and a KV cache of approximately 1.07 GB for a single sequence under FP16 with an 8192 context length.
60%UnverifiedDeepSeek 8B adopts the GQA (Grouped Query Attention) attention mechanism to reduce VRAM usage during inference.
60%UnverifiedLlama 3.1 70B uses 8 KV heads instead of 64 attention heads, reducing the KV cache size to about one-eighth of the original.
50%UnverifiedOn-device deployment often comes with mechanisms such as KV cache quantization (e.g., compressing to 4 or 8 bits), grouped-query attention (GQA), and sliding-window or sparse attention.
50%UnverifiedMQA (multi-query attention) has all heads share a single set of KVs; compared with GQA, GQA achieves a better balance between quality and efficiency.
50%UnverifiedRoPE, SwiGLU and GQA are structural components widely adopted by current mainstream large models (including the Qwen series).
50%UnverifiedThe architecture of LFM2 (Liquid AI) is 10 gated short convolution blocks plus 6 GQA blocks, discovered through hardware-in-the-loop search rather than human design.
50%UnverifiedFor a 70B model using GQA (n_layers=80, n_kv_heads=8, head_dim=128, FP16 cache), the KV Cache size per token is about 320 KB.
50%UnverifiedGQA compresses the KV Cache by about 8x relative to MHA and has become the standard choice for mainstream open-source models such as Llama 2/3, Qwen, and Mistral.
50%UnverifiedMethods for optimizing the KV cache include grouped-query attention (GQA), paged attention (such as vLLM's PagedAttention), and sliding window attention.
50%UnverifiedGrouped-query attention (GQA) was proposed by Ainslie et al. in 2023 and is a compromise between multi-head attention (MHA) and multi-query attention (MQA); Llama 3.1 8B uses 32 query heads divided into 8 groups, with each group of 4 query heads sharing 1 KV head, shrinking the KV cache by 4x compared to MHA.
50%