MQA
An attention mechanism optimization technique in which all query heads share a single set of key-value heads. Compared to Grouped Query Attention (GQA), MQA has smaller KV cache overhead but incurs relatively greater model quality loss, making it a more extreme variant of GQA.
Core Facts
Timeline (last 90 days)
The compressed KV sharing strategy is in the same spirit as Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), but with a coarser sharing granularity.
In MQA (Multi-Query Attention), all query heads share a single set of KVs, resulting in a smaller cache but with relatively greater quality degradation.
GQA can reduce KV Cache memory during inference to 1/4 of MHA while retaining stronger modeling capability than MQA
MQA has all Query heads share the same set of KV, while GQA groups several Query heads together to share one set of KV, serving as a compromise between MHA and MQA.
Flash models typically use GQA or MQA technology to compress the KV Cache by 4–8×, thereby supporting higher concurrency on the same hardware.
GQA (Grouped Query Attention) is an attention mechanism between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), which reduces KV Cache memory usage by grouping multiple Query heads to share Key-Value projections.
GQA 通过让多个 Query 头共享同一组 Key-Value 头，在保持模型表达力的同时降低了 KV Cache 的显存占用
Flash Attention reduces the number of memory accesses through block-wise computation, while MQA and GQA reduce computational overhead by sharing key-value heads.
All Facts (10)
GQA (Grouped-Query Attention) and MQA (Multi-Query Attention) were the industry's solutions prior to MLA for alleviating the KV cache problem by sharing Key/Value heads.
75%UnverifiedFlash系列模型通常采用更小的参数规模、多查询注意力(MQA)或分组查询注意力(GQA)等高效注意力机制，以及模型蒸馏技术
70%UnverifiedGQA (Grouped Query Attention) is an attention mechanism between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), which reduces KV Cache memory usage by grouping multiple Query heads to share Key-Value projections.
60%UnverifiedFlash Attention reduces the number of memory accesses through block-wise computation, while MQA and GQA reduce computational overhead by sharing key-value heads.
60%UnverifiedThe compressed KV sharing strategy is in the same spirit as Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), but with a coarser sharing granularity.
50%UnverifiedIn MQA (Multi-Query Attention), all query heads share a single set of KVs, resulting in a smaller cache but with relatively greater quality degradation.
50%UnverifiedGQA can reduce KV Cache memory during inference to 1/4 of MHA while retaining stronger modeling capability than MQA
50%UnverifiedMQA has all Query heads share the same set of KV, while GQA groups several Query heads together to share one set of KV, serving as a compromise between MHA and MQA.
50%UnverifiedFlash models typically use GQA or MQA technology to compress the KV Cache by 4–8×, thereby supporting higher concurrency on the same hardware.
50%UnverifiedGQA 通过让多个 Query 头共享同一组 Key-Value 头，在保持模型表达力的同时降低了 KV Cache 的显存占用
50%