MHA
Multi-Head Attention, a core component of the Transformer architecture. It processes sequence information in parallel through multiple independent attention heads, each maintaining its own Key/Value projections. MHA represents the standard implementation of the attention mechanism in Transformer models.
Core Facts
Timeline (last 90 days)
The core difference among MHA, GQA, and MQA lies in how key-value heads are shared: MHA has an independent key-value head for each query head, GQA has a group of query heads sharing one set of key-value heads, and MQA has all query heads sharing one set of key-value heads.
If this 70B model used classic multi-head attention (MHA, 64 KV heads), the KV Cache under the same conditions would expand to 312.5 GiB.
Compared to MHA, GQA can compress the KV cache size to one-quarter without noticeably degrading model quality.
GQA can reduce KV Cache memory during inference to 1/4 of MHA while retaining stronger modeling capability than MQA
MQA has all Query heads share the same set of KV, while GQA groups several Query heads together to share one set of KV, serving as a compromise between MHA and MQA.
GQA (Grouped Query Attention) is an attention mechanism between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), which reduces KV Cache memory usage by grouping multiple Query heads to share Key-Value projections.
All Facts (6)
GQA (Grouped Query Attention) is an attention mechanism between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), which reduces KV Cache memory usage by grouping multiple Query heads to share Key-Value projections.
65%UnverifiedThe core difference among MHA, GQA, and MQA lies in how key-value heads are shared: MHA has an independent key-value head for each query head, GQA has a group of query heads sharing one set of key-value heads, and MQA has all query heads sharing one set of key-value heads.
50%UnverifiedIf this 70B model used classic multi-head attention (MHA, 64 KV heads), the KV Cache under the same conditions would expand to 312.5 GiB.
50%UnverifiedCompared to MHA, GQA can compress the KV cache size to one-quarter without noticeably degrading model quality.
50%UnverifiedGQA can reduce KV Cache memory during inference to 1/4 of MHA while retaining stronger modeling capability than MQA
50%UnverifiedMQA has all Query heads share the same set of KV, while GQA groups several Query heads together to share one set of KV, serving as a compromise between MHA and MQA.
50%