Grouped Query Attention

An attention mechanism positioned between multi-head attention (MHA) and multi-query attention (MQA). It groups query heads that share KV heads, reducing KV cache size and computation while maintaining model quality close to that of MHA.