Grouped-Query Attention
A variant of the attention mechanism that groups query heads and shares key/value heads among them, significantly reducing KV cache memory usage while maintaining model quality.
Core Facts
Timeline (last 90 days)
Modern large models (such as Qwen, Llama 3, Mistral, etc.) generally use grouped query attention (GQA), and the number of KV heads is usually far smaller than the number of query heads.
Muse Spark Open has a hidden dimension of 8192 and uses grouped query attention (GQA) throughout.
All Facts (3)
The key technical innovations of Mistral 7B include Sliding Window Attention (SWA) and Grouped Query Attention (GQA).
75%UnverifiedModern large models (such as Qwen, Llama 3, Mistral, etc.) generally use grouped query attention (GQA), and the number of KV heads is usually far smaller than the number of query heads.
50%UnverifiedMuse Spark Open has a hidden dimension of 8192 and uses grouped query attention (GQA) throughout.
50%