Verified80% confidenceFactExact time
Grouped Query Attention (GQA) and Multi-Query Attention (MQA) reduce KV cache memory usage during inference by having multiple query heads share the same group of key/value heads, and are standard optimizations in models such as Llama 3 and Mistral.
10
Sources
80%
Confidence
Long-term
Relevance
7/16/2026
First Seen
Sources
Related Claims
Unverified自注意力通过将输入序列映射为查询Q、键K、值V三组向量，计算Q与K的点积相似度后经Softmax归一化得到注意力权重，再加权汇聚V向量79% similarUnverifiedGQA reduces the size of the KV Cache by having multiple Query heads share the same set of Key-Value heads.77% similarVerifiedGQA (Grouped-Query Attention) and MQA (Multi-Query Attention) were the industry's solutions prior to MLA for alleviating the KV cache problem by sharing Key/Value heads.77% similarVerifiedThe self-attention mechanism maps each token in the input sequence to three vectors—Query, Key, and Value—and computes attention weights by taking the dot product of Query and Key followed by Softmax normalization.73% similarUnverifiedMQA (multi-query attention) has all heads share a single set of KVs; compared with GQA, GQA achieves a better balance between quality and efficiency.72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/532362API
curl https://kongchang.com/api/v1/knowledge/claims/532362MCP
get_claim(id=532362)