MQA and GQA Explained: How KV Cache Reduces LLM Inference Memory Usage

Large language models face unavoidable challenges during inference: GPU memory consumption and bandwidth bottlenecks. KV Cache was introduced to solve the problem of redundant computation — keys and values only need to be computed once and can be reused repeatedly. But under standard Multi-Head Attention (MHA), every attention head maintains its own independent copy of keys and values in the cache. As sequences grow longer, the cache balloons rapidly, exhausting GPU memory. Even before memory runs out, each decoding step must transfer this ever-growing cache, creating a serious bandwidth bottleneck.

This article — based on a breakdown from the Visual AI channel on Bilibili — outlines the core ideas, trade-offs, and benefits of two optimization approaches: MQA (Multi-Query Attention) and GQA (Grouped-Query Attention), and explains why they've become the default configuration in today's mainstream LLMs.

The Root Cause: Why Does KV Cache Balloon?

Take the first token (BOS) as an example. With 4 attention heads, it immediately produces 4 key-value pairs in GPU memory. The question follows naturally: do we really need four independent sets of keys and values? Or can each head maintain a different perspective through its own Query, while sharing a common memory of the past?

The key is distinguishing the lifecycle of Queries from that of Keys/Values. Queries don't accumulate — they're created, used for a single decoding step, and then discarded. Keys and Values are different: they reside in memory and grow with every new token. This is the fundamental reason KV Cache keeps expanding.

In a 4-token, MHA setup, the model stores 8 head-cache entries per token (4 Keys + 4 Values), totaling 32 entries across 4 tokens. Cache overhead scales linearly with the number of heads.

KV Cache is fundamentally a space-for-time trade-off. In autoregressive generation, the model predicts one new token at a time, but computing attention requires "seeing" all historical tokens. Without caching, every new token would require recomputing the Keys and Values for all prior tokens — a computation that grows quadratically with sequence length. KV Cache stores these intermediate results in GPU memory, reducing redundant computation to zero at the cost of memory usage that grows linearly with sequence length.

Specifically, for a model with hidden dimension d, h attention heads, and sequence length L, the MHA KV Cache size is approximately 2 × h × L × (d/h) = 2 × L × d. When d reaches into the thousands, sequence lengths hit tens of thousands, and hundreds of concurrent requests are running simultaneously, total memory demand easily exceeds the capacity of a single GPU. This is precisely the engineering problem that MQA and GQA aim to solve.

MQA: Sharing One Set of Keys and Values Across All Heads

Multi-Query Attention (MQA) was first proposed in the 2019 paper Fast Transformer Decoding. Its approach is straightforward: instead of storing four separate sets of keys and values, use a single shared Key and Value.

During inference, tokens still arrive one at a time. When BOS arrives, we generate 4 different Query vectors for the 4 heads — but on the key-value side, only a single shared entry is written. Each head computes a dot product between its own Query and the shared Key, applies softmax to get attention weights, then computes a weighted sum over the shared Value to produce its output. Every subsequent token repeats this process: distinct Queries, but reading from the same shared Key/Value.

No matter how many heads you add, only one entry is written to the cache per step. Cache growth depends only on sequence length, not on the number of heads. For the same 4 tokens, MQA stores just 4 entries instead of MHA's 16 — an overall KV Cache memory reduction of 75%.

This is not just a memory win. Each decoding step reads far less data from GPU memory, relieving the bandwidth bottleneck and improving inference speed. The paper's empirical results are striking: under MHA, each decoding step takes roughly 46ms; with MQA, it drops to just 3.8ms. Under Beam4 search the gap is even larger — from 203ms down to 32ms.

The implementation change in MQA is minimal but far-reaching. In standard MHA, the Q, K, and V projection matrices all have shape [d_model, h × d_head]. In MQA, the K and V projection matrices are compressed to [d_model, d_head] — retaining only single-head size — while the Query projection matrix remains unchanged at [d_model, h × d_head]. This means MQA's parameter count changes very little, but KV Cache storage drops from scaling with the number of heads to being constant.

It's worth noting that MQA wasn't "invented" out of nowhere in 2019 — it's more of a systematic validation of an existing intuition (do Keys and Values really need to be fully independent across heads?). Later research found that training directly with MQA from scratch performs better than fine-tuning a converted MHA model, which influenced the direction of subsequent work on "checkpoint conversion" discussed in the GQA paper.

The Cost: How Shared Keys and Values Affect Model Quality

With so much saved, the natural question is: how much does sharing keys and values across all heads actually hurt model quality?

The paper measures this using perplexity — roughly, how confident the model is when predicting the next token. On the Billion Word benchmark, standard MHA scores 29.9; MQA scores 30.2 — an increase of just 0.3. This is a real cost, but given that KV Cache is reduced to one-quarter of its original size, it's remarkably small.

The speed gains are massive, and the quality loss is minimal — but it is still a cost. This raises a further question: does MQA go too far? Is it really necessary to force every head to share the exact same notebook, with no exceptions?

GQA: Finding Balance Between the Two Extremes

Grouped-Query Attention (GQA) was proposed by Google in a 2023 paper. Its idea is a compromise: rather than forcing all four heads to share one notebook, hand out two — two heads share one set of key-values, and the other two share another. It recovers much of the quality that MQA sacrifices, while preserving nearly all of the memory savings.

GQA gives us a dial: the number of groups, G. Turn G up, and groups become smaller — each head has more exclusive access to its own notebook, sliding toward full MHA. Turn G down, and groups become larger — more heads share fewer notebooks, sliding toward MQA. We don't have to choose between two extremes; we can precisely decide how much quality we're willing to trade for speed.

Mathematically, GQA is almost identical to standard MHA, with one small change: for any query head i, its keys and values come from the group G(i) it belongs to. When G equals 1, all heads share a single group — degenerating to MQA. When G equals the number of heads h, each head forms its own group — giving standard MHA. A single parameter lets you slide freely between maximum quality and maximum speed.

Beyond introducing the grouping mechanism itself, the GQA paper (Ainslie et al., 2023, Google Research) also solves a practical engineering problem: how to convert a checkpoint already trained with MHA into GQA, without retraining from scratch. The proposed method is to take the mean pooling of the Key/Value projection matrices of multiple MHA heads within the same group, then fine-tune on a small amount of data to recover quality. This "uptrain" strategy allows existing models to benefit from GQA's inference gains at relatively low cost — no need to pay the enormous price of retraining from scratch — which is one of the key reasons GQA was adopted so quickly in industry.

Real-World Performance and Industry Adoption

During generation, when BOS arrives, 4 independent Queries are still generated — but only 2 key-value pairs are projected (one per group). Each head reads from the notebook belonging to its group and performs the actual attention computation.

The impact on KV Cache is clear: for the same 4 tokens, MQA stores 4 entries, MHA stores 16, and GQA lands right in between at 8. Per token, MHA stores 8 head-cache entries, MQA drops to 2, and GQA sits in the middle at 4 (2 Keys + 2 Values). In this four-head example, GQA saves 50% memory compared to full MHA, while MQA saves 75%. As real model head counts grow larger, the savings only increase. GQA deliberately leaves some of the maximum savings on the table in order to buy back quality.

The paper doesn't just validate this on a four-head toy example — it uses a larger T5 model, divided into 8 groups rather than 2. The results are encouraging: MHA takes 1.51 seconds per step, MQA drops to 0.24 seconds, and GQA lands at 0.28 seconds — nearly as fast as MQA, while recovering almost all of MHA's quality.

This trade-off is good enough that within months it went from a research curiosity to an engineering standard. LLaMA adopted it, followed by Mistral and others. Today, GQA is closer to being the default choice than the exception.