Grouped Query Attention: How GQA Cuts KV Cache Size Without Sacrificing Quality

Every token a language model generates requires reading back the key and value vectors for every previous token in the context. Those vectors are stored in a structure called the KV cache, and at inference time the KV cache is the dominant consumer of GPU memory after the model weights themselves. For a model with 32 attention heads, a context of 4,096 tokens, and a per-head dimension of 128, the KV cache holds roughly 67 million floating-point values per layer. Multiply across 32 layers and the numbers add up quickly. Longer contexts make the problem worse: a 128,000-token context with the same configuration holds more than two billion values. The KV cache does not just consume memory; it also creates a memory-bandwidth bottleneck during the decode phase, because every autoregressive step must read the entire cache before computing the next token. Grouped Query Attention (GQA), introduced in a paper from Google Research in 2023, addresses this bottleneck with a structural change to how attention heads are organized.

Multi-head attention and why KV heads are expensive

Standard multi-head attention (MHA), as specified in the 2017 Transformer paper, assigns each attention head its own query, key, and value projections. If a model has H attention heads, it maintains H key matrices and H value matrices in the KV cache. The full cache size scales linearly with H. Most of that capacity is consumed by the keys and values, not the queries, because queries are only needed for the current decoding step and do not need to be cached across time steps. The keys and values grow with each new token generated, and bandwidth constraints mean that loading H separate KV matrices from GPU memory to compute a single decoding step becomes a throughput bottleneck as context length grows.

Multi-query attention: one KV head for all queries

The predecessor to GQA is Multi-Query Attention (MQA), introduced by Noam Shazeer at Google in 2019 (arXiv:1911.02150). MQA uses a single key head and a single value head shared across all query heads. Because there is only one KV head regardless of how many query heads exist, the KV cache size shrinks to 1/H of its MHA equivalent. The 2019 paper reports that MQA produces 10 to 100 times smaller key-value storage and approximately 12 times faster decoder inference compared to MHA. The quality trade-off is real but acceptable for many tasks: tasks requiring fine-grained positional distinctions or complex cross-token reasoning show larger degradation than simpler generation tasks. MQA was adopted in several production models and showed that the quality gap from collapsing all KV heads into one was manageable in practice, but researchers wanted a more precise tool that could be positioned between the two extremes.

The GQA paper: authors and contribution

Grouped Query Attention was introduced in 'GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints' (arXiv:2305.13245), submitted May 22, 2023 and accepted at EMNLP 2023. The authors are Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai, all at Google Research. The paper makes two distinct contributions. First, it introduces GQA as an architecture that sits between MHA and MQA by partitioning query heads into G groups, where each group shares a single key head and a single value head. Second, it proposes an uptraining recipe that converts existing MHA model checkpoints into GQA or MQA models using approximately 5 percent of the original pre-training compute, avoiding the cost of training from scratch.

How GQA groups queries and shares KV heads

In GQA, the H query heads are divided into G groups of equal size (H/G query heads per group). Each group is assigned one key head and one value head. All query heads within a group attend using the same key and value projections. The KV cache size is now proportional to G rather than H. Setting G equal to 1 recovers MQA (one shared KV head). Setting G equal to H recovers standard MHA (each query head has its own KV head). Any integer G between 1 and H exclusive is a GQA configuration.

• With H=32 query heads and G=8 KV heads, each group of 4 query heads shares one KV pair. KV cache is 1/4 the size of MHA.
• With H=64 query heads and G=8 KV heads (as in Llama 2 70B), each group of 8 query heads shares one KV pair. KV cache is 1/8 the size of MHA.
• Memory bandwidth at decode time scales with G, not H, because the attention kernel needs to load only G key and value matrices from cache per layer per step.
• The query projection remains full-rank with H heads, so the model retains H independent attention patterns at the query side while reducing the storage cost of the key-value side.
• GQA quality is consistently higher than MQA at the same G=1 reduction factor, and approaches MHA quality as G increases, giving practitioners a tunable trade-off between memory and accuracy.

Uptraining: converting MHA checkpoints with 5 percent compute

The paper's second major contribution is practical: many organizations have already trained large MHA models and cannot afford to retrain from scratch. The authors propose a two-step uptraining procedure. First, the G KV heads are initialized by mean-pooling the H existing KV projection weight matrices into G groups. Rather than randomly initializing new KV heads (which would require extensive retraining to converge) or selecting one head per group (which discards information from the others), mean-pooling produces a starting point that is already close to the final converged solution. Second, the model is continued on pre-training data for approximately 5 percent of the original training token count. The paper shows that uptrained GQA models achieve quality close to the original MHA checkpoint while matching or approaching the inference speed of MQA. Uptrained MQA models also improve over random-initialization MQA, but the quality gap to MHA is wider for MQA than for GQA at any practical group count.

Adoption in Llama 2, Llama 3, and Mistral

GQA moved quickly from paper to production. Meta first adopted GQA for the Llama 2 family when it released those models in July 2023. The 70B variant of Llama 2 uses 64 query heads and 8 KV heads, for a group size of 8 query heads per KV pair. The smaller Llama 2 models (7B and 13B) use standard MHA. Meta retained GQA throughout the Llama 3 model family released in 2024, applying it at all scales rather than only the largest variant. Mistral AI used GQA in the Mistral 7B model released in September 2023, with 32 query heads and 8 KV heads. The adoption pattern reflects the core trade-off: GQA is most valuable at larger scales, where the KV cache represents a larger fraction of total inference memory, and where the bandwidth savings translate into measurable throughput gains on the same hardware. For serving providers that run thousands of concurrent decoding sessions, the reduction in KV cache memory directly translates into higher concurrency per GPU, which is the primary operational reason GQA became the default architectural choice in modern open-weight models.

PromptingIndex tracks GQA configurations across open-weight model families, including the specific head counts used by Llama 2, Llama 3, Mistral, Gemma, and Falcon. The uptraining recipe the paper describes also remains relevant as older MHA models are candidates for conversion without full retraining, and practitioners applying it can get meaningful inference speedups from existing checkpoint investments.