Multi-Head Latent Attention: How DeepSeek-V2 Cut KV Cache by 93 Percent

Every autoregressive transformer stores key and value tensors for every token it has processed, layer by layer. As sequences grow, that KV cache balloons into the dominant memory cost at inference time, capping batch size and throttling throughput. The standard fixes, Multi-Query Attention (MQA, Shazeer 2019) and Grouped-Query Attention (GQA, Ainslie et al. 2023), shrink the cache by sharing key and value heads across query heads, but both methods cut into model quality. DeepSeek-V2 (arXiv:2405.04434, published May 2024 by DeepSeek-AI) introduces a different approach: Multi-head Latent Attention (MLA), which compresses the KV cache via low-rank joint projection rather than head sharing. The result is a 93.3% reduction in KV cache size relative to DeepSeek 67B and a 5.76x boost in maximum generation throughput, while the paper reports that MLA matches or outperforms standard Multi-Head Attention (MHA) on quality benchmarks. DeepSeek-V2 as a whole has 236B total parameters with 21B activated per token, supports a 128K-token context window, and was pretrained on 8.1T tokens.

The KV cache problem and where prior fixes fall short

In standard MHA, every token appends one key vector and one value vector per attention head per layer to the KV cache. With n_h heads, each of dimension d_h, the per-token cache cost per layer is 2 times n_h times d_h values stored in memory. At long contexts or large batch sizes this becomes prohibitive: the cache for a long-context request can match or exceed the parameter memory of the model itself. MQA addresses this by using a single shared key head and a single shared value head for all query heads, reducing per-token KV memory by a factor of n_h. GQA is a middle ground that groups query heads into g groups, each group sharing one key and one value head, reducing memory by a factor of n_h divided by g. The cost of both methods is quality regression. Experiments in the DeepSeek-V2 paper's ablation appendix (Section D) show that MLA outperforms MHA, MQA, and GQA under matched compute and training conditions on the paper's evaluation suite, meaning there is no quality trade-off for the compression.

How MLA compresses KV into a latent vector

MLA's core idea is low-rank joint compression of keys and values together, rather than head-count reduction. At each layer, instead of independently projecting the hidden state into n_h separate key vectors and n_h separate value vectors for caching, MLA first projects the hidden state into a single compressed latent vector c_KV of dimension d_c, where d_c is much smaller than the total key-plus-value dimension 2 times n_h times d_h. This compressed vector is the only thing stored in the KV cache per token per layer. At generation time, when attention needs to be computed, c_KV is up-projected back to the full key and value matrices via learned linear maps. The up-projection adds computation at inference, but the memory saving is large: the cache element per token is d_c values instead of 2 times n_h times d_h values. In DeepSeek-V2, this reduction in cache size is 93.3% compared to the equivalent dense model (DeepSeek 67B). Because all heads share a single c_KV rather than each head storing its own cache entry, no information is discarded by merging heads, as MQA and MQA do; instead, the compression is learned jointly across all head projections.

• Standard MHA cache per token per layer: 2 x n_h x d_h values (keys and values for every head stored separately).
• MQA cache per token per layer: 2 x d_h values (one shared key head and one shared value head).
• GQA cache per token per layer: 2 x g x d_h values where g is the number of groups, a tunable middle ground.
• MLA cache per token per layer: d_c values (single low-rank compressed latent vector, d_c much smaller than 2 x n_h x d_h).
• DeepSeek-V2 KV cache reduction vs. DeepSeek 67B: 93.3%, achieved entirely through the MLA compression.

Decoupled rotary position embedding

RoPE (Rotary Position Embedding, Su et al. 2021) encodes sequence position by rotating the query and key vectors by a position-dependent angle before computing attention scores. This interacts awkwardly with MLA: if RoPE is applied inside the compression matrix, the position encoding is fused into the compressed latent vector, and at cache-read time the up-projection must undo a rotation that depends on the query's position, not just the stored token's position. This makes caching the latent vector insufficient; the full rotated key must be stored or recomputed, eliminating the memory savings. MLA solves this with decoupled RoPE. Each attention head carries two components: a content key derived from c_KV (position-free, cacheable as part of the compressed latent), and a position key derived from a separate small projection of the hidden state with RoPE applied directly. Only the content keys flow through MLA's low-rank compression path. The position keys are kept in a small auxiliary cache of dimension d_c_prime per token, which is much smaller than the full key dimension. The attention score is then a sum of a content term and a position term. This decoupling lets RoPE work without corrupting the KV compression, preserving both long-context extrapolation and memory efficiency simultaneously.

DeepSeek-V2 model configuration and training cost

DeepSeek-V2 is a Mixture-of-Experts model combining MLA for attention with the DeepSeekMoE architecture for feed-forward layers. The total parameter count is 236B, but only 21B parameters are activated for each forward pass because the MoE routing selects a sparse subset of experts. The context window is 128K tokens, extended from a shorter base window using a long-context post-training phase. The pretraining corpus is 8.1T tokens, expanded and improved in quality compared to the 2T-token corpus used for DeepSeek 67B. For supervised fine-tuning, 1.5M conversational sessions were collected spanning math, code, writing, reasoning, and safety. Reinforcement learning used Group Relative Policy Optimization (GRPO, the same algorithm introduced in DeepSeekMath). On training cost, MLA and the sparse MoE together produce a 42.5% saving versus DeepSeek 67B for comparable pretraining, reflecting both fewer activated parameters per token and faster iteration from the reduced KV cache overhead during training runs.

Benchmark results and throughput numbers

On English open-ended evaluation, DeepSeek-V2 Chat (RL) achieves a 38.9 length-controlled win rate on AlpacaEval 2.0, an 8.97 overall score on MT-Bench, and a 7.91 score on AlignBench. On AlignBench (a Chinese instruction-following benchmark), the paper reports that DeepSeek-V2 Chat (RL) outperforms all open-source models and most closed-source models at evaluation time. On MMLU, DeepSeek-V2 reaches top-tier performance with only 21B activated parameters, which is substantially fewer activated parameters than competing open-source models of similar MMLU score. The throughput result is the most striking number from the paper: maximum generation throughput for DeepSeek-V2 is 5.76 times that of DeepSeek 67B. This gain comes from two compounding factors: the MLA KV cache reduction allows a much larger batch to fit in GPU memory, and the sparse MoE activation reduces the compute per token, both of which allow more tokens per second per GPU. The paper does not report wall-clock throughput in absolute tokens-per-second terms at a specific hardware configuration in the main text, but the 5.76x figure is relative to DeepSeek 67B on equivalent infrastructure.

• AlpacaEval 2.0 length-controlled win rate: 38.9 (DeepSeek-V2 Chat RL).
• MT-Bench overall score: 8.97 (DeepSeek-V2 Chat RL).
• AlignBench overall score: 7.91, outperforming all open-source and most closed-source models in Chinese.
• Maximum generation throughput: 5.76x versus DeepSeek 67B.
• KV cache size: 6.7% of DeepSeek 67B's KV cache (93.3% reduction).
• Training cost savings: 42.5% versus DeepSeek 67B for comparable pretraining.

What MLA means for the broader inference efficiency landscape

MLA is a third architectural path for KV cache compression, alongside the head-sharing approach (MQA and GQA) and the block-management approach (PagedAttention in vLLM). The three methods are orthogonal: a serving system can combine MLA for per-head compression, PagedAttention for block-level memory management, and prefix caching to reuse shared system-prompt KV entries. The DeepSeek-V2 paper's ablation in Section D shows MLA outperforming MHA on a held-out suite at the same training budget, suggesting the low-rank structure acts as a form of regularization on the key-value projections rather than merely a bottleneck. This result, if it holds broadly, challenges the common assumption that KV compression inherently trades quality for memory. The decoupled RoPE design also demonstrates a path for combining KV compression with position encodings that require per-position key transformation, resolving a technical obstacle that would otherwise prevent MLA from coexisting with RoPE-based long-context models. The technique has since influenced subsequent DeepSeek releases, including DeepSeek-V3 and the DeepSeek-R1 reasoning model, both of which carry forward the MLA architecture as a standard component. PromptingIndex covers MLA alongside GQA, KV cache mechanics, PagedAttention, and speculative decoding, each of which addresses a different layer of the LLM inference efficiency stack.