The KV Cache: How Language Models Remember What They Have Already Read

Every time an LLM generates a token, it needs to attend to everything it has already read. Without caching, that means recomputing the same keys and values for every prior token on every new generation step. A model generating a 2,000-token response would recompute the attention scores for the first token roughly 2,000 times. The key-value cache, almost universally called the KV cache, eliminates that redundancy. It is one of the most consequential engineering decisions in modern LLM inference, and understanding it explains a cluster of otherwise mysterious facts about model speed, cost, context limits, and the architectural choices in every major LLM released since 2023.

What the KV cache stores and why

In a transformer's self-attention mechanism, each input token produces three vectors: a query (Q), a key (K), and a value (V). The attention output for a given query is computed by comparing that query against all keys in the sequence, turning the comparison scores into weights via softmax, then taking a weighted sum of the corresponding values. The keys and values for any token that has already been processed are deterministic given the model weights and the token embedding. They will never change across generation steps. The KV cache simply stores them the first time they are computed and reads them from memory on every subsequent step, replacing recomputation with a cache lookup.

Without the cache, generating an N-token response requires O(N squared) total attention computations, because each of the N generation steps recomputes attention over all prior tokens. With the cache, the cost is O(N): each step computes keys and values only for the one new token, appends them to the cache, and reads all previously stored entries. The trade is memory for compute, and for any sequence longer than a handful of tokens the trade is overwhelmingly worth it.

How large the cache gets

The KV cache grows linearly with sequence length and scales with three model hyperparameters: the number of KV heads per layer, the head dimension, and the number of transformer layers. The size formula is: 2 (one tensor for keys, one for values) times the number of KV heads times the head dimension times the number of layers times the sequence length times the bytes per element.

For Llama 3 70B, which uses 8 KV heads, a head dimension of 128, and 80 layers, the cache occupies approximately 320 kilobytes per token in float16 precision. That sounds modest until you multiply by a long context: at the 128,000-token limit, the KV cache alone consumes around 40 gigabytes, roughly half the VRAM on a pair of H100 80 GB GPUs. Serving many concurrent users multiplies that by the batch size. The cache, not the model weights, is often the binding memory constraint in production inference systems.

Multi-query attention and grouped-query attention

The KV cache bottleneck motivated two related architectural changes. Multi-query attention (MQA) was proposed by Noam Shazeer of Google in a November 2019 paper (arXiv:1911.02150). Instead of one key head and one value head per query head, MQA uses a single shared key head and a single shared value head across all query heads in a layer. The KV cache shrinks by a factor equal to the number of query heads, with minimal accuracy loss on most tasks. The tradeoff is reduced model expressiveness: all query heads attend with the same keys and values, which limits the diversity of attention patterns each layer can represent.

Grouped-query attention (GQA), introduced by Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zeiler, Sumit Sanghai, and Yunhsuan Xu at Google Research in a paper submitted to arXiv in May 2023 (arXiv:2305.13245, accepted at EMNLP 2023), splits the query heads into groups where each group shares one KV head. With 32 query heads divided into 8 groups, each group of 4 query heads shares a single key and value projection. The KV cache is 8 times smaller than standard multi-head attention, versus 32 times smaller for full MQA. Ainslie et al. showed that GQA matches full multi-head attention quality far more closely than MQA does, making it the practical sweet spot.

• Llama 2 70B: 64 query heads, 8 KV heads (GQA with 8 groups), KV cache 8x smaller than standard MHA.
• Llama 3 8B: 32 query heads, 8 KV heads. Llama 3 70B: 64 query heads, 8 KV heads.
• Mistral 7B: 32 query heads, 8 KV heads, one of the first widely used open models with GQA.
• Standard multi-head attention (MHA): one KV head per query head, largest cache, used in older models like GPT-2 and the original Llama 1.

PagedAttention and the fragmentation problem

Even with GQA shrinking the cache, production serving systems faced a second problem: memory fragmentation. Before vLLM, most inference systems pre-allocated a contiguous block of GPU memory for each request's KV cache, sized to the maximum possible sequence length. Because requests finish at different times and have different actual lengths, large gaps of reserved but unused memory accumulate. The vLLM team measured that this fragmentation wasted 60 to 80 percent of GPU memory in existing systems, severely limiting the number of concurrent requests that could be served.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica at UC Berkeley addressed this with PagedAttention, submitted to arXiv on September 12, 2023 (arXiv:2309.06180) and presented at SOSP 2023, the top systems research venue. Inspired by virtual memory and paging in operating systems, PagedAttention stores KV cache entries in fixed-size non-contiguous blocks rather than one large contiguous buffer. A block table maps logical token positions to physical blocks. The approach achieves near-zero wasted memory, allows blocks to be shared across requests with common prefixes, and enables flexible memory allocation as sequences grow. The system built on top, vLLM, improved throughput by 2 to 4 times compared to prior state-of-the-art systems including FasterTransformer and Orca, with the gains increasing at longer sequence lengths.

Prefix caching across requests

PagedAttention's block sharing unlocks a further optimization: prefix caching. Many API requests from the same application share an identical system prompt. Without caching, every request pays the full prefill cost to process that prompt from scratch. With prefix caching, the KV cache blocks for the shared prefix are computed once and reused across all requests that start with the same tokens. For a long system prompt of several thousand tokens, this can cut prefill latency and cost by a large fraction on the first cache hit.

Anthropic, Google, and OpenAI all offer prompt caching as a paid feature, with cached input tokens billed at a reduced rate. Anthropic introduced prompt caching for Claude in August 2024, billing cached tokens at roughly one tenth the price of uncached input tokens and requiring a minimum cache block of 1,024 tokens. The underlying mechanism in all implementations is a form of KV cache sharing across requests, reusing previously computed blocks when the prefix matches.

What this means for inference costs and context design

The KV cache reframes how inference costs scale with context. The prefill phase (processing the input prompt) is compute-bound: the GPU must compute keys and values for all input tokens in parallel and store them. The decode phase (generating each new token) is memory-bandwidth-bound: the GPU reads the full KV cache for every generation step, so latency scales with cache size, not compute. A model generating at 128K context reads tens of gigabytes from memory on each token step, which is why longer contexts produce lower throughput even when the model weights have not changed.

For anyone designing applications on top of large context models, a few consequences follow directly. Long system prompts cost more than their token count alone suggests: the KV cache they occupy competes with batch capacity and increases decode latency for the entire sequence. Prompt caching returns the most value on fixed, frequently repeated content like system prompts and few-shot examples. Quantizing the KV cache to int8 or int4 precision, separate from quantizing model weights, can halve or quarter the memory footprint with typically small quality impact, and all major inference frameworks now support it. And the architectural choice of GQA versus standard MHA, made during model training, determines the cache size at every context length and every batch size for the model's entire serving lifetime. PromptingIndex covers model configurations, inference setups, and prompt structures tested across Claude, ChatGPT, and Gemini, so you can see how these system-level tradeoffs affect what arrives in the output.