Speculative Decoding: How LLMs Generate 2x to 3x More Tokens Without Changing Their Weights

Autoregressive decoding forces a large language model to generate one token at a time. Every token requires a complete forward pass through all model weights, reading hundreds of gigabytes from memory for each word produced. For a 70-billion-parameter model on an A100 GPU, that means streaming roughly 140 GB of parameters from high-bandwidth memory for every single token. The model spends most of its time waiting for memory, not computing. Speculative decoding, published by Yaniv Leviathan, Matan Kalman, and Yossi Matias of Google Research in November 2022 (arXiv:2211.17192, presented at ICML 2023), attacked this bottleneck head-on, cutting latency by 2x to 3x with a provable mathematical guarantee that output quality is unchanged.

The memory bottleneck that throttles LLM inference

Modern GPUs and TPUs are built for arithmetic, not memory access. An A100 GPU delivers around 312 trillion floating-point operations per second in BF16, but its memory bandwidth is roughly 2 terabytes per second. A transformer's decoding step reads nearly all model weights once per token and performs only a handful of multiply-accumulate operations per byte read, a ratio far below what the hardware is capable of. The practical result: the GPU's compute units sit largely idle while waiting for data from memory. The original paper identified this as the core opportunity. Spare compute is available during every autoregressive step, and the question is how to use it productively.

Two observations that define the solution

The Google Research paper rests on two observations. First, not all tokens are equally difficult. Predicting the word 'of' after 'square root' is straightforward for a 68-million-parameter model. Predicting the exact floating-point answer '2.646' is not. Small models handle easy, predictable tokens reliably; large models are needed mainly for the hard cases where precision matters. Second, because decoding is memory-bound rather than compute-bound, the large model has spare arithmetic capacity during each autoregressive step. Both observations point toward the same architecture: use a small model to handle easy tokens cheaply, and use the large model's idle compute to verify those guesses in parallel.

How the algorithm works

The algorithm runs two models in concert. A small draft model, typically 5 to 100 times fewer parameters than the target, autoregressively generates a short sequence of candidate tokens, often 4 to 8 at once. The draft model is fast because its weight volume is small. The large target model then receives all candidate tokens simultaneously and evaluates them in a single forward pass, which is equivalent to a prefill step rather than a sequential decoding step. Each candidate token is accepted or rejected using a statistically correct rule derived from both models' output distributions. Crucially, the rejection sampling procedure guarantees that accepted tokens come from exactly the same distribution as if the target model had generated them one by one. The speedup comes entirely from replacing N sequential decoding steps with one batched verification pass whenever the draft model's guesses hold up.

Leviathan et al. also introduced speculative sampling, a generalization of speculative execution to stochastic settings. Unlike plain speculative execution, which discards any mismatched guess entirely, speculative sampling accepts or rejects guesses probabilistically based on how close the draft and target distributions are. This makes the method efficient even when the draft model is only approximately right, which is the common case in practice.

Verified speedups and real-world deployments

• In the original 2022 paper, using T5-small (60M parameters) as the draft for T5-XXL (11B parameters) on a translation task produced roughly 3x wall-clock speedup with no measurable quality degradation.
• The authors also demonstrated 2x to 3x improvements on summarization tasks across the same model pair.
• Google Search's AI Overviews feature uses speculative decoding in production, as confirmed in a December 2024 retrospective published by Google Research. The technique is described as remaining a significant part of the serving optimizations for that large-scale product.
• Medusa (arXiv:2401.10774, January 2024), developed by Tianle Cai, Tri Dao, and colleagues, replaces the separate draft model with additional decoding heads added directly to the target model. Medusa-1 achieves 2.2x or better speedup with no quality loss; Medusa-2 reaches 2.3x to 3.6x through joint fine-tuning.
• EAGLE-3 (arXiv:2503.01840, March 2025), from Peking University and Microsoft Research, achieves up to 6.5x speedup by predicting tokens directly rather than at the feature level and using multi-layer feature fusion, about 1.4x better than its predecessor EAGLE-2.

Where it helps and where it does not

Speculative decoding provides the most benefit in latency-sensitive settings where batch size is small, typically 1 to a few concurrent requests. This covers the common case for interactive chatbots, code completions, and agent-based workloads where a single user is waiting for a response. Under those conditions, the model is memory-bandwidth bound on a single stream, and the draft-verify approach can exploit idle compute to produce multiple tokens per verification cycle.

The gains shrink at high batch sizes. When many requests are batched together, the target model's compute units are already occupied evaluating many independent sequences. Adding a draft model creates overhead without recovering meaningful idle time. For throughput-focused inference servers processing large concurrent batches, speculative decoding may offer little benefit and can in some cases reduce effective throughput by consuming additional memory for the draft model's KV cache.

Token acceptance rate is the key variable. When the draft and target models have closely aligned output distributions, most guesses are accepted and multiple tokens are produced per verification step. A comprehensive benchmarking study from the University of Wisconsin-Madison (arXiv:2402.01528) found that the draft model's latency is the dominant bottleneck, and that a model's accuracy on standard language modeling benchmarks does not reliably predict its usefulness as a speculative draft model. Matching draft model architecture to the inference latency target matters as much as matching it to the model family.

What this means in practice

Speculative decoding is now supported natively in vLLM, Hugging Face Text Generation Inference, and NVIDIA TensorRT-LLM. Enabling it typically requires only selecting a draft model and setting a draft length parameter, with no changes to the target model, no retraining, and no impact on output quality. For interactive single-user workloads on large models, it is one of the highest-leverage inference optimizations available at the infrastructure layer. The technique's core insight, that a hard sequential problem can be parallelized by running a cheap approximation ahead and verifying in bulk, is now a foundational principle in LLM serving, extending from text to image and speech generation as well. PromptingIndex tracks prompts and model configurations tested across the major inference providers, including setups that pair well with speculative decoding for common coding and reasoning tasks.