Speculative Decoding: How a Small Draft Model Makes Large LLMs 2 to 3x Faster

Large language models are slow at inference for a surprising reason: they are bottlenecked by memory bandwidth, not compute. Each autoregressive decoding step loads billions of parameters from high-bandwidth memory (HBM) to generate a single token, leaving the GPU's arithmetic units mostly idle. In 2022, Google researchers (Leviathan et al., arXiv:2211.17192) published a technique called speculative decoding that exploits this imbalance. A small, fast draft model proposes multiple tokens at once, and the large target model verifies all of them in a single parallel forward pass. The result: 2 to 3x faster inference with mathematically identical output quality. Chen et al. (2023) independently proposed the same core idea, confirming its robustness. Since then, speculative decoding has evolved from a research insight into standard production infrastructure.

The Memory Wall Behind Slow LLM Inference

During standard autoregressive decoding, the model generates one token per step. Each step requires loading the full model weights from HBM into the GPU's compute units. For a 70B-parameter model in FP16, that means moving roughly 140 GB of data per token. Modern GPUs like the H100 can perform hundreds of teraflops of computation per second, but their memory bandwidth tops out around 3.35 TB/s. The arithmetic intensity of single-token generation is so low that the compute units sit idle while waiting for data. This is the memory wall: adding more FLOPs does not help because the bottleneck is data movement, not computation. The critical insight behind speculative decoding is that verifying D tokens in parallel costs nearly the same wall-clock time as generating one, because the memory bandwidth cost dominates and the weights are loaded just once.

How Speculative Decoding Works

The algorithm has three phases. First, a small draft model (typically 10 to 100x smaller than the target) autoregressively generates D candidate tokens. Because the draft model is small, this is fast. Second, the large target model processes the entire draft sequence in a single forward pass, producing probability distributions for each position. Third, a verification step walks through the draft tokens from left to right, accepting each token whose probability under the target model passes a modified rejection sampling criterion. The target model accepts tokens until the first mismatch, then resamples a corrected token at that position. This guarantees at least one new token per verification pass.

The mathematical guarantee is important: the output distribution is identical to what the target model would produce under standard autoregressive decoding. There is no approximation. The rejection sampling scheme adjusts acceptance probabilities so that the combined draft-then-verify process exactly reproduces the target distribution. This makes speculative decoding lossless, a property that separates it from quantization or pruning approaches that trade quality for speed.

Draft Mechanisms: From Separate Models to Baked-In Heads

The original speculative decoding paper used a separate, smaller transformer as the draft model. Since then, researchers have developed increasingly efficient draft mechanisms:

• EAGLE: attaches a lightweight draft head (1 to 2 transformer layers) directly to the target model, reusing its hidden states. EAGLE achieves 2 to 3x speedups and is deployed in production serving frameworks including vLLM, SGLang, and TensorRT-LLM.
• P-EAGLE (AWS, 2025): generates all K draft tokens in a single forward pass instead of autoregressively, achieving up to 1.69x additional speedup over EAGLE-3. Available in vLLM v0.16.0.
• Medusa: trains multiple parallel draft heads, one per future position, to predict several tokens simultaneously without sequential dependence.
• Multi-Token Prediction (MTP): integrates prediction heads directly into the model during pretraining, eliminating the need for a separate draft model entirely. The model itself learns to predict multiple future tokens as part of its training objective.

MTP represents a shift in philosophy. Rather than bolting on a draft mechanism after training, models like DeepSeek V4, GLM-5.1, and Qwen3-Next ship with native MTP heads baked in. DeepSeek V4 includes one MTP head (num_nextn_predict_layers: 1), and on H200 SXM5 hardware it reaches approximately 3,400 tokens per second. DeepSeek-R1, Nemotron-3, and MiMo-V2-Flash also include native MTP heads, making speculative decoding a first-class model capability rather than an inference-time add-on.

Acceptance Rate and Real-World Speedups

The effectiveness of speculative decoding depends on the acceptance rate (alpha): the fraction of draft tokens that the target model accepts. When alpha is high, most draft tokens survive verification and the effective throughput multiplier approaches the draft length. Google's original paper showed 2 to 3x speedups on translation and summarization tasks. The general rule is that an acceptance rate of alpha >= 0.6 with a draft length >= 5 yields 2 to 3x speedups in practice.

Task type matters significantly. Coding tasks tend to benefit more than creative writing because code has higher token-level predictability: variable names, syntax patterns, and boilerplate are easier for a draft model to anticipate correctly. Creative writing, with its higher entropy per token, sees lower acceptance rates and correspondingly smaller speedups. NVIDIA's SPEED-Bench (2025, arXiv:2604.09557) provides a standardized evaluation framework for measuring these differences across tasks and model pairs, helping practitioners predict the speedup they can expect for their specific workloads.

Production Deployment and the MTP Era

Speculative decoding has moved from research papers to production infrastructure. Major serving frameworks (vLLM, SGLang, TensorRT-LLM) support EAGLE and P-EAGLE out of the box. Cloud providers use speculative decoding to improve throughput on their inference endpoints. The technique composes well with other optimizations like continuous batching, paged attention, and quantization.

The trend toward models shipping with native MTP heads suggests that speculative decoding is becoming a standard part of model architecture, not just inference optimization. When the model itself is trained to predict multiple future tokens, the draft quality improves and the integration overhead disappears. For practitioners choosing between approaches: if your model has native MTP heads, enable them in your serving framework. If not, EAGLE or P-EAGLE provide the best combination of speedup and integration simplicity. Either way, speculative decoding is one of the highest-leverage inference optimizations available today, delivering multiplicative speedups with zero quality degradation.