Speculative Decoding: How Draft Models Cut LLM Inference Latency Without Changing Outputs

Autoregressive decoding from a large language model is serial by construction. To generate K tokens, the model must execute K sequential forward passes, each producing one token whose value feeds the next step. The bottleneck is not arithmetic throughput. Modern GPUs have far more floating-point capacity than a single transformer forward pass needs. The bottleneck is memory bandwidth: each forward pass must read the entire set of model weights from GPU memory into compute units, and that transfer dominates latency at the token sizes where large models operate. Yaniv Leviathan, Matan Kalman, and Yossi Matias at Google Research identified this bandwidth constraint as an opportunity rather than a fixed cost. Their paper 'Fast Inference from Transformers via Speculative Decoding,' submitted to arXiv on November 30, 2022 (arXiv:2211.17192) and accepted as an Oral at ICML 2023, introduces an algorithm that generates multiple tokens per large-model forward pass without any change to the output distribution. Concurrently, Charlie Chen and colleagues at Google DeepMind independently developed the same idea under the name speculative sampling (arXiv:2302.01318, February 2023), demonstrating a 2 to 2.5x speedup on Chinchilla, a 70 billion parameter language model. Together the two papers established speculative decoding as the standard technique for latency reduction in production LLM inference.

The memory bandwidth bottleneck that makes draft-verify efficient

The reason speculative decoding works is a property of how transformer inference scales with sequence length under a fixed batch size. A single forward pass that scores a sequence of gamma plus one tokens costs only marginally more in wall time than a single forward pass that scores one token, because both reads must load the full model weight tensor from GPU memory regardless of input length. The compute for the attention and feedforward layers scales with input length, but on large models at inference batch sizes, the memory bandwidth cost of loading weights dominates and the compute overhead of extra tokens is comparatively small. This means that if a cheaper process can propose gamma draft tokens before the target model runs, and those proposals are plausible enough to be accepted most of the time, then the system can produce more than one verified token per target-model forward pass. Leviathan et al. describe the target model as 'often memory-bandwidth bound' and frame the additional compute from processing multiple draft tokens as unused capacity being reclaimed rather than a new expense being paid.

The algorithm: draft, score, accept, and correct

Speculative decoding works in three stages per iteration. First, a small and fast approximation model (the draft model) generates a sequence of gamma candidate tokens autoregressively. This draft generation is cheap because the draft model is far smaller than the target model and its forward passes are dominated by neither arithmetic cost nor the weight-loading penalty of the full model. Second, the target model runs one forward pass over the full context plus all gamma draft tokens in parallel, producing a probability distribution over the next token at each of the gamma positions simultaneously. Third, the algorithm applies a modified rejection sampling procedure to decide which draft tokens to accept. For each draft token in order, if the target model assigns at least as high a probability to that token as the draft model did, the token is accepted. If the target model assigns a lower probability, the token is rejected with probability proportional to the discrepancy, and the draft sequence is truncated at that position. After rejection, the target model's distribution at the rejection point is corrected so the residual distribution remains valid, and one token is sampled from the corrected distribution. The key guarantee, proved formally in both papers, is that the output of this process is drawn from exactly the same distribution as if the target model had run independently at each step. There is no approximation in the outputs: the correction step restores exactness. The papers demonstrate that the draft model can be any model with compatible vocabulary, including a much smaller version of the same architecture.

Results on T5-XXL and Chinchilla

Leviathan et al. evaluate speculative decoding on three settings: unconditional generation from a 97M parameter GPT-like model on lm1b, English-to-German translation and news summarization with T5-XXL at 11 billion parameters, and dialog with LaMDA at 137 billion parameters. For the T5-XXL experiments, they measure actual wall-clock latency against the standard T5X implementation and report a 2x to 3x speedup with identical outputs. Figure 1 in the paper shows a 38-token sequence generated using only 9 serial target-model calls rather than 38, using a 6 million parameter draft model alongside a 97 million parameter target. The Chen et al. paper reports a 2 to 2.5x decoding speedup on Chinchilla, a 70 billion parameter model, in a distributed multi-accelerator setup, with the modified rejection sampling preserving output distribution within hardware numerics. Both results represent speedup under realistic inference conditions, not synthetic benchmarks, and both confirm that the gains are reproducible across very different model sizes and task types.

• Leviathan et al. (arXiv:2211.17192, submitted Nov 30, 2022, ICML 2023 Oral): speculative decoding on T5-XXL achieves 2x to 3x wall-clock speedup over T5X with identical outputs.
• Chen et al. (arXiv:2302.01318, submitted Feb 2, 2023): speculative sampling on Chinchilla (70B) achieves 2 to 2.5x speedup in a distributed setup, provably preserving output distribution.
• The key insight: loading model weights from GPU memory dominates forward-pass latency; a longer input (gamma plus one tokens) costs little extra, so draft tokens are nearly free to verify.
• The rejection sampling correction guarantees the output distribution is mathematically identical to single-token greedy or temperature sampling from the target model alone.
• The draft model requires only a compatible vocabulary; it can be a smaller version of the same family, a specialized model, or any efficient approximator.
• Adopted in production by vLLM, TensorRT-LLM, llama.cpp, and the Hugging Face generate() API, where it is now a standard inference acceleration option.

Acceptance rate, gamma, and the expected tokens per call

The practical speedup from speculative decoding depends on two tunable factors. The first is gamma, the number of draft tokens proposed per iteration. Larger gamma means more potential tokens per target-model call but also more wasted draft computation when a rejection occurs early in the sequence. The second is the acceptance rate, the fraction of draft tokens the target model accepts on average. When the draft model produces tokens with high probability under the target model, the acceptance rate is high and the expected number of verified tokens per iteration approaches gamma plus one. When the draft model is a poor match, the acceptance rate is low and the algorithm degrades gracefully toward single-token generation, never performing worse than unaccelerated decoding. Both papers analyze this trade-off formally. The practical guidance that emerged from deployment experience is that draft models one to two orders of magnitude smaller than the target model, drawn from the same pretraining distribution, tend to achieve acceptance rates high enough to realize the 2x to 3x speedups the papers report, particularly on code generation and structured tasks where token probabilities are concentrated. Open-ended creative generation has more diverse next-token distributions and typically yields lower acceptance rates and more modest speedups.

From research to standard infrastructure

Speculative decoding moved from research papers to production infrastructure faster than most inference techniques. vLLM added speculative decoding support in 2023, allowing any draft model to be paired with any target model at the API level. NVIDIA TensorRT-LLM includes it as a first-class inference mode. llama.cpp implements it under the name speculative sampling with support for GGUF-format draft models. Hugging Face Transformers exposes it through the assistant_model parameter in generate(), requiring only a single extra argument to activate. The technique is especially impactful for latency-sensitive applications such as code completion, where the time between keystroke and suggestion directly affects user experience. GitHub Copilot and similar coding assistants run inference in conditions where a 2x to 3x latency reduction meaningfully changes perceived responsiveness. For batch throughput workloads, speculative decoding provides smaller benefits because the bottleneck shifts from memory bandwidth to compute, but for interactive single-stream generation it remains the most impactful drop-in latency optimization available without any change to the target model or its outputs. PromptingIndex covers speculative decoding alongside PagedAttention, GQA, FlashAttention, and continuous batching in its series on inference efficiency techniques that determine what LLM capabilities are deployable at practical cost and latency.