Speculative Decoding

Speculative Decoding is the inference acceleration technique that uses a small, fast draft model to generate multiple candidate tokens in parallel, which are then verified by the large target model in a single forward pass — achieving 2-3x speedup in autoregressive LLM inference without any change to the output distribution, because verification of K draft tokens costs approximately the same as generating one token from the large model.

The Autoregressive Bottleneck

Standard LLM inference generates one token at a time: each token requires a full forward pass through the model, and the next token depends on the previous one (sequential dependency). For a 70B parameter model, each forward pass takes ~30-50 ms on a single GPU, limiting throughput to ~20-30 tokens/second regardless of available compute — the process is memory-bandwidth bound, not compute bound.

How Speculative Decoding Works

1. Draft Phase: A small model (e.g., 1B parameters, 10x faster) generates K candidate tokens autoregressively: t₁, t₂, ..., tₖ.
2. Verification Phase: The large target model processes the original context plus all K draft tokens in a single forward pass (parallel evaluation, like processing a prompt). This produces the target model's probability distributions for each position.
3. Acceptance/Rejection: Starting from t₁, each draft token is accepted with probability min(1, p_target(tᵢ)/p_draft(tᵢ)). If a token is rejected, it is resampled from an adjusted distribution. All tokens after a rejection are discarded.
4. Guarantee: The acceptance-rejection scheme ensures the output distribution is mathematically identical to sampling directly from the target model — zero quality degradation.

Why It Works

LLM inference is memory-bandwidth bound: loading the model weights from GPU memory dominates the time, and the compute units are underutilized. Verifying K tokens requires loading the weights once (same as generating one token) but performs K times more useful compute. The speedup approaches K × acceptance_rate, where acceptance_rate depends on how well the draft model approximates the target.

Variants and Extensions

• Self-Speculative Decoding: The target model itself generates drafts using early exit (partial layers) or a smaller subset of its parameters, eliminating the need for a separate draft model.
• Medusa: Adds multiple prediction heads to the target model, each predicting tokens at different future positions. A tree-structured verification scheme evaluates multiple candidate sequences in a single forward pass.
• EAGLE: Uses a lightweight feature-level draft model that operates on the target model's hidden states rather than token embeddings, achieving higher acceptance rates.
• Lookahead Decoding: Generates N-gram candidates from Jacobi iteration trajectories without requiring a draft model at all.

Speculative Decoding is the key insight that LLM inference wastes most of its computational capacity generating one token at a time — and that parallel verification is essentially free, converting wasted compute into real throughput gains.