Speculative Decoding
Speculative Decoding is the inference acceleration technique that uses a smaller, faster draft model to propose multiple tokens in parallel, which the larger target model then verifies in a single forward pass — exploiting the fact that verification of N tokens (one forward pass through the target) is much cheaper than generating N tokens autoregressively (N forward passes), achieving 2-3× speedup with mathematically guaranteed identical output distribution to the original model, making it one of the few "free lunch" optimizations for LLM inference.
The Autoregressive Bottleneck
Standard autoregression (100 tokens):
Token 1 → [Full model forward pass] → Token 2 → [Full model forward pass] → ...
100 sequential forward passes, each memory-bandwidth-bound
Time: 100 × latency_per_token
Speculative decoding (100 tokens):
Draft model proposes K tokens in parallel
Target model verifies K tokens in one forward pass
Accept all correct tokens, regenerate from first wrong one
Time: ~(100/K) × latency_per_token (if acceptance rate is high)How It Works
1. Draft model generates K candidate tokens:
[The] → draft → [quick] [brown] [fox] [jumped] [over]
2. Target model scores ALL candidates in one forward pass:
P_target(quick|The) = 0.85 (draft said 0.80) → Accept
P_target(brown|The quick) = 0.90 (draft said 0.88) → Accept
P_target(fox|...brown) = 0.75 (draft said 0.70) → Accept
P_target(jumped|...fox) = 0.30 (draft said 0.60) → Reject!
3. Accept first 3 tokens, resample token 4 from adjusted distribution
Output: [The] [quick] [brown] [fox] [leaped]
Net gain: 3 tokens verified in 1 target pass instead of 3 passesMathematical Guarantee
- Acceptance criterion uses modified rejection sampling.
- If P_draft(x) ≤ P_target(x): Always accept.
- If P_draft(x) > P_target(x): Accept with probability P_target(x)/P_draft(x).
- On rejection: Sample from residual distribution (P_target - P_draft).
- Theorem: Output distribution is exactly P_target regardless of draft model quality.
Draft Model Strategies
| Strategy | Draft Model | Overhead | Acceptance Rate |
|---|---|---|---|
| Smaller same-family | Llama-3-8B drafts for Llama-3-70B | Low | 70-85% |
| Quantized self | INT4 version of target | Minimal | 75-90% |
| Early exit | First N layers of target | Minimal | 60-80% |
| Medusa heads | MLP heads on target model | Very low | 60-75% |
| Eagle | Feature-level autoregressive draft | Low | 75-85% |
| N-gram / retrieval | Statistical lookup | Near zero | 40-60% |
Performance Results
| Setup | Speedup | Use Case |
|---|---|---|
| 7B drafts for 70B | 2.0-2.5× | General text generation |
| Medusa heads | 2.0-2.8× | No separate draft model needed |
| Eagle-2 | 2.5-3.5× | Best draft architecture |
| Self-speculative (early exit) | 1.5-2.0× | Simplest to deploy |
When Speculative Decoding Helps Most
- Batch size 1 (interactive): Maximum benefit (memory-bandwidth bound).
- Code generation: High acceptance rate (code is predictable).
- Translation: Draft model easily approximates structure.
- Large batch: Less benefit (compute-bound, not bandwidth-bound).
Speculative decoding is the most important inference optimization for interactive LLM serving — by turning the sequential token-generation bottleneck into a parallel verify-and-accept loop, speculative decoding delivers 2-3× latency reduction with zero quality degradation, making it essential infrastructure for real-time AI applications from chatbots to code assistants, where every millisecond of response time directly impacts user experience.