Speculative Decoding

Speculative Decoding is an inference acceleration technique where a small draft model rapidly generates multiple candidate tokens, which a large model verifies in batch — achieving 2-4x speedup for large language models without changing outputs through acceptance/rejection sampling.

Core Algorithm:
- Draft Model Generation: small, fast model (e.g., 1B parameters) predicts γ tokens ahead (γ=3-5 typical) in single forward pass — takes 10-20ms on A100
- Batch Verification: large model (e.g., 70B Llama) verifies all γ candidate tokens simultaneously in one forward pass — computes attention over draft sequence
- Token Acceptance: comparing large model logits P_large(x_i) with draft logits P_draft(x_i), accept token if P_large(x_i) > P_draft(x_i) with probability adjustment — maintains exact output distribution
- Rejection Sampling: if token rejected, resampling from adjusted distribution P_new(x) = max(0, P_large(x) - P_draft(x)) / (1 - P_draft(x)) — preserves correctness

Speedup Mechanism:
- Latency Reduction: expected speedup γ_accept = Σ[i=1 to γ] P(accept all i) where P(accept_i) ≈ 0.7-0.9 per token — typical speedup 2-3.5x
- Large Model Efficiency: amortizing one large model call across multiple tokens (similar to batch size γ) — reduces relative overhead of attention computation
- Draft Model Overhead: small model adds 5-10% latency (10-20ms) but saves 50-100ms from large model — net gain 40-90ms per iteration
- Cache Reuse: KV cache from large model verification enables streamlined next iteration — minimal redundant computation

Practical Implementation:
- Model Pairing: Llama 70B with Llama 7B draft model achieves 3x speedup with <0.1% accuracy change — commercial services deploy this pattern
- Medusa Framework: leveraging shared Llama backbone with lightweight head predictors (1.2% parameters) — achieves 2.3x speedup over naive decoding
- HuggingFace Integration: "Assisted Generation" API enabling drop-in replacement with any fine-tuned draft model — compatible with transformers library
- Threshold Tuning: adjusting acceptance threshold to balance speed (higher threshold = lower acceptance rate) — critical for different quality requirements

Advanced Strategies:
- Multi-Draft Ensemble: using 2-3 different draft models and averaging predictions before verification — improves acceptance rate to 0.92-0.95
- Adaptive Gamma: dynamically adjusting lookahead tokens γ based on recent acceptance rates (increase if >0.8, decrease if <0.6) — auto-tuning for optimal throughput
- Prefix Sharing: caching draft model outputs for common prefixes in batch inference — 30-40% reduction in draft model compute
- Tree Attention: organizing draft proposals in tree structure enabling parallel verification of competing branches — enables 4-6x speedup with multiple valid continuations

Speculative Decoding is transforming inference economics — enabling production deployment of 70B parameter models on limited hardware while maintaining output quality through verification.