Speculative Decoding

Speculative decoding accelerates LLM inference by using a small draft model to rapidly propose multiple tokens, then having the larger target model verify them in a single forward pass, achieving 2-3× speedup while maintaining output quality. Traditional autoregressive: large model generates one token at a time; each token requires full forward pass; GPU often underutilized. Speculative approach: small draft model (2-4× smaller) generates k tokens quickly; target model processes all k tokens in one forward pass (verifies in parallel). Verification: target model computes probabilities for each position; accept tokens where draft matches or exceeds target quality; reject and resample from target otherwise. Acceptance rate: key efficiency metric; higher acceptance = fewer rejections = more speedup; depends on draft model quality. Speed math: if draft generates k tokens fast and acceptance rate is high, get (k × acceptance_rate) tokens per target model pass instead of 1. Draft model requirements: must be fast (smaller), must predict similar to target (same training data or distillation). Lossless property: carefully designed rejection sampling ensures output distribution equals target model exactly. Implementation: vLLM, TensorRT-LLM, and Hugging Face TGI support speculative decoding. Self-speculative: use draft heads on same model (Medusa-style) instead of separate model. Trade-off: need to host two models; memory overhead; most beneficial when target model is very large. Speculative decoding is standard optimization for production LLM serving.