Speculative Decoding

Speculative decoding accelerates LLM inference by drafting multiple tokens then verifying in parallel. Mechanism: Small "draft" model generates k candidate tokens quickly, large "target" model verifies all k tokens in single forward pass, accept verified prefix and regenerate from first rejection. Why it works: Single forward pass through target model processes k tokens in roughly same time as 1 token (attention parallelizes). If draft accepts 70% of tokens on average, effective 2-3x speedup. Draft model requirements: Much smaller (10-100x fewer parameters), trained on similar data or distilled from target, fast enough that drafting overhead is minimal. Variants: Medusa adds multiple prediction heads to single model, self-speculative uses early exit layers, parallel decoding with candidates from different strategies. Implementation: Careful handling of probability distributions during verification, tree-structured speculation for multiple candidates. Limitations: Overhead if draft quality poor, memory for draft model, complex implementation. Best use cases: Latency-sensitive applications, when draft model available, sequences where patterns are predictable. Used in production by major LLM providers.