Speculative Decoding
Speculative decoding is an inference-acceleration technique that produces several tokens per expensive forward pass of a large language model without changing its output distribution. A small, fast draft model proposes a short run of future tokens; the large target model then verifies all of them in a single parallel pass, accepts the longest prefix consistent with its own probabilities, and corrects the first token that disagrees. The result is the same text the target would have generated alone, produced in fewer of its costly passes.
Autoregressive decoding is memory-bound, so a pass has spare compute. Generating one token normally requires one full forward pass of the target, and because that pass is dominated by streaming the model's weights and KV cache from memory, the GPU's arithmetic units sit largely idle. Verifying K candidate tokens costs almost the same as generating one, since the extra tokens ride along in the same weight load. Speculative decoding exploits exactly this slack: it fills the underused compute of a single pass with the work of checking several guesses.
A draft proposes, the target verifies, and a sampling rule keeps it exact. The draft model (a smaller model, or the target with a cheaper head) autoregressively emits K tokens. The target scores all K in one batched pass and applies a rejection-sampling test: each drafted token is accepted with a probability that makes the accepted stream identical in distribution to pure target sampling. The first rejected token is resampled from a corrected distribution, and everything after it is discarded. Quality is provably unchanged — this is a pure speedup, not an approximation.
| Standard decoding | Speculative decoding | |
|---|---|---|
| Target passes | one per token | one per K-token block |
| Tokens per pass | 1 | 1 to K+1 (accepted+1) |
| Extra model | none | small draft model |
| Bottleneck used | memory bandwidth | reuses the same pass |
| Output quality | baseline | identical distribution |
| Speedup driver | — | draft acceptance rate |
Speedup tracks the acceptance rate, and variants remove the separate draft. If the draft agrees with the target a fraction of the time, the expected tokens per pass grow with that acceptance rate and the drafted length K, commonly giving two-to-three times faster generation. The catch is that a bad draft wastes passes, so the draft must be cheap yet well-aligned with the target. Self-speculative methods like Medusa and EAGLE attach extra prediction heads to the target itself, and lookahead decoding drafts from n-gram guesses — all avoiding a second model while keeping the verify-in-parallel core.
Read speculative decoding through a quant lens rather than a 'guess ahead' lens: it trades cheap draft compute for fewer memory-bound target passes, and the payoff is governed by one number — the acceptance rate times the draft length, minus the draft's own cost. The design question is matching a draft that is fast enough to be nearly free against one accurate enough to be accepted often, since the technique only wins while the tokens saved per target pass outrun the draft overhead and the wasted work of rejected guesses.