What Is Speculative Decoding
Speculative decoding is a way to generate multiple tokens per expensive decode step instead of just one, by letting a small, cheap "draft" model guess ahead and having the large model check its guesses in a single batched pass. Every earlier entry in this series established the same bottleneck: decode generates exactly one token at a time, and each of those steps is memory-bandwidth-bound because it has to re-read the model's weights and the growing KV cache from scratch. Speculative decoding doesn't shrink what has to be read — that's what quantization and grouped-query attention were for — instead it changes the math so one expensive read can verify several tokens' worth of work at once, which is a fundamentally different lever on the same throughput problem.
The trick works because verifying a guess is cheap relative to generating it, and checking many guesses at once is barely more expensive than checking one. A small draft model — trained on the same kind of data, often a distilled or scaled-down version of the target model — proposes a short run of several candidate next tokens, generated quickly because it's small. The large model then processes that entire candidate sequence in one parallel pass, exactly like the compute-heavy prefill phase covered earlier, checking whether each guessed token matches what the large model itself would have produced. Any tokens that match are accepted immediately at essentially the cost of one large-model pass; the moment a guess is wrong, everything after that point is discarded and the large model's own correct token is used instead, then drafting resumes from there.
The crucial guarantee is that speculative decoding never changes what the large model would have said — only how cheaply it gets said. Because every accepted token is exactly what the large model's own verification pass confirmed, and any wrong guess is replaced with the large model's actual correct token, the final output is identical to running the large model alone, token by token, the slow way. This is what separates speculative decoding from quantization or attention-sharing: those techniques trade a small amount of accuracy for memory savings, while speculative decoding trades none — it's a pure throughput optimization that works by restructuring how the expensive memory reads are scheduled, not by reading less or reading approximately.
| Approach | What It Reduces | Accuracy Cost | How It Helps Decode |
|---|---|---|---|
| Quantization | Bytes per weight/cache entry | Small, tunable | Less data moved per read |
| Grouped/Multi-Query Attention | Number of separate KV sets stored | Small, tunable | Smaller cache to read per step |
| Speculative Decoding | Number of large-model passes needed | None — output is exact | More tokens produced per expensive pass |
How much this actually speeds things up depends entirely on how often the draft model guesses right, which is why draft-model choice is its own engineering problem. Easy, predictable continuations — common phrases, boilerplate code, repeated structure — let the draft model guess long correct runs, multiplying decode throughput several times over; unpredictable or highly technical continuations cause frequent mismatches, shrinking the benefit toward zero extra cost versus running the large model alone. Because the technique adds compute (running a second small model) in exchange for saving memory-bandwidth-bound decode steps, it's a natural complement to everything else in this series: an accelerator with ample compute headroom but a memory-bandwidth ceiling — exactly the profile decode creates — is precisely the situation where trading a little extra compute for fewer expensive memory-bound steps pays off.