Draft Model Selection

Draft model selection is the process of choosing the proposer model used in speculative decoding to maximize acceptance rate and net speedup under quality constraints - selection quality determines whether speculative decoding delivers real benefit.

What Is Draft model selection?

• Definition: Model-pairing decision that balances draft speed against proposal accuracy.
• Selection Criteria: Includes draft latency, token agreement with target model, and serving cost.
• Compatibility Need: Tokenizer and vocabulary alignment are required for stable verification.
• Operational Role: Affects acceptance distribution, rejection overhead, and final throughput.

Why Draft model selection Matters

• Speedup Realization: Poor draft choices can erase expected speculative decoding gains.
• Cost Tradeoff: Draft inference must be cheap enough relative to target-model savings.
• Quality Stability: Misaligned draft behavior increases rejection churn and latency variance.
• Workload Fit: Different domains and output styles may favor different draft models.
• Platform Efficiency: Optimal pairing improves end-to-end tokens-per-second and SLA outcomes.

How It Is Used in Practice

• Candidate Benchmarking: Evaluate multiple draft models on acceptance and speed across traffic samples.
• Adaptive Routing: Use different draft models for distinct task classes when beneficial.
• Continuous Reassessment: Revalidate pairings after target-model or prompt-distribution changes.

Draft model selection is a key optimization step in speculative serving design - careful proposer selection is required to convert theory-level speedups into production gains.