Acceptance Rate

Acceptance rate is the fraction of draft-proposed tokens that pass target-model verification during speculative decoding - it is the primary efficiency metric for speculative inference.

What Is Acceptance rate?

• Definition: Ratio of accepted tokens to total proposed tokens over a decoding interval.
• Interpretation: Higher values indicate stronger draft-target alignment and better speed potential.
• Metric Scope: Can be measured globally, per model pair, or per traffic segment.
• Operational Link: Directly influences effective tokens generated per expensive target-model pass.

Why Acceptance rate Matters

• Performance Forecast: Acceptance rate predicts practical speculative speedup more accurately than raw draft speed alone.
• Model Pair Evaluation: Helps compare draft model candidates under real workload conditions.
• Tuning Feedback: Reveals whether proposal length or routing policies need adjustment.
• Cost Sensitivity: Low acceptance can increase overhead and negate expected savings.
• Stability Monitoring: Sudden drops can indicate distribution shift or prompt drift.

How It Is Used in Practice

• Segmented Dashboards: Track acceptance by endpoint, prompt type, and context length bands.
• Policy Adaptation: Dynamically shorten proposal windows when acceptance falls.
• Root-Cause Analysis: Inspect rejection hotspots to improve draft model or prompt normalization.

Acceptance rate is the key operational KPI for speculative decoding health - sustained high acceptance is required to realize stable inference acceleration benefits.