Token Dropping in MoE
Token dropping in MoE is the overflow handling behavior where tokens exceeding expert capacity are skipped or routed through fallback paths - it protects system stability when router assignments temporarily exceed per-expert processing limits.
What Is Token dropping in MoE?
- Definition: Capacity-control mechanism that limits tokens processed by each expert per step.
- Trigger Condition: Occurs when router assignments exceed configured expert capacity factor.
- Fallback Modes: Dropped tokens may pass through residual paths, backup experts, or deferred handling logic.
- Systems Context: Relevant in top-k routing schemes where load spikes can be highly uneven.
Why Token dropping in MoE Matters
- Stability Protection: Prevents runtime failures from expert buffer overflow.
- Throughput Control: Keeps step latency predictable under routing imbalance.
- Quality Risk: Excessive drops can hurt model accuracy and gradient quality.
- Capacity Planning: Drop rate is a key signal for tuning expert count and routing policy.
- Operational Monitoring: Persistent dropping indicates load-balancing or architecture issues.
How It Is Used in Practice
- Metric Tracking: Monitor drop fraction by layer, expert, and training phase.
- Router Tuning: Adjust capacity factors, auxiliary losses, and routing temperature to reduce overflow.
- Fallback Design: Implement robust residual or backup routing to limit quality degradation.
Token dropping in MoE is an important safeguard but also a diagnostic signal - controlled low drop rates indicate healthy routing and efficient expert utilization.