Load Balancing (MoE)

Load balancing in MoE ensures experts are used roughly equally, preventing underutilization and bottlenecks. The problem: Without balancing, router may send most tokens to few experts. Others underutilized, those overloaded become bottlenecks. Consequences of imbalance: Wasted parameters (unused experts), computation bottlenecks (overused experts), reduced effective capacity. Auxiliary loss: Add loss term penalizing imbalanced usage. Encourages router to spread tokens evenly. Loss proportional to variance of expert loads. Capacity factor: Set maximum tokens per expert (e.g., 1.25x fair share). Excess tokens dropped or rerouted. Expert choice routing: Let experts choose tokens rather than tokens choosing experts. Guarantees balance. Implementation challenges: Balance per-batch, per-sequence, or globally. Trade-offs with routing quality. Switch Transformer approach: Top-1 routing with capacity factor and aux loss. Current best practices: Combine auxiliary loss with capacity factors. Tune balance between routing quality and load balance. Monitoring: Track expert utilization during training. Imbalance indicates routing or loss tuning issues.