Mixture of Experts MoE

Mixture of Experts (MoE) is the sparse architecture paradigm where each input token is routed to only a small subset (typically 1-2) of many parallel "expert" sub-networks within each layer — enabling models with trillions of total parameters while activating only a fraction per token, achieving dramatically better quality-per-FLOP than equivalent dense models.

The Core Idea

A dense Transformer applies every parameter to every token. An MoE layer replaces the single feed-forward network (FFN) with N parallel FFN experts (e.g., 8, 16, or 64) and a lightweight gating network that decides which expert(s) each token should use. If only 2 of 64 experts fire per token, the active computation is ~32x smaller than a dense model with the same total parameter count.

Gating and Routing

• Top-K Routing: The gating network computes a score for each expert given the input token embedding. The top-K experts (typically K=1 or K=2) are selected, and their outputs are weighted by the softmax of their gate scores.
• Switch Transformer: Routes each token to exactly one expert (K=1), maximizing sparsity. The simplified routing reduces communication overhead and improves training stability.
• Expert Choice Routing: Instead of each token choosing experts, each expert selects its top-K tokens from the batch. This naturally balances load across experts but requires global coordination.

Load Balancing

Without intervention, the gating network tends to collapse — sending most tokens to a few "popular" experts while others receive no traffic (expert dropout). Mitigation strategies include auxiliary load-balancing losses that penalize uneven expert utilization, noise injection into gate scores during training, and capacity factors that cap the maximum tokens per expert.

Scaling Results

• GShard (2020): 600B parameter MoE with 2048 experts, trained with automatic sharding across TPUs.
• Switch Transformer (2021): Demonstrated that scaling to 1.6T parameters with simplified top-1 routing achieves 4x speedup over dense T5 at equivalent quality.
• Mixtral 8x7B (2024): 8 experts of 7B parameters each, with top-2 routing. Despite having ~47B total parameters, each forward pass activates only ~13B — matching or exceeding Llama 2 70B quality at ~3x lower inference cost.
• DeepSeek-V2/V3: Multi-head latent attention combined with fine-grained MoE (256 routed experts), pushing the efficiency frontier further.

Infrastructure Challenges

MoE models require expert parallelism — different experts reside on different GPUs, and all-to-all communication routes tokens to their assigned experts. This communication overhead can dominate training time if not carefully optimized with techniques like expert buffering, hierarchical routing, and capacity-aware placement.

Mixture of Experts is the architecture that broke the linear relationship between model quality and inference cost — proving that bigger models can actually be cheaper to run by activating only the knowledge each token needs.