Mixture of Experts MoE

Mixture of Experts (MoE) is the sparse model architecture that replaces each dense feed-forward layer with multiple parallel "expert" sub-networks and a learned gating function that routes each input token to only K of N experts (typically K=1-2 out of N=8-128) — enabling models with trillion-parameter total capacity while maintaining the per-token compute cost of a much smaller dense model, because only a fraction of parameters are activated for each input.

Why MoE Scales Efficiently

A dense 175B model requires 175B parameters of computation per token. An MoE model with 8 experts of 22B each has 176B total parameters but activates only 1-2 experts (22-44B) per token. The model has the capacity to specialize different experts for different input types while keeping inference cost comparable to a 22-44B dense model.

Architecture

In a transformer MoE layer:
1. Gating Network: A small linear layer maps each token's hidden state to a score for each expert: g(x) = softmax(W_g · x). The top-K experts with highest scores are selected.
2. Expert Computation: Each selected expert processes the token through its own feed-forward network (two linear layers with activation). Different experts can specialize in different token types.
3. Combination: The outputs of the K selected experts are weighted by their gating scores and summed: output = Σ g_k(x) · Expert_k(x).

Routing Challenges

• Load Imbalance: Without regularization, the gating network tends to route most tokens to a few "popular" experts, leaving others underutilized. An auxiliary load-balancing loss penalizes uneven expert utilization, encouraging uniform routing.
• Expert Collapse: In extreme imbalance, unused experts stop learning and become permanently dead. Hard-coded routing constraints (capacity factor limiting tokens per expert) prevent this.
• Token Dropping: When an expert exceeds its capacity budget, excess tokens are either dropped (skipping the MoE layer) or routed to a secondary expert. Dropped tokens lose representational quality.

Key Models

• Switch Transformer (Google, 2021): K=1 routing (only one expert per token), N=128 experts. Demonstrated 4-7x training speedup over dense T5 at equivalent compute.
• Mixtral 8x7B (Mistral, 2023): 8 experts, K=2 routing. 46.7B total parameters but 12.9B active per token. Matches or exceeds Llama 2 70B quality at fraction of compute.
• DeepSeek-V3 (2024): 256 experts with auxiliary-loss-free routing and multi-token prediction. 671B total / 37B active parameters.

Inference Challenges

MoE models require all N experts in memory even though only K are active per token. A 8x22B MoE needs the same memory as a 176B dense model. Expert parallelism distributes experts across GPUs, but the dynamic routing makes load balancing across GPUs non-trivial. Expert offloading (storing inactive experts on CPU/NVMe) enables single-GPU inference at the cost of latency.

Mixture of Experts is the architecture that breaks the linear relationship between model capacity and compute cost — proving that a model can know vastly more than it uses for any single input, selecting the relevant expertise on the fly.