Mixture of Experts MoE Architecture

Mixture of Experts (MoE) is the conditional computation architecture that routes each input token to a subset of specialized expert sub-networks rather than processing through all parameters — enabling models with massive parameter counts (hundreds of billions) while maintaining inference cost comparable to much smaller dense models by activating only 1-2 experts per token.

MoE Architecture:
- Expert Networks: each expert is a standard feed-forward network (FFN) with identical architecture but independent parameters; a Switch Transformer layer replaces the single FFN with E experts (typically 8-128), each containing the same hidden dimension
- Gating Network (Router): a learned linear layer that takes the input token embedding and produces a probability distribution over experts; top-K experts (K=1 or K=2) are selected per token based on highest gating scores
- Sparse Activation: with E=64 experts and K=2, each token uses 2/64 = 3.1% of the total parameters; total model capacity scales with E while per-token compute scales with K — decoupling capacity from compute cost
- Expert FFN Placement: MoE layers typically replace every other FFN layer in a Transformer; alternating dense and MoE layers provides a balance between shared representations (dense layers) and specialized processing (MoE layers)

Routing Mechanisms:
- Top-K Routing: select K experts with highest router logits; weight their outputs by normalized softmax probability; original Shazeer et al. (2017) approach used Top-2 routing with noisy gating
- Expert Choice Routing: instead of tokens choosing experts, each expert selects its top-K tokens based on router scores; guarantees perfect load balance (each expert processes exactly the same number of tokens) but some tokens may be dropped or processed by fewer experts
- Token Dropping: when an expert receives more tokens than its capacity buffer allows, excess tokens are dropped (assigned to a residual connection); capacity factor C (typically 1.0-1.5) determines buffer size as C × (total_tokens / num_experts)
- Auxiliary Load Balancing Loss: additional training loss penalizing uneven token distribution across experts; fraction of tokens assigned to each expert should approximate 1/E for uniform distribution; loss coefficient typically 0.01-0.1 to avoid overwhelming the main training objective

Training Challenges:
- Load Imbalance: without auxiliary loss, the majority of tokens route to a few "popular" experts while others receive minimal traffic (expert collapse); severe imbalance wastes capacity and starves unused experts of gradient signal
- Expert Parallelism: experts distributed across GPUs require all-to-all communication to route tokens to their assigned expert's GPU; communication volume = batch_size × hidden_dim × 2 (send + receive); bandwidth-intensive for large models
- Training Instability: router gradients can be noisy; expert competition creates reinforcement loops (popular experts improve faster, attracting more tokens); dropout on router logits and jitter noise stabilize training
- Batch Size Sensitivity: each expert sees batch_size/E effective tokens; larger global batch sizes ensure each expert receives sufficient gradient signal per step; MoE models typically require 4-8× larger batch sizes than equivalent dense models

Production Models:
- Mixtral 8×7B: 8 experts with 7B parameters each, Top-2 routing; total 47B parameters but only 13B active per token; matches or exceeds Llama 2 70B while being 6× faster at inference
- Switch Transformer: Top-1 routing to simplify training; scaled to 1.6 trillion parameters with 2048 experts; demonstrated that scaling expert count improves sample efficiency
- GPT-4 (Rumored): believed to use MoE architecture with ~16 experts; 1.8T total parameters with ~220B active per forward pass; demonstrates MoE viability at the frontier of AI capability
- DeepSeek-V2/V3: MoE with fine-grained expert segmentation (256+ experts, Top-6 routing); achieved competitive performance with significantly reduced training cost

Mixture of Experts is the architectural innovation that breaks the linear relationship between model capacity and inference cost — enabling the training of models with hundreds of billions of parameters at a fraction of the computational cost of equivalent dense models, fundamentally changing the economics of scaling AI systems.