Expert Parallelism

Expert parallelism is a distributed computing strategy specifically designed for Mixture of Experts (MoE) models, where different expert sub-networks are placed on different GPUs. This allows the model to scale to enormous sizes while keeping the compute cost per token manageable.

How Expert Parallelism Works

• Expert Assignment: In an MoE layer, each token is routed to a small subset of experts (typically 2 out of 8–64 experts) by a learned gating network.
• Physical Distribution: Different experts reside on different GPUs. When a token is routed to a specific expert, the token's data is sent to the GPU hosting that expert via all-to-all communication.
• Parallel Computation: Multiple experts process their assigned tokens simultaneously across different GPUs, then results are gathered back.

Comparison with Other Parallelism Strategies

• Data Parallelism: Replicates the entire model on each GPU, processes different data. Doesn't help with model size.
• Tensor Parallelism: Splits individual layers across GPUs. High communication overhead but fine-grained.
• Pipeline Parallelism: Splits the model into sequential stages across GPUs. Can cause pipeline bubbles.
• Expert Parallelism: Uniquely suited for MoE — splits the model along the expert dimension, with communication only needed for token routing.

Challenges

• Load Balancing: If the gating network sends too many tokens to experts on the same GPU, that GPU becomes a bottleneck. Auxiliary load-balancing losses are used during training to encourage even distribution.
• All-to-All Communication: The token shuffling between GPUs requires high-bandwidth interconnects (NVLink, InfiniBand) to avoid becoming a bottleneck.
• Token Dropping: When an expert receives more tokens than its capacity, excess tokens may be dropped, requiring careful capacity factor tuning.

Real-World Usage

Models like Mixtral 8×7B, GPT-4 (rumored MoE), and Switch Transformer use expert parallelism to achieve very large effective model sizes while only activating a fraction of parameters per token, making both training and inference more efficient.