Distributed Training Framework
Distributed Training Frameworks are the software systems that coordinate the training of large machine learning models across multiple GPUs and multiple machines — handling data distribution, gradient synchronization, communication optimization, and fault tolerance to enable training of models that exceed single-GPU memory capacity and to reduce training time from months to days through horizontal scaling.
Major Distributed Training Frameworks
| Framework | Developer | Key Feature | Typical Use |
|---|---|---|---|
| PyTorch DDP | Meta | Native PyTorch distributed | Standard multi-GPU training |
| DeepSpeed | Microsoft | ZeRO optimizer, pipeline parallelism | Large language models |
| Horovod | Uber → LF AI | Ring-allreduce, easy adoption | Multi-framework support |
| Megatron-LM | NVIDIA | Tensor + pipeline + data parallelism | GPT-scale training |
| JAX/pjit | XLA compiler, automatic sharding | TPU and GPU training | |
| ColossalAI | HPC-AI Tech | Heterogeneous, auto-parallelism | Research and production |
PyTorch DDP (DistributedDataParallel)
- Each GPU holds full model replica.
- Each GPU processes different data batch (data parallelism).
- Gradient synchronization: All-reduce across GPUs after backward pass.
- Bucket gradient all-reduce: Overlaps communication with computation.
- Scales to hundreds of GPUs efficiently for models that fit in single GPU memory.
DeepSpeed ZeRO Stages
| Stage | What's Partitioned | Memory Saving |
|---|---|---|
| ZeRO-1 | Optimizer states (Adam momentum, variance) | ~4x |
| ZeRO-2 | + Gradients | ~8x |
| ZeRO-3 | + Model parameters | ~Nx (N = GPU count) |
| ZeRO-Infinity | Offload to CPU/NVMe | Nearly unlimited |
- ZeRO-3 enables training models larger than single GPU memory.
- Communication cost: All-gather parameters before forward/backward, reduce-scatter gradients after.
Megatron-LM 3D Parallelism
- Data Parallelism: Replicate model, split data.
- Tensor Parallelism: Split individual layers across GPUs (within a node, needs fast NVLink).
- Pipeline Parallelism: Split model layers sequentially across GPUs.
- Combined: GPT-3 (175B parameters) trained on 1024 A100 GPUs using 3D parallelism.
Communication Patterns
| Pattern | Operation | Used By |
|---|---|---|
| All-Reduce | Sum gradients across all GPUs | DDP, Horovod |
| All-Gather | Collect full parameter from shards | ZeRO-3, FSDP |
| Reduce-Scatter | Reduce + distribute shards | ZeRO-2/3 |
| Point-to-Point | Send activation between pipeline stages | Pipeline parallelism |
Fault Tolerance
- Checkpointing: Save model/optimizer state periodically.
- Elastic training: Add/remove workers without restart (PyTorch Elastic, Horovod Elastic).
- Communication timeout: Detect and handle straggler or failed nodes.
Distributed training frameworks are the essential infrastructure for training modern AI — without them, training a GPT-4-class model (estimated > 1 trillion parameters on tens of thousands of GPUs) would be impossible, making these frameworks as critical to AI progress as the hardware itself.