Ring Allreduce

Ring AllReduce distributes gradient aggregation across GPUs by passing partial sums around a ring topology, achieving bandwidth-optimal communication for data parallel training. Tree topologies work better for hierarchical systems with varying interconnect speeds. Ring AllReduce algorithm: each GPU has gradients to aggregate; divide into N chunks (N = number of GPUs); each GPU sends one chunk to next GPU in ring, receives from previous, and adds to local sum; after N-1 steps, each GPU has complete sum of one chunk; then N-1 more steps broadcast results. Bandwidth optimality: each GPU sends and receives 2(N-1)/N of total data; approaches optimal as N grows; fully utilizes all links simultaneously. Tree AllReduce: hierarchical aggregation—reduce within nodes, then across nodes, then broadcast back; better for systems where intra-node bandwidth >> inter-node bandwidth. Recursive halving/doubling: alternative algorithm dividing GPUs into pairs; different communication pattern, same total data. Implementation: NCCL (NVIDIA), Gloo, and MPI provide optimized AllReduce implementations. Hardware topology awareness: modern systems detect topology and select optimal algorithm automatically. Gradient compression: reduce communication by compressing gradients (top-k, quantization); trades accuracy for bandwidth. Efficient AllReduce is foundational for scaling data parallel training.