Mpi Collective Communication
MPI Collective Communication Operations are the coordinated multi-process communication patterns where all (or a defined subset of) processes in a communicator participate simultaneously in data exchange — including broadcast, reduce, allreduce, scatter, gather, allgather, and alltoall — which are the dominant communication cost in most parallel scientific applications and whose algorithmic implementation determines whether communication scales efficiently to thousands of nodes.
Core Collective Operations
| Operation | Description | Data Movement |
|---|---|---|
| Broadcast | One process sends to all | 1 → N |
| Reduce | All contribute, one receives result | N → 1 |
| Allreduce | Reduce + broadcast result to all | N → N |
| Scatter | One distributes unique parts to each | 1 → N (unique) |
| Gather | Each sends unique part to one | N → 1 (concatenate) |
| Allgather | Each sends its part, all receive full | N → N (concatenate) |
| Alltoall | Each sends unique data to every other | N → N (personalized) |
Allreduce: The Most Critical Collective
Allreduce (sum/max/min across all processes, result available to all) dominates distributed deep learning (gradient synchronization) and iterative solvers (global residual computation). Its implementation determines training throughput.
Allreduce Algorithms
- Ring Allreduce: Processes are arranged in a logical ring. Data is segmented into P chunks. Each process sends one chunk to its right neighbor and receives from its left, accumulating partial sums. After 2(P-1) steps, all processes have the complete result. Bandwidth cost: 2(P-1)/P × N bytes — approaches 2N regardless of P. Optimal bandwidth utilization but latency grows as O(P).
- Recursive Halving-Doubling: Processes pair up, exchange and reduce data at each step. After log2(P) steps, each process has a portion of the result. Then a reverse (doubling) phase distributes the result. Total cost: O(log P × α + N × log P × β) — better latency than ring for small messages.
- Tree (Binomial) Reduce + Broadcast: Reduce to root via binomial tree, then broadcast the result. Simple but root becomes a bottleneck for large messages.
- NCCL (NVIDIA Collective Communications Library): Optimized for GPU clusters using NVLink/NVSwitch topology-aware algorithms. Uses ring or tree algorithms mapped to the physical NVLink rings, achieving near-peak NVLink bandwidth (900 GB/s on DGX H100).
Overlap with Computation
Non-blocking collectives (MPI_Iallreduce) allow computation to proceed while the collective executes in the background. This is essential for hiding communication latency: start the allreduce of layer N's gradients while computing layer N-1's backward pass.
MPI Collective Communication is the coordination language of parallel computing — every parallel algorithm that needs global agreement, global data redistribution, or global reduction depends on these primitives, and their efficient implementation is what separates a cluster that scales from one that saturates.