Mpi Collective Operations

MPI Collective Operations are communication patterns where all processes in a communicator participate simultaneously — implementing broadcast, scatter, gather, reduce, and all-to-all operations essential for distributed memory parallel computing.

Point-to-Point vs. Collective

• Point-to-point: `MPI_Send` / `MPI_Recv` between two specific processes.
• Collective: All processes in communicator participate — synchronization implied.
• Collective operations are more efficient and easier to reason about than manual P2P.

Core Collective Operations

MPI_Bcast (Broadcast):

MPI_Bcast(buffer, count, MPI_INT, root, MPI_COMM_WORLD);

• Root sends buffer to all other processes.
• Used for: Broadcasting parameters, model weights.

MPI_Scatter / MPI_Gather:
- Scatter: Root sends different data to each process (work distribution).
- Gather: Each process sends data to root (result collection).
- MPI_Scatterv / Gatherv: Variable-length messages per process.

MPI_Reduce:

MPI_Reduce(send, recv, count, MPI_DOUBLE, MPI_SUM, root, MPI_COMM_WORLD);

• Combine values from all processes using operation (SUM, MAX, MIN, PROD) → result at root.

MPI_Allreduce:
- Like Reduce but result available at ALL processes.
- Essential for distributed training: Sum gradients across all GPUs.
- Ring Allreduce: Most efficient algorithm — O(N) bandwidth, O(log N) latency.

MPI_Alltoall:
- Every process sends unique data to every other process.
- Used for: Matrix transpose, FFT butterfly, dense database joins.
- Most expensive collective: O(P²) messages in naive implementation.

Algorithm Implementations

• Butterfly (Recursive Halving/Doubling): Optimal for small counts.
• Ring: Optimal bandwidth for large messages (allreduce, allgather).
• Binomial Tree: Optimal for broadcast/reduce in latency-dominated regime.

Non-Blocking Collectives

MPI_Request req;
MPI_Iallreduce(sendbuf, recvbuf, count, dtype, op, comm, &req);
// Overlap computation here
MPI_Wait(&req, MPI_STATUS_IGNORE);

• Allows overlap of communication with computation — critical for scaling efficiency.

MPI collective operations are the communication backbone of HPC and distributed training — efficient collective implementations (MVAPICH, OpenMPI, NCCL) are what allow hundreds to thousands of GPUs to train LLMs together at near-linear efficiency.