NCCL

NCCL (NVIDIA Collective Communications Library) — a high-performance library for multi-GPU and multi-node collective communication operations, essential for distributed deep learning training.

What NCCL Does

• Optimized implementations of collective operations across GPUs:
• AllReduce: Sum/average gradients across all GPUs (most used in training)
• AllGather: Each GPU sends its data to all others
• ReduceScatter: Reduce + scatter result across GPUs
• Broadcast: One GPU sends to all
• AllToAll: Full exchange between all GPUs

Why NCCL Matters for Training

• Distributed training: Each GPU computes gradients on its data batch
• Before weight update: AllReduce to average gradients across all GPUs
• NCCL makes this AllReduce as fast as possible

Communication Backends

• NVLink: GPU-to-GPU direct (900 GB/s on H100)
• PCIe: Older/cheaper (25-64 GB/s)
• InfiniBand: Multi-node (400 Gbps HDR → 50 GB/s per link)
• NCCL automatically selects the best topology and algorithm

Ring vs Tree AllReduce

• Ring: Each GPU sends/receives to/from neighbors in a ring. Bandwidth-optimal for large messages
• Tree: Hierarchical reduce. Latency-optimal for small messages
• NCCL auto-selects based on message size and topology

Usage in Frameworks

• PyTorch DDP uses NCCL by default: `torch.distributed.init_process_group(backend='nccl')`
• Transparent to user code in most cases

NCCL is the hidden backbone of all large-scale GPU training — without it, multi-GPU training would be orders of magnitude slower.