Nvlink Nvswitch
GPU Interconnect Technologies (NVLink vs. PCIe vs. NVSwitch) are the communication fabrics that connect GPUs to each other and to CPUs — where the bandwidth, latency, and topology of these interconnects critically determine multi-GPU training performance, as gradient synchronization and tensor parallelism require moving terabytes of data between GPUs per second, making interconnect choice the primary bottleneck differentiator between consumer and data center GPU systems.
Interconnect Comparison
| Interconnect | Bandwidth (per direction) | Latency | Topology | Generation |
|---|---|---|---|---|
| PCIe 4.0 x16 | 32 GB/s | ~1 µs | Point-to-point via switch | 2017 |
| PCIe 5.0 x16 | 64 GB/s | ~0.8 µs | Point-to-point via switch | 2022 |
| NVLink 3 (A100) | 600 GB/s total (12 links) | ~0.5 µs | Mesh via NVSwitch | 2020 |
| NVLink 4 (H100) | 900 GB/s total (18 links) | ~0.3 µs | Full mesh via NVSwitch | 2022 |
| NVLink 5 (B200) | 1800 GB/s total | ~0.2 µs | Full mesh via NVSwitch | 2024 |
| AMD Infinity Fabric | 600 GB/s (MI300X) | ~0.5 µs | Mesh | 2023 |
NVLink Architecture
- NVLink is NVIDIA's proprietary high-speed GPU-to-GPU interconnect.
- Each NVLink lane: 25 GB/s (NVLink 3) → 50 GB/s (NVLink 4) → 100 GB/s (NVLink 5).
- H100: 18 NVLink 4 lanes = 900 GB/s bidirectional → 14× PCIe 5.0 bandwidth.
- Direct GPU-to-GPU memory access: GPU 0 can read/write GPU 1 memory at full NVLink speed.
NVSwitch
- NVSwitch: Dedicated switch chip that connects multiple GPUs via NVLink.
- DGX H100: 4 NVSwitch chips connect 8 H100 GPUs → any-to-any full bandwidth.
- Without NVSwitch: Only nearest-neighbor NVLink connections → limited topology.
- With NVSwitch: Full bisection bandwidth → AllReduce at full speed regardless of communication pattern.
Multi-Node: NVLink + InfiniBand
- Intra-node: NVLink (900 GB/s) → fast tensor/pipeline parallelism.
- Inter-node: InfiniBand (50-100 GB/s) → data parallelism gradient sync.
- Hierarchy: Optimize communication to keep most traffic intra-node.
Impact on ML Training
| Communication Pattern | PCIe Limited | NVLink Enabled |
|---|---|---|
| AllReduce (8 GPUs) | ~25 GB/s effective | ~700 GB/s effective |
| Tensor parallelism | Not feasible (too slow) | Standard approach |
| Pipeline parallelism | Limited | Good |
| Expert parallelism (MoE) | Bottleneck | Viable |
PCIe Still Matters
- CPU-GPU data transfer (dataset loading): PCIe 5.0 is sufficient.
- Consumer GPUs: NVLink not available → PCIe only.
- Inference serving: PCIe bandwidth often sufficient for batch inference.
- Cost: PCIe switches are commodity; NVSwitch is expensive and NVIDIA-exclusive.
GPU interconnect technology is the infrastructure that makes large-scale AI training possible — the 10-30× bandwidth advantage of NVLink over PCIe is what enables tensor parallelism across GPUs, without which training models larger than single-GPU memory would require prohibitively slow PCIe communication, and the NVSwitch full-mesh topology is what makes 8-GPU DGX systems behave like a single massive accelerator.