Nvlink
NVLink is NVIDIA's high-bandwidth GPU-to-GPU interconnect — providing substantially higher bandwidth and lower latency than PCIe for multi-GPU systems, enabling efficient large-scale training and inference across multiple GPUs.
PCIe vs. NVLink Comparison
| Feature | PCIe Gen4 x16 | NVLink 4.0 (H100) |
|---|---|---|
| Bandwidth (1 link) | 64 GB/s | 900 GB/s |
| Links per GPU | 1 | 18 |
| Total bi-directional | 128 GB/s | 900 GB/s |
| Latency | ~1.5 μs | ~1 μs |
| Topology | Star (via CPU) | Any (direct GPU-GPU) |
NVLink Generations
- NVLink 1.0 (P100, 2016): 160 GB/s.
- NVLink 2.0 (V100, 2018): 300 GB/s total.
- NVLink 3.0 (A100, 2020): 600 GB/s total.
- NVLink 4.0 (H100, 2022): 900 GB/s total + NVSwitch fabric.
NVSwitch
- Full all-to-all GPU interconnect fabric: Any GPU → any GPU at full bandwidth.
- NVIDIA DGX A100/H100: 8 GPUs + 6 NVSwitches → 300 GB/s all-to-all.
- NVLink Network (NVL72, 2024): 72 H100 GPUs in one NVLink domain.
Peer-to-Peer (P2P) Memory Access
// Enable P2P access between GPU 0 and GPU 1
cudaSetDevice(0);
cudaDeviceEnablePeerAccess(1, 0);
// Direct copy GPU0 → GPU1 (bypasses CPU)
cudaMemcpyPeerAsync(dst_on_gpu1, 1, src_on_gpu0, 0, size, stream);Impact on Distributed Training
- AllReduce within node: NVLink AllReduce ~10x faster than PCIe AllReduce.
- Tensor parallelism: Sharded matrix multiply requires high-bandwidth all-reduce every layer.
- Without NVLink: PCIe bottleneck limits GPU count for efficient tensor parallelism.
- With NVLink: Can tensor-parallelize across 8 GPUs efficiently.
NVLink is the critical infrastructure for large-scale LLM training — without it, inter-GPU communication would bottleneck all forms of model parallelism, and trillion-parameter models would be infeasible to train within reasonable time and cost budgets.