Rdma Infiniband
RDMA (Remote Direct Memory Access) and InfiniBand are the high-performance networking technologies that enable direct memory-to-memory data transfer between machines without involving the CPU or operating system — achieving latencies under 1 microsecond and throughputs over 400 Gbps, making them essential for HPC clusters, distributed training, and low-latency storage systems.
How RDMA Works
- Traditional networking: App → OS kernel → TCP/IP stack → NIC → wire → NIC → kernel → App.
- Each step: System calls, context switches, memory copies — adds latency.
- RDMA: App → NIC → wire → NIC → remote memory (bypasses both CPUs and kernels).
- Zero-copy: Data goes directly from wire to application buffer — no intermediate copies.
- Kernel bypass: NIC handles protocol processing in hardware — no OS involvement.
- CPU offload: CPU freed for computation while NIC handles transfers.
RDMA Operations
| Operation | Description | CPU Involvement |
|---|---|---|
| RDMA Write | Write to remote memory | None on remote side |
| RDMA Read | Read from remote memory | None on remote side |
| Send/Receive | Two-sided messaging | Both sides post buffers |
| Atomic (CAS, FetchAdd) | Atomic operation on remote memory | None on remote side |
RDMA Transports
| Transport | Fabric | Bandwidth | Latency | Deployment |
|---|---|---|---|---|
| InfiniBand (IB) | Dedicated IB fabric | HDR: 200 Gbps, NDR: 400 Gbps | < 0.6 μs | HPC, AI clusters |
| RoCE v2 | Standard Ethernet | 25-400 Gbps | 1-3 μs | Data centers |
| iWARP | Standard Ethernet (TCP) | 10-100 Gbps | 5-10 μs | Enterprise storage |
InfiniBand Generations
| Generation | Per-Lane Rate | 4x Port | Year |
|---|---|---|---|
| QDR | 10 Gbps | 40 Gbps | 2008 |
| FDR | 14 Gbps | 56 Gbps | 2012 |
| EDR | 25 Gbps | 100 Gbps | 2015 |
| HDR | 50 Gbps | 200 Gbps | 2019 |
| NDR | 100 Gbps | 400 Gbps | 2022 |
| XDR | 200 Gbps | 800 Gbps | 2024 |
RDMA in Distributed ML Training
- NCCL over InfiniBand: Default for multi-node GPU training.
- GPUDirect RDMA: NIC reads directly from GPU memory — no CPU staging buffer.
- 8× H100 DGX pods connected via 8× NDR400 (3.2 Tbps per node).
- Gradient AllReduce: Ring/tree AllReduce over RDMA achieves near-wire-speed.
- Without RDMA: Multi-node training bandwidth drops 5-10x → scaling becomes impractical beyond 2-4 nodes.
Programming RDMA
- libibverbs: Low-level C API for RDMA operations (complex — ~200 LOC for simple send).
- UCX (Unified Communication X): Higher-level library abstracting RDMA transports.
- NCCL / Gloo: ML-specific collective communication over RDMA.
RDMA and InfiniBand are the networking foundation of modern AI supercomputers — the ability to move data between machines at hardware speed without CPU involvement is what makes it possible to train trillion-parameter models across thousands of GPUs with near-linear scaling efficiency.