Rdma Remote Direct Memory Access
Remote Direct Memory Access (RDMA) is the high-performance networking technology that allows one computer to read from or write to another computer's memory directly, bypassing the remote CPU and OS kernel entirely — achieving latencies under 2 microseconds and bandwidths exceeding 400 Gbps (50 GB/s) per port, making it the foundation of interconnect fabrics in HPC clusters, AI training systems, and high-frequency trading networks.
Why RDMA Is Transformative
Traditional TCP/IP networking involves multiple software layers: application → socket API → kernel TCP/IP stack → NIC driver → hardware. Each layer adds latency (context switches, buffer copies, protocol processing). A typical TCP round-trip takes 20-50 microseconds. RDMA eliminates all intermediate software — the sending application posts a descriptor to the NIC hardware, which DMA-reads the data from the source memory and sends it directly to the remote NIC, which DMA-writes it into the destination memory. Total latency: 1-2 microseconds.
RDMA Technologies
- InfiniBand (IB): Purpose-built RDMA fabric. The dominant interconnect in HPC and AI training clusters. Current generation: NDR (400 Gbps / 50 GB/s per port). Provides full RDMA semantics natively. Mellanox/NVIDIA ConnectX-7 adapters.
- RoCE (RDMA over Converged Ethernet): RDMA over standard Ethernet infrastructure using the InfiniBand transport protocol encapsulated in UDP/IP. Requires lossless Ethernet (PFC flow control, ECN). Lower cost than InfiniBand but more complex network configuration.
- iWARP: RDMA over TCP. Works on any IP network without special configuration but has higher latency than IB or RoCE due to TCP processing.
RDMA Operations (Verbs)
| Operation | Description | Remote CPU Involved? |
|---|---|---|
| RDMA Write | Write data to remote memory | No |
| RDMA Read | Read data from remote memory | No |
| Send/Receive | Message passing (two-sided) | Yes (receive posted) |
| Atomic | Fetch-and-add, compare-and-swap on remote memory | No |
One-sided operations (Read/Write/Atomic) are the key innovation — the remote CPU is completely uninvolved. This enables millions of operations per second per core because each operation completes in hardware without any remote software execution.
GPU-Direct RDMA
NVIDIA GPUDirect RDMA allows the NIC to read/write GPU memory directly, eliminating the CPU staging buffer. Data flows: GPU memory → NIC → network → remote NIC → remote GPU memory. Critical for distributed deep learning where gradient tensors must move between GPUs on different nodes at NVLink-like speeds.
RDMA is the technology that makes distributed computing feel like shared memory — providing memory-to-memory data movement so fast that the network interconnect becomes nearly invisible to the application, enabling clusters of machines to cooperate as efficiently as processors on a single motherboard.