Rdma Remote Direct Memory Access

RDMA (Remote Direct Memory Access) is the networking capability that allows a computer to access memory on a remote machine without involving the remote CPU or operating system — the data transfer happens directly between network adapter and application memory, bypassing kernel, protocol stack, and remote CPU, achieving latencies of 1-2 µs and bandwidths of 200+ Gbps that are impossible with conventional TCP/IP socket programming.

Why RDMA Exists

TCP/IP socket communication: data copies user buffer → kernel socket buffer → NIC → network → NIC → kernel buffer → user buffer. Each copy takes CPU cycles and memory bandwidth. OS overhead (system calls, interrupts, scheduling) adds 10-50 µs latency. For HPC and distributed ML (gradient allreduce, parameter server), this overhead dominates.

RDMA Operation Model

• Two-sided (send/receive): both sides involved. Sender posts a send work request (WR); receiver must pre-post a receive WR. The NIC delivers directly to receiver's pre-registered memory buffer. Similar semantics to MPI messaging.
• One-sided (read/write): initiator specifies remote memory address (rkey + virtual address obtained via out-of-band exchange). RDMA Write: push data to remote memory without remote CPU involvement. RDMA Read: pull data from remote memory. Atomic (Compare-and-Swap, Fetch-and-Add) operations.

Verbs API

Low-level RDMA programming interface:
- Protection Domain (PD): namespace for memory registrations and queue pairs.
- Memory Registration: ``ibv_reg_mr()`` pins and registers buffer (virtual → physical mapping given to NIC), returns lkey/rkey.
- Queue Pair (QP): pair of send queue (SQ) and receive queue (RQ). Types: RC (Reliable Connected — in-order delivery, acknowledgments), UC (Unreliable Connected), UD (Unreliable Datagram — broadcast/multicast).
- Completion Queue (CQ): NIC posts completion events; application polls CQ (busy-poll for low latency vs event-driven interrupt for efficiency).
- Work Request (WR): descriptor posted to SQ/RQ specifying operation, buffer, length, remote address.

Transport Technologies

• InfiniBand: native RDMA, lossless fabric (credit-based flow control), industry standard in HPC (Frontier, Summit, Aurora).
• RoCE v2 (RDMA over Converged Ethernet): RDMA semantics over UDP/IPv4, requires priority-flow control (PFC) or DCQCN for lossless operation. Lower cost than IB, adopted by hyperscalers.
• iWARP: RDMA over TCP, preserves TCP reliability/NAT traversal but higher latency.

High-Level Abstractions

• UCX (Unified Communication X): portable RDMA API (used by OpenMPI, OpenSHMEM), selects best transport automatically.
• libfabric (OFI): OpenFabrics Interface, provider model (verbs, psm2, CXI for Slingshot).
• NCCL, Gloo: distributed ML collective libraries using RDMA.

Performance

• Latency: ~1.0-1.5 µs (IB HDR) vs 50-200 µs (TCP/IP loopback).
• Bandwidth: 200 Gbps per port (IB NDR: 400 Gbps).
• CPU offload: near-zero CPU involvement for data path.

RDMA is the networking technology that removes the CPU from the data movement critical path — enabling distributed HPC and AI systems to exchange data at memory-bus speeds across a cluster, making large-scale parallel computing economically and technically feasible by eliminating the latency and CPU overhead of conventional networking.