Infiniband

InfiniBand is a high-bandwidth, low-latency networking technology using RDMA for GPU cluster communication — providing 200-400 Gbps per port with microsecond latencies, InfiniBand is the interconnect of choice for large-scale AI training where multi-node communication efficiency determines scaling effectiveness.

What Is InfiniBand?

  • Definition: High-performance networking fabric for clusters.
  • Technology: RDMA (Remote Direct Memory Access).
  • Vendor: NVIDIA/Mellanox (dominant).
  • Use Case: HPC, AI training, storage networks.

Why InfiniBand for AI

  • Bandwidth: 400 Gbps (NDR) vs. 100 Gbps Ethernet.
  • Latency: ~1 μs vs. ~10-50 μs Ethernet.
  • RDMA: Bypass CPU for GPU-to-GPU transfers.
  • Scaling: Efficient all-reduce across thousands of GPUs.
  • Proven: Used in largest AI training runs.

InfiniBand Generations

Speed Evolution:

Generation | Speed (per port) | Year
-----------|------------------|------
EDR        | 100 Gbps         | 2014
HDR        | 200 Gbps         | 2019
NDR        | 400 Gbps         | 2022
XDR        | 800 Gbps         | 2024
GDR        | 1600 Gbps        | Future

Comparison with Ethernet:

Aspect        | InfiniBand NDR | 400G Ethernet
--------------|----------------|---------------
Bandwidth     | 400 Gbps       | 400 Gbps
Latency       | ~1 μs          | ~10-50 μs
RDMA          | Native         | RoCE (extra)
Congestion    | Credit-based   | Drop-based
CPU overhead  | Minimal        | Higher
AI training   | Optimized      | Improving
Cost          | Higher         | Lower

RDMA Explained

How RDMA Works:

Traditional Network:
CPU → Copy to buffer → NIC → Network → NIC → Copy to buffer → CPU

RDMA:
GPU Memory → NIC → Network → NIC → GPU Memory
(CPU not involved, zero-copy)

GPU Direct RDMA:

InfiniBand — The AI Cluster Network RDMA networking: bypass the kernel, write directly to remote GPU memory — 400-800 Gbps per port Fat-Tree Topology (AI training cluster) Spine Spine Spine Spine Leaf Leaf Leaf Leaf Leaf DGX nodes (8 GPU each) DGX nodes non-blocking fat-tree: every node can talk to every other at full BW simultaneously RDMA — Zero-Copy, Kernel-Bypass TCP/IP: app → kernel → copy → NIC → wire → NIC → copy → kernel → app RDMA: app → NIC → wire → NIC → app (direct) no kernel involvement, no data copies latency: 1-2 µs (vs 50+ µs TCP) GPU-Direct RDMA: NIC writes directly to GPU HBM (skips CPU entirely for AllReduce) InfiniBand Generations QDR (2008): 40 Gbps FDR (2011): 56 Gbps EDR (2014): 100 Gbps HDR (2018): 200 Gbps NDR (2022): 400 Gbps XDR (2025): 800 Gbps NVIDIA/Mellanox ConnectX HCA Quantum switches current production next gen Why InfiniBand Dominates AI Training AllReduce (gradient sync): 100K GPUs each send ~1GB/step → network must deliver TB/s aggregate GPU-Direct RDMA + SHARP (in-network reduction): hardware-accelerated collective ops inside the switch Meta 24K GPU cluster: 3 tiers of NDR IB switches | xAI Colossus: 100K H100 on IB fabric Alternative: RoCE v2 (RDMA over Ethernet) — cheaper but worse tail latency. Broadcom/Arista compete here. InfiniBand is the backplane of AI: 400K+ connections at microsecond latency — the highway between every GPU.

AI Training Infrastructure

Typical Large Cluster:

┌─────────────────────────────────────────────────────────┐│ Spine Switches ││ (InfiniBand NDR, high-radix, non-blocking) │└─────────────────────────────────────────────────────────┘ │ │ │ │ ▼ ▼ ▼ ▼ ┌─────────┐┌─────────┐┌─────────┐┌─────────┐ │ Leaf ││ Leaf ││ Leaf ││ Leaf │ │ Switch ││ Switch ││ Switch ││ Switch │ └────┬────┘└────┬────┘└────┬────┘└────┬────┘ │ │ │ │ ┌─────┼─────┐ ... ... ... │ │ │┌──────┐┌──────┐┌──────┐│DGX 1 ││DGX 2 ││DGX 3 │ (8 H100s each)└──────┘└──────┘└──────┘

NCCL with InfiniBand

import torch
import torch.distributed as dist
import os

# Set NCCL environment for InfiniBand
os.environ["NCCL_IB_DISABLE"] = "0"  # Enable InfiniBand
os.environ["NCCL_NET_GDR_LEVEL"] = "5"  # Enable GPUDirect

# Initialize distributed
dist.init_process_group(
    backend="nccl",
    init_method="env://",
)

# Training code - NCCL uses InfiniBand automatically
model = DistributedDataParallel(model)

Checking InfiniBand

# List InfiniBand devices
ibstat

# Show port status
ibstatus

# Check link speed
ibstat mlx5_0 | grep Rate

# Performance test
ib_write_bw -d mlx5_0

InfiniBand vs. Alternatives

Use Case              | Best Choice
----------------------|------------------
AI training (1000+ GPU) | InfiniBand NDR
Small clusters (<64 GPU)| Either (cost-dependent)
Cloud/flexibility     | Ethernet (easier)
Maximum performance   | InfiniBand
Budget constrained    | 400G Ethernet + RoCE

Cost Considerations

Component          | InfiniBand | 400G Ethernet
-------------------|------------|---------------
NIC/HCA            | $3-5K      | $1-2K
Switch (port)      | $500-1K    | $200-400
Total system cost  | Higher     | Lower
Performance/$      | Better at scale | Better for small

InfiniBand is the performance backbone of large-scale AI training — when training frontier models across thousands of GPUs, the efficiency of collective operations enabled by InfiniBand's low latency and RDMA capabilities directly determines how well training scales.

Take infiniband further

Ask the copilot about this term, or have our engineers assess it against your process.

Ask CFSGPT about this → Request a technical assessment → Open the CFS app