Peer to Peer GPU

Peer-to-Peer (P2P) GPU Communication is the hardware and software capability that allows one GPU to directly access another GPU's memory without routing data through CPU main memory — eliminating the host memory copy bottleneck in multi-GPU systems, reducing transfer latency by 2-5×, and enabling programming models where GPUs transparently share data, essential for multi-GPU deep learning training, scientific simulation, and real-time rendering.

P2P Communication Paths

Path	How	Bandwidth	Latency
Traditional (staged)	GPU A → CPU RAM → GPU B	Limited by PCIe + memcpy	~10-20 µs
P2P over PCIe	GPU A → PCIe switch → GPU B	PCIe speed (32-64 GB/s)	~3-5 µs
P2P over NVLink	GPU A → NVLink → GPU B	NVLink speed (600-900 GB/s)	~1-2 µs
GPUDirect RDMA	Network → GPU (bypass CPU)	Network speed (25-100 GB/s)	~2-5 µs

CUDA P2P API

// Check P2P support
int canAccess;
cudaDeviceCanAccessPeer(&canAccess, gpu0, gpu1);

// Enable P2P access
cudaSetDevice(gpu0);
cudaDeviceEnablePeerAccess(gpu1, 0);

// Direct copy between GPUs (no CPU staging)
cudaMemcpyPeer(dst_ptr_gpu1, gpu1, src_ptr_gpu0, gpu0, size);

// Async P2P copy
cudaMemcpyPeerAsync(dst, gpu1, src, gpu0, size, stream);

// Direct pointer access (Unified Virtual Addressing)
// GPU 0 kernel can dereference pointer to GPU 1 memory
my_kernel<<<grid, block>>>(gpu1_ptr);  // Access remote GPU memory

GPUDirect Technologies (NVIDIA)

Technology	What	Bypass
GPUDirect P2P	GPU-to-GPU over PCIe/NVLink	CPU memory
GPUDirect RDMA	Network NIC → GPU directly	CPU memory + CPU
GPUDirect Storage	NVMe SSD → GPU directly	CPU memory + filesystem
GPUDirect Async	Async control of all above	CPU involvement

GPUDirect RDMA (Network → GPU)

InfiniBand NIC reads/writes GPU memory directly.
MPI_Send from GPU → NIC grabs data directly from GPU memory → sends over network → remote NIC writes directly to remote GPU.
No CPU copies in the data path → critical for distributed training.
Requires: NVIDIA GPU + Mellanox/NVIDIA NIC + GPUDirect-aware driver.

P2P Topology Awareness

GPU 0 ←NVLink→ GPU 1
      ↑               ↑
    NVLink           NVLink
      ↓               ↓
    GPU 2 ←NVLink→ GPU 3
      ↑               ↑
    PCIe             PCIe
      ↓               ↓
    CPU Socket 0    CPU Socket 1

GPU 0 → GPU 1 (NVLink): ~600 GB/s, ~1 µs.
GPU 0 → GPU 3 (NVLink via switch): ~600 GB/s, ~1.5 µs.
GPU 0 → GPU on remote socket (PCIe): ~25 GB/s, ~5 µs.
Training frameworks (PyTorch, DeepSpeed) should be topology-aware → minimize cross-socket transfers.

Impact on ML Training

AllReduce: P2P NVLink → ring topology at 600 GB/s → fast gradient sync.
Tensor parallelism: Each GPU holds fraction of layer → P2P required for activations.
Expert parallelism (MoE): Tokens routed to expert GPUs → P2P for token transfer.
Without P2P: All traffic goes through CPU → 10× slower → multi-GPU training impractical.

Peer-to-peer GPU communication is the physical foundation of multi-GPU computing — by enabling GPUs to share data at NVLink or PCIe speeds without CPU intermediation, P2P transforms a collection of discrete GPUs into a unified computational fabric where tensor and pipeline parallelism can operate at the bandwidth required by modern large-scale AI training.

Peer to Peer GPU

Take peer to peer gpu further