Zero Copy Data Transfer GPU

Zero-Copy and Pinned Memory for GPU Data Transfer is the memory management technique that eliminates redundant data copies between CPU and GPU memory by using page-locked (pinned) host memory that the GPU can access directly via DMA or memory-mapped access — achieving maximum PCIe/NVLink transfer bandwidth (25-900 GB/s), enabling overlap of data transfer with computation, and in some cases allowing the GPU to read CPU memory directly without any explicit copy.

The Data Transfer Bottleneck

Standard cudaMemcpy involves:
1. OS copies data from user-space pageable memory to a pinned staging buffer.
2. DMA engine transfers from pinned buffer to GPU memory via PCIe.

The extra copy through the staging buffer halves effective bandwidth and prevents DMA/compute overlap.

Pinned (Page-Locked) Memory

cudaMallocHost() or cudaHostAlloc() allocates memory that is locked in physical RAM (cannot be swapped to disk):
- DMA Direct: The GPU DMA engine can directly access pinned memory via its physical address — no intermediate staging copy.
- Full PCIe Bandwidth: Pinned transfers achieve 12-25 GB/s on PCIe Gen4 x16 vs. 6-12 GB/s for pageable memory.
- Async Transfer: cudaMemcpyAsync() with pinned memory enables overlap of transfer and kernel execution using CUDA streams: while stream 1 transfers batch N+1 to GPU, stream 0 computes on batch N.
- Cost: Pinned memory reduces available pageable memory for the OS. Excessive pinned allocation can cause swapping of other processes. Best practice: pin only what's actively transferred.

Write-Combined (WC) Memory

cudaHostAlloc(... cudaHostAllocWriteCombined) allocates pinned memory with write-combining:
- CPU writes bypass L1/L2 cache and coalesce in WC buffers before flushing to memory.
- Transfer to GPU is faster (PCIe bus is write-optimized).
- CPU reads are very slow (not cached) — use only for CPU-write, GPU-read patterns.

Mapped (Zero-Copy) Memory

cudaHostAlloc(... cudaHostAllocMapped) maps pinned host memory into GPU's address space:
- GPU accesses host memory directly via PCIe load/store — no explicit cudaMemcpy needed.
- Each GPU load translates to a PCIe read (high latency: 1-5 μs per random access).
- Useful for: sparse access patterns (GPU reads only a small fraction of a large host array), host-side data that changes between kernel launches, and systems where GPU memory is insufficient.
- Performance: terrible for bulk access (PCIe bandwidth << GPU memory bandwidth). Good for meta-data access, small lookups, signaling between CPU and GPU.

NVLink and RDMA

• NVLink CPU-GPU (Grace Hopper): 450-900 GB/s CPU-GPU bandwidth. Coherent memory sharing — both CPU and GPU access the same physical memory with hardware cache coherence. Zero-copy becomes the default — no transfer needed.
• GPUDirect RDMA: Network adapter directly reads/writes GPU memory without going through CPU memory. Eliminates a CPU-memory hop for MPI and NCCL communication between GPUs on different nodes.

Zero-Copy and Pinned Memory are the memory optimization techniques that maximize the effective bandwidth of the CPU-GPU data pipeline — ensuring that data movement, the dominant bottleneck for GPU-accelerated applications, operates at the theoretical limits of the interconnect rather than being throttled by unnecessary copies.