GPU Memory Management
GPU Memory Management is the system-level discipline that governs how data is allocated, transferred, and accessed across the discrete address spaces of CPU (host) and GPU (device) — where the latency and bandwidth of host-device data transfers often dominate total application time, making memory management the primary performance concern for GPU-accelerated workloads.
The Host-Device Memory Architecture
Discrete GPUs have their own memory (VRAM: HBM or GDDR) connected via a PCIe or NVLink bus to the CPU's system memory:
| Memory Type | Bandwidth | Latency | Capacity |
|---|---|---|---|
| GPU VRAM (HBM3e) | 3-8 TB/s | ~200 ns | 24-192 GB |
| PCIe 5.0 x16 | 64 GB/s | ~2-5 us | - |
| NVLink 5.0 | 900 GB/s | ~1 us | - |
| CPU DDR5 | 50-100 GB/s | ~80 ns | 128-2048 GB |
The PCIe bus is 50-100x slower than GPU VRAM bandwidth — every unnecessary host-device transfer is catastrophic for performance.
Memory Types and Their Uses
- Device Memory (cudaMalloc): Allocated in GPU VRAM. Accessible only from GPU kernels. Maximum bandwidth. Must be explicitly copied to/from host.
- Host Pinned (Page-Locked) Memory (cudaMallocHost): CPU memory that is pinned (prevented from being paged to disk). Enables DMA transfers between host and device without an intermediate copy through the OS page cache. Achieves full PCIe bandwidth (~25 GB/s PCIe 4.0) vs. pageable memory (~10 GB/s with the extra copy).
- Unified Virtual Memory (UVM / cudaMallocManaged): Creates a single virtual address space accessible from both CPU and GPU. The runtime automatically migrates pages between host and device on demand (page faults). Simplifies programming but can suffer from migration latency on first access — careful prefetching (cudaMemPrefetchAsync) is essential for performance.
- Zero-Copy (Mapped) Memory: Host pinned memory mapped into GPU address space. GPU accesses traverse the PCIe bus per-access. Useful for sparse access patterns where transferring the entire buffer would waste bandwidth.
Transfer Optimization Techniques
- Asynchronous Transfers: cudaMemcpyAsync on a non-default stream enables overlap of data transfer with kernel execution. Double-buffering: while the GPU processes batch N, the CPU transfers batch N+1.
- Pinned Memory Pools: Pre-allocating a pool of pinned memory avoids the overhead of pinning/unpinning on every transfer (pinning is expensive — ~1 ms per call).
- Compression: Hardware-accelerated memory compression (NVIDIA Ampere+) reduces effective transfer size by 2-4x for compressible data patterns.
- GPUDirect RDMA: Enables direct transfer from NIC or NVMe storage to GPU memory without CPU involvement, eliminating the CPU bottleneck for I/O-heavy workloads.
GPU Memory Management is the performance-critical infrastructure that determines whether a GPU application achieves 10% or 90% of theoretical hardware throughput — because the fastest GPU in the world is idle if it spends most of its time waiting for data to arrive from the host.