GPU Memory Pool

GPU Memory Pool Allocators are the caching memory management systems that maintain pre-allocated pools of GPU memory to eliminate the overhead of frequent cudaMalloc/cudaFree calls — reducing allocation latency from milliseconds to microseconds, preventing memory fragmentation, and enabling the rapid tensor allocation/deallocation patterns required by deep learning frameworks.

The Problem with Raw CUDA Allocation

• `cudaMalloc()`: ~1-10 ms per call — extremely slow (requires GPU driver interaction, page table updates).
• Deep learning: Each training iteration allocates/frees hundreds of tensors.
• Without pooling: 200 allocations × 5 ms = 1 second of pure allocation overhead per iteration.
• With pooling: 200 allocations × 5 μs = 1 ms — 1000x faster.

How Caching Allocators Work

1. First allocation: Pool calls `cudaMalloc` for a large block (e.g., 2GB).
2. User requests 256MB: Pool carves out 256MB from the large block — returns pointer.
3. User frees 256MB: Pool marks the segment as available — does NOT call `cudaFree`.
4. Next 256MB request: Pool reuses the freed segment — zero allocation overhead.
5. Pool grows: If existing blocks are insufficient, allocate another large block.

PyTorch CUDA Caching Allocator

• Default allocator for all PyTorch GPU tensors.
• Maintains separate pools for small (< 1MB) and large (≥ 1MB) allocations.
• Uses best-fit strategy with block splitting to minimize fragmentation.
• `torch.cuda.memory_summary()`: Shows allocated, reserved, and fragmented memory.
• `torch.cuda.empty_cache()`: Returns unused cached blocks to CUDA (but doesn't help with fragmentation).

Memory Fragmentation

• Even with pooling, fragmentation occurs: Many small free blocks but no contiguous space for a large allocation.
• Example: 8GB reserved, 2GB in use, but largest free block is only 500MB → cannot allocate 1GB tensor.
• Mitigation: PyTorch 2.x uses `expandable_segments` configuration to reduce OS-level fragmentation.

CUDA Memory Pool API (CUDA 11.2+)

• `cudaMemPool_t`: Native CUDA memory pool support.
• `cudaMallocAsync()` / `cudaFreeAsync()`: Stream-ordered allocation — allocation tied to CUDA stream.
• Benefit: GPU hardware manages allocation ordering — further reduces synchronization overhead.

Memory Management Best Practices

• Pre-allocate: Allocate maximum-size tensors once at startup, reuse buffers.
• Gradient accumulation: Process smaller micro-batches to reduce peak memory.
• Mixed precision: FP16/BF16 tensors use half the memory of FP32.
• Activation checkpointing: Trade compute for memory by recomputing activations during backward.

GPU memory pool allocators are essential infrastructure for all GPU computing frameworks — without them, the rapid tensor allocation patterns of modern deep learning and scientific computing would be throttled by driver-level allocation overhead, making interactive and training workloads impractically slow.