GPU Occupancy Optimization
GPU Occupancy Optimization is the process of maximizing the ratio of active warps to the maximum possible warps per Streaming Multiprocessor (SM) — achieved by carefully choosing thread block sizes and managing resource usage (registers, shared memory) to ensure enough warps are resident on each SM to hide memory latency through warp switching, though maximum occupancy does not always yield maximum performance.
Understanding Occupancy
- $\text{Occupancy} = \frac{\text{Active Warps per SM}}{\text{Max Warps per SM}}$
- Example (A100): Max 64 warps per SM. If kernel runs with 32 active warps → 50% occupancy.
What Limits Occupancy?
| Resource | A100 Limit per SM | How It Limits |
|---|---|---|
| Threads/block | 1024 max | Limits threads per block |
| Warps per SM | 64 max | Hard cap on active warps |
| Registers per SM | 65536 | If kernel uses 64 regs/thread, 256 threads max → 8 warps |
| Shared memory per SM | 164 KB (configurable) | If block uses 48 KB → only 3 blocks fit |
| Blocks per SM | 32 max | Even tiny blocks: max 32 |
Register Pressure Example
- Kernel uses 32 registers per thread.
- 65536 registers / 32 = 2048 threads max = 64 warps → 100% occupancy.
- Kernel uses 64 registers per thread.
- 65536 / 64 = 1024 threads = 32 warps → 50% occupancy.
- Kernel uses 128 registers per thread.
- 65536 / 128 = 512 threads = 16 warps → 25% occupancy.
Shared Memory Example
- SM has 164 KB shared memory.
- Block uses 48 KB → 164/48 = 3 blocks max.
- If block has 256 threads = 8 warps → 24 active warps → 37.5% occupancy.
Choosing Block Size
| Block Size | Warps/Block | Pros | Cons |
|---|---|---|---|
| 32 (1 warp) | 1 | Minimal shared memory | Max 32 blocks = 32 warps |
| 128 (4 warps) | 4 | Good balance | Common default |
| 256 (8 warps) | 8 | High occupancy | Higher shared memory/block |
| 512 (16 warps) | 16 | Fewer blocks needed | Limits block count per SM |
| 1024 (32 warps) | 32 | Max threads/block | Only 2 blocks possible per SM |
Occupancy vs. Performance
- Higher occupancy → more warps to switch between → better latency hiding.
- BUT: Higher occupancy may force fewer registers → more register spilling → slower.
- Sweet spot: Often 50-75% occupancy. Going from 75% to 100% rarely helps.
- Profile-driven: Use Nsight Compute to measure actual performance vs. occupancy.
Tools
- CUDA Occupancy Calculator: Spreadsheet/API that computes occupancy from kernel resource usage.
- `cudaOccupancyMaxPotentialBlockSize()`: API to auto-select block size for max occupancy.
- Nsight Compute: Reports achieved occupancy, register/shared memory usage, and limiting factor.
GPU occupancy optimization is a necessary but not sufficient condition for high GPU performance — while insufficient occupancy leaves the SM unable to hide memory latency, blindly maximizing occupancy at the cost of register spilling or reduced per-thread work can actually decrease performance, requiring empirical tuning guided by profiling.