L2 Cache

L2 cache is the shared on-chip cache level that serves all streaming multiprocessors before traffic reaches high-latency HBM - it acts as a global reuse and coherence layer for data exchanged across blocks and kernels on the same GPU.

What Is L2 cache?

• Definition: Unified cache between per-SM caches and off-chip memory, visible to all compute units.
• Role: Absorbs repeated global-memory accesses and reduces expensive HBM transactions.
• Coherence Point: Writes from one SM can be observed by others through L2-backed memory consistency behavior.
• Capacity Context: Larger than L1 or shared memory but slower than those nearest compute tiers.

Why L2 cache Matters

• Bandwidth Relief: High L2 hit rate reduces pressure on external memory channels.
• Cross-SM Reuse: Common tensors accessed by many blocks can be served with lower latency.
• Kernel Throughput: Memory-heavy kernels often scale with effective L2 behavior.
• Energy Efficiency: On-chip reuse consumes less energy than repeated off-chip fetches.
• System Balance: Optimized L2 utilization improves overall compute to memory balance.

How It Is Used in Practice

• Access Locality: Design kernels so nearby threads and successive blocks touch overlapping address regions.
• Working-Set Tuning: Adjust tile size and launch strategy to fit hot data within L2 residency windows.
• Profiling: Track L2 hit rate and throughput counters to identify memory hierarchy bottlenecks.

L2 cache is the shared memory-traffic stabilizer for GPU-wide execution - strong L2 locality can significantly raise end-to-end kernel performance.