GPU Performance Profiling Nsight

GPU Performance Profiling encompasses systematic measurement and analysis of kernel execution, memory access patterns, and hardware utilization using Nsight tools, roofline models, and application-specific metrics to identify bottlenecks and guide optimization.

Nsight Compute and Nsight Systems Overview

  • Nsight Compute: Kernel-centric profiler. Analyzes single kernel execution: register/shared memory usage, L1/L2 cache hit rates, warp stall reasons, SM efficiency.
  • Nsight Systems: System-wide profiler. Timeline view of entire application: kernel launches, memory transfers, CPU-GPU synchronization, context switches, power consumption.
  • Guided Analysis Workflow: Nsight Compute recommends optimizations based on measured metrics (e.g., "warp occupancy 50%, increase shared memory usage to 75%").
  • Overhead: Profiling adds ~5-50% runtime overhead depending on metric set. Light profiling (SM efficiency) minimal; heavy profiling (register spills) substantial.

NVTX Annotations for Custom Metrics

  • NVTX (NVIDIA Tools Extension): API to annotate application code. Marks user-defined ranges, domains, events with custom names.
  • Range Annotation: nvtxRangePush/Pop() delineate code sections. Nsight timeline shows annotated regions, enabling user-level performance tracking.
  • Domain Separation: nvtxDomainCreate() organizes related annotations. Example: separate domains for preprocessing, compute, postprocessing.
  • Color and Category: Annotations assigned colors (visual grouping) and categories (filtering). Facilitates timeline analysis of complex multi-threaded applications.

Roofline Model for GPU Analysis

  • Roofline Concept: 2D plot of achievable GFLOP/s vs arithmetic intensity (FLOP per byte transferred). Machine peak provides "roofline" ceiling.
  • Peak Compute Roofline: GPU compute peak (theoretical FP32 FLOP/s). Ampere A100: 312 TFLOP/s peak.
  • Peak Bandwidth Roofline: GPU memory bandwidth (theoretical throughput). A100 HBM2e: 2 TB/s peak. Roofline ceiling = MIN(peak_compute, intensity × peak_bandwidth).
  • Application Characterization: Measure kernel arithmetic intensity (FLOP count / memory bytes transferred). Points below roofline indicate under-utilization.

Achieved Occupancy and Bottleneck Analysis

  • Occupancy Metric: Percentage of SM warp slots filled. Occupancy = (resident_warps / max_warps_per_sm) × 100%. Max warp/SM: 64 (Volta), 48 (Ampere).
  • Limiting Factors: Register pressure (32k limit per SM), shared memory allocation (96KB per SM), thread blocks per SM (varies by GPU).
  • Occupancy vs Performance: Higher occupancy generally improves performance (more warps hide memory latency), but not always. Some high-register kernels benefit from lower occupancy.
  • Warp Stall Reasons: Nsight reports stall causes (memory, dependency, execution resource, synchronization). Prioritize fixing most-common stall.

Memory Bandwidth Utilization

  • Effective Bandwidth: Measured memory bytes (profiler) vs theoretical peak. Typical ratios: 50-90% depending on access pattern.
  • Coalescing Efficiency: Consecutive threads accessing consecutive memory addresses coalesce into single transaction. Scattered access wastes bandwidth (cache-only reuse).
  • Bank Conflicts: Shared memory bank conflicts serialize accesses. All 32 threads accessing same bank → 32x slowdown. Proper access pattern avoids conflicts.
  • L2 Cache Effectiveness: L2 cache hit rate impacts bandwidth. Reuse distance (iterations between data access) determines cache utility.

Cache Utilization and Patterns

  • L1 Cache: Per-SM cache (32-96KB depending on config). Caches load/store operations if enabled. Bank conflicts similar to shared memory.
  • L2 Cache: Shared across all SMs (4-40 MB depending on GPU). Victim cache for L1, also receives uncached loads.
  • Hit Rate Interpretation: High L1 hit rate (>80%) indicates locality; low ratio indicates poor spatial/temporal locality.
  • Profiler L2 Analysis: Misses per 1k instructions metric. Aim for <2-5 misses/1k instructions for well-optimized kernels.

SM Efficiency and Load Balancing

  • SM Efficiency: Percentage of SM slots executing useful instructions. Idle slots due to warp stalls, divergence, or under-occupancy.
  • Warp Divergence Analysis: Branch divergence metrics show divergence frequency and impact. Serialization within warp reduces throughput.
  • Grid-Level Load Balancing: Blocks distributed unevenly → some SMs idle while others compute. Profiler shows block-per-SM histogram.
  • Dynamic Parallelism Overhead: Child kernels launched from kernel require synchronization overhead. Impacts SM efficiency if child kernels small.

Optimization Workflows

  • Memory-Bound Analysis: If roofline point below bandwidth line, kernel memory-bound. Optimize: improve coalescing, increase data reuse, prefetching.
  • Compute-Bound Analysis: If roofline point below compute line, kernel compute-bound. Optimize: reduce instruction count, use tensor cores, improve ILP.
  • Iterative Refinement: Profile → identify bottleneck → optimize → re-profile. Typical 5-10 iteration cycle for 2-5x speedup.

Take gpu performance profiling nsight further

Ask the copilot about this term, or have our engineers assess it against your process.

Ask CFSGPT about this → Request a technical assessment → Open the CFS app