Samples per Second

Samples per second is the throughput metric measuring how many training examples are processed each second - it is a core indicator for image and tabular workloads where progress is naturally measured in sample count.

What Is Samples per second?

• Definition: Number of individual training samples consumed by the model per second.
• Computation: Typically derived from global batch size divided by step time, aggregated across workers.
• Sensitivity: Affected by data loading speed, communication overhead, and kernel efficiency.
• Interpretation Caveat: Higher throughput is valuable only if model convergence quality remains acceptable.

Why Samples per second Matters

• Performance Tracking: Provides immediate signal of system-level throughput improvements or regressions.
• Scaling Analysis: Helps assess how close distributed training is to linear speedup.
• Cost Efficiency: More samples per second generally lowers training wall time and infrastructure cost.
• Bottleneck Diagnosis: Drops often indicate data or communication stalls rather than compute saturation.
• Capacity Planning: Useful for estimating runtime and cluster demand for new experiments.

How It Is Used in Practice

• Consistent Measurement: Report using standardized warm-up handling and averaging windows.
• Pipeline Profiling: Correlate throughput changes with dataloader and network telemetry.
• Optimization Loop: Tune batch, prefetch, and parallelism settings while monitoring convergence impact.

Samples per second is a primary throughput KPI for data-parallel training workflows - when tracked with quality metrics, it drives practical performance optimization.