Tokens per Second

Tokens per second is the throughput metric for language-model workloads representing text units processed or generated per second - it is the standard speed measure for both LLM training and inference economics.

What Is Tokens per second?

• Definition: Count of tokenizer output units handled each second during training or generation.
• Training vs Inference: Training tracks processed tokens per step, while inference tracks generated-token rate.
• Determinants: Model size, context length, batching strategy, and hardware communication efficiency.
• Economic Link: Higher token throughput improves tokens-per-dollar and reduces time-to-result.

Why Tokens per second Matters

• LLM Performance Baseline: Provides common cross-model and cross-cluster speed benchmark.
• Cost Management: Token throughput strongly influences cloud spend for training and serving.
• Capacity Forecasting: Supports planning for latency and throughput service-level targets.
• Optimization Visibility: Reveals impact of kernel, quantization, and parallelism improvements.
• Product Relevance: Inference token rate directly affects user-perceived responsiveness.

How It Is Used in Practice

• Measurement Consistency: Standardize tokenizer, sequence length, and warm-up treatment when reporting.
• System Correlation: Analyze token rate alongside GPU utilization and memory bandwidth metrics.
• Policy Tuning: Adjust batching, sequence packing, and model parallel strategy for better token throughput.

Tokens per second is the operational speed currency of language-model systems - precise tracking connects engineering tuning directly to user experience and cost efficiency.