Throughput

Throughput in LLM deployment measures the number of tokens or requests processed per unit time, a key metric determining serving capacity, cost efficiency, and scalability. Metrics: (1) Tokens per second (TPS)—total output tokens generated across all concurrent requests; (2) Requests per second (RPS)—completed inference requests per second; (3) Time to first token (TTFT)—latency for first output token (prefill phase); (4) Time per output token (TPOT)—average inter-token latency (decode phase). Throughput bottlenecks: (1) Prefill phase—compute-bound, processing all input tokens in parallel (matrix multiplication limited); (2) Decode phase—memory-bandwidth-bound, autoregressive token generation reads entire KV cache per token; (3) KV cache memory—limits concurrent requests (batch size). Throughput optimization: (1) Batching—process multiple requests simultaneously to saturate GPU compute; (2) Continuous batching—dynamically add/remove requests from batch as they complete; (3) PagedAttention—efficient KV cache memory management enables larger batches; (4) Tensor parallelism—split model across GPUs for faster per-request latency; (5) Quantization—smaller model footprint enables larger batch sizes; (6) Speculative decoding—draft model proposes multiple tokens, verified in parallel. Throughput vs. latency tradeoff: larger batches increase throughput but may increase per-request latency. SLO (service level objectives) define acceptable latency bounds. Hardware impact: GPU memory bandwidth (HBM3: 3.35 TB/s on H100) directly limits decode throughput; compute (FLOPS) limits prefill throughput. Scaling: horizontal scaling (more GPU instances) for more total throughput; vertical scaling (better hardware, optimization) for per-instance efficiency. Throughput optimization is critical for economic viability of LLM services—higher throughput directly translates to lower cost per token.