Latency

Latency optimization is the systematic reduction of response time in LLM inference — minimizing the delay between user input and AI response through techniques like quantization, KV cache optimization, speculative decoding, and model architecture choices, critical for real-time interactive applications.

What Is Latency in LLM Inference?

  • Definition: Time from request submission to complete response.
  • Components: Queue time + prefill (TTFT) + decode (TPOT × tokens).
  • Target: Interactive applications need <100ms TTFT, <50ms TPOT.
  • Challenge: Balance latency with throughput and cost.

Why Latency Matters

  • User Experience: Slow responses frustrate users (<200ms feels instant).
  • Conversational Flow: Real-time chat requires low latency.
  • Competitive Advantage: Faster AI feels smarter and more capable.
  • Use Cases: Autocomplete, coding assistants, voice need sub-second.
  • Throughput Trade-off: Lower latency often means lower throughput.

Latency Breakdown

Key Metrics:

LLM Latency Breakdown & Performance Metrics Time-To-First-Token (TTFT) · Time-Per-Output-Token (TPOT) · Prefill vs Decode Phases 1. TTFT vs TPOT Execution Timeline Inference Timeline Phases: Phase 1: Prefill (Prompt Processing) TTFT Phase Compute-Bound (Gemm) Phase 2: Autoregressive Decode TPOT (Mem-Bound) End-to-End Latency = TTFT + (TPOT × Output Tokens) Core Latency Definitions: • TTFT: Time to first token (perceived responsiveness) • TPOT: Time per output token (reading speed ~30-80 tok/s) Prefill = GPU Compute Limited | Decode = HBM Bandwidth Limited 2. Latency Reduction Techniques Optimization Stack: 1. Speculative Decoding: Draft model predicts K tokens → Target model verifies in 1 step. 2. FP8 / INT4 Quantization: Halves memory footprint → 2x faster decode bandwidth. 3. Chunked Prefill: Splits long prompts to prevent spiking TPOT latency. Target Production SLAs: Interactive Chat: TTFT < 400ms | TPOT < 25ms/tok Code Autocomplete: TTFT < 150ms Optimized by vLLM, TensorRT-LLM, & SGLang

Latency Targets by Use Case:

Use Case           | TTFT Target | TPOT Target
-------------------|-------------|-------------
Voice assistant    | <300ms      | <40ms
Chat interface     | <500ms      | <50ms
Code completion    | <200ms      | <30ms
Batch processing   | N/A         | Maximize throughput

Optimization Techniques

Quantization:
- INT8/INT4 weights reduce memory bandwidth requirements.
- 2-4× speedup with minimal quality loss.
- AWQ, GPTQ, bitsandbytes implementations.
- FP8 on modern GPUs (H100) for best speed/quality.

KV Cache Optimizations:
- PagedAttention: Reduce memory fragmentation.
- Quantized KV: INT8/INT4 cache values.
- Prefix Caching: Reuse KV for common system prompts.
- Sliding Window: Limit attention span (Mistral).

Speculative Decoding:

1. Small "draft" model generates N candidate tokens quickly
2. Large "target" model verifies all N in parallel
3. Accept matching tokens, reject at first mismatch
4. Net speed: ~2-3× faster for matching drafts

Example: 7B draft + 70B verify = faster than 70B alone

Model Architecture:
- GQA/MQA: Fewer Key-Value heads = faster decode.
- Smaller Models: Latency scales with model size.
- MoE: Only activate subset of parameters.
- Early Exit: Stop at confident predictions.

Attention Optimizations:
- Flash Attention: Fused kernel, IO-aware.
- Flash Attention 2/3: Further optimized versions.
- Paged Attention: Memory-efficient for variable lengths.

Infrastructure Optimizations

Hardware Selection:

GPU             | Memory BW  | Typical TPOT (7B)
----------------|------------|------------------
RTX 4090        | 1 TB/s     | 15-25ms
A100 (80GB)     | 2 TB/s     | 10-15ms
H100 (80GB)     | 3.35 TB/s  | 6-10ms
H200 (141GB)    | 4.8 TB/s   | 4-7ms

Network & Infrastructure:
- Deploy close to users (edge, CDN).
- Use gRPC over REST for lower overhead.
- Connection pooling, keep-alive.
- Streaming responses (SSE) for perceived speed.

Measurement & Monitoring

  • P50/P95/P99 Latencies: Distribution matters, not just average.
  • Real-time Dashboards: Monitor TTFT, TPOT, queue depth.
  • Load Testing: Stress test before production.
  • Alerting: Detect latency regressions quickly.

Latency optimization is essential for user-facing AI applications — the difference between a 500ms and 2000ms response time determines whether AI feels like a helpful assistant or a frustrating bottleneck, making latency engineering critical for any interactive AI product.

Take latency further

Ask the copilot about this term, or have our engineers assess it against your process.

Ask CFSGPT about this → Request a technical assessment → Open the CFS app