Latency Monitoring

Latency monitoring is the practice of continuously tracking the time it takes for an AI system to process requests and deliver responses. For LLM applications, latency directly impacts user experience — slow responses feel broken, while fast responses feel like natural conversation.

Key Latency Metrics

• TTFT (Time to First Token): Time from request submission to receiving the first token of the response. Critical for perceived speed in streaming applications.
• TPOT (Time Per Output Token): Average time to generate each subsequent token. Determines the speed of streaming text appearance.
• Total Latency: End-to-end time from request to complete response. Important for non-streaming and API-to-API calls.
• Queue Wait Time: Time spent waiting in the request queue before inference begins.
• Preprocessing Latency: Time for input validation, tokenization, and prompt construction.
• Retrieval Latency: Time for RAG vector search and document retrieval.

Percentile Metrics

• p50 (Median): The typical user experience — 50% of requests are faster than this.
• p95: 95% of requests are faster — captures most users' experience.
• p99: 99% of requests are faster — captures the worst common experience.
• p99.9: Extreme tail latency — important for SLA compliance.

Why Percentiles Matter More Than Averages

Averages mask problems — a system with 100ms average latency might have p99 of 5,000ms. One in 100 users experiences a 50× slower response. Averages look fine; percentiles reveal the truth.

Monitoring Best Practices

• Set SLOs: Define Service Level Objectives (e.g., "p95 TTFT < 500ms, p99 total latency < 10s").
• Alert on SLO Breaches: Trigger alerts when latency SLOs are violated for a sustained period.
• Break Down by Component: Monitor latency at each pipeline stage to identify bottlenecks.
• Segment by Request Type: Simple queries vs. complex reasoning, short vs. long responses.
• Dashboard Visualization: Time-series graphs of p50/p95/p99 with deployment annotations.

Common Latency Issues in LLM Systems

• Cold Start: First request after scaling up is slow due to model loading.
• Long Contexts: Latency scales with context length (quadratically for attention).
• Batch Contention: Large batch sizes improve throughput but increase individual request latency.

Latency monitoring is the most user-visible metric for AI applications — users forgive occasional errors but not consistent slowness.