Timeout

Timeouts and SLAs for LLM Services

Why Timeouts Matter
LLM operations can be slow. Without proper timeouts, resources get exhausted and user experience suffers.

Timeout Layers

Client timeout (shortest)
    |
    v
API Gateway timeout
    |
    v
Application timeout
    |
    v
LLM Provider timeout (longest)

Setting Timeouts

import httpx
import asyncio

async def call_with_timeout(prompt, timeout=30):
    async with httpx.AsyncClient(timeout=timeout) as client:
        try:
            response = await client.post(
                "https://api.openai.com/v1/completions",
                json={"prompt": prompt},
                timeout=timeout
            )
            return response.json()
        except httpx.TimeoutException:
            raise TimeoutError(f"LLM call exceeded {timeout}s")

Deadline Propagation
Pass remaining time through the stack:

def process_with_deadline(request):
    deadline = time.time() + 30  # 30 second deadline

    # Pass remaining time to each step
    remaining = deadline - time.time()
    result1 = step1(request, timeout=remaining * 0.3)

    remaining = deadline - time.time()
    result2 = step2(result1, timeout=remaining * 0.5)

    remaining = deadline - time.time()
    return step3(result2, timeout=remaining)

SLA Levels
| Tier | P50 | P99 | Timeout |
|------|-----|-----|---------|
| Fast | 1s | 5s | 10s |
| Standard | 5s | 30s | 60s |
| Batch | 30s | 120s | 300s |

Monitoring SLA Compliance

import time
from prometheus_client import Histogram

latency = Histogram(
    "llm_request_latency_seconds",
    "LLM request latency",
    buckets=[0.5, 1, 2, 5, 10, 30, 60]
)

@latency.time()
def call_llm(prompt):
    return llm.generate(prompt)

# Alert on SLA breach
def check_sla(latencies, sla_p99=30):
    p99 = percentile(latencies, 99)
    if p99 > sla_p99:
        alert("P99 latency exceeds SLA!")

Timeout Strategies
| Strategy | Description |
|----------|-------------|
| Hard timeout | Cancel after limit |
| Soft timeout | Return partial result |
| Adaptive | Adjust based on load |
| Budget-based | Split time across steps |

Handling Timeout

async def call_with_retry_on_timeout(prompt, max_attempts=2):
    for attempt in range(max_attempts):
        try:
            return await call_with_timeout(prompt, timeout=30)
        except TimeoutError:
            if attempt == max_attempts - 1:
                # Return cached or fallback
                return get_fallback_response(prompt)
            await asyncio.sleep(1)

Best Practices
- Set timeouts at every layer
- Make timeouts configurable
- Log timeout occurrences
- Monitor and alert on SLA breaches
- Consider tail latency (P99, P999)