Prefix Caching

Prefix caching stores computed KV cache for common prompt prefixes (like system prompts), enabling reuse across requests that share the same prefix to reduce first-token latency and computation for repetitive patterns. Why it matters: system prompts are often identical across requests (instructions, context documents); recomputing their KV cache wastefully repeats work. Implementation: hash prompt prefix, store computed KV cache keyed by hash, and check cache before computing. Cache hit: if prefix matches cached entry, load pre-computed KV cache and begin generation from where prefix ends. Latency reduction: time-to-first-token dramatically reduced for cache hits; no prefill computation for cached portion. Memory trade-off: storing KV caches consumes GPU/system memory; cache management needed. Cache invalidation: when system prompt changes, old cache entries become stale; versioning or TTL policies. Radix tree approach: vLLM and similar systems use radix trees to share common prefixes across even partially overlapping prompts. Page-level caching: combine with paged attention for efficient memory management of cached blocks. Use cases: chatbots (same system prompt), multi-turn conversations (shared context), and batch processing (same instructions). Production systems: vLLM, SGLang, and TensorRT-LLM support prefix caching. Prefix caching is essential optimization for production LLM serving.