Kv Cache

KV cache stores computed key-value pairs to accelerate autoregressive LLM inference. How it works: During generation, each token attends to all previous tokens. Rather than recomputing K and V for all past tokens, cache and reuse them. Only compute K, V for the new token. Memory cost: Cache grows linearly with sequence length and batch size: batch_size × num_layers × 2 × seq_len × hidden_dim × precision_bytes. For 70B model with 32K context, can be 40GB+. Optimization techniques: KV cache quantization (FP8, INT8), paged attention (vLLM) for dynamic allocation, sliding window for bounded memory, grouped-query attention reduces K, V heads, shared KV layers. Implementation: Pre-allocate for max sequence length or dynamic growth. Store per-layer. Handle variable batch sizes. Impact: Enables 10-100x faster generation vs naive recomputation. Critical for production LLM serving. Memory-speed trade-off: Larger caches enable faster generation but limit batch size. Optimize based on latency vs throughput requirements.