Kv Cache Management

KV cache management is the process of efficiently storing, reusing, and evicting the key-value pairs computed during transformer attention in LLM inference. Each time a token is generated, the model computes attention over all previous tokens — storing these KV pairs in a cache avoids redundant recomputation and is essential for efficient autoregressive generation.

How the KV Cache Works

• During Generation: Each transformer layer computes key (K) and value (V) vectors for each token. These are stored in the KV cache.
• Autoregressive Reuse: When generating the next token, the model only computes K and V for the new token, then concatenates them with the cached K and V from all previous tokens to compute attention.
• Without KV Cache: The model would need to reprocess the entire sequence for every new token — making generation O(n²) instead of O(n).

Memory Challenge

The KV cache grows linearly with sequence length and linearly with model size:
- For a 70B parameter model with 128K context: the KV cache can consume 40+ GB of GPU memory per request.
- For batch serving, KV cache memory scales with batch_size × sequence_length, often becoming the primary memory bottleneck.

Management Techniques

• Paged Attention (vLLM): Manages KV cache as virtual memory pages, eliminating fragmentation and enabling efficient memory sharing across requests.
• Multi-Query Attention (MQA): Shares K and V heads across attention heads, reducing KV cache size by the number of heads (e.g., 8× reduction).
• Grouped-Query Attention (GQA): Groups multiple query heads to share K and V heads — a middle ground between MHA and MQA. Used in Llama 2 and later.
• KV Cache Compression: Quantize cached K and V values to lower precision (FP16 → INT8 or INT4) to reduce memory.
• Sliding Window Attention: Only cache the last N tokens, limiting memory to a fixed window. Used in Mistral models.
• Token Eviction (H2O, StreamLLM): Evict less important KV entries based on attention scores to maintain a fixed cache budget.

Efficient KV cache management is the single most impactful optimization for LLM serving throughput and is the core innovation behind high-performance inference engines like vLLM, TensorRT-LLM, and SGLang.