Memory

KV cache is the memory buffer storing previously computed key and value tensors during autoregressive LLM inference — avoiding redundant computation by caching intermediate results, but requiring significant GPU memory that scales with sequence length and batch size, making cache management critical for efficient serving.

What Is KV Cache?

• Definition: Cached key-value pairs from attention computation.
• Purpose: Avoid recomputing previous token representations each step.
• Growth: Linear with sequence length × layers × batch size.
• Challenge: Major memory bottleneck for long contexts and batching.

Why KV Cache Matters

• Efficiency: Without caching, cost would be O(n²) per token.
• Memory: Can exceed model weights for long sequences.
• Throughput: KV cache size limits batch size.
• Long Context: 100K+ contexts need cache optimization.
• Cost: Memory management directly impacts inference cost.

How KV Cache Works

Autoregressive Generation:

Without KV Cache (naive):
Step 1: Compute K,V for [token1]
Step 2: Recompute K,V for [token1, token2]
Step 3: Recompute K,V for [token1, token2, token3]
        ...each step recomputes everything!

With KV Cache:
Step 1: Compute K,V for [token1], cache it
Step 2: Compute K,V for [token2] only, append to cache
Step 3: Compute K,V for [token3] only, append to cache
        ...only compute new token each step

Memory Layout:

Memory Calculation

KV Cache Size = 2 × L × H × S × B × dtype_size

Where:
- 2 = keys and values
- L = number of layers
- H = hidden dimension
- S = sequence length
- B = batch size
- dtype = FP16 (2 bytes) or FP8 (1 byte)

Example (Llama-70B, 4K context, batch=1, FP16):
= 2 × 80 layers × 8192 hidden × 4096 seq × 1 × 2 bytes
= 10.7 GB per sequence!

Batch of 8 = 86 GB just for KV cache

KV Cache Optimizations

PagedAttention (vLLM):

Traditional: Contiguous memory per sequence (fragmentation)
PagedAttention: Memory in fixed-size pages (like OS virtual memory)

Benefits:
- No fragmentation
- Share pages across requests (prefix caching)
- Dynamic allocation
- 2-4× higher throughput

Quantized KV Cache:

Store cache in INT8 or INT4 instead of FP16
Memory reduction: 2-4×
Quality impact: Minimal for most models

FP16: 16 bits/value
INT8: 8 bits/value (2× reduction)
INT4: 4 bits/value (4× reduction)

Grouped Query Attention (GQA):

Standard MHA: heads_k = heads_q = 32
GQA: heads_k = 8, heads_q = 32

KV cache 4× smaller with GQA
Most modern models use GQA

Multi-Query Attention (MQA):

MQA: heads_k = 1, heads_q = 32
Even smaller cache, some quality trade-off

Prefix Caching:

System prompt: "You are a helpful assistant..."
This is same across requests → compute once, share KV

First request: Compute full KV for system prompt
Later requests: Reuse cached system prompt KV
Savings: Skip prefill for common prompts

Memory Comparison

Optimization      | Memory | Implementation
------------------|--------|------------------
Baseline FP16     | 100%   | Standard
INT8 KV           | 50%    | Most frameworks
INT4 KV           | 25%    | Some frameworks
GQA (4 groups)    | 25%    | Model architecture
GQA + INT8        | 12.5%  | Combined
PagedAttention    | ~60-80%| vLLM (less fragmentation)

Sliding Window Attention

Instead of attending to full history:
- Only attend to last W tokens
- KV cache capped at W entries
- Used in Mistral (W=4096)

Trade-off: Bounded memory vs. long-range attention

KV cache management is the critical bottleneck in LLM inference — as context windows grow to 100K+ tokens and users expect real-time responses, efficient cache strategies determine whether serving is practical and affordable, making KV optimization essential infrastructure.