Page-Attention

Page-attention is the paged attention mechanism that stores KV cache in fixed-size memory blocks to reduce fragmentation and enable efficient dynamic batching - it is a key innovation in high-throughput LLM serving systems.

What Is Page-attention?

• Definition: Attention runtime that manages KV tensors using virtual-memory-like paging abstractions.
• Memory Layout: Requests map to pages, allowing non-contiguous physical allocation and fast reuse.
• Serving Benefit: Reduces allocator overhead and improves utilization under variable request lengths.
• System Context: Commonly associated with modern engines such as vLLM.

Why Page-attention Matters

• Fragmentation Reduction: Fixed pages avoid severe memory waste from variable-length sequences.
• Batching Efficiency: Paged layouts support dynamic request interleaving and continuous batching.
• Latency Stability: Allocator predictability lowers p95 spikes under mixed traffic.
• Capacity Gains: Higher effective memory utilization increases concurrent session count.
• Operational Simplicity: Page-level accounting improves debugging and resource governance.

How It Is Used in Practice

• Page Size Tuning: Choose block size based on sequence profile and hardware characteristics.
• Runtime Instrumentation: Track page occupancy, churn, and compaction pressure.
• Scheduler Integration: Coordinate paging with batching and priority policies for stable throughput.

Page-attention is a foundational memory-management technique for modern inference engines - paged attention enables scalable decode throughput with better memory utilization.