Continuous Batching

Continuous Batching is an LLM serving optimization that dynamically inserts new requests into a running inference batch as soon as individual sequences complete — replacing static batching (where the entire batch waits for the longest sequence to finish) with iteration-level scheduling that fills freed GPU capacity immediately, achieving up to 20× higher throughput by eliminating the GPU idle time caused by variable-length sequence generation.

What Is Continuous Batching?

• Definition: A scheduling strategy for LLM inference where the serving system operates at the granularity of individual decoding iterations rather than complete requests — when one sequence in the batch finishes generating (hits the end-of-sequence token), a new request from the queue immediately takes its slot in the next iteration, keeping the GPU fully utilized.
• Static Batching Problem: In static (naive) batching, a batch of N requests starts together and finishes only when the longest sequence completes — if one request generates 10 tokens and another generates 2000 tokens, the GPU sits idle for the short request's slot during 1990 iterations.
• Iteration-Level Scheduling: Continuous batching makes scheduling decisions at every decoding step — checking if any sequence has finished, removing completed sequences, and inserting waiting requests into the freed slots.
• Also Called: In-flight batching, dynamic batching, or iteration-level batching — all refer to the same concept of per-iteration request management.

Why Continuous Batching Matters

• Throughput: Continuous batching achieves 5-20× higher throughput than static batching for workloads with variable output lengths — the improvement is proportional to the variance in sequence lengths.
• Latency Fairness: Short requests complete quickly without waiting for long requests in the same batch — eliminating "head-of-line blocking" where a single long generation delays all other requests.
• GPU Utilization: Keeps GPU compute units occupied at every iteration — static batching wastes GPU cycles on padding tokens for completed sequences, while continuous batching fills those slots with real work.
• Cost Efficiency: Higher throughput per GPU means fewer GPUs needed to serve the same request volume — directly reducing infrastructure cost for LLM serving.

Continuous Batching with PagedAttention

• Memory Challenge: Each active request maintains a KV cache that grows with sequence length — continuous batching requires efficient memory management to handle requests entering and leaving the batch dynamically.
• PagedAttention (vLLM): Manages KV cache memory like virtual memory pages — allocating and freeing cache blocks dynamically as requests enter and leave the batch, eliminating memory fragmentation.
• Memory Efficiency: PagedAttention + continuous batching achieves near-zero memory waste — compared to static batching which must pre-allocate maximum sequence length for every request.

Continuous batching is the essential serving optimization for production LLM deployment — dynamically managing request lifecycles at the iteration level to maximize GPU utilization and throughput, eliminating the idle time waste of static batching and enabling cost-efficient serving of variable-length LLM generation workloads.