Continuous Batching

Continuous Batching

The Problem with Static Batching
With static batching, all requests in a batch must complete before new requests can start:

Static Batch:
Request 1: [====]         (short)
Request 2: [============] (long)
Request 3: [======]       (medium)

All must wait for Request 2 to finish.

Resources wasted while shorter requests are complete but waiting.

How Continuous Batching Works
Process requests as they complete, immediately adding new ones:

Continuous Batching:
Request 1: [====]
                ↳ Request 4: [===]
Request 2: [============]
                        ↳ Request 6: [==]
Request 3: [======]
              ↳ Request 5: [====]

Iteration-Level Scheduling
At each decoding iteration:
1. Generate one token for all active requests
2. Check if any request is complete (hit EOS or max tokens)
3. Remove completed requests
4. Add waiting requests from queue (if GPU memory available)

# Pseudocode
while requests_pending:
    # Run one forward pass for current batch
    for request in active_batch:
        new_token = model.generate_one_token(request)
        request.append(new_token)

    # Remove completed
    active_batch = [r for r in active_batch if not r.is_complete()]

    # Add new requests
    while has_capacity() and waiting_queue:
        active_batch.append(waiting_queue.pop())

Benefits
| Metric | Static Batching | Continuous Batching |
|--------|-----------------|---------------------|
| GPU Utilization | Variable | Consistently high |
| Latency (short requests) | Blocked by long | Minimal waiting |
| Throughput | Lower | 2-3x higher |
| Memory efficiency | Poor | Good (with paging) |

Implementation in Inference Servers
| Server | Support |
|--------|---------|
| vLLM | Built-in |
| TGI | Built-in |
| TensorRT-LLM | Built-in |
| Triton + TensorRT | Configurable |

Configuration Considerations

Max Batch Size

# Limit concurrent requests
max_batch_size = 64  # Adjust based on GPU memory

Preemption
When memory is tight, may need to preempt (pause) low-priority requests:

preemption_mode = "swap"  # swap to CPU, or "recompute"

Queue Management
- FIFO: First-in, first-out
- Priority: Based on request importance
- Deadline-based: Prioritize requests nearing SLA

Continuous batching is essential for production LLM serving with variable-length requests.