Batch Processing Optimization

Batch Processing Optimization is the practice of maximizing throughput and resource utilization when processing multiple inference requests simultaneously — through careful batch size selection, padding strategies, memory management, and scheduling policies that balance GPU utilization, memory constraints, and latency requirements to achieve optimal cost-efficiency for offline and high-throughput workloads.

Batch Size Selection:
- GPU Utilization: larger batches improve GPU utilization by amortizing kernel launch overhead and increasing arithmetic intensity; utilization typically plateaus at batch size 32-128 depending on model size and GPU memory
- Memory Constraints: batch size limited by GPU memory; memory usage = model_weights + batch_size × (activations + gradients); for inference (no gradients), can use 2-4× larger batches than training
- Latency vs Throughput Trade-off: larger batches increase throughput (requests/second) but also increase per-request latency; batch_size=1 minimizes latency, batch_size=max_memory maximizes throughput; application requirements determine optimal point
- Optimal Batch Size Search: profile throughput at batch sizes [1, 2, 4, 8, 16, 32, 64, 128, ...]; plot throughput vs batch size; select batch size where throughput plateaus (diminishing returns beyond this point)

Padding and Sequence Length Handling:
- Static Padding: pads all sequences to maximum length in batch; simple but wasteful for variable-length inputs; batch with lengths [10, 50, 100, 500] pads all to 500, wasting 85% of computation
- Bucketing: groups sequences into length buckets (0-64, 64-128, 128-256, ...); processes each bucket separately with appropriate padding; reduces wasted computation by 50-80% compared to static padding
- Pack and Unpack: concatenates sequences into single long sequence without padding; processes as single batch; unpacks outputs to original sequences; eliminates padding overhead but requires custom attention masks
- Dynamic Shape Batching: batches sequences of similar length together; minimizes padding within each batch; requires sorting or binning incoming requests by length

Memory Management:
- Activation Checkpointing: recomputes activations during backward pass instead of storing; not applicable to inference (no backward pass) but relevant for training large batches
- Gradient Accumulation: simulates large batch by accumulating gradients over multiple small batches; enables training with effective batch size larger than GPU memory allows; inference equivalent is processing large dataset in chunks
- Mixed Precision: uses FP16 or BF16 for activations, FP32 for weights; reduces memory usage by 50% for activations; enables 1.5-2× larger batch sizes; requires hardware support (Tensor Cores)
- Memory Pooling: pre-allocates memory pools to avoid repeated allocation/deallocation; reduces memory fragmentation; PyTorch caching allocator and TensorFlow BFC allocator implement this

Parallel Batch Processing:
- Data Parallelism: splits batch across multiple GPUs; each GPU processes subset of batch; no communication during forward pass; all-reduce gradients during training (not needed for inference)
- Multi-Stream Processing: uses multiple CUDA streams to overlap computation and memory transfer; stream 1 processes batch while stream 2 loads next batch; hides data transfer latency
- Pipeline Parallelism: different layers on different GPUs; processes multiple batches in pipeline; batch 1 in layer 1, batch 2 in layer 2, etc.; improves GPU utilization but adds complexity
- Asynchronous Processing: submits batches to GPU asynchronously; CPU continues preparing next batch while GPU processes current batch; overlaps CPU and GPU work

Batching Strategies for Different Workloads:
- Offline Batch Processing: processes large dataset (millions of samples); maximizes throughput, latency not critical; use largest batch size that fits in memory; process dataset in parallel across multiple GPUs
- Online Serving with Batching: accumulates requests over short time window (1-10ms); processes accumulated requests as batch; balances latency and throughput; dynamic batching in TorchServe, Triton
- Streaming Processing: processes continuous stream of data; maintains steady-state batch size; buffers incoming data to form batches; used for video processing, real-time analytics
- Priority-Based Batching: high-priority requests processed in smaller batches (lower latency); low-priority requests batched more aggressively (higher throughput); requires separate queues and scheduling

Autoregressive Generation Batching:
- Static Batching: all sequences generate same number of tokens; wastes computation when some sequences finish early (EOS token); simple but inefficient
- Dynamic Batching with Early Stopping: removes finished sequences from batch; batch size decreases over time; more efficient but requires dynamic shape handling
- Continuous Batching (Iteration-Level): adds new sequences to batch as others finish; maintains constant batch size; maximizes GPU utilization; vLLM, TGI implement this; 10-20× throughput improvement
- Speculative Batching: batches draft model generation and verification separately; draft model uses large batch (cheap), verification uses smaller batch (expensive); optimizes for different computational characteristics

Throughput Optimization Techniques:
- Kernel Fusion: fuses multiple operations into single kernel; reduces memory traffic and kernel launch overhead; Conv+BN+ReLU fusion common; 1.5-2× speedup for memory-bound operations
- Operator Scheduling: reorders operations to maximize parallelism; independent operations executed concurrently; requires careful dependency analysis
- Quantization: INT8 quantization enables 2× larger batch sizes (half the memory per activation); 2-4× throughput improvement from both larger batches and faster compute
- Pruning: structured pruning reduces memory per sample; enables larger batch sizes; 30-50% pruning allows 1.5-2× larger batches

Profiling and Optimization:
- Throughput Profiling: measure samples/second at various batch sizes; identify optimal batch size where throughput plateaus; consider both GPU and CPU bottlenecks
- Memory Profiling: track peak memory usage vs batch size; identify memory bottlenecks (activations, weights, KV cache); optimize memory layout and allocation
- Bottleneck Analysis: profile to identify compute-bound vs memory-bound operations; compute-bound benefits from larger batches (amortize overhead); memory-bound benefits from kernel fusion and quantization
- End-to-End Latency: measure total latency including data loading, preprocessing, inference, and postprocessing; optimize entire pipeline, not just model inference

Framework-Specific Features:
- PyTorch DataLoader: multi-process data loading with prefetching; pin_memory for faster CPU-to-GPU transfer; num_workers=4-8 typical; persistent_workers reduces process spawn overhead
- TensorFlow tf.data: parallel data loading and preprocessing; prefetch() overlaps data loading with computation; map() with num_parallel_calls for parallel preprocessing
- ONNX Runtime: dynamic batching and shape inference; optimized execution providers for different hardware; supports INT8 quantization and graph optimization
- TensorRT: automatic batch size optimization; layer fusion and precision calibration; dynamic shape support for variable batch sizes

Batch processing optimization is the key to cost-effective AI deployment at scale — maximizing GPU utilization and throughput through intelligent batching, padding, and scheduling strategies that can reduce inference costs by 10-100× compared to naive single-sample processing, making the difference between economically viable and prohibitively expensive AI services.