Model Serving Inference

ML Model Serving and Inference Optimization is the engineering discipline of deploying trained models into production systems that process real-time requests at scale — where the challenges shift from training accuracy to inference latency, throughput, cost, and reliability, requiring specialized optimization techniques (quantization, batching, graph optimization, hardware-specific compilation) to achieve millisecond-level response times at thousands of requests per second.

The Inference Challenge

Training is batch-oriented (maximize GPU utilization over hours/days). Inference is request-oriented (minimize latency for each query while maximizing throughput). A model that takes 50 ms per request on a V100 GPU needs to serve 1,000 requests/second — requiring batching, pipelining, multi-GPU deployment, and aggressive optimization.

Model Optimization Techniques

• Quantization: Reduce weight and activation precision from FP32 to FP16/INT8/INT4. Post-Training Quantization (PTQ) converts a trained model without retraining. INT8 quantization provides ~2-4x speedup on GPUs (Tensor Core INT8) and CPUs (VNNI). For LLMs, GPTQ and AWQ achieve 4-bit quantization with minimal quality loss.
• Graph Optimization: Fuse operations (Conv+BN+ReLU → single kernel), eliminate redundant operations, constant folding. TensorRT, ONNX Runtime, and XLA apply these automatically.
• Pruning: Remove weights (unstructured) or entire neurons/channels (structured) that contribute minimally to output. Structured pruning directly reduces computation; unstructured pruning requires sparse-aware hardware.
• Knowledge Distillation: Train a smaller model to mimic the larger one. DistilBERT is 60% the size, 2x faster, 97% accuracy of BERT.

Serving Frameworks

• NVIDIA Triton Inference Server: Multi-framework (PyTorch, TensorFlow, ONNX, TensorRT), multi-model serving with dynamic batching, model ensembles, and GPU sharing. The standard for GPU-based inference at scale.
• ONNX Runtime: Cross-platform inference engine. Export models to ONNX format, optimize with graph transformations and execution providers (CUDA, TensorRT, DirectML, CoreML). Single model format deployable across GPUs, CPUs, and edge devices.
• vLLM: High-throughput LLM serving engine using PagedAttention for memory-efficient KV cache management. Achieves 2-4x higher throughput than naive HuggingFace serving.
• TensorRT-LLM: NVIDIA's optimized LLM inference library. In-flight batching, quantization-aware kernels, tensor parallelism for multi-GPU LLM serving.

Key Serving Patterns

• Dynamic Batching: Accumulate incoming requests and batch them together for GPU processing. Wait up to a configurable deadline (e.g., 10 ms) to form larger batches. Throughput increases dramatically (8x-32x) with batching at modest latency cost.
• Continuous/In-Flight Batching: For autoregressive LLMs, new requests join the batch as existing requests complete tokens. Avoids waiting for the longest sequence in the batch to finish. vLLM and TensorRT-LLM implement this.
• Model Parallelism for Serving: Large models that exceed single-GPU memory are split across GPUs using tensor or pipeline parallelism. Inference-time parallelism trades latency for the ability to serve models that won't fit on one device.

ML Model Serving is the bridge between trained models and real-world impact — the engineering that transforms a research artifact consuming GPU-hours for a single prediction into a production system handling millions of requests per day at sub-100-millisecond latency and dollars-per-million-requests cost.