Auto-Scaling

Auto-scaling is the capability to automatically adjust the number of compute resources (instances, containers, GPUs) allocated to a service based on real-time demand. It ensures that AI systems have enough capacity during peak loads while minimizing costs during low-traffic periods.

How Auto-Scaling Works

• Monitoring: Continuously track metrics like CPU usage, GPU utilization, request queue depth, latency, or token throughput.
• Scaling Policy: Define rules that trigger scaling actions — e.g., "add 2 instances when average GPU utilization exceeds 80% for 5 minutes."
• Scale Out: When demand increases, automatically launch new instances to handle the load.
• Scale In: When demand decreases, automatically terminate excess instances to reduce costs.
• Cooldown Period: Wait a defined period after a scaling action before evaluating again to prevent oscillation.

Scaling Metrics for AI Systems

• GPU Utilization: Scale when GPUs are highly utilized across existing instances.
• Request Queue Depth: Scale when pending requests exceed a threshold — indicates the current fleet can't keep up.
• Inference Latency: Scale when the p95 or p99 latency exceeds SLA targets.
• Tokens Per Second: Scale based on token throughput demand.
• Concurrent Requests: Scale based on the number of simultaneous active requests.

Auto-Scaling Challenges for LLMs

• Cold Start: Loading a large model onto a new GPU takes minutes (model download, weight loading, CUDA initialization). This makes rapid scaling difficult.
• GPU Availability: Cloud GPU instances are often scarce — scaling may fail if instances aren't available.
• Cost Spikes: Auto-scaling during unexpected demand surges can cause dramatic cost increases.
• Minimum Scale: Large models may require a minimum number of GPUs even at zero traffic, creating a high cost floor.

Solutions

• Warm Pools: Keep standby instances with models pre-loaded, ready to serve immediately.
• Scheduled Scaling: Pre-scale for known traffic patterns (business hours, marketing campaigns).
• Spot/Preemptible Instances: Use cheaper interruptible instances for burst capacity.
• Serverless Inference: Services like AWS SageMaker, Replicate, and Modal handle scaling automatically.

Auto-scaling is essential for cost-effective production AI — GPU compute is expensive, and paying for idle GPUs during off-peak hours is a significant waste.