Horizontal Scaling
Horizontal scaling (scaling out) is the practice of adding more machines to a system to handle increased load, distributing work across a growing fleet of servers. It is the primary scaling strategy for production AI systems because LLM inference is compute-intensive and single machines have physical limits.
How Horizontal Scaling Works
- Add Instances: Deploy additional server instances running the same service.
- Load Balancer: A load balancer distributes incoming requests across all instances.
- Shared Nothing: Each instance is independent — no shared memory or local state between instances.
- Stateless Design: Services must be designed to be stateless (or use external state stores) so any instance can handle any request.
Horizontal vs. Vertical Scaling
| Aspect | Horizontal (Scale Out) | Vertical (Scale Up) |
|---|---|---|
| Method | Add more machines | Add more resources to existing machine |
| Limit | Practically unlimited | Physical hardware limits |
| Downtime | No downtime to add instances | Often requires restart |
| Cost | Many cheaper machines | Single expensive machine |
| Complexity | Higher (distributed systems) | Lower (single machine) |
Horizontal Scaling for AI/ML
- Inference Scaling: Deploy multiple GPU servers running the same model, distribute inference requests across them.
- Data Parallelism: Distribute training data across GPUs/machines, each computing gradients on a subset.
- Pipeline Parallelism: Split model layers across machines, processing different microbatches simultaneously.
- RAG Scaling: Distribute vector database shards across multiple nodes for higher query throughput.
Challenges
- Model Size: Large models (70B+ parameters) may not fit on a single GPU, requiring model parallelism before horizontal scaling applies.
- Consistency: Ensuring all instances serve the same model version during rolling deployments.
- Cost Efficiency: GPU machines are expensive, so right-sizing instance count is critical.
Horizontal scaling is the industry standard approach for handling production LLM traffic — all major AI API providers (OpenAI, Anthropic, Google) use large fleets of GPU servers behind load balancers.