Vertical Scaling

Vertical scaling (scaling up) is the practice of adding more resources to an existing machine — more CPU cores, more RAM, faster or more GPUs, faster storage — to handle increased workload. It is the simplest scaling approach but has physical upper limits.

How Vertical Scaling Works

• Upgrade Hardware: Add more GPU memory, upgrade to a faster GPU, increase RAM, add NVMe storage.
• Same Application: The application code doesn't need to change — it simply has more resources available.
• No Distribution Complexity: No need for load balancers, distributed state, or inter-node communication.

Vertical Scaling for AI/ML

• Larger GPU: Move from A10G (24GB) to A100 (80GB) to H100 (80GB) for more VRAM and faster inference.
• Multi-GPU Single Node: Use NVIDIA DGX or cloud instances with 4–8 GPUs connected via NVLink for high-bandwidth inter-GPU communication.
• More RAM: Increase system RAM to support larger batch sizes, bigger KV caches, or more concurrent requests.
• Faster Storage: Use NVMe SSDs for faster model loading and checkpointing.

Advantages

• Simplicity: No distributed systems complexity — the application runs on one machine.
• Low Latency: No network hops between components — all communication is in-memory.
• Easy Management: One machine to monitor, maintain, and debug.
• Better for Large Models: Models that require multi-GPU parallelism benefit from high-bandwidth intra-node GPU connections (NVLink) rather than inter-node networking.

Limitations

• Hardware Ceiling: Even the most powerful single machine has limits — currently an 8×H100 DGX with 640GB total GPU memory.
• Single Point of Failure: If the machine goes down, the entire service is unavailable.
• Cost Scaling: High-end hardware has non-linear cost — a 2× more powerful machine often costs 3–4× more.
• Downtime for Upgrades: Hardware upgrades typically require system downtime.

When to Choose Vertical Scaling

• Early-stage development and prototyping.
• Models that fit on a single (possibly multi-GPU) machine.
• Latency-sensitive applications where network hops are unacceptable.

Vertical scaling is often the right first step before investing in the complexity of horizontal scaling — maximize what a single machine can do before distributing.