Load Balancer

Load Balancing for ML Services

Why Load Balance?
Distribute traffic across multiple model instances for reliability, scalability, and efficient resource utilization.

Load Balancing Strategies

Round Robin
Distribute requests evenly:

upstream llm_servers {
    server llm1.example.com:8000;
    server llm2.example.com:8000;
    server llm3.example.com:8000;
}

Least Connections
Route to server with fewest active connections:

upstream llm_servers {
    least_conn;
    server llm1.example.com:8000;
    server llm2.example.com:8000;
}

Weighted Distribution
Allocate based on server capacity:

upstream llm_servers {
    server gpu-a100.example.com:8000 weight=10;
    server gpu-t4.example.com:8000 weight=3;
}

Nginx Configuration

http {
    upstream llm_api {
        least_conn;
        server 10.0.0.1:8000 weight=5;
        server 10.0.0.2:8000 weight=5;

        # Health checks
        keepalive 32;
    }

    server {
        listen 80;

        location /api/v1/completions {
            proxy_pass http://llm_api;
            proxy_http_version 1.1;
            proxy_set_header Connection "";

            # Timeouts for LLM
            proxy_read_timeout 300s;
            proxy_connect_timeout 10s;
        }
    }
}

ML-Specific Considerations
| Consideration | Solution |
|---------------|----------|
| Long requests | Extended timeouts |
| Streaming | HTTP/1.1, chunked transfer |
| GPU memory | Session affinity if stateful |
| Warm-up | Gradual traffic increase |

Health Checks

upstream llm_servers {
    server llm1:8000;
    server llm2:8000;

    # Active health check
    health_check interval=5s fails=2 passes=1;
}

Session Affinity
For stateful models (e.g., with KV cache):

upstream llm_servers {
    ip_hash;  # Same IP -> same server
    server llm1:8000;
    server llm2:8000;
}

Cloud Load Balancers
| Cloud | Service |
|-------|---------|
| AWS | ALB, NLB |
| GCP | Cloud Load Balancing |
| Azure | Load Balancer |
| Cloudflare | Load Balancing |

Best Practices
- Use health checks to remove unhealthy servers
- Set appropriate timeouts for LLM operations
- Consider GPU utilization in routing
- Implement graceful shutdown