Request Scheduling

Request scheduling in inference servers manages the queue of incoming model requests to optimize throughput, latency, and fairness according to service level agreements (SLAs). Scheduling policies: FCFS (First-Come First-Served), LCFS (Last-Come, good for real-time dropping), and Shortest Job First (if duration known). Priority queues: VIP users or critical endpoints get faster processing. Batching integration: scheduler groups compatible requests into batches; waits for batch window or max size. Preemption: pause long-running request to serve high-priority short one (requires sophisticated memory management). Fairness: ensure heavy users don't starve others (Max-Min fairness). Overload handling: load shedding (drop requests) when queue full or latency targets unreachable; better to fail fast than timeout. Concurrency control: limit max simultaneous requests to prevent OOM. Multi-model: schedule requests across different models sharing same GPU(s); model switching overhead considerations. Smart scheduling improves perceived performance and hardware utilization without changing the model itself.