Slurm Scheduler
HPC Job Scheduling (SLURM, PBS) is the cluster resource management and workload scheduling system that allocates compute nodes, GPUs, memory, and time to user-submitted batch jobs — enabling fair, efficient sharing of expensive HPC and AI training clusters among hundreds of users by queuing jobs, enforcing resource limits, managing priorities, and automating node allocation, with SLURM being the dominant scheduler powering most of the world's top supercomputers and AI training clusters.
Why Job Schedulers Are Needed
- HPC clusters: 100-10,000+ nodes, $10M-$1B+ investment.
- Multiple users/projects competing for resources.
- Without scheduler: Users fight for nodes, waste resources, no fairness.
- With scheduler: Automated allocation, queuing, priority, accounting.
Major Schedulers
| Scheduler | Full Name | Prevalence | Key Strength |
|---|---|---|---|
| SLURM | Simple Linux Utility for Resource Mgmt | ~65% of Top500 | Scalability, GPU support |
| PBS Pro | Portable Batch System | ~15% | Enterprise features |
| LSF | IBM Spectrum LSF | ~10% | Commercial support |
| HTCondor | High-Throughput Condor | ~5% | Cycle scavenging |
| Kubernetes | Container orchestration | Growing | Cloud-native, elastic |
SLURM Job Submission
#!/bin/bash
#SBATCH --job-name=train_llm
#SBATCH --partition=gpu
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=12
#SBATCH --mem=1024G
#SBATCH --time=72:00:00
#SBATCH --output=train_%j.log
module load cuda/12.2 nccl/2.18
srun torchrun --nproc_per_node=8 --nnodes=4 train.py# Submit job
sbatch train_job.sh
# Check queue
squeue -u $USER
# Cancel job
scancel 12345
# Check cluster status
sinfoSLURM Architecture
- slurmctld: Central controller daemon — manages queue, scheduling decisions.
- slurmd: Node daemon — runs on each compute node, launches/monitors jobs.
- slurmdbd: Database daemon — stores accounting, job history.
- srun: Launch parallel tasks within an allocation.
- sbatch: Submit batch job scripts.
- salloc: Interactive allocation request.
Scheduling Policies
| Policy | How | Best For |
|---|---|---|
| FIFO | First come, first served | Simple, small clusters |
| Fair-share | Historical usage determines priority | Multi-group fairness |
| Backfill | Small jobs fill gaps around large jobs | Improve utilization |
| Preemption | High-priority jobs evict lower | Urgent workloads |
| Gang scheduling | Time-share nodes among jobs | Oversubscription |
GPU Scheduling with SLURM
- GRES (Generic Resources): SLURM tracks GPUs as generic resources.
- Allocation: ``--gpus-per-node=8`` reserves 8 GPUs per node.
- GPU binding: SLURM sets CUDA_VISIBLE_DEVICES automatically.
- MIG (Multi-Instance GPU): SLURM can allocate MIG instances as separate resources.
- Topology-aware: SLURM can consider NVLink topology when co-locating tasks.
AI Training Cluster Patterns
- Large-scale training: Reserve entire nodes exclusively (no sharing).
- Inference serving: Pack multiple inference jobs per node (GPU sharing).
- Checkpointing: Jobs save state periodically → can be preempted and restarted.
- Elastic training: Jobs can scale up/down as resources become available.
HPC job scheduling is the operating system of supercomputing — SLURM and similar schedulers transform a collection of individual compute nodes into a coherent, shared computing resource that can run thousands of concurrent jobs while ensuring fair access, efficient utilization, and the multi-node coordination essential for training modern AI models that span hundreds of GPUs.