Hyperparameter Optimization
Hyperparameter Optimization (HPO) is the systematic search for the best configuration of training settings (learning rate, batch size, architecture choices, regularization) that maximizes model performance — automating what was traditionally a manual trial-and-error process, with methods ranging from simple grid search to sophisticated Bayesian optimization that can efficiently explore high-dimensional configuration spaces.
Common Hyperparameters
| Category | Parameters | Typical Range |
|---|---|---|
| Optimization | Learning rate, weight decay, momentum | LR: 1e-5 to 1e-1 |
| Architecture | Hidden size, num layers, num heads | Problem-dependent |
| Regularization | Dropout, label smoothing, data augmentation | 0.0 to 0.5 |
| Training | Batch size, epochs, warmup steps | 16 to 4096 |
| LR Schedule | Cosine, linear, step decay | Schedule type + params |
Search Strategies
Grid Search
- Evaluate all combinations of pre-specified values.
- Cost: Exponential in number of hyperparameters — $O(V^D)$ for V values per D dimensions.
- Effective only for 1-3 hyperparameters.
Random Search (Bergstra & Bengio 2012)
- Sample configurations randomly from distributions.
- Provably more efficient than grid search: Better at finding narrow optima.
- Widely used as a strong baseline.
Bayesian Optimization
- Build a surrogate model (Gaussian Process, Tree-structured Parzen Estimator) of the objective function.
- Acquisition function (Expected Improvement, UCB) selects next configuration to try.
- After each trial: Update surrogate model with new result.
- Efficient: Finds good configurations in 20-100 trials — 10-50x fewer than random search.
Multi-Fidelity Methods
- Hyperband / ASHA: Train many configurations for a few epochs → prune bad ones → train survivors longer.
- Successive halving: Start 81 configs for 1 epoch → keep top 27 for 3 epochs → top 9 for 9 epochs → top 3 for 27 epochs → best 1 for 81 epochs.
- Dramatically reduces total compute compared to full training of each configuration.
HPO Frameworks
| Framework | Backend | Highlights |
|---|---|---|
| Optuna | TPE, CMA-ES | Pythonic, pruning, visualization |
| Ray Tune | Any (Optuna, BO, PBT) | Distributed, multi-GPU support |
| Weights & Biases Sweeps | Bayes, Random, Grid | Integrated experiment tracking |
| Ax (Meta) | Bayesian (BoTorch) | Multi-objective, neural BO |
Population-Based Training (PBT)
- Run multiple training runs in parallel.
- Periodically: Poorly performing runs copy weights and hyperparameters from top performers, with random perturbation.
- Hyperparameters evolve during training — adapts LR schedule automatically.
Hyperparameter optimization is a critical but often undervalued component of ML development — a well-tuned baseline model frequently outperforms a poorly-tuned novel architecture, making systematic HPO one of the highest-ROI investments in any machine learning project.