Hyperparameter Optimization

Hyperparameter Optimization and AutoML — Automating the Design of Deep Learning Systems

Hyperparameter optimization (HPO) and Automated Machine Learning (AutoML) systematically search for optimal model configurations, replacing manual trial-and-error with principled algorithms. These techniques automate decisions about learning rates, architectures, regularization, and training schedules, enabling practitioners to achieve better performance with less expert intervention.

— Search Space Definition and Strategy —

Effective hyperparameter optimization begins with carefully defining what to search and how to explore:

• Continuous parameters include learning rate, weight decay, dropout probability, and momentum coefficients
• Categorical parameters encompass optimizer choice, activation functions, normalization types, and architecture variants
• Conditional parameters create hierarchical search spaces where some choices depend on others
• Log-scale sampling is essential for parameters spanning multiple orders of magnitude like learning rates
• Search space pruning removes known poor configurations to focus computational budget on promising regions

— Optimization Algorithms —

Various algorithms balance exploration of the search space with exploitation of promising configurations:

• Grid search exhaustively evaluates all combinations on a predefined grid but scales exponentially with dimensions
• Random search samples configurations uniformly and often outperforms grid search in high-dimensional spaces
• Bayesian optimization builds a probabilistic surrogate model of the objective function to guide intelligent sampling
• Tree-structured Parzen Estimators (TPE) model the density of good and bad configurations separately for efficient search
• Evolutionary strategies maintain populations of configurations that mutate and recombine based on fitness scores

— Neural Architecture Search (NAS) —

NAS extends hyperparameter optimization to automatically discover optimal network architectures:

• Cell-based search designs repeatable building blocks that are stacked to form complete architectures
• One-shot NAS trains a single supernetwork containing all candidate architectures and evaluates subnetworks by weight sharing
• DARTS relaxes the discrete architecture search into a continuous optimization problem using differentiable relaxation
• Hardware-aware NAS incorporates latency, memory, and energy constraints directly into the architecture search objective
• Zero-cost proxies estimate architecture quality without training using metrics computed at initialization

— Practical AutoML Systems and Frameworks —

Production-ready tools make hyperparameter optimization accessible to practitioners at all skill levels:

• Optuna provides a define-by-run API with pruning, distributed optimization, and visualization capabilities
• Ray Tune offers scalable distributed HPO with support for diverse search algorithms and early stopping schedulers
• Auto-sklearn wraps scikit-learn with automated feature engineering, model selection, and ensemble construction
• BOHB combines Bayesian optimization with Hyperband's early stopping for efficient multi-fidelity optimization
• Weights & Biases Sweeps integrates hyperparameter search with experiment tracking for reproducible optimization

Hyperparameter optimization and AutoML have democratized deep learning by reducing the expertise barrier for achieving state-of-the-art results, enabling both researchers and practitioners to systematically explore vast configuration spaces and discover optimal model designs that would be impractical to find through manual experimentation alone.