Hyperparameter

Hyperparameter tuning systematically searches for optimal values of learning rate, batch size, regularization, and architecture choices, using grid search, random search, Bayesian optimization, or population-based approaches to maximize model performance. Common hyperparameters: learning rate (most important), batch size, weight decay, dropout rate, architecture choices (layers, hidden size), and optimizer settings (beta1, beta2). Grid search: exhaustive search over predefined values; expensive but thorough; exponential cost with number of hyperparameters. Random search: sample hyperparameters randomly within ranges; often more efficient than grid—finds good values faster because not all hyperparameters equally important. Bayesian optimization: model relationship between hyperparameters and performance; use model to suggest promising configurations; efficient for expensive evaluations. Population-based training (PBT): evolve population of models; copy weights from good performers, mutate hyperparameters; adaptive throughout training. Search space design: use log scale for LR and weight decay; categorical for architecture choices; appropriate ranges based on prior knowledge. Early stopping: terminate poor runs early; use successive halving (Hyperband) to allocate resources efficiently. Multi-fidelity: evaluate on small data/epochs first, full training only for promising configurations. Tools: Optuna, Ray Tune, Weights & Biases sweeps, and cloud HPO services. Reproducibility: log all hyperparameters and results; enable others to reproduce or extend. Systematic hyperparameter tuning often yields larger gains than architecture changes.