Random Search
Random Search is a hyperparameter optimization method that samples random combinations from specified distributions — proven by Bergstra & Bengio (2012) to be more efficient than Grid Search for most ML problems because it explores more unique values of important hyperparameters, can be stopped at any time with a "good enough" result, is embarrassingly parallel (every trial is independent), and requires no assumptions about the objective function landscape.
What Is Random Search?
- Definition: A hyperparameter tuning strategy that randomly samples configurations from the search space (e.g., learning rate drawn from log-uniform [1e-5, 1e-1], batch size drawn from {16, 32, 64, 128}) and evaluates each combination independently, keeping the best result.
- The Key Insight: In high-dimensional hyperparameter spaces, not all hyperparameters matter equally. Learning rate might determine 80% of performance while weight decay matters only 5%. Grid Search wastes time testing many weight decay values while keeping learning rate fixed. Random Search samples more unique learning rate values per trial.
- Why It Beats Grid Search: For a grid of 9 points (3×3), Grid Search tries only 3 unique values per dimension. Random Search with 9 points tries 9 unique values per dimension — 3× more exploration of each axis.
Random Search vs Grid Search (Visual Explanation)
| Dimension | Grid Search (3×3 = 9 trials) | Random Search (9 trials) |
|---|---|---|
| Learning Rate | Tests 3 values: [0.001, 0.01, 0.1] | Tests 9 unique values: [0.0023, 0.0071, 0.014, ...] |
| Weight Decay | Tests 3 values: [1e-4, 1e-3, 1e-2] | Tests 9 unique values: [3.2e-4, 7.1e-4, ...] |
| Coverage of LR | 3 unique values ❌ | 9 unique values ✓ |
If learning rate is the important parameter, Random Search explores 3× more of its range.
When to Use Which
| Method | Best For | Pros | Cons |
|---|---|---|---|
| Grid Search | ≤3 hyperparams, known good ranges | Exhaustive, reproducible | Exponential cost, wastes time on unimportant params |
| Random Search | 3-10 hyperparams, broad ranges | More efficient, parallelizable, stoppable | May miss optimal region by chance |
| Bayesian Optimization | Expensive evaluations (hours per trial) | Most sample-efficient | Sequential, harder to parallelize |
| Manual Tuning | Expert intuition, few key params | Very fast for experienced practitioners | Not systematic, hard to document |
Python Implementation
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform, randint
param_distributions = {
'learning_rate': loguniform(1e-4, 1e-1),
'max_depth': randint(3, 12),
'n_estimators': randint(100, 1000),
'subsample': [0.6, 0.7, 0.8, 0.9, 1.0],
}
search = RandomizedSearchCV(
model, param_distributions,
n_iter=100, cv=5, scoring='accuracy',
random_state=42, n_jobs=-1
)
search.fit(X_train, y_train)
print(f"Best: {search.best_params_}")Practical Guidelines
| Budget | Strategy |
|---|---|
| 10-20 trials | Random Search with broad ranges (exploration) |
| 50-100 trials | Random Search → narrow ranges → more Random Search |
| 100+ trials | Random Search for exploration → Bayesian for exploitation |
| Unlimited | Still start with Random Search to understand the landscape |
Random Search is the practical default for hyperparameter optimization — providing better coverage of important hyperparameters than Grid Search at the same computational budget, supporting any distribution (continuous, discrete, categorical), and offering the unique advantages of being embarrassingly parallel (run on 100 GPUs simultaneously) and anytime-stoppable (the best result so far is always valid).