Hyperparameter

Hyperparameter Tuning

Key Hyperparameters for LLMs

Learning Rate
| Setting | Typical Range |
|---------|---------------|
| Pretraining | 1e-4 to 3e-4 |
| Full fine-tuning | 1e-5 to 5e-5 |
| LoRA | 1e-4 to 3e-4 |
| LoRA rank | 8, 16, 32, 64 |

Training
| Hyperparameter | Considerations |
|----------------|----------------|
| Batch size | Larger = more stable, memory permitting |
| Warmup steps | 1-5% of total steps |
| Weight decay | 0.01 to 0.1 |
| Max sequence length | Task-dependent |
| Epochs | 1-5 for fine-tuning |

Tuning Strategies

Grid Search
Try all combinations:

learning_rates = [1e-5, 5e-5, 1e-4]
batch_sizes = [8, 16, 32]

for lr in learning_rates:
    for bs in batch_sizes:
        result = train_and_eval(lr=lr, batch_size=bs)

Exhaustive but expensive.

Random Search
Sample randomly from distributions:

import random

lr = 10 ** random.uniform(-5, -3)  # Log-uniform
bs = random.choice([8, 16, 32, 64])

More efficient than grid search for most problems.

Bayesian Optimization
Use past results to guide search:

from optuna import create_study

def objective(trial):
    lr = trial.suggest_float("lr", 1e-5, 1e-3, log=True)
    bs = trial.suggest_int("batch_size", 8, 64, step=8)
    return train_and_eval(lr=lr, batch_size=bs)

study = create_study(direction="minimize")
study.optimize(objective, n_trials=20)

Tools for HP Sweeps

Weights & Biases Sweep

# sweep.yaml
method: bayes
metric:
  name: val_loss
  goal: minimize
parameters:
  learning_rate:
    min: 0.00001
    max: 0.001
  batch_size:
    values: [8, 16, 32]

wandb sweep sweep.yaml
wandb agent <sweep_id>

Best Practices

Start Simple
1. Use published hyperparameters as baseline
2. Tune one hyperparameter at a time
3. Focus on learning rate first

Resource Allocation
- Use smaller model/dataset for initial sweeps
- Verify best settings transfer to full scale
- Budget compute for tuning (10-20% of total)

Common Mistakes
- Tuning on test set (data leakage!)
- Not setting random seeds
- Comparing runs with different # of steps
- Ignoring variability across runs