Boosting
Boosting is an ensemble technique where models are trained sequentially, with each new model specifically targeting the errors made by the previous models — unlike bagging (which trains independent models in parallel to reduce variance), boosting builds an additive chain where Model 2 focuses on the examples Model 1 got wrong, Model 3 focuses on what Models 1+2 still get wrong, and so on, progressively reducing both bias and variance to produce the most powerful supervised learning algorithms available for structured/tabular data (XGBoost, LightGBM, CatBoost).
What Is Boosting?
- Definition: A family of ensemble algorithms that convert many "weak learners" (models slightly better than random) into a single "strong learner" by training them sequentially — each weak learner focuses on the mistakes of the previous ones, and the final prediction is a weighted combination of all learners.
- The Intuition: Imagine a student (Model 1) takes a test and gets 30% of questions wrong. A tutor (Model 2) then specifically drills those 30% of hard questions. A second tutor (Model 3) drills the remaining errors. After 100 tutoring sessions, the student masters the entire test.
- Key Difference from Bagging: Bagging trains independent models to reduce variance. Boosting trains dependent models (each one depends on previous errors) to reduce bias and variance.
How Gradient Boosting Works
| Step | Process | What the Model Learns |
|---|---|---|
| 1. Train Tree 1 | Fit to the target y | Rough overall pattern |
| 2. Compute residuals | $r_1 = y - \hat{y}_1$ (what Tree 1 got wrong) | Errors of Tree 1 |
| 3. Train Tree 2 | Fit to residuals $r_1$ | How to fix Tree 1's errors |
| 4. Update prediction | $\hat{y} = \hat{y}_1 + eta \cdot \hat{y}_2$ (η = learning rate) | Combined prediction |
| 5. Compute new residuals | $r_2 = y - (\hat{y}_1 + eta \cdot \hat{y}_2)$ | Remaining errors |
| 6. Repeat N times | Each tree fixes the remaining residual | Progressively better fit |
Boosting Algorithms Timeline
| Algorithm | Year | Key Innovation | Status |
|---|---|---|---|
| AdaBoost | 1997 | Reweight misclassified examples | Historic, still used for simple tasks |
| Gradient Boosting (GBM) | 1999 | Fit residuals using gradient descent in function space | Foundation of modern boosting |
| XGBoost | 2014 | Regularization + parallelized splits + missing value handling | Dominated Kaggle 2014-2020 |
| LightGBM | 2017 | Histogram binning + leaf-wise growth + GOSS | Fastest, most memory-efficient |
| CatBoost | 2017 | Native categorical encoding + ordered boosting | Best for categorical-heavy data |
Critical Hyperparameters
| Parameter | Effect | Too Low | Too High |
|---|---|---|---|
| n_estimators (# trees) | Number of sequential models | Underfitting | Overfitting (mitigated by early stopping) |
| learning_rate (η) | Shrinkage per tree | Needs many more trees | Overfits quickly |
| max_depth | Individual tree complexity | Weak learners (good for boosting) | Each tree overfits |
| subsample | Fraction of data per tree | More regularization | Less regularization |
Rule of thumb: Use a low learning rate (0.01-0.1) with many trees (500-5000) and early stopping.
Boosting is the most powerful supervised learning paradigm for structured data — sequentially building an additive ensemble where each model corrects the errors of its predecessors, powering the XGBoost/LightGBM/CatBoost family that dominates tabular data competitions and production systems, with the critical requirement of proper learning rate and early stopping tuning to prevent the overfitting that sequential error-correction can cause.