Ensemble
Ensemble Learning is the strategy of combining multiple machine learning models to produce better predictive performance than any single model alone — based on the "wisdom of crowds" principle that independent errors from different models cancel each other out when aggregated, with three major paradigms: Bagging (train models in parallel on random subsets to reduce variance — Random Forest), Boosting (train models sequentially to fix predecessors' errors — XGBoost), and Stacking (train a meta-model to optimally combine diverse base models).
What Is Ensemble Learning?
- Definition: A machine learning approach that combines the predictions of multiple "base learners" (individual models) through voting, averaging, or learned combination to produce a final prediction that is more accurate, robust, and stable than any individual model.
- Why It Works: If Model A makes mistakes on cases 1-10 and Model B makes mistakes on cases 11-20, combining them eliminates mistakes on all 20 cases. The key requirement is that models make different errors (diversity).
- The Math: For N independent models each with error rate ε, the ensemble error rate (majority vote) drops exponentially: $P(error) = sum_{k=lceil N/2 \rceil}^{N} \binom{N}{k} \varepsilon^k (1-\varepsilon)^{N-k}$. With 21 models at 40% individual error, majority vote achieves ~18% error.
Three Paradigms
| Paradigm | Training | Goal | Key Algorithm |
|---|---|---|---|
| Bagging | Parallel (independent models on bootstrap samples) | Reduce variance (overfitting) | Random Forest |
| Boosting | Sequential (each model fixes previous errors) | Reduce bias (underfitting) | XGBoost, LightGBM, AdaBoost |
| Stacking | Layered (meta-model combines base predictions) | Optimal combination of diverse models | Stacked generalization |
Bagging vs Boosting
| Property | Bagging | Boosting |
|---|---|---|
| Training | Parallel (independent) | Sequential (dependent) |
| Focus | Reduce variance | Reduce bias + variance |
| Overfitting risk | Low (averaging reduces it) | Higher (sequential fitting can overfit) |
| Typical base model | Full decision trees | Shallow trees (stumps) |
| Speed | Parallelizable | Sequential (harder to parallelize) |
| Example | Random Forest | XGBoost, LightGBM |
Aggregation Methods
| Method | Task | How |
|---|---|---|
| Hard Voting | Classification | Majority class label wins |
| Soft Voting | Classification | Average predicted probabilities, pick highest |
| Averaging | Regression | Mean of all model predictions |
| Weighted Averaging | Both | Models with higher validation scores get more weight |
| Stacking | Both | Meta-model learns optimal combination |
Why Ensembles Dominate Competitions
| Competition | Winning Solution |
|---|---|
| Netflix Prize ($1M) | Ensemble of 800+ models |
| Most Kaggle tabular competitions | XGBoost/LightGBM ensemble |
| ImageNet 2012+ | Ensemble of multiple CNNs |
Ensemble Learning is the most reliable strategy for maximizing predictive performance — combining the diverse strengths of multiple models through parallel training (bagging), sequential error correction (boosting), or learned combination (stacking) to produce predictions that are more accurate, more robust, and more stable than any single model can achieve alone.