Bagging
Bagging (Bootstrap Aggregating) is an ensemble technique that reduces variance and overfitting by training multiple models independently on random bootstrap samples of the training data and averaging their predictions — based on the insight that while a single decision tree might overfit to noise in the training data, averaging 100 trees trained on different random subsets cancels out the individual trees' noise, producing a stable, robust predictor that is the foundation of Random Forest, one of the most successful algorithms in machine learning.
What Is Bagging?
- Definition: An ensemble method that (1) creates M different training sets by sampling N items with replacement from the original data (bootstrap sampling), (2) trains a separate model on each bootstrap sample independently, and (3) aggregates predictions by averaging (regression) or majority voting (classification).
- Bootstrap Sampling: Sampling N items with replacement from N items — each bootstrap sample contains ~63.2% of unique original examples (some appear multiple times, ~36.8% are left out). The left-out examples form the "Out-of-Bag" (OOB) set, which can be used for validation without a separate holdout set.
- Why "Aggregating": The power comes from combining multiple unstable models into a stable one — each individual tree is "wrong" in a different way, and averaging cancels out the individual errors.
How Bagging Works
| Step | Process | Example |
|---|---|---|
| 1. Bootstrap | Sample N with replacement from N | Original: [1,2,3,4,5] → Sample 1: [1,1,3,4,5] |
| 2. Train | Fit independent model on each sample | Tree 1 on Sample 1, Tree 2 on Sample 2, ... |
| 3. Repeat | Create M bootstrap samples + models | M = 100 trees trained independently |
| 4. Aggregate | Combine predictions | Classification: majority vote; Regression: average |
Variance Reduction
| Single Tree | Bagged Ensemble (100 Trees) |
|---|---|
| High variance — changing one training example changes the tree | Low variance — changing one example affects ~1 tree |
| Unstable — small data changes → very different predictions | Stable — consistent predictions across data perturbations |
| Overfits easily | Resistant to overfitting |
| Interpretable (one tree) | Less interpretable (100 trees) |
Out-of-Bag (OOB) Evaluation
Each bootstrap sample leaves out ~36.8% of the data. For each training example, ~37% of the trees never saw it during training. These trees provide honest predictions for that example — no need for a separate validation set.
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier
# Generic Bagging (any base estimator)
bagging = BaggingClassifier(
n_estimators=100, max_samples=1.0,
bootstrap=True, oob_score=True
)
bagging.fit(X_train, y_train)
print(f"OOB Score: {bagging.oob_score_:.3f}")
# Random Forest = Bagging + Feature Subsampling
rf = RandomForestClassifier(n_estimators=100, oob_score=True)Bagging vs Random Forest
| Feature | Bagging (Decision Trees) | Random Forest |
|---|---|---|
| Bootstrap samples | Yes | Yes |
| Feature subsampling per split | No (uses all features) | Yes ($\sqrt{p}$ features per split) |
| Tree diversity | From bootstrap only | From bootstrap + feature randomization |
| Performance | Good | Better (more diverse trees) |
Bagging is the foundational variance-reduction ensemble technique — demonstrating that averaging many unstable, overfitting models produces a stable, accurate predictor, serving as the theoretical basis for Random Forest, and proving the counterintuitive principle that combining many "wrong" models can produce a "right" ensemble when their errors are independent.