Stacking
Stacking (stacked generalization) is the ensemble learning technique that trains a meta-model to optimally combine predictions from multiple diverse base models, learning through cross-validation which base learners to trust for different types of inputs — consistently outperforming simple averaging or voting by discovering complementary strengths across algorithms, making it the dominant ensemble strategy in machine learning competitions and a robust approach for production systems where no single model excels across all data patterns.
What Is Stacking?
- Architecture: Layer 0 (base models: RF, XGBoost, SVM, Neural Net) → Layer 1 (meta-model: logistic regression or linear model) → Final prediction.
- Key Insight: Different models make different mistakes — a meta-learner can identify which model to trust for which inputs.
- Cross-Validation Requirement: Base model predictions used for meta-training must come from out-of-fold predictions to prevent data leakage and overfitting.
- Meta-Features: The meta-model's input features are the predictions (or probabilities) from each base model.
Why Stacking Matters
- Superior Performance: Typically beats any individual base model and outperforms simple averaging by 1-5% on benchmarks.
- Diversity Exploitation: A random forest might excel on categorical features while a neural network handles continuous interactions — stacking learns to route decisions appropriately.
- Competition Dominance: Nearly every top Kaggle submission uses stacking or its variants.
- Robustness: Less sensitive to individual model failures since the meta-learner can down-weight unreliable base models.
- Flexible Architecture: Any combination of models can serve as base learners — mixing paradigms (tree-based, linear, neural) maximizes diversity.
How Stacking Works
Step 1 — Generate Out-of-Fold Predictions:
- Split training data into K folds.
- For each base model, train on K-1 folds and predict on the held-out fold.
- Concatenate held-out predictions to create meta-features for the full training set.
Step 2 — Train Meta-Model:
- Use the out-of-fold predictions as features and original labels as targets.
- Fit a simple meta-model (logistic regression is standard) to learn optimal combination.
Step 3 — Final Prediction:
- Train all base models on full training data.
- Generate predictions on test data from each base model.
- Feed base predictions through the trained meta-model for final output.
Stacking Variants
| Variant | Description | Use Case |
|---|---|---|
| Standard Stacking | Single-layer meta-model on base predictions | Default approach |
| Multi-Level Stacking | Multiple meta-model layers (stack of stacks) | Competitions (diminishing returns) |
| Blending | Uses hold-out set instead of cross-validation | Faster, simpler, slightly less optimal |
| Feature-Weighted Stacking | Meta-model also receives original features | When base models miss important signals |
| Stacking with Diversity | Deliberately train weaker but diverse base models | Maximum complementarity |
Best Practices
- Meta-Model Simplicity: Use logistic regression or ridge — complex meta-models overfit to the small number of meta-features.
- Base Model Diversity: Maximize architectural diversity (trees, linear, neural, nearest-neighbor) — correlated base models add no value.
- Sufficient Folds: Use 5-10 fold CV to generate reliable out-of-fold predictions.
- Probability Outputs: Feed predicted probabilities (not classes) to the meta-model for maximum information transfer.
Stacking is the principled way to let models vote on the answer — going beyond democratic averaging to intelligent weighting where a meta-learner discovers exactly when to trust each expert, consistently producing the most robust predictions achievable from a given set of base models.