Feature Selection
Feature Selection is the process of identifying and keeping only the most informative variables for a machine learning model while discarding noisy, redundant, or irrelevant features — improving model accuracy (less noise = better signal), reducing overfitting (fewer parameters = better generalization), speeding up training and inference (fewer features = less computation), and improving interpretability (fewer features = easier to explain), making it a critical preprocessing step that sits between feature engineering and model training.
What Is Feature Selection?
- Definition: The systematic identification of the subset of input features that contribute most to prediction accuracy — using statistical tests, model-based importance scores, or iterative search to separate signal from noise.
- Why Not Keep Everything?: More features aren't always better. Irrelevant features add noise that models can overfit to. Redundant features (height_cm and height_inches) waste computation without adding information. The "curse of dimensionality" means that as features increase, the data becomes increasingly sparse in high-dimensional space.
- Feature Selection vs. Feature Extraction: Selection keeps a subset of original features. Extraction (PCA, autoencoders) creates new features that are combinations of originals. Selection preserves interpretability; extraction may not.
Three Categories of Methods
| Category | Approach | Speed | Quality | Example |
|---|---|---|---|---|
| Filter Methods | Rank features by statistical score, independent of model | Very fast | Good | Correlation, Chi-Square, Mutual Information |
| Wrapper Methods | Train model with different feature subsets, select the best | Slow | Best | Recursive Feature Elimination (RFE), Forward Selection |
| Embedded Methods | Model selects features during training | Moderate | Very good | L1 (Lasso), Tree Feature Importance, ElasticNet |
Filter Methods (Model-Independent)
| Method | Feature Type | What It Measures |
|---|---|---|
| Pearson Correlation | Continuous vs Continuous | Linear relationship strength |
| Chi-Square (χ²) | Categorical vs Categorical | Statistical independence |
| Mutual Information | Any | Non-linear dependency between feature and target |
| Variance Threshold | Any | Remove features with near-zero variance |
| ANOVA F-test | Continuous vs Categorical | Difference in means across classes |
Wrapper Methods (Model-Dependent)
| Method | Process | Trade-off |
|---|---|---|
| Forward Selection | Start empty, add best feature one at a time | Greedy, may miss feature interactions |
| Backward Elimination | Start with all, remove worst feature one at a time | Expensive for many features |
| RFE (Recursive Feature Elimination) | Train model, remove least important, repeat | Good balance, sklearn built-in |
Embedded Methods (During Training)
| Method | How It Selects | Best For |
|---|---|---|
| L1 Regularization (Lasso) | Drives weak feature coefficients to exactly zero | Linear/logistic regression |
| Tree Feature Importance | Features used in early splits are most important | Random Forest, XGBoost |
| ElasticNet (L1 + L2) | Combines L1 sparsity with L2 grouping | Correlated features |
Feature Selection is the essential preprocessing step that ensures models learn from signal rather than noise — using statistical tests, model-based importance, or iterative search to identify the features that actually matter, improving accuracy, reducing overfitting, speeding up training, and producing models that are easier to interpret and deploy.