Normalization
Normalization and Standardization are feature scaling techniques that transform numeric features to comparable ranges — essential preprocessing for distance-based algorithms (KNN, SVM) and gradient-based methods (neural networks, logistic regression) because unscaled features with different magnitudes (Age 0-100 vs Salary 0-200,000) cause the larger-magnitude features to dominate distance calculations and gradient updates, leading to biased models and slow convergence.
Why Scale Features?
- The Problem: If you measure distances between data points using Age (0-100) and Salary (0-200,000), Salary dominates the distance calculation because its values are 2,000× larger — a difference of $10,000 in salary overwhelms a difference of 10 years in age, even though both might be equally important.
- Which Algorithms Need Scaling: Distance-based (KNN, SVM, K-Means), gradient-based (Neural Networks, Logistic Regression, Linear Regression with regularization). Tree-based models (Random Forest, XGBoost) do NOT need scaling because they split on individual features independently.
Standardization (Z-Score Normalization)
- Formula: $X_{new} = \frac{X - \mu}{\sigma}$
- Result: Mean = 0, Standard Deviation = 1
- Range: Unbounded (typically -3 to +3, but outliers can be ±10+)
- Best For: Most ML algorithms — robust to outliers because outliers don't affect the mean/std as severely as they affect min/max
| Feature | Original | Standardized |
|---|---|---|
| Age = 25 | 25 | -1.2 |
| Age = 50 | 50 | 0.0 |
| Age = 75 | 75 | +1.2 |
| Salary = $30K | 30,000 | -1.0 |
| Salary = $60K | 60,000 | 0.0 |
| Salary = $90K | 90,000 | +1.0 |
Normalization (Min-Max Scaling)
- Formula: $X_{new} = \frac{X - X_{min}}{X_{max} - X_{min}}$
- Result: All values mapped to [0, 1]
- Best For: Neural networks (bounded activations), image data (pixels 0-255 → 0-1), algorithms requiring bounded input
| Feature | Original | Normalized |
|---|---|---|
| Age = 25 | 25 | 0.25 |
| Age = 50 | 50 | 0.50 |
| Age = 75 | 75 | 0.75 |
Comparison
| Property | Standardization (Z-Score) | Normalization (Min-Max) |
|---|---|---|
| Output range | Unbounded (~-3 to +3) | Fixed [0, 1] |
| Outlier sensitivity | Moderate (outliers shift mean/std slightly) | High (one outlier compresses all other values) |
| Best for | General ML, regression, SVM | Neural networks, image data |
| Preserves zero | Yes (sparse data friendly) | No |
| Rule of thumb | "When in doubt, standardize" | When bounded input is required |
Critical Rule: Fit on Train, Transform Both
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) # Learn mean/std from train
X_test_scaled = scaler.transform(X_test) # Apply train's mean/std to testNever call `fit_transform` on test data — that would leak test statistics into the scaler, causing data leakage.
Normalization and Standardization are the essential preprocessing steps for fair feature comparison — ensuring that all features contribute proportionally to model learning regardless of their original scale, with standardization as the safe default for most algorithms and min-max normalization for neural networks and bounded-input requirements.