Encoding
One-Hot Encoding is the standard technique for converting categorical variables into a binary matrix representation that machine learning models can process — where each unique category becomes its own column with values 0 or 1 (Red → [1,0,0], Blue → [0,1,0], Green → [0,0,1]), avoiding the false ordinal assumption that Label Encoding introduces (Red=0, Blue=1, Green=2 implies Blue is "between" Red and Green), making it the default encoding for linear models and neural networks.
What Is One-Hot Encoding?
- Definition: A transformation that converts a single categorical column with K unique values into K binary columns — each row has exactly one "1" (hot) and K-1 "0"s (cold), creating a sparse binary representation.
- Why Not Just Numbers?: If you encode Red=0, Blue=1, Green=2 (Label Encoding), a linear model learns weights where Blue is literally "between" Red and Green mathematically. This is nonsensical for nominal categories. One-hot encoding gives each category its own independent coefficient.
Example
| Original | Red | Green | Blue |
|---|---|---|---|
| Red | 1 | 0 | 0 |
| Blue | 0 | 0 | 1 |
| Green | 0 | 1 | 0 |
| Red | 1 | 0 | 0 |
When to Use One-Hot Encoding
| Model Type | Use One-Hot? | Reason |
|---|---|---|
| Linear Regression / Logistic | Yes (required) | Cannot handle nominal categories as integers |
| Neural Networks | Yes (standard) | Independent dimensions for each category |
| SVM | Yes | Distance-based, needs proper encoding |
| KNN | Yes | Distance calculation needs binary dimensions |
| Decision Trees / Random Forest | Optional | Trees split on individual features, can use label encoding |
| XGBoost / LightGBM | Optional | LightGBM has native categorical support |
The High-Cardinality Problem
| Feature | Unique Values | One-Hot Columns | Problem |
|---|---|---|---|
| Color | 3 | 3 | Fine |
| Country | 195 | 195 | Manageable |
| Zip Code | 41,000+ | 41,000+ | Too many columns — model becomes slow, sparse, overfitting |
| User ID | 1,000,000+ | 1,000,000+ | Completely impractical |
Solutions for high cardinality:
- Target Encoding: Replace category with mean of target variable.
- Frequency Encoding: Replace category with its count.
- Embeddings: Learn dense vector representations (standard in deep learning).
- Hash Encoding: Map categories to a fixed number of buckets.
The Dummy Variable Trap
- Problem: With K one-hot columns, the last column is perfectly predictable from the first K-1 (if all are 0, the last must be 1). This creates multicollinearity in linear models.
- Solution: Drop one column (`drop_first=True` in pandas). Use K-1 columns instead of K.
import pandas as pd
pd.get_dummies(df["color"], drop_first=True)One-Hot Encoding is the default categorical encoding for most machine learning models — providing each category with an independent dimension that prevents false ordinal assumptions, with the key trade-off being dimensionality explosion for high-cardinality features that requires alternative encoding strategies like target encoding or embeddings.