Class Weight
Class Weights is a technique for handling imbalanced datasets that modifies the loss function to penalize misclassifying the minority class more heavily — instead of manipulating the data (oversampling or undersampling), class weights make the model "care more" about getting minority examples right by multiplying their loss contribution by a factor inversely proportional to their frequency, so misclassifying 1 fraud case costs as much as misclassifying 100 legitimate ones.
What Are Class Weights?
- Definition: A modification to the training loss function where each class receives a weight inversely proportional to its frequency — the minority class gets a higher weight (bigger penalty for errors) and the majority class gets a lower weight, making the model optimize equally for both classes despite their unequal representation.
- The Intuition: In a dataset with 100 cats and 1 dog, a standard model learns "always predict cat" (99% accuracy). With class weights, misclassifying the dog costs 100× more than misclassifying a cat — forcing the model to actually learn to recognize dogs.
- No Data Manipulation: Unlike SMOTE (creates synthetic examples) or undersampling (removes examples), class weights don't change the training data at all — they only change how the loss function weights errors from different classes.
How Class Weights Work
| Class | Count | Standard Loss Weight | Balanced Weight | Effect |
|---|---|---|---|---|
| Legitimate | 10,000 | 1.0 | 0.05 | Low penalty per error |
| Fraud | 100 | 1.0 | 5.0 | 100× higher penalty per error |
The Balanced Weight Formula: $w_c = \frac{N}{k \times n_c}$ where N = total samples, k = number of classes, $n_c$ = samples in class c.
For the example above: $w_{fraud} = \frac{10100}{2 \times 100} = 50.5$ and $w_{legit} = \frac{10100}{2 \times 10000} = 0.505$.
Implementation Across Frameworks
| Framework | Code | Notes |
|---|---|---|
| Scikit-learn | `LogisticRegression(class_weight='balanced')` | Automatic weight calculation |
| XGBoost | `scale_pos_weight=100` | Ratio of negative to positive |
| PyTorch | `nn.CrossEntropyLoss(weight=torch.tensor([0.05, 5.0]))` | Manual weight tensor |
| Keras | `model.fit(class_weight={0: 0.05, 1: 5.0})` | Dict per class |
| LightGBM | `is_unbalance=True` | Automatic handling |
Class Weights vs Other Imbalance Techniques
| Technique | Modifies Data? | Modifies Loss? | Pros | Cons |
|---|---|---|---|---|
| Class Weights | No | Yes | Simplest, no data change | Can't add new information |
| SMOTE | Yes (adds synthetic) | No | Expands decision boundary | Can create noisy examples |
| Undersampling | Yes (removes majority) | No | Reduces training time | Loses information |
| Focal Loss | No | Yes (down-weights easy examples) | Focuses on hard examples | More complex to tune |
| Threshold Tuning | No | No (post-processing) | Adjusts precision/recall after training | Model unchanged |
The Precision-Recall Trade-off
| Higher Minority Weight | Effect on Recall | Effect on Precision |
|---|---|---|
| More aggressive weight | Recall ↑ (catches more minority examples) | Precision ↓ (more false positives) |
| Less aggressive weight | Recall ↓ | Precision ↑ |
| Balanced weight | Good balance | Good balance |
Class Weights is the simplest and most universally supported technique for handling imbalanced datasets — requiring just one parameter change (class_weight="balanced") to make any classifier treat minority examples as equally important as majority examples, with the trade-off that it increases recall for the minority class at the cost of some precision, and cannot add new information the way oversampling techniques can.