Label Smoothing
Label Smoothing is the regularization technique that replaces hard one-hot target labels with soft labels that distribute a small amount of probability mass to non-target classes — preventing the model from becoming overconfident in its predictions, improving calibration, and acting as an implicit regularizer that encourages the model to learn more generalizable representations rather than memorizing the exact training labels.
How Label Smoothing Works
- Hard label (standard): y = [0, 0, 1, 0, 0] (one-hot for class 2).
- Soft label (smoothing ε=0.1, K=5 classes): y = [0.02, 0.02, 0.92, 0.02, 0.02].
- Formula: $y_{smooth} = (1 - \varepsilon) \times y_{one-hot} + \varepsilon / K$
- Target class gets probability (1 - ε + ε/K), others get ε/K each.
Implementation
def label_smoothing_loss(logits, targets, epsilon=0.1):
K = logits.size(-1) # number of classes
log_probs = F.log_softmax(logits, dim=-1)
# NLL loss for true class
nll = -log_probs.gather(dim=-1, index=targets.unsqueeze(1)).squeeze(1)
# Uniform loss (smooth part)
smooth = -log_probs.mean(dim=-1)
loss = (1 - epsilon) * nll + epsilon * smooth
return loss.mean()Why Label Smoothing Helps
| Effect | Without Smoothing | With Smoothing |
|---|---|---|
| Logit magnitude | Grows unbounded (push toward ±∞) | Bounded (no need for extreme confidence) |
| Calibration | Overconfident (99%+ on everything) | Better calibrated probabilities |
| Generalization | May memorize noisy labels | More robust to label noise |
| Representation | Clusters collapse to single point | Clusters have finite spread |
Typical ε Values
| Task | ε | Notes |
|---|---|---|
| ImageNet classification | 0.1 | Standard since Inception v2 |
| Machine translation | 0.1 | Default in Transformer paper |
| Speech recognition | 0.1-0.2 | Common in ASR systems |
| Fine-tuning | 0.0-0.05 | Lower to preserve pre-trained knowledge |
| Knowledge distillation | 0.0 | Soft targets from teacher serve similar purpose |
Relationship to Other Techniques
- Knowledge distillation: Teacher's soft predictions serve as implicit label smoothing.
- Mixup/CutMix: Create soft labels by mixing examples → similar regularization effect.
- Temperature scaling: Can be applied post-training for calibration (label smoothing does it during training).
When NOT to Use Label Smoothing
- When exact probabilities matter (some ranking/retrieval tasks).
- When combined with knowledge distillation (redundant smoothing).
- When label noise is already high (smoothing adds more uncertainty).
Label smoothing is one of the simplest and most effective regularization techniques available — adding just one hyperparameter (ε) that consistently improves generalization and calibration across vision, language, and speech models, making it a default inclusion in most modern training recipes.