Label Smoothing in ViT

Label smoothing in ViT is the regularization method that replaces hard one-hot targets with softened distributions to reduce overconfidence and improve calibration - instead of forcing probability one for a single class, it reserves small mass for other classes and encourages less extreme logits.

What Is Label Smoothing?

• Definition: Modify target vector so true class gets 1 - epsilon and remaining classes share epsilon.
• Regularization Mechanism: Penalizes overly sharp probability outputs.
• Typical Values: Epsilon around 0.05 to 0.2 depending on dataset and augmentation strength.
• Loss Integration: Applied directly in cross entropy computation.

Why Label Smoothing Matters

• Generalization: Reduces overfitting by discouraging memorization of hard labels.
• Calibration: Produces more realistic confidence scores at inference time.
• Stability: Limits extreme logits that can destabilize mixed precision optimization.
• Noise Tolerance: Slightly reduces impact of mislabeled samples.
• Recipe Synergy: Works well with mixup, CutMix, and strong augmentation policies.

Smoothing Configurations

Fixed Epsilon:
- Constant smoothing value throughout training.
- Simple and commonly effective.

Scheduled Epsilon:
- Start higher then reduce near end for sharper decision boundaries.
- Useful in long training runs.

Class-Aware Smoothing:
- Different epsilon values by class frequency.
- Can improve rare class handling.

How It Works

Step 1: Build softened label distribution for each sample by allocating most probability to target class and small residual across others.

Step 2: Compute cross entropy against softened targets, producing gradients that discourage extreme certainty.

Tools & Platforms

• PyTorch cross entropy: Supports label smoothing parameter directly.
• timm recipes: Includes tuned defaults for ViT families.
• Calibration metrics: ECE and reliability diagrams validate impact.

Label smoothing is a simple but effective calibration tool that helps ViTs generalize better by reducing pathological confidence spikes - it keeps classifier behavior more realistic under real world variation.