Confidence Penalty

Confidence Penalty is a regularization technique that penalizes the model for making overconfident predictions — adding a penalty term to the loss that discourages the model from outputting predictions with very low entropy (highly concentrated probability distributions).

Confidence Penalty Formulation

• Penalty: $L = L_{task} - \beta H(p)$ where $H(p) = -\sum_c p(c) \log p(c)$ is the entropy of the predicted distribution.
• Effect: Maximizing entropy encourages spreading probability across classes — prevents overconfidence.
• $\beta$ Parameter: Controls the penalty strength — larger $\beta$ = more uniform predictions.
• Relation: Equivalent to label smoothing with a uniform target distribution.

Why It Matters

• Calibration: Overconfident models are poorly calibrated — confidence penalty improves calibration.
• Exploration: In active learning and RL, confidence penalty encourages exploration of uncertain regions.
• Distillation: Better-calibrated teacher models produce more informative soft labels for distillation.

Confidence Penalty is punishing overconfidence — explicitly penalizing low-entropy predictions to produce better-calibrated, more honest models.