Sharpening in Self-Supervised

Sharpening in self-supervised learning is the temperature-based target transformation that makes teacher probability distributions more confident and less uniform - by lowering temperature before softmax, training receives clearer discrimination signals across semantic dimensions.

What Is Sharpening?

• Definition: Applying low-temperature softmax to teacher logits to reduce entropy of target distributions.
• Core Effect: Higher probability mass on a few dimensions and lower mass on irrelevant dimensions.
• Primary Role: Improve supervisory signal strength in self-distillation losses.
• Common Pairing: Typically used with centering to avoid trivial dominant channels.

Why Sharpening Matters

• Signal Clarity: Student receives less ambiguous targets and learns faster semantic structure.
• Collapse Prevention: Uniform targets are discouraged, reducing non-informative solutions.
• Feature Separation: Encourages sharper clusters in embedding space.
• Downstream Benefit: Improves linear evaluation and retrieval ranking consistency.
• Stability Balance: Proper temperature prevents both noisy and overconfident extremes.

How Sharpening Works

Step 1:
- Compute centered teacher logits and divide by temperature value T below 1.
- Lower T yields sharper target distribution, higher T yields softer distribution.

Step 2:
- Apply softmax to obtain teacher probabilities and train student to match targets.
- Tune temperature schedule across epochs to balance stability and discrimination.

Practical Guidance

• Temperature Range: Values around 0.04 to 0.2 are common depending on architecture and objective.
• Schedule Design: Warm temperature early can help stability, sharper temperature later can improve separation.
• Diagnostics: Track target entropy and feature collapse indicators during training.

Sharpening in self-supervised learning is the decisiveness control that turns flat targets into informative supervision - with centering and momentum updates, it becomes a core ingredient for high-quality representation learning.