Supervised Contrastive Learning

Supervised Contrastive Learning (SupCon) is an extension of contrastive learning that leverages label information — treating all samples of the same class as positives and samples of different classes as negatives, producing better-structured representations than standard cross-entropy training.

How Does SupCon Work?

- Positive Set: All augmented views of all samples with the same label (not just augmented views of the same instance).
- Loss: $mathcal{L} = -sum_{i} frac{1}{|P(i)|} sum_{p in P(i)} log frac{exp(z_i cdot z_p / \tau)}{sum_{a
eq i} exp(z_i cdot z_a / \tau)}$
- Contrast: Pull same-class representations together, push different-class representations apart.
- Training: Two-stage — SupCon on the encoder, then cross-entropy on a linear classifier.

Why It Matters

• Better Representations: Produces more structured, class-aware feature spaces than cross-entropy alone.
• Robustness: More robust to natural corruptions, label noise, and hyperparameter sensitivity.
• Transfer: Better linear probe performance than cross-entropy-trained features.

Supervised Contrastive Learning is SimCLR with labels — using class supervision to define positive pairs more accurately and learn cleaner decision boundaries.