Contrastive Learning

Contrastive Learning / Self-Supervised Learning — training models to learn useful representations from unlabeled data by contrasting similar (positive) and dissimilar (negative) pairs.

Core Idea

• Create two augmented views of the same image (positive pair)
• Pull their representations together in embedding space
• Push representations of different images apart
• No labels needed — the augmentation defines the learning signal

Key Methods

• SimCLR: Simple framework. Augment → encode → project → contrastive loss (InfoNCE). Needs large batches (4096+)
• MoCo (Momentum Contrast): Maintains a momentum-updated queue of negatives. Works with normal batch sizes
• BYOL (Bootstrap Your Own Latent): No negatives at all — uses a momentum target network. Surprisingly effective
• DINO/DINOv2: Self-distillation with no labels. Produces exceptional image features
• MAE (Masked Autoencoder): Mask 75% of image patches → reconstruct. Vision analog of BERT

Why It Matters

• Labeled data is expensive and limited
• Self-supervised models trained on billions of unlabeled images learn better features than supervised training
• Foundation models (CLIP, DINOv2) are self/weakly supervised

Performance

• DINOv2 features match or beat supervised features on downstream tasks
• Self-supervised pretraining is now the default for large vision models

Self-supervised learning is how modern AI escapes the bottleneck of labeled data.