Contrastive Learning
Contrastive Learning is the self-supervised and supervised representation learning framework that trains models by pulling similar (positive) pairs close together and pushing dissimilar (negative) pairs apart in embedding space — producing high-quality feature representations without requiring labeled data, forming the foundation of CLIP, SimCLR, and modern embedding models.
Core Principle
- Given an anchor sample, create a positive pair (augmented version of same sample) and negative pairs (different samples).
- Loss function encourages: $sim(anchor, positive) >> sim(anchor, negative)$.
- Result: Model learns semantic features that capture what makes samples similar or different.
InfoNCE Loss (Standard Contrastive Loss)
$L = -\log \frac{\exp(sim(z_i, z_j^+)/\tau)}{\sum_{k=0}^{K} \exp(sim(z_i, z_k)/\tau)}$
- $z_i$: Anchor embedding.
- $z_j^+$: Positive pair embedding.
- K negatives in denominator.
- τ: Temperature parameter (typically 0.07-0.5).
- Denominator = positive + all negatives → softmax over similarity scores.
SimCLR (Visual Self-Supervised)
1. Take an image, create two random augmentations (crop, color jitter, flip).
2. Encode both through a ResNet backbone → projector MLP → embeddings z₁, z₂.
3. These two views are the positive pair.
4. All other images in the mini-batch are negatives.
5. Minimize InfoNCE loss.
6. After training: Discard projector, use backbone features for downstream tasks.
CLIP (Vision-Language Contrastive)
- Positive pairs: Matching (image, text) pairs from the internet.
- Negative pairs: Non-matching (image, text) combinations within the batch.
- Image encoder (ViT) and text encoder (Transformer) trained jointly.
- Batch of N pairs → N² possible pairings → N positives, N²-N negatives.
- Result: Unified vision-language embedding space enabling zero-shot classification.
Key Design Choices
| Factor | Impact | Best Practice |
|---|---|---|
| Batch size | More negatives → better | Large batches (4096-65536) |
| Temperature τ | Lower = sharper distinctions | 0.07-0.1 for vision |
| Augmentation strength | Determines what's "invariant" | Strong augmentation essential |
| Projection head | Improves representation quality | MLP projector, discard after training |
| Hard negatives | Training signal quality | Mine semi-hard negatives |
Beyond SimCLR
- MoCo: Momentum-updated encoder + queue of negatives → doesn't need huge batches.
- BYOL/SimSiam: No negatives at all — positive pairs only + stop-gradient trick.
- DINO/DINOv2: Self-distillation with no labels → exceptional visual features.
Contrastive learning is the dominant paradigm for learning general-purpose representations — its ability to leverage unlimited unlabeled data to produce embeddings that transfer across tasks has made it the foundation of modern embedding models, multimodal AI, and self-supervised pretraining.