Contrastive Learning Self Supervised

Contrastive Learning is the self-supervised representation learning paradigm where a model learns to distinguish between similar (positive) and dissimilar (negative) pairs of data augmentations — producing embeddings where semantically similar inputs are mapped nearby and dissimilar inputs are pushed apart, all without requiring human-annotated labels.

Core Principles:
- Positive Pairs: two augmented views of the same image — random crop, color jitter, Gaussian blur, horizontal flip applied independently to create two correlated views (x_i, x_j) that should have similar embeddings
- Negative Pairs: augmented views from different images — all other images in the mini-batch serve as negatives; more negatives provide better coverage of the representation space but require more memory
- InfoNCE Loss: L = -log(exp(sim(z_i,z_j)/τ) / Σ_k exp(sim(z_i,z_k)/τ)) — maximizes agreement between positive pair relative to all negatives; temperature τ controls how hard negatives are emphasized (typical τ=0.07-0.5)
- Projection Head: non-linear MLP applied after the backbone encoder — maps representations to a space where contrastive loss is applied; the pre-projection representations transfer better to downstream tasks

Major Frameworks:
- SimCLR: end-to-end contrastive learning within a mini-batch — requires large batch sizes (4096-8192) to provide sufficient negatives; uses NT-Xent loss with cosine similarity; simple but compute-intensive
- MoCo (Momentum Contrast): maintains a queue of negatives from recent mini-batches — momentum-updated encoder produces consistent negative representations; decouples negative count from batch size enabling smaller batches (256)
- BYOL (Bootstrap Your Own Latent): eliminates negative pairs entirely — online network predicts the representation of a target network (momentum-updated); avoids mode collapse through asymmetric architecture and momentum update
- SwAV (Swapping Assignments): assigns augmented views to learned prototype clusters — enforces consistency: view 1's assignment should match view 2's assignment; combines contrastive learning with clustering for multi-crop efficiency

Training and Transfer:
- Pre-Training Scale: competitive contrastive learning requires 200-1000 training epochs on ImageNet — compared to 90 epochs for supervised training; long training compensates for weaker per-sample supervision
- Linear Evaluation Protocol: freeze pre-trained backbone, train only a linear classifier on top — standard benchmark for representation quality; SimCLR achieves 76.5%, supervised achieves 78.2% on ImageNet
- Fine-Tuning Transfer: pre-trained representations fine-tuned on downstream tasks — contrastive pre-training often outperforms supervised pre-training for transfer learning, especially with limited labeled data (10-100× improvement at 1% label fraction)
- Multi-Modal Contrastive (CLIP): contrasts image-text pairs from internet data — learns aligned vision-language representations enabling zero-shot classification; 400M image-text pairs produces representations that transfer broadly without fine-tuning

Contrastive learning has fundamentally changed the deep learning landscape by demonstrating that high-quality visual representations can be learned without any human labels — enabling AI systems trained on vast unlabeled data to match or exceed the performance of fully supervised methods.