Simsiam
SimSiam (Simple Siamese Networks) is a self-supervised representation learning method that learns useful visual representations without requiring negative sample pairs, large batches, or momentum encoders — achieving competitive performance with contrastive methods through a remarkably minimal architecture consisting of two weight-sharing encoders and a stop-gradient operation that prevents representational collapse — published by Kaiming He et al. (Facebook AI Research, 2021) as a theoretical and empirical demonstration that the seemingly essential components of contrastive self-supervised learning were not actually necessary.
What Is SimSiam?
- Siamese Network: Two identical encoder networks (sharing weights) process two differently augmented views of the same image — each producing a feature vector.
- Predictor MLP: One branch passes its representation through an additional small MLP (the predictor) before computing the similarity loss.
- Stop-Gradient: The other branch's representation is treated as a constant — gradients are not propagated through it during the backward pass.
- Loss Function: Negative cosine similarity between the predicted representation of one branch and the stopped-gradient representation of the other — minimized to encourage the two views to agree.
- No Negatives, No Momentum: Unlike SimCLR (needs large batches of negatives) and BYOL/MoCo (needs momentum encoder), SimSiam needs neither.
Why Stop-Gradient Prevents Collapse
The critical question: why doesn't SimSiam collapse to a trivial solution (all representations identical)?
- Intuitive Answer: Stop-gradient creates an implicit expectation-maximization (EM) algorithm — one branch optimizes the predictor to match a "target" (the other branch), while the target is periodically updated by weight sharing. Neither branch fully controls the target.
- Theoretical Analysis: With stop-gradient, SimSiam alternates between optimizing the predictor (E-step: find best prediction of fixed representations) and updating the encoder (M-step: improve representations to make them more predictable).
- Symmetrical Update: The loss is computed in both directions (both branches act as predictor and target), stabilizing the asymmetric update.
Comparison with Alternatives
| Method | Negatives Needed | Momentum Encoder | Large Batch | Stop-Gradient |
|---|---|---|---|---|
| SimCLR | Yes (2×4096) | No | Yes | No |
| MoCo v2 | Yes (queue 65536) | Yes | No | No |
| BYOL | No | Yes | No | No |
| SimSiam | No | No | No | Yes |
| Barlow Twins | No | No | No | No (cross-corr) |
Performance and Impact
- ImageNet Linear Evaluation: SimSiam achieves ~71% top-1 with ResNet-50 — competitive with SimCLR and BYOL despite the greater simplicity.
- Transfer Learning: Features transfer well to detection and segmentation on COCO and Pascal VOC.
- Theoretical Impact: SimSiam drove the field to understand *why* self-supervised methods work — the stop-gradient analysis revealed implicit EM optimization hiding in SSL methods.
SimSiam is the minimalist proof that self-supervised learning needs less than we thought — its discovery that useful representations emerge from simple similarity maximization with stop-gradient reshaped the theory of SSL and inspired a generation of even simpler, more scalable methods.