Collapse Prevention
Collapse prevention in self-supervised learning is the set of architectural and loss-level constraints that stop encoders from mapping all inputs to identical or low-information embeddings - without these constraints, training can reach deceptively low loss while producing useless representations.
What Is Collapse?
- Definition: Degenerate solution where embeddings lose discriminative variation across samples.
- Trivial Outcome: All images map to same vector or narrow manifold.
- Diagnostic Symptom: Very low training loss with poor linear probe accuracy.
- Common Risk Areas: Non-contrastive objectives and weak regularization settings.
Why Collapse Prevention Matters
- Representation Utility: Preventing collapse is required for any transfer performance.
- Training Reliability: Early detection avoids wasted compute on failed runs.
- Scalability: Collapse risk increases in long, high-capacity training regimes.
- Method Comparison: Stable anti-collapse design differentiates robust SSL methods.
- Production Readiness: Guarantees learned features contain usable information.
Core Prevention Techniques
Architectural Asymmetry:
- Use stop-gradient, predictor heads, and momentum teachers.
- Prevent mutual shortcut updates to constant outputs.
Distribution Controls:
- Apply centering and sharpening on teacher outputs.
- Maintain entropy and avoid uniform or single-channel dominance.
Statistical Regularizers:
- Enforce variance floors and covariance decorrelation.
- Preserve dimensional capacity in embedding space.
Practical Monitoring
- Variance Metrics: Track per-dimension standard deviation across batches.
- Covariance Metrics: Watch off-diagonal magnitude for redundancy buildup.
- Probe Checks: Periodic linear probe confirms semantic information retention.
Collapse prevention in self-supervised learning is the non-negotiable foundation of usable unlabeled representation training - every high-quality SSL recipe includes explicit mechanisms to preserve feature diversity and information content.