Centering in Self-Supervised

Centering in self-supervised learning is the target normalization strategy that subtracts a running mean from teacher logits so one class channel does not dominate training - this keeps target distributions balanced and prevents trivial fixed-output solutions in non-label supervision pipelines.

What Is Centering?

• Definition: A moving-average correction applied to teacher outputs before softmax target generation.
• Core Mechanism: Subtract running center vector from teacher logits to remove persistent bias.
• Primary Goal: Prevent output collapse where every sample maps to nearly identical teacher probabilities.
• Typical Use: DINO-like student-teacher setups with multi-view consistency objectives.

Why Centering Matters

• Collapse Resistance: Reduces risk of constant class preference across all images.
• Target Diversity: Preserves spread across output dimensions for richer supervision.
• Training Stability: Smooths batch-to-batch drift in teacher target statistics.
• Representation Quality: Improves feature separability in downstream linear probing.
• Recipe Compatibility: Works with sharpening, momentum encoders, and multi-crop views.

How Centering Works

Step 1:
- Compute teacher logits for current batch and update an exponential moving average center vector.
- Keep the center update slow enough to avoid noisy oscillation.

Step 2:
- Subtract center vector from teacher logits before temperature scaling and softmax.
- Feed normalized soft targets to student loss for cross-view alignment.

Practical Guidance

• Momentum Choice: High center momentum improves stability in large-batch runs.
• Monitoring: Track per-dimension target entropy to detect imbalance early.
• Numerics: Compute center updates in float32 even when model trains in mixed precision.

Centering in self-supervised learning is a small normalization step that prevents target bias from derailing representation learning - it is one of the highest leverage stabilizers in modern non-label student-teacher training.