Dino (self-Distillation with No Labels)

DINO (Self-DIstillation with NO labels) is a self-supervised learning approach for Vision Transformers — demonstrating that self-supervised ViT features explicitly contain scene layout and object segmentation information, which usually requires supervised learning.

What Is DINO?

• Definition: A self-supervised method using knowledge distillation without labels.
• Architecture: Student and Teacher networks with the same architecture but different parameters.
• Update Rule: Teacher is updated as an exponential moving average (EMA) of the student.
• Key Insight: Self-supervision on ViTs automatically leads to class-specific features.

Why DINO Matters

• Emergent Segmentation: Attention maps automatically segment objects without supervision.
• k-NN Performance: Features work incredibly well with simple k-nearest neighbor classifiers.
• No Labels Needed: Unlocks learning from massive uncurated image datasets.
• Teacher-Student Stability: Solves collapse issues common in self-supervised learning without negative pairs.

How It Works

• Multi-Crop Strategy: Feeds global and local crops to student, only global to teacher.
• Cross-Entropy Loss: Minimizes distance between student and teacher probability distributions.
• Centering & Sharpening: Prevents mode collapse (outputting same class for everything).

DINO is a landmark in unsupervised vision — proving that supervision is not necessary for models to "understand" object boundaries and semantic categories.