Ibot Pre-Training

iBOT pre-training is the self-supervised vision transformer method that combines masked patch prediction with online token-level self-distillation - it aligns global and local representations across views, producing strong semantic features without manual labels.

What Is iBOT?

• Definition: Image BERT style training that uses teacher-student framework with masked tokens and patch-level targets.
• Dual Objective: Global view alignment plus masked patch token prediction.
• Online Distillation: Teacher network updates by momentum from student weights.
• Token Supervision: Encourages meaningful patch embeddings, not only image-level embeddings.

Why iBOT Matters

• Dense Feature Quality: Patch-level targets improve segmentation and localization transfer.
• Label-Free Learning: Learns high-level semantics from unlabeled data.
• Strong Benchmarks: Delivers competitive results on linear probe and fine-tuning tasks.
• Representation Diversity: Combines global invariance with local detail modeling.
• Modern Influence: Informs many later token-centric self-supervised methods.

Training Mechanics

View Augmentation:
- Generate multiple crops and perturbations of each image.
- Feed views to student and teacher branches.

Teacher-Student Targets:
- Teacher produces soft targets for global and token-level outputs.
- Student matches targets with masked and unmasked inputs.

Momentum Update:
- Teacher parameters follow exponential moving average of student.
- Stabilizes targets during training.

Implementation Notes

• Temperature Settings: Critical for stable soft target distributions.
• Mask Ratio: Influences balance between local reconstruction and global alignment.
• Batch Diversity: Large and diverse batches improve representation quality.

iBOT pre-training is a powerful blend of masked modeling and self-distillation that yields highly transferable ViT representations without labels - it is especially effective when dense token quality is a priority.