Masked Image Modeling

Masked image modeling (MIM) is the self-supervised training paradigm where a model reconstructs hidden image patches from visible context - this forces ViT encoders to learn semantic and structural representations instead of memorizing local texture shortcuts.

What Is Masked Image Modeling?

• Definition: Randomly mask a subset of patches and train model to predict pixel or token targets for masked regions.
• Mask Ratio: Often high, such as 40 to 75 percent, to create meaningful reconstruction challenge.
• Target Choices: Raw pixels, quantized tokens, or latent features.
• Backbone Fit: ViT token structure makes masking straightforward and efficient.

Why MIM Matters

• Unlabeled Learning: Extracts supervision from raw image structure.
• Context Reasoning: Encourages understanding of global layout and object relationships.
• Transfer Performance: Pretrained encoders perform strongly on many downstream tasks.
• Data Scalability: Benefits from large unlabeled corpora.
• Architectural Flexibility: Supports lightweight or heavy decoders depending on objective.

MIM Variants

Pixel Reconstruction:
- Predict normalized pixel values for masked patches.
- Simple but can emphasize low-level detail.

Token Reconstruction:
- Predict discrete visual tokens from tokenizer.
- Often yields stronger semantic abstraction.

Feature Reconstruction:
- Match teacher or latent feature targets.
- Balances detail and semantic fidelity.

Training Flow

Step 1:
- Sample mask pattern, remove masked patches from encoder input, and process visible tokens.

Step 2:
- Decoder predicts masked targets and optimization minimizes reconstruction loss over masked positions.

Masked image modeling is a versatile and scalable self-supervised framework that teaches ViTs to infer missing visual context from surrounding evidence - it is now a core building block for modern vision pretraining.