Inpainting as Pretext

Inpainting as Pretext is a self-supervised learning task where the model is trained to reconstruct missing regions of an image — requiring the network to understand scene context, object structure, and texture patterns to fill in the blanks convincingly.

How Does Inpainting Work?

• Process: Mask out a patch (or multiple patches) of the image. The network predicts the missing pixels.
• Architecture: Typically encoder-decoder (U-Net or similar) with adversarial loss.
• Loss: L2 reconstruction + perceptual loss + GAN discriminator loss.
• Paper: Pathak et al., "Context Encoders" (2016).

Why It Matters

• Context Understanding: To fill in a missing region, the model must understand what should be there based on surrounding context.
• Generative Features: Learns representations useful for both discriminative and generative downstream tasks.
• MAE Connection: Masked Autoencoders (MAE) are a modern evolution of the inpainting pretext concept using Vision Transformers.

Inpainting is the fill-in-the-blank test for vision — teaching networks to understand images by challenging them to reconstruct what they can't see.