Zero-Shot Segmentation

Zero-Shot Segmentation is the ability to segment objects that were not seen during training — allowing models to identify and delineate novel categories based on visual similarity, textual descriptions, or generic objectness properties.

What Is Zero-Shot Segmentation?

• Definition: Segmenting classes $C_{test}$ disjoint from training classes $C_{train}$.
• Mechanism: Usually aligns visual features with semantic word embeddings (like CLIP).
• Goal: Eliminate the need to collect masks for every possible object in the world.
• Input: Image + Category Name (e.g., "armadillo") -> Output: Mask of armadillo.

Why This Matters

• Long Tail: Standard datasets cover ~80 classes (COCO); real world has millions.
• Cost: Pixel-level annotation is expensive and slow to acquire.
• Scalability: New categories can be added simply by changing the text vocabulary.

Approaches

• Open-Vocabulary Segmentation: Using CLIP-based text encoders to classify pixel regions.
• Class-Agnostic Segmentation: Models like SAM that segment "objects" regardless of class.
• Generative Approaches: Using text-to-image diffusion models to generate masks.

Zero-Shot Segmentation is the key to open-world understanding — breaking the constraints of closed-set fixed-vocabulary computer vision systems.