Video Object Segmentation

Video Object Segmentation (VOS) is a dense prediction task that assigns a pixel-level mask to specific objects in every frame of a video — requiring the model to propagate a segmentation mask from the first frame (semi-supervised) or automatically discover primary objects (unsupervised).

What Is VOS?

• Task: Tracking the exact shape (mask) of an object as it moves and deforms.
• Semi-Supervised (One-Shot): User provides mask for Frame 1 -> Model segments Frames 2 to $N$.
• Zero-Shot: Model automatically segments the most salient moving object.

Why It Matters

• VFX & Rotoscoping: Automating the tedious task of cutting out actors for special effects.
• Video Editing: "Change the color of this car" throughout the entire video clip.
• Robotics: Precise manipulation of moving objects (catching a ball).

Challenges:
- Occlusion: Object gets covered -> Re-appears later (Re-identification).
- Drift: Small errors in frame 2 accumulate to massive errors by frame 100.

Video Object Segmentation is the "Green Screen" of AI — capable of digitally extracting moving subjects from their environment without a studio setup.