Vivit

ViViT is the video vision transformer family that tokenizes clips into spatiotemporal tubelets and applies transformer attention over space and time - it extends ViT principles to video with multiple factorization options for efficiency.

What Is ViViT?

• Definition: Vision transformer architecture for video with tubelet embedding and temporal modeling modules.
• Tokenization Strategy: Tubelets capture local motion by grouping pixels across consecutive frames.
• Model Variants: Joint space-time attention or factorized spatial-then-temporal encoders.
• Output Tasks: Action recognition and video understanding benchmarks.

Why ViViT Matters

• Transformer Transfer: Brings strong image-transformer design into video domain.
• Flexible Scaling: Factorized variants support larger clips under memory limits.
• Long-Range Modeling: Better global temporal context than short-kernel 3D CNNs in many settings.
• Research Influence: Helped establish transformer-first direction for video.
• Extensibility: Compatible with self-supervised pretraining and multimodal fusion.

ViViT Design Options

Joint Encoder:
- Attend over all spatiotemporal tokens together.
- Strong but expensive for long clips.

Factorized Encoder:
- Apply spatial transformer then temporal transformer.
- Better efficiency with minimal quality loss in many tasks.

Hybrid Heads:
- Combine global pooled tokens with temporal heads.
- Useful for long-video adaptation.

How It Works

Step 1:
- Split clip into tubelets, project to embeddings, and add positional encodings for space and time.

Step 2:
- Process tokens with selected ViViT attention scheme and classify actions with final head.

ViViT is a foundational video-transformer formulation that made tubelet tokenization and factorized attention mainstream - it remains a key reference for modern transformer video architecture design.