Timesformer

TimeSformer is the factorized video transformer that separates spatial attention and temporal attention to reduce computation while preserving long-range modeling - by decomposing full 3D attention into two simpler steps, it scales better to longer clips and higher resolutions.

What Is TimeSformer?

• Definition: Transformer architecture applying attention across spatial tokens within each frame and across time for each spatial location.
• Factorized Scheme: Spatial attention followed by temporal attention or alternative ordering.
• Token Format: Frame patches converted to sequence with positional encodings.
• Design Goal: Approximate full space-time reasoning at lower cost.

Why TimeSformer Matters

• Compute Reduction: Avoids quadratic blow-up of joint attention over all space-time tokens.
• Temporal Reach: Can model longer clips with practical memory usage.
• Transformer Flexibility: Retains global interactions along each axis.
• Benchmark Strength: Competitive on action recognition datasets.
• Architecture Clarity: Easy to reason about and ablate due to factorized blocks.

Factorization Variants

Space Then Time:
- Capture per-frame visual structure first.
- Then align dynamics across frames.

Time Then Space:
- Prioritize temporal traces at each patch location.
- Then integrate per-frame context.

Divided Attention Blocks:
- Alternate temporal and spatial blocks through depth.
- Improves receptive field with controlled complexity.

How It Works

Step 1:
- Patchify frames into tokens and apply spatial attention in each frame.

Step 2:
- Reorganize tokens by spatial index and apply temporal attention across frames, then classify actions.

TimeSformer is a divide-and-conquer transformer design that delivers strong video modeling without full joint attention cost - factorized attention makes long-clip processing significantly more practical.