Axial Attention for Video

Axial attention for video is the factorized attention method that applies separate attention passes along temporal, height, and width axes - this decomposition reduces complexity while still enabling broad space-time context exchange.

What Is Axial Attention in Video?

• Definition: Attention computed one axis at a time instead of over full flattened spatiotemporal sequence.
• Axis Sequence: Temporal pass, height pass, and width pass in configurable order.
• Complexity Benefit: Lower cost than full joint attention at comparable receptive reach.
• Use Cases: Long clips, high resolution inputs, and memory-constrained training.

Why Axial Attention Matters

• Scalable Context: Preserves long-range dependencies with manageable token operations.
• Modular Design: Axis-specific blocks are easy to tune and analyze.
• Hardware Friendliness: Smaller attention matrices improve throughput.
• Quality Retention: Often close to joint-attention accuracy when layered effectively.
• Hybrid Compatibility: Works well with local windows and multiscale backbones.

Axial Video Pipeline

Temporal Axis Pass:
- Connect corresponding spatial tokens across frames.
- Capture motion and event progression.

Spatial Axis Passes:
- Height and width attention propagate contextual structure within frames.
- Build spatial coherence after temporal update.

Residual Integration:
- Residual and normalization layers stabilize multi-pass composition.
- Deep stacking increases effective receptive field.

How It Works

Step 1:
- Reshape token tensor to isolate one axis and run attention for that axis only.

Step 2:
- Repeat for remaining axes, merge outputs with residual paths, and continue through network depth.

Axial attention for video is a practical decomposition that approximates global spatiotemporal reasoning at much lower cost - it is a strong option for long-form or high-resolution video transformers.