Joint Space-Time Attention

Joint space-time attention is the full attention strategy that computes interactions across all tokens from every frame simultaneously - it provides maximal modeling flexibility but incurs very high memory and compute cost as clip length and resolution increase.

What Is Joint Space-Time Attention?

• Definition: Self-attention over flattened token set combining temporal and spatial dimensions in one sequence.
• Token Count: N equals T times H times W tokenized patches.
• Complexity: Quadratic in N, causing rapid scaling bottlenecks.
• Primary Benefit: Direct global coupling between any two space-time positions.

Why Joint Attention Matters

• Expressive Power: Captures arbitrary long-range dependencies without factorization constraints.
• Modeling Simplicity: One unified attention operation avoids hand-designed decomposition.
• Research Baseline: Useful upper-bound reference for architectural studies.
• Short-Clip Strength: Effective when token count is moderate.
• Interpretability: Full attention map shows direct global relation structure.

Practical Limitations

Memory Explosion:
- Attention matrix size grows with square of token count.
- Quickly becomes infeasible for long videos.

Throughput Constraints:
- Large quadratic operations reduce training and inference speed.
- Demands high-end hardware and careful batching.

Scaling Workarounds:
- Downsample input, shorten clips, or use sparse approximations.
- Often replaced by factorized or windowed attention designs.

How It Works

Step 1:
- Flatten all spatiotemporal tokens and project to queries, keys, and values.

Step 2:
- Compute full attention matrix, aggregate values globally, and pass through transformer blocks.

Joint space-time attention is the most expressive but most expensive transformer video operation - it is valuable as a conceptual gold standard and as a practical option only under constrained token budgets.