Temporal Shift Module

Temporal Shift Module (TSM) is the parameter-free operation that shifts a fraction of feature channels across neighboring timesteps to inject temporal context into 2D backbones - it adds motion awareness with almost zero additional FLOPs.

What Is TSM?

• Definition: Channel shift operator where some channels move forward in time and some move backward, while remaining channels stay unchanged.
• Design Goal: Provide temporal interaction without expensive 3D convolutions.
• Placement: Inserted into residual blocks of standard 2D CNNs.
• Cost Profile: No learned parameters, minimal arithmetic overhead.

Why TSM Matters

• Efficiency Breakthrough: Near-free temporal modeling for edge and real-time systems.
• Backbone Reuse: Existing image models can become video models with light modification.
• Strong Accuracy-Speed Tradeoff: Good performance under tight latency budgets.
• Deployment Simplicity: Uses basic tensor shift operations supported by common runtimes.
• Scalable Integration: Can be combined with segment sampling and transformer heads.

TSM Mechanics

Channel Partitioning:
- Split channels into forward-shift, backward-shift, and static groups.
- Typical ratio keeps most channels static to preserve spatial signal.

Temporal Mixing:
- Forward-shift channels import previous-step context.
- Backward-shift channels import next-step context.

Residual Compatibility:
- Shift operation wraps around convolution blocks and maintains shape consistency.
- Easy insertion into existing ResNet-like pipelines.

How It Works

Step 1:
- Reshape features by time dimension and apply deterministic channel shifts between adjacent timesteps.

Step 2:
- Process shifted features with standard 2D convolutions and aggregate predictions across clips.

Temporal Shift Module is a lightweight temporal context mechanism that upgrades image backbones for video with minimal compute overhead - it is a practical option when efficiency is a primary deployment constraint.