Shifted Window Attention

Shifted window attention is the cross-window communication mechanism in Swin Transformer that shifts the window partition grid by half a window size between consecutive transformer layers — enabling information flow across window boundaries while maintaining the computational efficiency of local window attention, effectively providing global context through alternating local computations.

What Is Shifted Window Attention?

• Definition: A technique where the spatial partitioning of attention windows is offset by (M/2, M/2) pixels between consecutive transformer layers, so that tokens at the boundary of one layer's windows are placed in the interior of the next layer's windows, enabling cross-boundary information exchange.
• Swin Transformer Core: The defining innovation of the Swin Transformer (Liu et al., 2021, "Hierarchical Vision Transformer using Shifted Windows") that solves the isolation problem of non-overlapping window attention.
• Alternating Pattern: Layer L uses regular window partition. Layer L+1 shifts the partition by (⌊M/2⌋, ⌊M/2⌋). Layer L+2 returns to regular partition. This alternation continues through all layers.
• Effective Global Receptive Field: After just a few alternating layers, information can propagate across the entire image through successive cross-window connections.

Why Shifted Window Attention Matters

• Breaks Window Isolation: Without shifting, tokens in different windows can never interact — shifted windows create bridges between previously isolated regions.
• Maintains Linear Complexity: The shifting operation itself has zero computational cost — it's just a change in how tokens are grouped, not an additional attention computation.
• Equivalent to Cross-Window Attention: Mathematically, alternating regular and shifted windows achieves similar information flow to overlapping windows or cross-window attention, but with lower implementation complexity.
• Hierarchical Global Context: Combined with patch merging (spatial downsampling), shifted windows enable global context to emerge naturally — early layers handle local features, later layers (with reduced spatial resolution) handle global relationships.
• SOTA Performance: Swin Transformer with shifted window attention achieved state-of-the-art results on ImageNet classification, COCO detection, and ADE20K segmentation upon release.

How Shifted Window Attention Works

Regular Window (Layer L):
- Feature map partitioned into non-overlapping M×M windows.
- Example with M=4 on an 8×8 map: 4 windows, each 4×4 = 16 tokens.
- Self-attention computed independently within each window.

Shifted Window (Layer L+1):
- Window grid shifted by (M/2, M/2) = (2, 2) pixels.
- New window boundaries now cross the centers of the previous windows.
- Tokens that were at the edges of regular windows are now in the middle of shifted windows.
- Cross-boundary information flows naturally through attention within the new windows.

Efficient Masking Implementation:
- Naive shifting creates irregular windows at image borders (different sizes).
- Cyclic Shift: Instead of padding, the feature map is cyclically shifted, creating full-size windows everywhere.
- Attention Mask: A mask prevents tokens from different original spatial regions from attending to each other within the same shifted window.
- This approach maintains a uniform window count and avoids padding overhead.

Information Flow Example

Swin Transformer Architecture

Performance Impact

Shifted window attention is the elegant solution to the locality-efficiency tradeoff in Vision Transformers — by simply alternating window positions between layers, Swin Transformer achieves global information flow with purely local computation, proving that cleverness in architecture design can be more powerful than brute-force compute.