Strided Attention

Strided Attention is a sparse attention pattern where each token attends to every $s$-th token in the sequence — creating a dilated attention pattern that efficiently captures long-range dependencies without computing full $O(N^2)$ attention.

How Does Strided Attention Work?

• Pattern: Token $i$ attends to tokens ${i - s, i - 2s, ...}$ (every $s$-th previous token).
• Stride $s$: Typically $s = \sqrt{N}$ so each token attends to $\sqrt{N}$ positions.
• Combined: Often paired with local attention — local captures nearby context, strided captures distant context.
• Paper: Child et al. (2019, Sparse Transformer).

Why It Matters

• Long-Range: Captures dependencies across the full sequence length with only $O(\sqrt{N})$ attention per token.
• Complementary: Combined with local attention, provides both fine-grained local and coarse global context.
• Image Generation: Originally designed for autoregressive image generation (attending to spatially distant pixels).

Strided Attention is dilated convolution for attention — skipping tokens at regular intervals to efficiently reach across the entire sequence.