Sliding Window Attention Patterns
Sliding Window Attention is a sparse attention pattern that restricts each token to attending only to nearby tokens within a fixed local window — reducing the computational complexity from O(n²) to O(n × w) where w is the window size (e.g., 512 or 4096 tokens), enabling processing of much longer sequences with bounded memory while capturing the local dependencies that dominate most natural language and code understanding tasks.
What Is Sliding Window Attention?
- Definition: An attention pattern where each token at position i can only attend to tokens in the range [i-w, i] (for causal/autoregressive) or [i-w/2, i+w/2] (for bidirectional), where w is the window size. Tokens outside the window receive zero attention weight.
- The Motivation: Full attention is O(n²) — for a 100K token sequence, that's 10 billion attention computations per layer. But most relevant context for any given token is nearby (within a few hundred to a few thousand tokens). Sliding window exploits this locality.
- The Key Insight: Even with local-only attention, information can propagate across the full sequence through multiple layers. With window size w=4096 and L=32 layers, the effective receptive field is w × L = 131,072 tokens — covering the full context through cascading local interactions.
Complexity Comparison
| Attention Type | Memory | Compute | Effective Receptive Field |
|---|---|---|---|
| Full Attention | O(n²) | O(n²) | Full sequence (every token sees all others) |
| Sliding Window | O(n × w) | O(n × w) | w per layer, w × L across L layers |
| Global + Sliding | O(n × (w + g)) | O(n × (w + g)) | Full (via global tokens) |
For n=100K, w=4096: Full attention = 10B operations; Sliding window = 410M operations (24× less).
How It Works
| Position | Attends To (w=4, causal) | Cannot See |
|---|---|---|
| Token 1 | [1] | — |
| Token 2 | [1, 2] | — |
| Token 3 | [1, 2, 3] | — |
| Token 5 | [2, 3, 4, 5] | Token 1 (outside window) |
| Token 10 | [7, 8, 9, 10] | Tokens 1-6 |
| Token 1000 | [997, 998, 999, 1000] | Tokens 1-996 |
Combining Sliding Windows with Other Patterns
| Combination | How It Works | Used In |
|---|---|---|
| Sliding + Global tokens | Special tokens (CLS, task tokens) attend to ALL positions | Longformer, BigBird |
| Sliding + Dilated | Additional attention to every k-th token for long-range | Longformer (upper layers) |
| Sliding + Random | Random attention connections for probabilistic global coverage | BigBird |
| Different window sizes per layer | Lower layers: small window (local); Upper layers: large window (broader) | Many efficient transformers |
| Sliding + Full attention layers | Every N-th layer uses full attention | Mistral design choice |
Models Using Sliding Window Attention
| Model | Window Size | Approach | Max Context |
|---|---|---|---|
| Mistral 7B | 4,096 | Sliding window in every layer | 32K (via rolling KV-cache) |
| Longformer | 256-512 | Sliding + global + dilated | 16K |
| BigBird | 256-512 | Sliding + global + random | 4K-8K |
| Gemma-2 | 4,096 (alternating) | Alternating sliding/full layers | 8K |
Sliding Window Attention is the foundational sparse attention pattern for efficient transformers — exploiting the locality of language by restricting each token to attend only within a fixed neighborhood, reducing memory and compute from quadratic to linear in sequence length, while maintaining full-sequence information flow through multi-layer receptive field expansion and combination with global attention tokens.