Dilated Attention in Vision
Dilated Attention is the sparse attention pattern that spreads tokens apart to cover a larger effective field while keeping the number of attended neighbors low — by stepping through spatial positions with a stride greater than one, the model reaches distant tokens with fewer dot products, letting it process ultra-high-resolution inputs without computing full dense matrices.
What Is Dilated Attention?
- Definition: An attention variant that attends to keys and values sampled every d tokens along height and width, mirroring the dilation trick from dilated convolutions.
- Key Feature 1: Stride d controls how sparse the attention grid becomes; d=1 recovers regular attention, while larger d zooms out to global cues.
- Key Feature 2: Dilation can increase across layers so that early layers see local patches and later layers capture global structure.
- Key Feature 3: Relative positional encodings pair with dilation to keep token alignment in check even with sparse sampling.
- Key Feature 4: Works naturally with hybrid attention where some heads use dilation and others stay dense.
Why Dilated Attention Matters
- Receptive Field Growth: Each layer jumps farther without costing extra tokens, making it ideal for megapixel segmentation or remote sensing.
- Compute Savings: Because keys/values are subsampled, complexity drops proportionally to 1/d^2.
- Multi-Scale Fusion: Dilation matches multi-scale features such as edges, corners, and textures by sampling at appropriate granularities.
- Generalization: Sparse yet structured sampling prevents overfitting to local noise while still preserving global alignment.
- Compatibility: Dilation plays well with axial and windowed patterns, letting architects mix and match patterns per head.
Dilation Strategies
Constant Dilation:
- Use a fixed d per layer for predictable receptive field.
- Suitable when the entire dataset has similar spatial scale requirements (e.g., microscopy slides).
Progressive Dilation:
- Increase d with depth, like 1, 2, 4, to gradually enlarge coverage.
- Matches the way CNNs increase receptive field by stacking dilated convolutions.
Hybrid Dilation:
- Assign dilation to half of the heads while keeping other heads dense, providing both detail and overview simultaneously.
How It Works / Technical Details
Step 1: Build attention neighborhoods by striding across the flattened spatial grid with step d, gathering keys and values only at those positions through strided slicing.
Step 2: Compute scaled dot product attention on the collected subsets, apply softmax, and gather weighted sums; combine with residual projections and feed-forward blocks.
Comparison / Alternatives
| Aspect | Dilated Attention | Local Window (Swin) | Global Attention |
|---|---|---|---|
| Coverage | Sparse global | Local with shifts | Dense global |
| Compute | O(N/k^2) depending on d | O(Nw^2) | O(N^2) |
| Detail | Varies with d | Fixed by w | Full detail |
| Best Use | Very high resolution | Balanced performance | Moderate sizes |
Tools & Platforms
- OpenMMLab: Offers dilated attention modules for detection and segmentation backbones.
- ConvNeXt / Timm: Provide dilation parameters inside attention_config dictionaries.
- TensorRT: Can fuse dilated attention with other vision blocks for inference.
- Visualization Tools: Use attention rollout maps to ensure dilation still covers critical objects.
Dilated attention is the sparse yet powerful lens that makes Vision Transformers see far without lifting a heavy quadratic burden — it jumps over nearby redundancy and attends to tokens that truly matter at distant locations.