Neighborhood Attention
Neighborhood Attention is the locally dynamic attention pattern that slides a window over the feature map while keeping each center token attention-aware of its immediate neighbors — unlike fixed convolution kernels, the attention weights change with content, so edges, textures, and small objects get adaptive emphasis without computing full global maps.
What Is Neighborhood Attention?
- Definition: A structured attention that restricts each query to a fixed-size K×K neighborhood around itself, optionally grouping the neighborhood into horizontal and vertical stripes for efficiency.
- Key Feature 1: Windows overlap so that every pixel participates as both query and key, ensuring continuity.
- Key Feature 2: Dynamic weights are recalculated per token, making the operation content-sensitive rather than purely geometric.
- Key Feature 3: Block-sparse implementation exploits the regular grid to batch gather operations effectively on GPUs.
- Key Feature 4: Windows can grow with depth or be dilated to widen the effective receptive field.
Why Neighborhood Attention Matters
- Local Detail: Maintains the precise local structure necessary for segmentation, detection, and medical imaging.
- Cost Control: Complexity is O(HWk^2) with small k (e.g., 7), so it scales linearly with the number of patches.
- Receptive Field Expansion: Dilation or stacking multiple layers gradually expands the contextual footprint without global cost.
- Smooth Transitions: Overlapping windows prevent blocking artifacts because tokens appear in multiple local neighborhoods.
- Plug-In Flexibility: Can replace Swin or other windowed attention modules without rewiring offset computations.
Neighborhood Configurations
Regular Window:
- Use K=3 or 5, with padding to keep spatial shapes constant.
- Each token attends to its immediate neighbors in a square layout.
Dilated Neighborhood:
- Skip tokens using dilation factor d, allowing the kernel to cover a broader area while still remaining local.
- Good for high-resolution tasks where receptive field must grow smoothly.
Grouped Neighborhoods:
- Partition channels or heads into groups focusing on different neighborhood ranges (e.g., near vs far neighbors).
How It Works / Technical Details
Step 1: Extract the K×K patch around each query via unfolding or strided gather, producing per-query keys and values.
Step 2: Compute attention scores using scaled dot product, apply softmax within the patch, and aggregate the values. Optionally add relative positional biases to encode spatial shifts.
Comparison / Alternatives
| Aspect | Neighborhood | Swin (Window) | Global |
|---|---|---|---|
| Context | Local adaptive | Local static geometry | Global |
| Learnable | Yes | Only via biases | Yes |
| Blocking | Negative but mitigated by overlap | Possible if shifts missing | None |
| Complexity | O(Nk^2) | O(Nw^2) | O(N^2) |
Tools & Platforms
- MMCViT / timm: Provide NeighborhoodAttention modules with dilation controls.
- MMSegmentation: Uses neighborhood attention for high-resolution segmentation heads.
- Detectron2: Supports custom attention heads for both heads and FPN features.
- TVM / Triton: Can compile neighborhood attention kernels for efficient inference on GPUs.
Neighborhood attention is the adaptive local focus that keeps transformers precise on small structures without blowing up compute — it gives every token a neighborhood-aware view while keeping the cost linear with image size.