Deformable Attention

Deformable Attention is an attention mechanism that attends to a small set of key sampling points around a reference point — instead of attending to all spatial positions, reducing the $O(N^2)$ complexity of full attention to $O(N \cdot K)$ where $K$ is the number of sampling points.

How Does Deformable Attention Work?

• Reference Points: Each query has a reference point (e.g., grid position or predicted object center).
• Sampling Offsets: $K$ learnable offsets from the reference point (typically $K = 4-8$).
• Attention Weights: Learned attention weights for each of the $K$ sampling points.
• Multi-Scale: Can sample from multiple feature map scales simultaneously.
• Paper: Zhu et al., "Deformable DETR" (2021).

Why It Matters

• Efficiency: $O(N \cdot K)$ vs. $O(N^2)$ for full attention -> enables high-resolution feature maps.
• DETR Acceleration: Deformable DETR converges 10× faster than vanilla DETR.
• Detection Standard: The standard attention mechanism in modern detection transformers.

Deformable Attention is attention that samples smartly — attending to a few learnable positions instead of everything, making transformer detection practical.