Axial Attention

Axial Attention is a factored attention mechanism that decomposes 2D self-attention into two sequential 1D attention operations — first along the height axis, then along the width axis, reducing complexity from $O(N^2)$ to $O(N \sqrt{N})$.

How Does Axial Attention Work?

• Height Attention: Each position attends to all positions in its column.
• Width Attention: Each position then attends to all positions in its row.
• Sequential: Apply height attention, then width attention (or vice versa).
• Position Encoding: Relative position embeddings added to queries and keys.
• Paper: Ho et al. (2019), Wang et al. (2020, Axial-DeepLab).

Why It Matters

• Scalability: Enables self-attention on high-resolution images (512×512 and above).
• Segmentation: Axial-DeepLab achieves strong panoptic segmentation results.
• Image Generation: Used in efficient attention for image generation models.

Axial Attention is 2D attention factored into 1D — decomposing full spatial attention into efficient row-then-column operations.