Axial Attention in ViT

Axial Attention is the factorized attention strategy that alternates row-wise and column-wise self-attention to cover entire images without quadratic compute — by sweeping first along the height axis and then along the width axis, the layer retains full-field context while shrinking complexity to O(HW(H+W)), which lets Vision Transformers scale to megapixel inputs for satellite, microscopy, and clinical imagery without blowing up memory.

What Is Axial Attention?

• Definition: A transformer block that splits multi-head attention into two sequential passes, one attending to each row and the other attending to each column, with interleaved projections and residual merges.
• Key Feature 1: Row pass aggregates information within each horizontal stripe of patches while keeping positional bias along the other axis constant.
• Key Feature 2: Column pass then propagates those summaries vertically so every pixel eventually receives contributions from all directions.
• Key Feature 3: Multi-head projections in each pass reuse the same heads so parameter count stays similar to standard attention.
• Key Feature 4: Relative or axial positional encodings keep track of sequence order along the active axis without full 2D tables.

Why Axial Attention Matters

• Resolution Scalability: Complexity reduces from quadratic in HW to linear in the sum (H+W), enabling 1,000+ patch grids.
• Hardware Friendliness: Each pass performs dense matrix multiplications of shape (N, C) rather than (N, N), keeping GPU memory stable.
• Global Receptive Field: Alternating passes allow even distant patches to exchange information in two hops, preserving global context.
• Gradient Stability: Two smaller attention operations avoid the extreme softmax behavior of a single huge matrix, improving training stability.
• Fine-Grain Control: Designers can mix axis order or skip one axis occasionally for dynamic sparsity without rewiring the entire backbone.

Axis Configurations

Row-then-Column:
- Row Stage: Attends to H long sequences of length W, capturing textures and horizontal edges.
- Column Stage: Attends to W sequences of length H, aggregating vertical context.
- Fusion: Residual addition merges both stages before the feedforward sublayer.

Column-then-Row:
- Order Swap: Useful when vertical semantics dominate (e.g., document pages).
- Symmetry: Maintains the same compute budget with axes swapped.

Hybrid:
- Local Axial Blocks: Combine with window attention to focus networks on both near neighbors and distant patches by alternating axial/global passes every few layers.

How It Works

Step 1: Project tokens to queries, keys, and values and reshape them into (axis_length, channel), then run the first attention pass along rows, normalizing by sqrt(dk) and applying softmax with per-row masks.

Step 2: Feed row outputs into the second pass that attends along columns, optionally including learned relative offsets, before adding the standard feed-forward module and layer norm.

Comparison / Alternatives

Tools & Platforms

• PyTorch / timm: AxialTransformer and ViT variants expose axial_config dictionaries for quick swapping.
• DeiT / Timm scripts: Support axial blocks as drop-in replacements for standard attention.
• DeepSpeed / Fairscale: Mesh-Tensor-Parallel training runs axial blocks with large batch support.
• Model Zoo: Axial-DeepLab and Axial-ResNet use the same axis-splitting principle outside of pure transformers.

Axial attention is the existential tool for scaling transformers to dense, high-resolution imaging tasks — it keeps every patch in play without ever materializing an enormous attention matrix, so practical deployments can see the whole field without compromising training budgets.