Fixed Attention Patterns

Fixed Attention Patterns are predetermined, static sparsity patterns for self-attention — where the set of positions each token can attend to is defined before training and does not depend on the input content, enabling efficient implementation.

Types of Fixed Patterns

• Block Diagonal: Divide sequence into blocks. Each token attends only within its block.
• Dilated/Strided: Regular stride patterns across the sequence.
• Axial: Attend along one dimension at a time (for 2D data).
• Global Tokens: Designate a few tokens as "global" that attend to and are attended by all tokens.
• Combination: Longformer/BigBird combine local windows + global tokens + random connections.

Why It Matters

• Predictable: Fixed patterns enable highly optimized CUDA kernels and hardware-aware implementations.
• Proven: Longformer and BigBird demonstrate that fixed patterns can match full attention on long document tasks.
• Scalable: Complexity is $O(N)$ for most fixed patterns (linear in sequence length).

Fixed Attention Patterns are the predetermined wiring diagrams for attention — trading flexibility for efficiency with hand-designed connectivity structures.