Split Attention

Split Attention is the attention mechanism used in ResNeSt (Split-Attention Networks) — which splits feature map channels into cardinal groups and further into radix splits, applying attention across splits within each group to dynamically weight different representations.

How Does Split Attention Work?

• Cardinal Groups: Like ResNeXt, divide channels into $K$ cardinal groups.
• Radix Splits: Within each group, further split into $R$ radix branches (different kernel sizes or transformations).
• Attention: Apply softmax attention across the $R$ radix splits within each group based on global channel statistics.
• Paper: Zhang et al. (2020, ResNeSt).

Why It Matters

• Dynamic Selection: Adaptively weights different feature representations within each group.
• ResNeSt: ResNeSt-50 significantly outperforms ResNet-50 and matches ResNet-152 accuracy.
• Downstream: Strong backbone for detection and segmentation when combined with Feature Pyramid Networks.

Split Attention is attention within groups — dynamically selecting the best representation from multiple radix splits within each cardinal group.