Triplet Attention

Triplet Attention is a lightweight attention mechanism that computes cross-dimension interactions between channel and spatial dimensions — using three parallel branches to capture (C×H), (C×W), and (H×W) attention, without any dimensionality reduction.

How Does Triplet Attention Work?

• Branch 1: Rotate tensor to (H, C, W) -> compute attention on (C, W) plane.
• Branch 2: Rotate tensor to (W, H, C) -> compute attention on (H, C) plane.
• Branch 3: Standard spatial attention on (H, W) plane.
• Aggregate: Average the outputs of all three branches.
• Paper: Misra et al. (2021).

Why It Matters

• No Reduction: Unlike SE/CBAM, uses no dimensionality reduction (MLP bottleneck) -> preserves all information.
• Cross-Dimension: Captures interactions between channel and spatial dimensions that separate attention misses.
• Negligible Cost: Almost zero additional parameters (only uses 7×7 convolutions for attention).

Triplet Attention is three-way cross-dimensional attention — capturing every possible interaction between channel, height, and width dimensions.