Squeeze-and-Excitation

Squeeze-and-Excitation (SE) is a channel attention mechanism that adaptively recalibrates channel-wise feature responses — by globally summarizing each channel (squeeze) and then learning inter-channel dependencies to produce per-channel importance weights (excitation).

How Does SE Work?

• Squeeze: Global average pooling across spatial dimensions: $z_c = \frac{1}{HW}\sum_{h,w} x_{c,h,w}$.
• Excitation: Two FC layers with ReLU and sigmoid: $s = \sigma(W_2 \cdot \text{ReLU}(W_1 \cdot z))$.
• Scale: Multiply each channel by its learned importance: $\hat{x}_c = s_c \cdot x_c$.
• Paper: Hu et al. (2018). Won ILSVRC 2017.

Why It Matters

• Universal: Can be inserted into any CNN (ResNet-SE, MobileNet-SE, EfficientNet uses SE).
• Minimal Cost: Only ~1% more parameters and ~0.5% more FLOPs for 1-2% accuracy improvement.
• Attention Pioneer: One of the first channel attention mechanisms, inspiring CBAM, ECA, and GE-Net.

SE is the channel importance learner — teaching the network to amplify useful channels and suppress uninformative ones.