Switchable Normalization

Switchable Normalization is a meta-normalization technique that learns to combine BatchNorm, InstanceNorm, and LayerNorm — using learnable weights to adaptively select the optimal normalization method for each layer and each channel during training.

How Does Switchable Normalization Work?

• Three Statistics: Compute BN, IN, and LN statistics simultaneously.
• Learnable Weights: $\hat{\mu} = \lambda_{BN}\mu_{BN} + \lambda_{IN}\mu_{IN} + \lambda_{LN}\mu_{LN}$ (and same for variance).
• Softmax: Weights are softmax-normalized -> always sum to 1.
• Learning: The network learns which normalization is best for each layer.
• Paper: Luo et al. (2019).

Why It Matters

• Automatic Selection: No need to manually choose between BN, IN, LN — the network decides.
• Task-Adaptive: Different tasks (classification, style transfer, detection) benefit from different normalizations.
• Insight: Analysis of learned weights reveals which normalization is preferred at different depths and for different tasks.

Switchable Normalization is letting the network choose its own normalization — a meta-learning approach that adapts normalization strategy per layer.