Spectral Normalization

Spectral Normalization is a weight normalization technique that constrains each weight matrix's spectral norm (largest singular value) to a target value — controlling the Lipschitz constant of each layer to stabilize training and improve adversarial robustness.

How Spectral Normalization Works

• Spectral Norm: $\sigma(W) = \max_{|v|=1} |Wv|$ — the largest singular value of the weight matrix.
• Normalization: $\hat{W} = W / \sigma(W)$ — divide by the spectral norm so each layer has Lipschitz constant ≤ 1.
• Power Iteration: Estimate $\sigma(W)$ efficiently using one step of power iteration per training step.
• Application: Applied to every weight matrix (linear, conv) in the network.

Why It Matters

• GAN Stability: Originally introduced for stabilizing GAN discriminator training (Miyato et al., 2018).
• Robustness: Constraining spectral norms improves adversarial robustness by limiting sensitivity.
• Lightweight: Power iteration adds negligible computational cost — one extra matrix-vector product per layer.

Spectral Normalization is capping the sensitivity of each layer — normalizing weight matrices to control how much each layer amplifies perturbations.