Spectral Normalization

Spectral Normalization is a weight normalization technique that constrains the spectral norm (largest singular value) of each weight matrix to 1 — enforcing a 1-Lipschitz constraint on the layer, which stabilizes GAN discriminator training without gradient penalty's computational cost.

How Does Spectral Normalization Work?

• Normalization: $\bar{W} = W / \sigma(W)$ where $sigma(W)$ is the largest singular value of $W$.
• Power Iteration: $sigma(W)$ is estimated efficiently using one step of power iteration per training step.
• Cost: Negligible — one matrix-vector multiply per layer per step.
• Paper: Miyato et al. (2018).

Why It Matters

• GAN Stability: Stabilizes discriminator training without the per-sample cost of gradient penalty.
• Efficiency: Much cheaper than WGAN-GP (which requires gradient computation through the discriminator).
• Universal: Applied in BigGAN, StyleGAN, and most modern GANs as a default technique.

Spectral Normalization is the singular value leash — keeping each layer's transformation gentle enough to produce stable, high-quality GAN training.