Gradient Penalty

Gradient Penalty is a regularization technique used primarily in GAN training (WGAN-GP) — penalizing the norm of the discriminator's gradient with respect to its input, enforcing the Lipschitz constraint required by the Wasserstein distance formulation.

How Does Gradient Penalty Work?

• WGAN-GP: $\mathcal{L}_{GP} = \lambda \cdot \mathbb{E}_{\hat{x}}[( \nabla_{\hat{x}} D(\hat{x}) _2 - 1)^2]$
• Interpolation: $\hat{x} = \alpha x_{real} + (1-\alpha) x_{fake}$ with $\alpha \sim U(0,1)$.
• Target: The gradient norm should be 1 everywhere along interpolation paths.
• Paper: Gulrajani et al., "Improved Training of Wasserstein GANs" (2017).

Why It Matters

• GAN Stability: Replaced weight clipping in WGAN, dramatically improving training stability and sample quality.
• Lipschitz Constraint: Provides a soft, differentiable enforcement of the 1-Lipschitz constraint.
• Widely Adopted: Standard in most modern GAN architectures (StyleGAN, BigGAN, etc.).

Gradient Penalty is the smoothness enforcer for GANs — ensuring the discriminator function changes gradually, preventing the adversarial training from becoming unstable.