Weight Normalization

Weight Normalization is a reparameterization technique that decouples the magnitude and direction of weight vectors — representing each weight vector as $w = g \cdot v/ v $, where $g$ is a learnable scalar (magnitude) and $v/ v $ is the unit direction.

How Does Weight Normalization Work?

• Decomposition: $w = g \cdot \hat{v}$ where $\hat{v} = v / v $.
• Parameters: Learn $g$ (scalar magnitude) and $v$ (direction vector) instead of $w$ directly.
• Gradient: Gradients with respect to $v$ are projected orthogonal to $v$, decoupling magnitude from direction updates.
• Paper: Salimans & Kingma (2016).

Why It Matters

• Faster Convergence: Decoupling magnitude and direction improves conditioning of the optimization.
• No Batch Statistics: Unlike BatchNorm, weight normalization doesn't depend on batch statistics -> works for any batch size.
• Inference: No difference between training and inference behavior (unlike BatchNorm).

Weight Normalization is polar coordinates for neural network weights — separating how big the weights are from which direction they point for smoother optimization.