Orthogonal Initialization

Orthogonal Initialization is a weight initialization method that initializes weight matrices as orthogonal (or near-orthogonal) matrices — ensuring that the linear transformation preserves the norm of the input at initialization, providing optimal signal propagation through deep networks.

How Does Orthogonal Initialization Work?

• Process: Generate a random matrix $A$ -> compute QR decomposition $A = QR$ -> use $Q$ (orthogonal matrix) as the initial weight.
• Property: $ Qx = x $ — an orthogonal matrix preserves vector norms.
• Gain: Optionally multiply by a gain factor to account for the activation function (e.g., $\sqrt{2}$ for ReLU).

Why It Matters

• Perfect Propagation: At initialization, signals neither grow nor shrink through orthogonal layers.
• RNNs: Particularly important for recurrent networks where weights are applied repeatedly over time steps.
• Theory: Theoretically optimal for signal propagation in linear networks (all singular values = 1).

Orthogonal Initialization is the norm-preserving start — beginning training with transformations that perfectly preserve signal magnitude through every layer.