Scaled Initialization

Scaled Initialization is a weight initialization strategy that scales initial weights based on network depth — ensuring that the variance of activations and gradients remains stable as they propagate through many layers, preventing signal explosion or vanishing.

How Does Scaled Initialization Work?

• Principle: Scale weights by $1/\sqrt{n}$ where $n$ depends on the method (fan-in, fan-out, or both).
• Depth Scaling: For very deep networks, additionally scale by $1/\sqrt{L}$ or $1/\sqrt{2L}$ where $L$ is the number of layers.
• Examples: GPT-2 scales residual projections by $1/\sqrt{N}$ where $N$ is the number of layers.
• Goal: Maintain unit variance of activations and gradients throughout the entire network at initialization.

Why It Matters

• Deep Networks: Without proper scaling, signals either explode or vanish after many layers.
• Foundation Models: Scaled initialization is critical for training 100+ layer transformers.
• Theory: Connects to mean field theory — maintaining criticality at initialization.

Scaled Initialization is the volume control for deep networks — setting initial weights at exactly the right level so signals propagate cleanly through every layer.