Zero-Init Residual

Zero-Init Residual is an initialization strategy where the last layer (e.g., final BN or convolution) of each residual branch is initialized to zero — making each residual block output the identity function at initialization, similar in spirit to SkipInit.

How Does Zero-Init Residual Work?

• Standard: Initialize all layers' weights with standard initialization (He, Xavier).
• Zero-Init: Set the weight (or BN $gamma$) of the last layer in each residual block to 0.
• Effect: At initialization, $F(x) = 0$ so $y = x + F(x) = x$ (identity).
• Used In: ResNet (zero-init last BN $gamma$), GPT-2 (zero-init last linear in each block).

Why It Matters

• Training Stability: Enables stable training of very deep networks from the start.
• Standard Practice: Used by default in modern ResNet implementations and GPT-style transformers.
• Gradient Flow: Identity initialization ensures gradients flow directly through skip connections at the start.

Zero-Init Residual is the default start for residual networks — making each block initially transparent so gradients flow freely from the very first step.