Skipinit

SkipInit is an initialization technique for residual networks that multiplies each residual path by a learnable scalar initialized to zero — ensuring that at initialization, the network is equivalent to a shallow network (identity function), enabling training of extremely deep networks without BatchNorm.

How Does SkipInit Work?

• Standard Residual: $y = x + F(x)$
• SkipInit: $y = x + \alpha \cdot F(x)$ where $\alpha$ is initialized to 0.
• At Init: $y = x$ (identity mapping). The network is effectively 1 layer deep.
• During Training: $\alpha$ grows from 0, gradually introducing the residual contributions.
• Paper: De & Smith (2020).

Why It Matters

• No BatchNorm Needed: Enables training 10,000+ layer ResNets without any normalization.
• Simplicity: One scalar parameter per residual block. Trivial to implement.
• Theory: Connects to the insight that deep networks train best when they start as shallow networks and gradually deepen.

SkipInit is starting as nothing — initializing each residual pathway to zero so the model begins as a simple identity and gradually builds complexity.