Metainit

MetaInit is a meta-learning-based initialization method that uses gradient descent to find weight initializations that minimize the curvature of the loss landscape — searching for starting points where training dynamics will be most favorable.

How Does MetaInit Work?

• Objective: Find initial weights $\theta_0$ that minimize the trace of the Hessian $\text{tr}(H(\theta_0))$ (surrogate for loss landscape curvature).
• Process: Use gradient descent on the initialization itself — not on the loss, but on a meta-objective about the loss landscape.
• Effect: Produces starting points in flat, well-conditioned regions of the loss landscape.
• Paper: Dauphin & Schoenholz (2019).

Why It Matters

• Principled: Directly optimizes the quantity that determines training difficulty (curvature).
• BatchNorm-Free: Can enable training of deep networks without BatchNorm by finding better starting points.
• Theory: Connects initialization to the loss landscape geometry literature (flat vs. sharp minima).

MetaInit is learning how to start — using meta-learning to find the optimal initial conditions for neural network training.