Lazy Training Regime

Lazy Training Regime is a theoretical configuration where neural network weights barely change from their random initialization during training — the network acts essentially as a linear model in the feature space defined at initialization, as predicted by NTK theory.

What Is Lazy Training?

• Condition: Very wide networks with small learning rate and/or large initialization scale.
• Feature Freeze: The features (hidden representations) remain approximately fixed. Only the output layer's linear combination changes.
• NTK Regime: This is the regime described by Neural Tangent Kernel theory.
• Kernel Method: In lazy training, the network is equivalent to kernel regression with the NTK.

Why It Matters

• Theoretical Clarity: Lazy training is mathematically tractable — convergence and generalization can be proven.
• Poor Features: Lazy training doesn't learn features — it relies on random features from initialization. This limits performance.
• Practical: Real networks that achieve SOTA performance operate in the *feature learning* regime, not lazy training.

Lazy Training is the couch potato of neural networks — barely moving from initialization and relying on random features rather than learned ones.