Neural Tangent Kernel

Neural Tangent Kernel (NTK) is a theoretical framework that describes the training dynamics of infinitely wide neural networks — showing that in the infinite-width limit, neural networks behave like linear models in a fixed feature space defined by the kernel at initialization.

What Is the NTK?

• Definition: $Theta(x, x') = \nabla_\theta f(x, \theta)^T \nabla_\theta f(x', \theta)$ where $f$ is the network output.
• Key Result: In the infinite-width limit, the NTK is constant during training.
• Implication: Training dynamics become equivalent to kernel regression with the NTK.
• Paper: Jacot, Gabriel & Hongler (2018).

Why It Matters

• Theory: Provides the first rigorous characterization of when and why neural network training converges.
• Lazy Training: In the NTK regime, weights barely change from initialization (lazy training).
• Limitation: Real networks operate in the feature learning regime, not the lazy regime — NTK describes the easier, less interesting case.

NTK is the theoretical microscope on neural network training — revealing the elegant mathematics hidden in the dynamics of gradient descent.