Neural Network Gaussian Process

NNGP (Neural Network Gaussian Process) is a theoretical result showing that infinitely wide neural networks with random weights converge to Gaussian Processes — the distribution over functions defined by the random initialization becomes exactly a GP in the infinite-width limit.

What Is NNGP?

• Result: A single hidden-layer network with $n \rightarrow \infty$ neurons and random weights defines a GP with a specific kernel.
• Kernel: The NNGP kernel is determined by the activation function and the weight/bias distributions.
• Deep Networks: Each layer's GP kernel is defined recursively from the previous layer.
• Papers: Neal (1996), Lee et al. (2018), Matthews et al. (2018).

Why It Matters

• Bayesian DL: Provides exact Bayesian inference for infinitely wide networks (no MCMC needed).
• Uncertainty: Inherits GP's calibrated uncertainty estimates.
• Theory: Connects deep learning to the well-understood GP framework, enabling analytical results.

NNGP is the bridge between neural networks and Gaussian Processes — revealing that infinitely wide random networks are, mathematically, just kernel machines.