Infinite-Width Limit
The Infinite-Width Limit is a theoretical idealization in deep learning where the number of neurons in each hidden layer is taken to infinity — revealing that at this limit, randomly initialized neural networks become Gaussian processes, and gradient descent training becomes kernel regression in the Neural Tangent Kernel space — providing tractable mathematical models of neural network behavior that yield convergence guarantees, generalization bounds, and insights into scaling laws — while simultaneously highlighting that practical neural networks operate away from this limit, relying on finite-width feature learning that the infinite-width regime cannot capture.
What Happens at Infinite Width?
- Gaussian Process at Initialization: As hidden layer width n → ∞ (with independent random parameter initialization), by the Central Limit Theorem, the pre-activation distribution at each layer becomes Gaussian — and the function computed by the network becomes a Gaussian Process (GP) with covariance determined by the activation function and architecture.
- NTK Freezes During Training: As shown by NTK theory (Jacot et al., 2018), as width → ∞ trained with small learning rates, the Neural Tangent Kernel remains constant throughout training. Training dynamics simplify to linear kernel regression.
- No Bad Local Minima: In the infinite-width limit with overparameterization, gradient descent converges to a global minimum — the loss landscape becomes convex in function space.
- No Feature Learning: In the kernel regime, the network's internal representations do not change — only the output head weights (effectively) change. The network does not learn progressively better features; it performs fixed-basis function approximation.
Mathematical Framework
| Quantity | Finite Width | Infinite Width |
|---|---|---|
| Pre-activations | Correlated (non-Gaussian) | Independent Gaussians (CLT) |
| Network at init | Complex non-GP function | Exact Gaussian Process |
| Training dynamics | Nonlinear ODE in weight space | Linear ODE in function space (kernel regression) |
| Feature representations | Evolve (feature learning) | Fixed (no representation learning) |
| Generalization | Complex, architecture-dependent | RKHS norm regularization (kernel theory) |
Practical Relevance and Limitations
Where the limit helps:
- Initialization Design: Infinite-width analysis motivates proper weight initialization (e.g., He initialization for ReLU, LeCun for tanh) to ensure stable signal propagation and full-rank NTK at training start.
- Architecture Comparison: Comparing infinite-width GP/NTK kernels of different architectures provides insight into their inductive biases before training.
- Neural Scaling Theory: Infinite-width limit is the starting point for understanding how performance scales with width — corrections at finite width produce scaling law models.
- Bayesian Deep Learning: Infinite-width GP correspondence enables exact posterior inference tractable for small datasets.
Where the limit fails:
- Feature Learning: Real transformer and CNN performance relies on learning increasingly abstract and task-relevant representations — absent at infinite width.
- Sparse Representations: Finite-width networks develop sparse features; infinite-width representations are dense Gaussian.
- Generalization on Large Data: Kernel methods (infinite-width equivalent) often underperform finite-width networks on large-scale tasks — evidence they lack the inductive biases arising from finite-width training dynamics.
- Emergent Capabilities: The emergent capabilities of large language models (in-context learning, chain-of-thought reasoning) have no analog in the infinite-width regime.
Research Frontiers
- Mean-Field Theory: Studies the 1/n corrections to the infinite-width limit — capturing first-order feature learning effects.
- Tensor Programs (Greg Yang): A unified framework computing the limiting behavior of any architecture as width → ∞, enabling systematic analysis of Transformers, LSTMs, and normalization layers.
- Maximal Update Parameterization (muP): Derived from infinite-width analysis — enables training hyperparameters (learning rate, initialization) to transfer cleanly from small to large width, used in practice for scaling up LLMs efficiently.
The Infinite-Width Limit is the theoretical microscope for deep learning — an idealized mathematical lens that, while not accurately describing production neural networks, reveals the structural principles governing convergence, generalization, and architectural inductive biases, grounding practical design decisions in rigorous theory.