Random Matrix Theory

Random Matrix Theory (RMT) applied to deep learning is the mathematical study of the eigenvalue distributions of weight matrices and Hessian matrices — providing insights into network training dynamics, generalization, and the structure of the loss landscape.

What Does RMT Tell Us About DNNs?

• Weight Matrices: Well-trained networks develop heavy-tailed eigenvalue distributions (not the Marchenko-Pastur distribution of random matrices).
• Hessian Spectrum: The eigenvalue distribution of the Hessian reveals the curvature of the loss landscape — many near-zero eigenvalues + a few large ones.
• Generalization: The heavy-tail exponent $alpha$ of weight matrix eigenvalues correlates with generalization quality.

Why It Matters

• Diagnostics: Analyzing weight eigenspectra can predict model quality without validation data.
• Double Descent: RMT provides theoretical explanations for the double descent phenomenon.
• Pruning: Eigenvalue analysis identifies which weight matrices are over-parameterized (pruneable).

Random Matrix Theory is spectral analysis for neural networks — reading the eigenvalue fingerprints of weight matrices to understand what the network has learned.