Mean Field Theory

Mean Field Theory applied to deep learning is a mathematical framework that analyzes how signals propagate through randomly initialized neural networks — determining the conditions under which forward signals and backward gradients neither explode nor vanish.

What Is Mean Field Theory for DNNs?

• Approach: Treat each neuron's pre-activation as a random variable. Compute the mean and variance of activations layer by layer.
• Order Parameters: Track the mean ($q^*$) and variance ($chi^*$) of activations across layers.
• Critical Point: At the "edge of chaos," signals propagate stably — neither growing nor shrinking.
• Initialization: Use mean field theory to derive optimal weight initialization (e.g., He, Xavier).

Why It Matters

• Trainability: Networks initialized at the critical point train fastest (gradients propagate well).
• Depth: Explains why very deep networks are hard to train without careful initialization.
• Batch Normalization: Mean field theory explains why BatchNorm works — it dynamically maintains criticality.

Mean Field Theory is the physics of signal propagation through deep networks — determining whether information flows or dies as it passes through millions of parameters.