Mode Connectivity

Mode Connectivity is the observation that distinct minima of neural network loss functions are often connected by low-loss paths — meaning there exist smooth trajectories in parameter space between different trained solutions without crossing high-loss barriers.

What Is Mode Connectivity?

• Discovery: Garipov et al. (2018) showed that independently trained networks can be connected by simple curves (quadratic Bezier) with near-constant loss along the path.
• Linear Connectivity: A stronger form where a straight line between two minima has low loss everywhere.
• Loss Barriers: Traditional view (convex optimization) expected high barriers between minima. Mode connectivity shows the landscape is more benign.

Why It Matters

• Ensemble Understanding: Explains why model averaging and snapshot ensembles work well.
• Landscape Geometry: Reveals that the loss landscape has a connected low-loss manifold, not isolated valleys.
• Training: Models trained with different initializations find solutions in the same "basin."

Mode Connectivity is the hidden highways between solutions — low-loss tunnels connecting apparently different minima in the vast parameter landscape.