Mode Interpolation

Mode Interpolation (Linear Mode Connectivity) is a model merging technique based on the observation that fine-tuned models from the same pre-trained checkpoint are connected by a linear path of low loss — enabling simple weight interpolation between models.

How Does Mode Interpolation Work?

• Two Models: $\theta_A$ and $\theta_B$, both fine-tuned from the same pre-trained $\theta_0$.
• Interpolate: $\theta_\alpha = (1-\alpha) \theta_A + \alpha \theta_B$ for $\alpha \in [0, 1]$.
• Low Loss Path: The loss along the interpolation path is roughly constant (linear mode connectivity).
• Paper: Frankle et al. (2020), Neyshabur et al. (2020).

Why It Matters

• Model Soup: Linear mode connectivity is the theoretical foundation for model soup working.
• Multi-Task: Interpolating between task-specific models creates multi-task models.
• Pre-Training Matters: Models fine-tuned from different random initializations are NOT linearly connected — shared pre-training is key.

Mode Interpolation is the straight line between fine-tuned models — the remarkable finding that models from the same checkpoint live in the same loss valley.