Model Soup

Model Soup is a model merging technique that averages the weights of multiple fine-tuned models — taking several models fine-tuned with different hyperparameters from the same pre-trained checkpoint and averaging their parameters, often outperforming the best individual model.

How Does Model Soup Work?

• Fine-Tune: Train multiple models from the same pre-trained checkpoint with different hyperparameters (learning rate, augmentation, etc.).
• Average: $\theta_{soup} = \frac{1}{K}\sum_k \theta_k$ (simple weight averaging).
• Greedy Soup: Iteratively add models to the soup only if they improve validation accuracy.
• Paper: Wortsman et al. (2022).

Why It Matters

• Free Accuracy: Outperforms the best individual model without additional inference cost.
• CLIP: Greedy model soup of CLIP fine-tunes achieved SOTA on ImageNet (2022).
• No Ensemble Cost: Unlike model ensembles ($K \times$ compute at inference), model soup has the same cost as one model.

Model Soup is the recipe for better models — averaging multiple fine-tuned models into one that is better than any individual ingredient.