Weight Averaging

Weight averaging is a model combination technique that averages parameters from multiple trained models — creating merged models that often outperform individual components through ensemble-like effects.

What Is Weight Averaging?

• Definition: Average corresponding weights from multiple models.
• Formula: w_merged = (w_A + w_B) / 2, or weighted average.
• Requirement: Models must share same architecture.
• Result: Single model combining capabilities.
• No Training: Merge without additional compute.

Why Weight Averaging Matters

• Improved Performance: Often beats individual models.
• Combine Strengths: Merge specialist models.
• Regularization: Averaging smooths weight space.
• Community: Foundation of Stable Diffusion model merging.
• Efficiency: No training required.

Averaging Methods

• Simple Average: (A + B) / 2.
• Weighted Average: α*A + (1-α)*B, control contribution.
• SLERP: Spherical interpolation in weight space.
• Task Arithmetic: Add/subtract task-specific directions.

When It Works

• Models trained on same architecture.
• Models fine-tuned from same base.
• Similar training data distributions.
• Complementary specializations.

Example

merged = {}
for key in model_a.keys():
    merged[key] = 0.7 * model_a[key] + 0.3 * model_b[key]

Weight averaging is the simplest and often effective model merging — combining capabilities without training.