Fisher-Weighted Averaging

Fisher-Weighted Averaging is a model merging technique that weights each parameter by its Fisher information — parameters that are more important for a task (higher Fisher information) are weighted more heavily during averaging, preserving critical task-specific knowledge.

How Does Fisher-Weighted Averaging Work?

• Fisher Information: $F_i = \mathbb{E}[(\nabla_{\theta_i} \log p(y|x, \theta))^2]$ — measures how sensitive the loss is to each parameter.
• Weighted Average: $\theta_{merged,i} = \frac{\sum_k F_i^{(k)} \cdot \theta_i^{(k)}}{\sum_k F_i^{(k)}}$ (Fisher-weighted).
• Intuition: If parameter $i$ is crucial for task $A$ but unimportant for task $B$, use task $A$'s value.
• Paper: Matena & Raffel (2022).

Why It Matters

• Importance-Weighted: Not all parameters are equally important — Fisher weighting respects this.
• Better Than Uniform: Outperforms simple averaging by preserving each task's critical parameters.
• EWC Connection: Related to Elastic Weight Consolidation, using Fisher information to prevent catastrophic forgetting.

Fisher-Weighted Averaging is importance-aware merging — using information theory to determine which task's version of each parameter matters most.