Dare

DARE (Drop and Rescale) is a model merging technique that randomly drops (zeros out) a fraction of fine-tuned parameter changes and rescales the remaining ones — reducing parameter interference between merged models while preserving the overall magnitude of task-specific updates.

How Does DARE Work?

• Task Vector: Compute $\tau = \theta_{fine} - \theta_{pre}$ (the fine-tuning delta).
• Drop: Randomly set a fraction $p$ of $\tau$'s elements to zero (Bernoulli mask).
• Rescale: Multiply remaining elements by $1/(1-p)$ to maintain expected magnitude.
• Merge: Average the dropped-and-rescaled task vectors from multiple models.
• Paper: Yu et al. (2024).

Why It Matters

• Less Interference: Dropping parameters reduces overlap and conflict between task vectors.
• Better Merging: DARE + TIES or DARE + simple averaging significantly outperforms naive averaging.
• LLM Merging: Widely used in the open-source LLM community for merging fine-tuned models.

DARE is dropout for model merging — randomly sparsifying task vectors before merging to reduce destructive interference between models.