Model Stitching

Model Stitching is a technique that combines layers from different pre-trained models into a single network — inserting a small "stitching layer" (typically a 1×1 convolution or linear layer) between layers from different models to align their representations.

How Does Model Stitching Work?

• Source Models: Two or more pre-trained models trained independently.
• Cut Points: Select layer $i$ from model $A$ and layer $j$ from model $B$.
• Stitch: Insert a trainable stitching layer between layer $i$ and layer $j$.
• Train Stitch: Train only the stitching layer (freeze source model weights).
• Result: Front of model $A$ + stitch + back of model $B$.

Why It Matters

• Representation Analysis: Reveals how similar representations are between different models at different layers.
• Efficiency: Create models with novel accuracy-efficiency trade-offs by combining parts of different architectures.
• Transfer: Transfer the "front end" of one model with the "back end" of another.

Model Stitching is Frankenstein assembly for neural networks — combining parts of different models with minimal adaptation layers.