Network Morphism

Network Morphism is a technique for transforming a trained neural network into a larger or differently structured network — while preserving its learned function exactly, allowing the new network to continue training from a warm start rather than from random initialization.

What Is Network Morphism?

• Definition: Function-preserving transformations on neural networks.
• Operations:
• Widen: Add more neurons/filters to a layer (pad with zeros).
• Deepen: Insert a new identity layer (initialized as pass-through).
• Reshape: Change kernel size while preserving learned features.
• Guarantee: $f_{new}(x) = f_{old}(x)$ for all inputs immediately after morphism.

Why It Matters

• NAS (Neural Architecture Search): Efficiently explore architectures by morphing one into another without retraining from scratch.
• Transfer Learning: Grow a small model into a larger one if more capacity is needed.
• Curriculum: Start small, grow as data or task complexity increases.

Network Morphism is neural evolution — growing neural networks organically like biological brains rather than rebuilding them from scratch.