Progressive Unfreezing

Progressive Unfreezing is a fine-tuning strategy where layers are gradually unfrozen from top to bottom during training — starting by training only the classifier head, then progressively unfreezing deeper layers, allowing each layer to adapt without catastrophically disrupting the pre-trained features.

How Does Progressive Unfreezing Work?

• Phase 1: Train only the classification head (all layers frozen).
• Phase 2: Unfreeze the last block/layer. Train with small learning rate.
• Phase 3: Unfreeze the next deeper block. Continue training.
• Phase N: Eventually all layers are unfrozen, training end-to-end with very small learning rate for deep layers.

Why It Matters

• Catastrophic Forgetting Prevention: Gradually exposing pre-trained layers to gradients prevents sudden destruction of learned features.
• Small Datasets: Especially beneficial when downstream data is limited — avoids overfitting early layers.
• ULMFiT: Howard & Ruder (2018) demonstrated this technique for NLP transfer learning.

Progressive Unfreezing is gentle adaptation — slowly waking up each layer of the network to let it adjust to the new task without forgetting what it already knows.