Gradual Unfreezing

Gradual Unfreezing is an alternative name for Progressive Unfreezing — the fine-tuning strategy where pre-trained layers are incrementally unfrozen from top to bottom over the course of training, preventing catastrophic forgetting while allowing deep adaptation.

Gradual Unfreezing in Practice

• Identical To: Progressive Unfreezing. The terms are used interchangeably in the literature.
• Process: Start with classifier only -> unfreeze one layer group per epoch -> eventually train all layers.
• Key Setting: The number of epochs per unfreezing phase and the learning rate schedule during each phase.
• Context: Part of the ULMFiT framework alongside discriminative fine-tuning and STLR.

Why It Matters

• Robust Transfer: Prevents the "forgetting cliff" where aggressive fine-tuning destroys useful pre-trained features.
• Curriculum: Creates a natural curriculum from task-specific (top layers) to general (bottom layers).
• Best Practice: Recommended for any transfer learning scenario with limited downstream data.

Gradual Unfreezing is the same concept as progressive unfreezing — a careful, layer-by-layer approach to adapting pre-trained models to new tasks.