Discriminative Fine-Tuning

Discriminative Fine-Tuning is the combined strategy of using different learning rates for different layers (layer-wise LR) during fine-tuning — a term coined by the ULMFiT paper to describe the practice of discriminating between layers based on their depth when setting hyperparameters.

What Is Discriminative Fine-Tuning?

• Core Idea: Each layer group gets a different learning rate: $eta_l = eta_{base} \cdot \gamma^{(L-l)}$ where $l$ is the layer index and $\gamma$ is the decay factor.
• Motivation: Different layers encode different levels of abstraction and require different amounts of adaptation.
• ULMFiT: Introduced as part of the ULMFiT framework (Howard & Ruder, 2018) alongside progressive unfreezing and slanted triangular LR.

Why It Matters

• Transfer Learning Standard: Now a standard practice in NLP and increasingly in vision fine-tuning.
• Robust: Reduces sensitivity to the overall learning rate choice by using relative scaling between layers.
• Synergy: Works best when combined with progressive unfreezing and warm-up scheduling.

Discriminative Fine-Tuning is personalized training for each layer — acknowledging that not all layers need the same amount of adaptation for a new task.