Layer-Wise Learning Rates

Layer-Wise Learning Rates is a fine-tuning technique where different learning rates are applied to different layers of a pre-trained network — typically using lower rates for earlier (more general) layers and higher rates for later (more task-specific) layers.

How Does It Work?

• Decay Schedule: LR decreases exponentially from the top layer to the bottom. E.g., if top layer LR = $10^{-3}$, each layer below uses LR × decay factor (e.g., 0.95).
• Intuition: Early layers learn general features (edges, textures) that should change little. Later layers learn task-specific features that need more adaptation.
• Implementation: Assign separate parameter groups with different learning rates in the optimizer.

Why It Matters

• Better Fine-Tuning: Consistently outperforms uniform learning rate across all layers.
• Feature Preservation: Protects valuable low-level features from being overwritten during fine-tuning.
• Combined: Often used with progressive unfreezing for maximum transfer learning performance.

Layer-Wise Learning Rates are the gradient speed limits for neural layers — letting each level adapt at its own pace based on how much it needs to change.