Slanted Triangular Learning Rates

Slanted Triangular Learning Rates (STLR) is a learning rate schedule introduced in ULMFiT — that quickly increases the learning rate early in training (warm-up) and then linearly decays it for the remainder, creating a skewed triangular shape that balances fast convergence with careful fine-tuning.

How Does STLR Work?

• Shape: Sharp rise to peak (5-10% of training), then gradual linear decay (90-95%).
• Intuition: High LR early to quickly adapt to the new task's loss landscape. Low LR later for fine-grained optimization.
• Parameters: Peak LR, cut fraction (fraction of iterations spent warming up), and ratio (LR at start vs. peak).

Why It Matters

• Fast Convergence: The warm-up phase helps escape the pre-trained loss basin quickly.
• Stability: The long decay phase prevents overshooting and allows careful fine-tuning.
• Widely Adopted: The warm-up + decay paradigm (now often called "linear warmup + cosine/linear decay") is standard in transformer training.

STLR is the ramp-up-then-slow-down schedule — a simple but effective learning rate policy that became the blueprint for modern training schedules.