Curriculum Learning
Curriculum Learning is the training strategy that presents training examples to a neural network in a meaningful order — typically from easy to hard — rather than in random order — inspired by how humans learn progressively, this approach can improve convergence speed, final model quality, and training stability by initially building a foundation on simple patterns before tackling complex examples that require compositional understanding.
Core Idea (Bengio et al., 2009)
- Standard training: Shuffle data randomly, present uniformly.
- Curriculum learning: Define a difficulty measure → present easy examples first → gradually increase difficulty.
- Analogy: Students learn arithmetic before calculus, not randomly mixed.
Curriculum Strategies
| Strategy | Difficulty Measure | Scheduling |
|---|---|---|
| Loss-based | Training loss on each example | Start with low-loss samples |
| Confidence-based | Model prediction confidence | Start with high-confidence samples |
| Length-based | Sequence/sentence length | Short sequences first |
| Complexity-based | Label noise, class rarity | Clean, common examples first |
| Teacher-guided | Pre-trained model scores | Teacher ranks examples |
Pacing Functions
- Linear: Fraction of data available increases linearly over training.
- Exponential: Quick ramp → most data available early.
- Step: Discrete difficulty levels added at specific epochs.
- Root: Slow ramp → spends more time on easy examples.
Self-Paced Learning (SPL)
- Automatic curriculum: Model itself decides what's "easy."
- At each step, include samples with loss below threshold λ.
- Gradually increase λ → more difficult samples included.
- No need for external difficulty annotation.
Applications
| Domain | Curriculum Strategy | Benefit |
|---|---|---|
| Machine Translation | Short sentences → long sentences | 10-15% faster convergence |
| Object Detection | Easy (clear) images → hard (occluded) | Better mAP |
| NLP Pre-training | Simple text → complex text | Improved perplexity |
| RL | Easy tasks → hard tasks | Solves otherwise unlearnable tasks |
| LLM Fine-tuning | Simple instructions → complex reasoning | Better reasoning capability |
Anti-Curriculum (Hard Examples First)
- Counterintuitively, some tasks benefit from emphasizing hard examples.
- Focal loss (object detection): Down-weight easy examples, focus on hard ones.
- Online hard example mining (OHEM): Select hardest examples per batch.
- Works when the model is already competent (fine-tuning) and needs to improve on tail cases.
Practical Implementation
1. Pre-compute difficulty scores for all training examples.
2. Sort by difficulty (or assign curriculum bins).
3. Training loop: Sample from easy subset initially, gradually expand to full dataset.
4. Alternative: Weight sampling probability by difficulty level.
Curriculum learning is a simple yet powerful meta-strategy for improving training dynamics — by respecting the natural difficulty structure of training data, it can accelerate convergence and improve final quality, particularly for tasks with wide difficulty ranges where random sampling wastes early training capacity on examples the model cannot yet benefit from.