Curriculum Learning

Curriculum Learning

What is Curriculum Learning?
Training models on examples ordered by difficulty, starting with easy examples and progressing to harder ones, mimicking human learning.

Curriculum Types

Predefined Curriculum
Order by known difficulty:

def difficulty_score(example):
    return len(example["text"])  # Simple: shorter is easier

# Sort by difficulty
curriculum = sorted(data, key=difficulty_score)

# Train in batches of increasing difficulty
for epoch in range(epochs):
    current_data = curriculum[:epoch_fraction * len(curriculum)]
    train(model, current_data)

Self-Paced Learning
Model determines what is easy:

def self_paced_weights(losses, threshold):
    # Easy examples have low loss
    weights = (losses < threshold).float()
    return weights

# Increase threshold over training
for epoch in range(epochs):
    threshold = initial + epoch * increment
    losses = model.get_losses(data)
    weights = self_paced_weights(losses, threshold)
    train(model, data, weights)

Difficulty Metrics
| Metric | Description |
|--------|-------------|
| Length | Shorter sequences are easier |
| Vocabulary | Common words are easier |
| Syntax complexity | Simple grammar is easier |
| Model loss | Low loss = easy for current model |
| Human annotation | Expert-labeled difficulty |

Curriculum Strategies
| Strategy | Description |
|----------|-------------|
| Baby Steps | Very gradual difficulty increase |
| One-pass | Single sweep from easy to hard |
| Interleaved | Mix difficulties, weighted toward easy |
| Anti-curriculum | Hard first (sometimes works) |

Benefits
- Faster convergence
- Better generalization
- More stable training
- Can help with difficult examples

Implementation Example

class CurriculumDataLoader:
    def __init__(self, data, difficulty_fn, pacing_fn):
        self.data = sorted(data, key=difficulty_fn)
        self.pacing_fn = pacing_fn

    def get_epoch_data(self, epoch):
        fraction = self.pacing_fn(epoch)
        cutoff = int(fraction * len(self.data))
        return self.data[:cutoff]

Use Cases
- Training LLMs (simple to complex examples)
- Computer vision (clear to ambiguous images)
- Reinforcement learning (easy to hard tasks)
- Low-resource scenarios (maximize data efficiency)