Lifelong Distillation

Lifelong Distillation is a continual learning technique that uses knowledge distillation to prevent catastrophic forgetting — when learning a new task, the model distills its own knowledge of previous tasks from its earlier version, maintaining old competencies while acquiring new ones.

How Does Lifelong Distillation Work?

• Process: Before learning task $t+1$, save a copy of the model trained on tasks $1..t$.
• Loss: $\mathcal{L} = \mathcal{L}_{new\_task} + \alpha \cdot \mathcal{L}_{KD}(model_{new}, model_{old})$
• Effect: The KD term penalizes the model for changing its outputs on old-task inputs.
• Related: Learning without Forgetting (LwF) uses this exact approach.

Why It Matters

• No Data Storage: Unlike experience replay, lifelong distillation doesn't need to store old training data.
• Privacy: Old data may be unavailable or confidential — distillation preserves knowledge without data.
• Scalability: Works for arbitrarily many sequential tasks.

Lifelong Distillation is self-teaching through time — a model using its past self as a teacher to remember what it learned before.