Catastrophic Forgetting Prevention

Catastrophic Forgetting Prevention encompasses techniques that prevent a neural network from losing previously learned knowledge when trained on new tasks — a critical challenge in continual learning, transfer learning, and fine-tuning scenarios.

Key Prevention Techniques

• Regularization-Based:
• EWC (Elastic Weight Consolidation): Penalize changes to weights important for previous tasks.
• L2-SP: Regularize toward the pre-trained weights.
• Architecture-Based:
• Progressive Networks: Add new columns for new tasks, freeze old columns.
• PackNet: Prune and freeze subnetworks for each task.
• Replay-Based:
• Experience Replay: Store and replay examples from previous tasks.
• Generative Replay: Use a generative model to synthesize past data.

Why It Matters

• Continual Learning: The #1 obstacle to lifelong learning in neural networks.
• Fine-Tuning: Aggressive fine-tuning on small datasets can destroy pre-trained knowledge.
• Practical: Any system deployed over time (recommendation engines, autonomous vehicles) faces catastrophic forgetting.

Catastrophic Forgetting Prevention is the art of learning new tricks without forgetting old ones — the central challenge in making neural networks truly adaptable over time.