Critical Learning Periods

Critical Learning Periods in neural networks are early training phases where the network's future performance and representation quality are largely determined — exposure to particular data distributions or training conditions during these critical periods has a lasting, often irreversible effect on the final model.

Critical Period Evidence

• Early Deficit: Degrading training data quality briefly during early training permanently damages final model performance.
• Late Deficit: The same degradation later in training has minimal lasting effect — the model recovers.
• Fisher Information: The Fisher information matrix peaks during critical periods — the network is maximally sensitive to data.
• Representation Crystallization: Internal representations "crystallize" during critical periods — becoming resistant to change later.

Why It Matters

• Data Quality: Ensuring high-quality data during early training is crucial — early corruption causes permanent damage.
• Curriculum Design: The order and timing of training data exposure matters — not just the data itself.
• Biology Analogy: Mirrors critical periods in biological development — early sensory experience shapes brain connectivity permanently.

Critical Learning Periods are the formative moments of training — early phases that irreversibly determine the model's representational capacity and final performance.