Phase Transitions in Training

Phase Transitions in Training are sudden, discontinuous changes in model behavior during training — analogous to physical phase transitions (ice → water), neural networks can undergo abrupt shifts in their learned representations, capabilities, or performance metrics.

Types of Training Phase Transitions

• Grokking: Sudden generalization after prolonged memorization.
• Capability Emergence: Sudden appearance of new capabilities at certain model scales or training durations.
• Loss Spikes: Sharp, temporary increases in loss followed by rapid improvement to a new, lower plateau.
• Representation Change: Discontinuous reorganization of internal representations — features suddenly restructure.

Why It Matters

• Predictability: Phase transitions make model behavior hard to predict — capabilities appear suddenly.
• Scaling Laws: Some capabilities emerge only at specific scales — phase transitions define threshold model sizes.
• Safety: Sudden capability emergence complicates AI safety analysis — capabilities can appear without warning.

Phase Transitions are sudden leaps in learning — discontinuous changes in model behavior that challenge smooth, predictable training assumptions.