Grokking

Grokking is a training phenomenon where a model suddenly generalizes long after memorizing the training data — the model first achieves perfect training accuracy (memorization), then after many more training steps, test accuracy suddenly jumps from near-random to near-perfect, exhibiting delayed generalization.

Grokking Characteristics

• Memorization First: Training loss drops to zero quickly — the model memorizes all training examples.
• Delayed Generalization: Test accuracy remains at chance for many epochs after memorization.
• Phase Transition: Generalization appears suddenly — a sharp, discontinuous improvement in test accuracy.
• Weight Decay: Grokking is strongly influenced by regularization — weight decay encourages the transition from memorization to generalization.

Why It Matters

• Understanding: Challenges the assumption that generalization happens gradually alongside training loss reduction.
• Training Duration: Models may need training far beyond overfitting to achieve generalization — premature stopping can miss grokking.
• Mechanistic: Research reveals grokking involves learning structured, generalizable algorithms that replace memorized lookup tables.

Grokking is generalization after memorization — the surprising phenomenon where models learn to generalize long after perfectly memorizing their training data.