Double Descent

Double descent is the phenomenon where test error follows a non-monotonic curve as model complexity increases—first decreasing (classical regime), then increasing (interpolation threshold), then decreasing again (modern regime). Classical U-curve: traditional bias-variance tradeoff predicts test error decreases with model complexity (reducing bias) then increases (increasing variance)—optimal at intermediate complexity. Double descent observation: (1) Under-parameterized regime—classical behavior, more parameters reduce bias; (2) Interpolation threshold—model just barely fits training data, very sensitive to noise, peak test error; (3) Over-parameterized regime—model has far more parameters than needed, test error decreases again despite perfectly fitting training data. Interpolation threshold: occurs when model capacity approximately equals training set size—the model is forced to fit every training point exactly but has no spare capacity for smooth interpolation. Why over-parameterization helps: (1) Implicit regularization—gradient descent on over-parameterized models finds smooth, low-norm solutions; (2) Multiple solutions—many parameter settings fit training data, optimizer selects generalizable one; (3) Effective dimensionality—not all parameters are used effectively. Double descent manifests in: (1) Model-wise—increasing parameters with fixed data; (2) Epoch-wise—increasing training epochs with fixed model; (3) Sample-wise—can occur with increasing data at certain model sizes. Practical implications: (1) Bigger models can be better—don't stop scaling at interpolation threshold; (2) More training can help—epoch-wise double descent argues against aggressive early stopping; (3) Standard ML intuition breaks—over-parameterized models generalize well despite memorizing training data. Connection to modern LLMs: large language models operate deep in the over-parameterized regime where double descent theory predicts good generalization despite massive parameter counts.