Overfitting

Overfitting and Underfitting — the two fundamental failure modes in machine learning, related to the bias-variance tradeoff.

Underfitting (High Bias)

• Model is too simple to capture the data pattern
• High training error AND high validation error
• Fix: Increase model capacity, train longer, reduce regularization

Overfitting (High Variance)

• Model memorizes training data including noise
• Low training error BUT high validation error
• Fix: More data, regularization (dropout, weight decay), data augmentation, early stopping

Diagnosis

• Plot training vs. validation loss curves
• If both high: underfitting
• If training low but validation high: overfitting
• If both low and converging: good fit

Bias-Variance Tradeoff

• Bias: Error from overly simple assumptions
• Variance: Error from sensitivity to training data fluctuations
• Total error = Bias$^2$ + Variance + Irreducible noise
• Goal: Minimize total error, not just one component

Modern deep learning often defies the classical tradeoff — very large models can generalize well with proper regularization (double descent phenomenon).