Overtraining

Overtraining is the training regime where additional optimization yields little generalization benefit and may overfit data idiosyncrasies - it can consume large compute while delivering minimal or negative practical return.

What Is Overtraining?

• Definition: Model continues training beyond efficient convergence point for target objectives.
• Symptoms: Validation gains flatten while compute cost and potential memorization risk increase.
• Context: Can occur when token budget is too high for model size or data novelty is low.
• Detection: Observed through diminishing downstream gains and unstable generalization metrics.

Why Overtraining Matters

• Compute Waste: Overtraining can consume budget better spent on data or architecture improvements.
• Safety: Extended exposure to repeated data may increase memorization and leakage risks.
• Opportunity Cost: Delays exploration of alternative training strategies.
• Benchmark Drift: May over-optimize narrow metrics without broad capability gains.
• Operational Efficiency: Timely stop criteria improve program throughput.

How It Is Used in Practice

• Stop Rules: Define multi-metric early-stop criteria beyond training loss alone.
• Data Refresh: Introduce new high-quality data if additional training is still required.
• Budget Reallocation: Shift compute to evaluation and targeted fine-tuning when plateau appears.

Overtraining is a common scaling inefficiency in large-model training programs - overtraining should be prevented with explicit stopping governance and cross-metric monitoring.