Loss Function

Cross-entropy loss is the standard objective function for language model training, measuring the difference between predicted token probability distributions and actual (one-hot) target distributions, with minimization corresponding to maximizing likelihood of correct tokens. Mathematical form: L = -Σ log(p(y_true | context)), where p is model's predicted probability for the correct next token. Equivalently, cross-entropy between one-hot target and predicted distribution. Why cross-entropy: information-theoretic foundation (measures bits needed to encode true distribution using predicted one), equivalent to maximum likelihood estimation (minimizing cross-entropy = maximizing log-likelihood), and provides meaningful gradients (pushes probability mass toward correct tokens). Perplexity connection: perplexity = exp(cross-entropy loss)—interpretable as effective vocabulary size of uncertainty. Training dynamics: early training sees rapid loss decrease (learning common patterns); later training shows slower improvement (learning rare patterns). Label smoothing: softening one-hot targets (0.9 correct, 0.1/V others) can improve generalization. Cross-entropy variants: teacher-forced (standard), scheduled sampling (gradually using model predictions), and reinforcement learning objectives (optimizing non-differentiable metrics). Understanding loss dynamics—plateaus, spikes, divergence—is essential for diagnosing training issues.