Loss Function Basics

Loss Function — the mathematical function that measures how wrong the model's predictions are, providing the signal that guides training through gradient descent.

Classification Losses

• Cross-Entropy Loss: $L = -\sum y_i \log(\hat{y}_i)$ — standard for classification. Penalizes confident wrong predictions heavily
• Binary Cross-Entropy (BCE): For two-class problems or multi-label classification
• Focal Loss: Down-weights easy examples, focuses on hard ones. Developed for object detection with class imbalance

Regression Losses

• MSE (Mean Squared Error): $L = \frac{1}{n}\sum(y - \hat{y})^2$ — penalizes large errors quadratically
• MAE (Mean Absolute Error): $L = \frac{1}{n}\sum|y - \hat{y}|$ — more robust to outliers
• Huber Loss: MSE for small errors, MAE for large errors (best of both)

Other Important Losses

• Contrastive Loss: Pull similar pairs together, push dissimilar apart (CLIP, SimCLR)
• Triplet Loss: Anchor closer to positive than negative by margin
• KL Divergence: Measure difference between two probability distributions (used in VAE, knowledge distillation)
• CTC Loss: For sequence-to-sequence without alignment (speech recognition)

Choosing the right loss function is one of the most impactful design decisions — it directly defines what the model optimizes for.