Noisy Labels Learning

Noisy labels learning (also called learning from noisy labels or robust training) encompasses machine learning techniques designed to train accurate models despite errors in the training labels. Since real-world datasets almost always contain some mislabeled examples, these methods are critical for practical ML.

Key Approaches

• Robust Loss Functions: Replace standard cross-entropy with losses that are less sensitive to mislabeled examples:
• Symmetric Cross-Entropy: Combines standard CE with a reverse CE term.
• Generalized Cross-Entropy: Interpolates between CE and mean absolute error.
• Truncated Loss: Caps the loss for examples with very high loss (likely mislabeled).

• Sample Selection: Identify and down-weight or remove likely mislabeled examples:
• Co-Teaching: Train two networks simultaneously, each selecting "clean" examples for the other based on small-loss criterion — examples with high loss are likely mislabeled.
• Mentornet: Use a separate "mentor" network to guide the main network's training by weighting examples.
• Confident Learning: Estimate the noise transition matrix and use it to identify mislabeled examples.

• Regularization-Based: Prevent the model from memorizing noisy labels:
• Mixup: Blend training examples together, smoothing decision boundaries and reducing overfitting to noise.
• Early Stopping: Stop training before the model starts memorizing noisy labels.
• Label Smoothing: Soften hard labels to reduce the impact of any single mislabeled example.

• Noise Transition Models: Explicitly model the probability of label corruption:
• Learn a noise transition matrix T where $T_{ij}$ = probability that true class i is labeled as class j.
• Use T to correct the loss function or the predictions.

When to Use

• Large-Scale Web Data: Datasets scraped from the internet invariably contain label errors.
• Distant Supervision: Programmatically generated labels have systematic noise patterns.
• Crowdsourced Data: Worker quality varies, producing noisy annotations.

Noisy labels learning is an important practical concern — methods like DivideMix and SELF have shown that models can achieve near-clean-data performance even with 20–40% label noise.