Label Shift

Label shift (also called prior probability shift or target shift) is a type of distribution shift where the distribution of output labels P(Y) changes between training and deployment, while the class-conditional input distribution P(X|Y) remains the same.

Intuitive Example

• A spam detector is trained when 10% of emails are spam. At deployment, spam increases to 40%. The characteristics of spam and non-spam emails haven't changed — but their proportions have shifted.
• A disease classifier trained on hospital data where 2% of patients have the disease, deployed in a screening program where 15% have it.

Why Label Shift Matters

• Models implicitly learn class prior probabilities from training data. If the prior changes, the model's calibration and decision boundaries become suboptimal.
• Precision and recall are affected — a model tuned for rare positives will under-predict when positives become more common.
• Threshold-based decisions break — the optimal classification threshold depends on class priors.

Detection

• Monitor Class Proportions: Track the distribution of predicted classes over time. Significant changes in prediction proportions may indicate label shift.
• Black Box Shift Detection (BBSD): Use model predictions to estimate whether the label distribution has changed.
• Confusion Matrix Monitoring: Track precision, recall, and other metrics across time windows.

Correction Methods

• Importance Weighting: Re-weight training examples based on the ratio of target-to-source class proportions. If class A is 2× more common in deployment, upweight class A training examples by 2×.
• Expectation Maximization: Iteratively estimate the new class prior and adjust the model's outputs accordingly.
• Threshold Adjustment: Modify the classification threshold to account for the new class balance without retraining.
• Calibration: Re-calibrate model probabilities on data representative of the deployment distribution.

Label Shift vs. Other Shifts

• Covariate Shift: Input P(X) changes, P(Y|X) stays same.
• Label Shift: Output P(Y) changes, P(X|Y) stays same.
• Concept Drift: P(Y|X) itself changes — fundamentally different and harder to handle.

Label shift is one of the simpler forms of distribution shift to correct because the fundamental input-output relationship hasn't changed — only the proportions have.