Domain Shift

Domain shift (also called distribution shift) occurs when the statistical distribution of test/deployment data differs from the distribution of training data. It is one of the most common and impactful causes of model performance degradation in real-world AI deployments.

Types of Domain Shift

• Covariate Shift: The input distribution P(X) changes, but the relationship P(Y|X) stays the same. Example: A model trained on professional photos struggles with smartphone photos — the subjects are the same but the image quality differs.
• Label Shift (Prior Probability Shift): The output distribution P(Y) changes. Example: A disease diagnostic model trained when prevalence was 5% deployed when prevalence rises to 20%.
• Concept Drift: The relationship P(Y|X) itself changes — the same inputs should now produce different outputs. Example: Fraud patterns evolve over time.
• Dataset Shift: A general term encompassing any distributional difference between training and deployment data.

Why Domain Shift Happens

• Temporal Changes: The world changes over time — user behavior, language, trends, and data distributions evolve.
• Geographic Differences: A model trained in one region encounters different demographics, languages, or cultural contexts in another.
• Platform Changes: Data collected from different devices, sensors, or software versions has different characteristics.
• Selection Bias: Training data was collected differently than deployment data (e.g., hospital data vs. field data).

Detecting Domain Shift

• Performance Monitoring: Track model accuracy on labeled production data — degradation suggests shift.
• Distribution Comparison: Compare input feature distributions between training and production data using KL divergence, MMD, or statistical tests.
• Drift Detection Algorithms: DDM, ADWIN, and other algorithms detect distributional changes in data streams.

Mitigating Domain Shift

• Domain Adaptation: Explicitly adapt the model to the new domain using techniques like fine-tuning or domain-adversarial training.
• Domain Generalization: Train the model to be robust across domains from the start.
• Continuous Learning: Periodically retrain or update the model on recent data.
• Data Augmentation: Expose the model to diverse conditions during training.

Domain shift is the primary reason ML models degrade after deployment — monitoring for and adapting to distribution shifts is essential for maintaining production model quality.