Rollback

Rollback

Rollback mechanisms provide an essential safety net for AI system deployments, enabling immediate reversion to a previously known stable version if production issues are detected with a new model or configuration. Strategy: Blue/Green or Canary deployment facilitates instant rollback; keep old (Blue) environment running until new (Green) is verified. Triggers: automated alerts (error rate spike, latency increase) or manual intervention. State consistency: unlike code, model rollbacks must consider data schema changes or vector store compatibility. Artifact management: model registry (MLflow, W&B) tracks precise versions of weights, code, and config; "latest" tag should point to stable. Mean Time To Recovery (MTTR): efficient rollback minimizes downtime impact. Database migrations: if model update included schema change, rollback plan must include database reversion script. Communication: automated notification to stakeholders when rollback occurs. Testing: regular game day exercises where rollbacks are triggered intentionally ensure process works. Fast, reliable rollback encourages deployment confidence and innovation velocity.