Post-Mortem

A post-mortem (also called a retrospective or incident review) is a structured after-incident analysis conducted to understand what happened, why it happened, and what changes will prevent recurrence. It is the primary mechanism for organizational learning from production failures.

Post-Mortem Structure

• Incident Summary: What happened, when, and who was affected. Include duration, severity, and blast radius.
• Timeline: Chronological sequence of events from detection through resolution. Include timestamps, actions taken, and who did what.
• Root Cause Analysis: The underlying cause(s) — not just "the server crashed" but why it crashed and why safeguards didn't prevent impact.
• Impact Assessment: Quantified impact — users affected, revenue lost, SLO budget consumed, safety implications.
• What Went Well: Highlight things that worked — effective alerts, fast response, good runbooks.
• What Went Poorly: Areas where the response was slow, confused, or ineffective.
• Action Items: Specific, assigned, time-bound improvements to prevent recurrence. Each action item has an owner and deadline.

Core Principles

• Blameless: Focus on systemic issues, not individual mistakes. "Why did the system allow this to happen?" not "Who made the mistake?"
• Thorough: Dig deep into root causes using the "Five Whys" technique or other root cause analysis methods.
• Actionable: Every post-mortem produces concrete action items, not vague promises.
• Shared: Post-mortems are shared widely to spread learning across the organization.

Post-Mortems for AI Systems

• Model Regression Post-Mortem: Why did the new model version perform worse? What evaluation gap allowed it through?
• Safety Incident Post-Mortem: How did harmful content bypass safety filters? What guardrails need strengthening?
• Cost Post-Mortem: What caused unexpected spending? How can cost controls prevent recurrence?

Best Practices

• Schedule Within 48 Hours: Conduct the post-mortem while details are fresh.
• Include All Participants: Everyone involved in the incident response should attend.
• Track Action Items: Use a tracking system to ensure action items are completed — unfinished action items from post-mortems undermine the entire process.

Post-mortems are the highest-leverage activity for improving system reliability — each incident, properly analyzed, makes the system and team stronger.