Postmortems without blame theatre

core5-2020-5050-150150+

Blameless isn't a tone, it's a method: keep asking what made the wrong action look reasonable at the time. Every incident gets one, including the near misses.

Also available as a standalone Playbook →

Blameless gets misunderstood as a tone. People think it means being nice about the outage, avoiding names, and saying "the system" a lot.

It's a method, and the method is one question asked repeatedly: what made the wrong action look like the right action at the time.

Because the person who ran the command believed it was safe. The reviewer who approved it thought it was fine. Everyone involved was doing what looked correct from where they stood, with the information they had. If your analysis ends at "they made a mistake," you've stopped exactly one step before the finding, which is whatever made the mistake look reasonable.

That's not generosity. It's the only version that produces a fix, because you can't prevent a person from having been wrong, and you can prevent a system from making wrongness easy.

Every incident, including the near misses

Mandatory, for everything that met the bar, and specifically including the ones where nothing bad actually happened.

Near misses are the most valuable and the most commonly skipped. The deploy that would have taken production down if someone hadn't noticed in time. The backup that turned out to be empty, discovered during a test rather than during a restore. All the same information, none of the damage, and no urgency forcing anyone to look at it. So they get skipped, and the same conditions produce a real incident eight weeks later.

Making it mandatory removes the judgment call about whether this one was serious enough. That judgment always drifts toward "no," particularly in a busy quarter, and particularly for the near misses.

What "mandatory" doesn't mean is heavyweight. A small incident deserves half a page and twenty minutes. Keeping the ritual proportional is what keeps it universal.

Timeline first, and keep interpretation out of it

The first section is a timeline of facts with timestamps, and nothing else.

09:14 deploy of version X. 09:22 error rate on checkout begins rising. 09:31 first customer report. 09:34 alert fires. 09:41 engineer begins investigating.

What doesn't belong: "the alert was too slow," "we should have caught this," "unfortunately nobody was watching the dashboard." All of those are interpretations, and if they enter the timeline the discussion becomes an argument about interpretations rather than a shared view of what happened.

Build it from the incident channel log, which is why that log matters. Reconstruction from memory produces a timeline bent toward the conclusion people have already reached.

The gaps in the timeline are usually where the finding is. The nine minutes between the error rate rising and the alert firing. The twelve minutes between the alert and someone investigating. Those numbers are the actual material, and nobody argues with a timestamp.

Counterfactuals hide the cause

The most useful piece of technique I know for these: strike out every sentence containing "should have," "could have," or "failed to."

"The engineer should have noticed the migration was still running." That sentence feels like an analysis and contains none. It describes a world that didn't happen instead of the one that did, and it terminates the investigation, because once you've established what someone should have done, there's nothing left to ask.

Should haveStops hereWhat they sawThe finding
Once you have established what someone should have done, there is nothing left to ask.

Replace it with the real question. What did they see? What was on their screen? What did the tool tell them? Why did the migration's state look complete when it wasn't?

Now you're somewhere useful, and the finding will usually be something structural: the status endpoint reports complete when the job is queued, and everyone has misread it, and this is the second time.

Same for "root cause," which I'd mostly avoid as a phrase. Real incidents have three or four contributing conditions that had to line up, and fixing only the last one leaves the rest waiting for the next coincidence.

Action items that actually happen

Most postmortem actions never get done, and the reason is that they're written as intentions rather than as work.

Three rules fix most of it.

Each action has one named person, not a team. "The platform team will improve alerting" is nobody's Tuesday.

Each has a date, and it's soon. An action item due in three months will be done in three months' time by someone who has forgotten the incident, if at all.

And each goes into the same tracker as everything else, competing with normal work, visible in the same place. Actions kept in the postmortem document are actions in a graveyard.

Then, the part people skip: someone reviews the open ones monthly and closes the ones that aren't going to happen. Explicitly, with a note saying why. That sounds like giving up and it's the opposite: a list of eighty open action items teaches everyone the process is fiction, and a list of six real ones gets done.

The bar for what becomes an action item should also be higher than most teams set it. Not everything noticed in a postmortem deserves work. Some things are genuinely fine to leave, and saying so out loud is healthier than an action item nobody intends to do.

When individual judgment is fair to discuss

The one place blameless gets stretched into something it isn't: it doesn't mean individual performance is never a topic.

If the same person repeatedly makes the same category of error, that's a real thing and it belongs in a conversation with their manager. Privately, as a performance discussion, on the same timeline as any other. What it doesn't belong in is the postmortem document, and the reason is practical rather than diplomatic: the moment a postmortem can affect someone's standing, everyone starts writing defensively, and you've destroyed the information the process exists to collect.

Two separate channels, both real. The postmortem asks what the system permitted. The manager asks whether this person is doing the job. Neither substitutes for the other, and an organisation that only has the first one is avoiding something.

Worth being explicit about this rather than leaving it implicit, because the version of "blameless" where nobody is ever accountable for anything is a real failure mode, and people are right to be suspicious of it.

The meeting itself

Small: the people involved, plus whoever is running it. Not an audience.

Someone who wasn't in the incident should run it, because the people who were are still carrying it, and they'll defend decisions rather than examine them.

Read the document in silence for the first ten minutes rather than presenting it. Presenting produces a narrative, and narratives smooth over exactly the confusing parts that are worth talking about.

And write it up in a place others can read. The best onboarding material an engineering org has is a folder of its own postmortems, in order, because it's the honest history of how the system actually behaves rather than how it's documented.