Incident response under pressure

The instinct that makes someone a good engineer is the one that damages an incident: the urge to understand what's happening before acting.

In normal work that's correct. In an incident it's expensive, because every minute spent diagnosing is a minute customers are broken, and quite often there's a way to make the pain stop that doesn't require knowing the cause at all.

Roll back the deploy. Turn off the flag. Fail over. Shed load. Any of those might resolve the symptom in ninety seconds, leaving the interesting question for a calmer afternoon.

Mitigate first, understand second. It's the single most useful thing to say out loud during an incident, and it usually needs saying, because the room will be full of people who want to know why.

Three roles, held by different people

Even at ten engineers, the separation matters, and it matters most when there are only two of you and one is doing everything.

The incident commander runs the incident and does not fix anything. They decide what's being tried, they keep track of who's doing what, and they're the one asking "is this mitigating or diagnosing." The moment the commander gets into the code, the incident loses its coordinator, and you get three people independently trying things on the same system.

Communications handles everyone who isn't fixing it. The status page, the customer-facing teams, the executive asking for an update. This role exists specifically so the people working on the problem are not the people answering questions about it, because context-switching mid-incident is where the mistakes come from.

Operations is the people doing the hands-on work.

At a small company, one person can hold commander and comms, and even that is a stretch. What can't work is the same person fixing and commanding, which is the default arrangement in most small teams and the reason small-team incidents run long.

The commander doesn't have to be senior, incidentally. It's a coordination job, and some of the best I've seen were done by people who couldn't have fixed the problem themselves, which is arguably an advantage.

Severity that changes behaviour

Severity levels are useful only if each one triggers a different response. If they just categorise things for a report afterwards, they're paperwork.

Three is enough for most companies. The top one means customers are broken, wake people up, incident commander now, status page immediately. The middle one means degraded, business hours, one person owns it, tell support. The bottom means annoying, ticket, no ceremony.

The thing to get right is the declaration threshold, and the useful rule is: declare early and downgrade freely. Most organisations do the opposite. Nobody wants to over-react, so everyone waits to be sure, and the average incident starts twenty minutes late.

Make downgrading routine and public, so it costs nothing. "I've declared this a sev one, we'll drop it if it turns out to be one customer" should be an unremarkable sentence, said often.

The other tell of a healthy system: engineers can declare incidents without asking a manager. Any approval step in the declaration path adds delay in the exact place delay is most expensive.

Talking to customers while you're still confused

The rule is that you can say what you know without knowing the cause, and the thing customers want is not an explanation. It's evidence that a competent human is aware and working on it.

So the first update goes out within minutes and says: something is broken, here's what's affected, we're on it, next update at a specific time. That's it. No cause, no estimate, no speculation.

Then keep the promise about the next update, even when there's nothing new. "Still working on it, no change, next update in thirty minutes" maintains trust in a way that silence destroys. Silence is read as absence, and absence is what makes people escalate through other channels, which costs you more time.

Two things not to do. Don't guess at the cause in public, because you'll be wrong in the first hour reasonably often, and the correction reads worse than the original problem. And don't give a restoration estimate you can't stand behind. "We expect to be back within the hour" that turns into four hours is the thing customers remember.

The tone that works is plain and unhedged. Corporate softening ("some users may be experiencing intermittent difficulties") when the whole product is down convinces nobody and costs you credibility that you'll want later.

Knowing when to stop

Long incidents are lost on fatigue, not on difficulty.

After a few hours, the people who've been on it are the worst-placed people in the building to make decisions about it, and they're also the least likely to notice. They have the most context, which is exactly what makes handing over feel wasteful and makes staying feel responsible.

So make the handover a rule rather than a judgment call. Four hours, or whatever number you pick, and the commander changes. The outgoing commander does a proper handover: what's known, what's been tried, what's currently running, what the working theory is.

A written running log makes this possible and it needs to exist from minute one. One channel, everything in it, timestamped: what we tried, what happened, what we decided. It costs almost nothing during the incident and it's the difference between a real handover and a fifteen-minute verbal summary that omits the important thing.

It's also, later, your postmortem timeline, reconstructed for free rather than from memory and screenshots.

The half-hour that decides everything

The first thirty minutes set how the rest goes, and they're mostly about restraint.

Get the roles named out loud. Get one channel and put everyone in it. Get the first customer update out. Ask the two questions that resolve a surprising share of incidents: what changed recently, and can we undo it.

Then, before diagnosing, ask whether there's a mitigation available that doesn't require understanding. Usually there is. Take it.

After it's over

Two things, on the same day, before anyone writes anything.

Tell the people who fixed it to stop working. Not tomorrow. The instinct after a long incident is to immediately start on the permanent fix, exhausted, on the system that just broke.

And say publicly, to the whole company, that it's resolved and who did it. The incident was visible to everyone; the resolution frequently isn't, and the people who spent six hours on it deserve better than silence.

The postmortem comes next, and it's a different discipline with its own lesson.