The deploy pipeline as a safety net

Given a choice between a careful release process and a fast rollback, take the rollback every time.

They get presented as complements. They behave as substitutes, because they compete for the same attention and they encode opposite assumptions.

Careful release assumes you can determine in advance whether a change is safe. Fast rollback gives up on that and invests in making it cheap to be wrong.

The second assumption is the true one.

Every organisation with a heavyweight release process still has incidents caused by releases. That's the evidence, and it's everywhere.

Every change behind a flag

No exceptions, and I mean that more literally than most people who say it.

A flag turns a deploy into a non-event.

The code goes to production dark. Enabled for you, then the team, then one percent of users, then everyone.

If something is wrong it's off in seconds, switched by somebody who doesn't need to understand the code, and nobody is performing a rollback under pressure at 2am.

It also separates two things that get conflated and shouldn't be.

Deploying is an engineering action, happening continuously. Releasing is a product decision, happening when it's ready.

Once those come apart, the entire category of "the release is on Thursday" disappears, and so does the batch that was accumulating for it.

The objection is flag debt, and it's legitimate.

A codebase with three hundred live flags is unreadable and untestable. The number of possible states is enormous, and nobody knows which combinations actually exist in production.

So flags get an expiry date at creation, and removing them counts as part of the work.

A weekly report of flags older than the limit, with owners against each. A norm that they get removed on sight.

And treat a permanent flag as a configuration option, which is a genuinely different thing, with a much higher bar for creating one.

No persistent staging

I don't run a staging environment, and I'd argue against having one in most companies.

Staging is always wrong.

Different data, different scale, different traffic patterns, different third-party configurations, and a drift that grows every single week.

It gives you a confident green signal about a system that doesn't exist, and it costs real money and real maintenance to go on lying to you.

Worse, it becomes a queue. Somebody's change is in staging, so yours waits.

Then it becomes a gate. "Has this been through staging" turns into a checklist item, and the checklist item stands in for thought.

What replaces it is an ephemeral preview environment per change, created automatically when the branch opens and destroyed when it merges.

It holds the change under test. It's isolated. Nobody queues behind anybody.

Combine that with flags and you've covered the case staging was supposed to cover, minus the permanent drift.

Some honest exceptions, though. Integrating with a third party that has a sandbox and no other way to test, you need somewhere to do that. Working in a regulated environment where pre-production sign-off is a legal requirement, that's a debate engineering arguments don't win. And if your product ships to customers who deploy it themselves, the whole model changes.

What I'd still refuse is a chain of them.

Dev, then QA, then staging, then pre-prod, then production, each with a gate and an owner. That's an organisational structure pretending to be a technical one.

The lead time cost is enormous. The defect-catching benefit past the first one is close to zero.

Rollback as a rehearsed operation

Rollback is a capability you've tested recently, or it's a story you tell yourself.

One command, or one button. Never a runbook with eleven steps and a database restore in the middle of it.

A rollback requiring judgment goes unused at exactly the moment judgment is scarce.

Rehearse it.

Roll back a real deploy in production, deliberately, once a quarter, at 3pm on a Tuesday when everybody is calm.

The first time you try, something will be broken. Finding that out on a Tuesday afternoon is the entire point.

The hard part is data.

Code rolls back cleanly. Migrations don't. And a rollback leaving the database in a state the old code can't read is worse than having no rollback at all.

Which forces a discipline on migrations. Every schema change has to be compatible with the version of code before it and the version after it.

Add columns, never rename them. Deploy the migration separately, ahead of the code using it. Backfill in its own step. Remove the old column weeks later, in its own deploy.

Expand-migrate-contract. It's the price of being able to undo anything.

Progressive delivery, in the order worth adopting

Not all at once. There's a sequence, and every step is useful on its own.

Feature flags first, because they give the most safety per unit of effort and they're a prerequisite for everything else.

Then automated rollback on your key metrics. Error rate crosses a threshold within some window of a deploy, it reverts itself.

Machines are much better than humans at noticing that in the first two minutes.

Then canaries. Send a small share of traffic to the new version, compare, promote.

That's where you start needing decent observability, which is why it comes third.

Rings (internal users, then beta customers, then everyone) come later still, and mostly matter when you have distinct user populations who can absorb different amounts of risk.

Most companies get 80% of the value from the first two and spend their energy on the fourth.

The release process that should just be a merge

The end state.

An engineer merges a pull request, the pipeline runs, it goes to production with the flag off, and nothing happens. Later, somebody turns the flag on for a fraction of users.

That's the whole ceremony.

What it removes: the release manager, the change advisory board for routine changes, the deploy window, the freeze, the Thursday.

All of them exist to manage risk that flags and fast rollback manage better, and much more cheaply.

I'd keep risk classification, so that the small share of genuinely dangerous changes (anything touching money, auth, or data destruction) gets a second pair of eyes and a slower rollout.

And I'd keep a real record of what went out and when, which you need during incidents and which the pipeline should produce for you automatically.

The deploy freeze deserves its own paragraph, because it's near-universal and usually counterproductive.

Freezing changes over a high-traffic period sounds prudent. What it produces is a large batch of accumulated changes released immediately afterwards, into a system nobody has deployed to in three weeks, with everybody out of practice.

If your pipeline is safe you don't need the freeze. If it isn't, the freeze is treating a symptom.

The number that tells you where you are

Time from merge to production, median, measured over a month.

Under an hour means you have a pipeline. A day means you have a process.

A week means you have a release, and everything upstream of it (batch size, review latency, fear of changing things) is being set by that number, whether or not anybody has noticed.