A testing strategy for people who ship

Ask a room of engineers why they write tests and you'll get answers about coverage, about catching bugs before production, about professionalism.

All beside the point.

A test suite exists to make a team willing to deploy without dread. That's the whole of it. Everything else is a proxy, and proxies drift away from the thing they were supposed to measure.

Coverage percentage is the worst offender, because it counts lines touched while claiming to count confidence earned.

A team can sit at 95% coverage and still flinch every time somebody opens a pull request against the billing service, because half those tests only check that a getter returns what was set on it.

A team at 60% can ship on a Friday afternoon without a second thought, because the tests that exist cover the paths that actually break.

The real unit to optimise is confidence per second of runtime. How much dread the suite removes, divided by how long you wait for it to run.

A test that's slow, brittle, and guards against a mistake nobody would ever make in a way that matters is overhead wearing a lab coat.

Delete it. The system stays exactly as safe as it was and the suite tells the truth faster.

The pyramid still holds. The volume assumption doesn't.

The old pyramid still describes the right cost curve. Many fast unit tests, a thinner layer of integration tests, a handful of slow end-to-end ones at the top.

Unit tests are cheap to write, cheap to run, and narrow enough that a failure tells you almost exactly where to look.

End-to-end tests are expensive in every dimension, worth it only where a real user, in a real browser, hitting a real dependency, is the only way to know something works.

That physics holds.

The volume moving through the pipeline on top of it is another matter.

The pyramid's assumptions were built for a world where a human wrote a pull request, thought about it for twenty minutes, and opened it. Agent-driven development doesn't run at that pace.

An agent can produce five plausible diffs in the time a person produces one, and every one of them runs through the same suite.

Fine, as long as the suite is honest.

Serious trouble if it carries even a little flakiness, because a flake that fires once a week at human volume fires several times a day once the volume climbs an order of magnitude.

Team size alone already multiplies this before AI enters the picture. Merges that break the main branch happen at roughly 12.5% of merges once a team crosses about 40 engineers, against roughly 0.77% for teams of two to five.

Sixteen times worse. All of it a coordination story, and it gets harder as more of that change volume starts coming from agents working in parallel.

Then the part that should surprise you less than it does.

In the same dataset, AI-assisted pull requests broke the main branch about 1.9% of the time, against 4.4% for everything else, despite the AI-assisted diffs generally being larger.

So much for the fear that AI-generated code is inherently riskier.

The risk lives somewhere else entirely. In whether your suite can still tell truth from noise at the volume you're now asking it to handle.

Flaky tests are not a minor annoyance

A test that fails for reasons unrelated to the change under review teaches your team something false. And teams learn fast.

First time a red build turns out to be nothing, an engineer clicks retry and moves on. Second time, they don't even look at what failed. By the fifth, red means nothing at all, and the suite has quietly stopped doing its job while still eating everybody's time.

Worse than having no suite at all. A team with no suite knows it's flying without instruments. A team with a lying suite believes it's covered.

So treat flakiness as an emergency. Never a backlog item.

Quarantine the test the day somebody notices. Fix it or delete it.

Every day it stays is another day teaching people that red doesn't mean stop, and unlearning that takes a great deal longer than the fix would have.

Production is not where testing ends

Pre-deploy testing can't replicate production, however thorough your team is, because production has real traffic, real data distributions, real third-party latency, and combinations nobody ever wrote a fixture for.

Testing in production, done responsibly, is the honest acknowledgment of that limit.

A canary taking a small slice of traffic before the rest gets it. A feature flag gating exposure where a binary deploy used to be. Rollback measured in seconds, with no war room attached.

All of it is the second half of the same job.

Pre-deploy testing tries to keep the bad change from landing at all. Production testing accepts that some will get through anyway, and makes sure you can reverse them faster than they can do damage.

A team with fast rollback and no test suite is reckless. A team with a great test suite and no rollback path is fragile in a quieter way that takes much longer to notice.

You want both.

Leaders underinvest in the second, because on a slide it reads as infrastructure, and infrastructure gets no applause until the day it's the only thing standing between you and an outage.