Observability before you need it
You cannot retrofit visibility during an outage. Instrument the few things that answer "is it working for users right now", and do it before you need the answer.
Also available as a standalone Playbook →
The moment you need observability is the moment you can't add it.
It's 2am. Something is wrong. The question is whether the broken thing is the payment provider, your queue, or a change that went out four hours ago.
Answering that requires data collected before the incident started.
If it wasn't, your options are guessing, or adding instrumentation to a production system under pressure. Which is how a one-hour incident turns into a four-hour one.
So the argument for doing it early has nothing to do with thoroughness. This particular capability either exists in advance or it doesn't exist at all.
The minimum for a small team
Four things. They'll cover most of what you need for a long time.
One measurement, taken from outside your infrastructure, of whether the main user journey works.
A synthetic check doing a real transaction every minute against a test account. Fails twice in a row, somebody gets paged.
A day of work, and the highest-value monitoring you will ever set up, because it's the only check that fails when the whole thing is broken in a way nobody anticipated.
Error rate and latency on your main entry points, as a time series you can look at.
Structured logs with a request identifier flowing through everything.
Key-value, machine-parseable, with one identifier letting you follow a single request across every component it touched.
Retrofitting that identifier later is genuinely painful, which is exactly why it belongs on day one.
And a deployment marker on your graphs.
Knowing the error rate rose four minutes after a deploy resolves an enormous share of incidents in seconds. It's a line on a chart.
That's it.
Anything past that, at a small size, is usually somebody building a monitoring practice when they meant to answer a question.
What each of the three is actually for
The logs-metrics-traces framing is useful once you know what each is good at. Expensive the moment you collect all three of everything.
Metrics answer "is something wrong, and when did it start."
Cheap, aggregated, all detail lost. Build your alerting on these, because they're fast and they cost very little per unit.
Logs answer "what happened to this specific request."
Expensive at volume, precious when you need them, and the first thing people over-collect. Log the events that matter, meaning a decision made, a state change, an error with context, and leave every step of every function alone.
Traces answer "where did the time go across all these services."
Genuinely necessary once a request crosses more than about three systems. Mostly overkill before that.
If you're a monolith and a database, tracing is a nice-to-have and the money goes further elsewhere.
The failure mode worth avoiding: collecting all three at full fidelity, everywhere, discovering the bill, then turning off the wrong ones in a panic.
Instrument on the way in, not on the way out
The habit that makes this cheap is instrumenting a system while you build it, as part of building it. Never as a follow-up.
Concretely, on day one of any new service.
Entry points emit latency and error counts. Important business events get logged with context. The request identifier propagates. And there's a health check meaning something more than "the process is running."
Two of those deserve expanding.
A health check returning 200 because the web server is up tells you nothing useful. It should check what the service actually depends on, starting with whether it can reach its database.
And "important business events" means the ones you'd want to count later, in business terms. Order placed. Payment failed. User invited.
Those are the logs that answer the questions you'll actually be asked, and they're much harder to add retroactively, because by then the code has moved on.
Dashboards people actually open
Most dashboards get built once, during an incident or just after, then never looked at again. An archaeological layer, sitting where a monitoring strategy should be.
The one dashboard that survives answers "is it working right now" in under five seconds, carries very few things, and lives on a screen somebody walks past daily.
Four to six graphs, maximum. Traffic, errors, latency, and whatever your product's one key business number happens to be. Orders per minute. Messages sent. Whatever moves when customers are actually using the thing.
That last one is the most valuable graph you'll have, and the least commonly present.
Business metrics catch entire categories of failure the technical ones miss. Everything green, error rate zero, and nobody has completed a checkout in eleven minutes because a flag turned the button off.
Everything else gets built during an investigation and thrown away afterwards, or kept as a saved query instead of a permanent dashboard.
A library of forty dashboards is a cost carried as though it were an asset. Nobody knows which one to look at.
Controlling the bill before it surprises you
Observability spend has a nasty property. It scales with traffic and with code changes, so it grows without anybody deciding to grow it, and it usually gets noticed as a finance question long before anyone treats it as an engineering one.
The usual form of the surprise. A debug log left on in a hot path. Or high-cardinality labels on a metric (user identifiers, request identifiers, anything unbounded) multiplying the stored series by a factor nobody predicted.
Sample aggressively on the high-volume, low-value data, and keep everything for errors.
Set a cardinality rule and enforce it in review, because that's the one producing ten-times bills.
And put the cost on a dashboard beside the technical metrics, where the engineers making the decisions can see it, since a cost nobody can see is a cost nobody manages.
Worth an annual review, too. Go through what you're collecting and ask, for each stream, when it was last used in an actual investigation.
For a meaningful share of it, the answer will be never.
The test
Pick a service. Ask somebody to answer three questions using only your existing telemetry.
Is it healthy right now. Was it healthy at 4am last Tuesday. And what changed just before the last time it wasn't.
If they can't answer all three within a few minutes, you have your list, and you have it on a day when nothing is on fire.