Writing a check right after an incident only encodes the failure you already caught. A real eval constrains behavior nobody has observed yet, which means it has to exist before, not after.
By JP LeBlanc
An eval written after an incident isn't a new safeguard. It's the incident, rendered as a pass/fail check, which is a description of a regression test, not an eval. The distinction is not pedantic: an eval's entire value is that it constrains behavior you haven't observed yet, and a check written the week after a postmortem constrains exactly the one behavior you already know about, which the postmortem's own action item was already going to cover.
Watch what happens after most incidents involving an agent. Someone writes a check: verify the migration doesn't drop this specific index. It gets added to the suite, it passes forever, and the team feels safer. They are safer, against that exact failure. They've learned nothing about the failures sitting next to it, adjacent conditions that never actually happened, because a test named after last month's incident describes last month's incident and nothing else.
Postmortems, done properly, keep asking what made the wrong action look reasonable at the time, and the answer usually lands on a specific gap: a status endpoint reporting "complete" while the job is still queued, an index dropped from a schema migration that held up fine against test data and buckled under production volume. Naming that gap and fixing it is real work, and it belongs in the action items.
Calling the fix an eval doesn't make it something new. It's the same fix wearing a different word, and the word matters, because it lets a team believe it built evaluation infrastructure when it actually wrote one more assertion into a suite that was already green.
The sharper problem isn't the mislabeling. It's where evals exist at all. A customer-facing AI feature has a product manager checking its output daily, a support team escalating anything strange, a dashboard somebody actually watches. It gets evaluated from more directions than it strictly needs.
An agent quietly rewriting internal migrations, editing config, or triaging a queue has none of that. Nobody's job is to notice when it's subtly wrong, because nobody's job was ever to notice when the equivalent human-written change was subtly wrong, and the org never built the function to begin with. Where agents earn their keep is precisely this kind of bounded, checkable internal work, which means it's also exactly the surface running with the least scrutiny once it's live.
That's backwards. The customer-facing feature has a redundant set of eyes on it. The internal one, doing arguably riskier work against production infrastructure, has none.
Ask which of your internal, agent-touched workflows has an owner who could tell you, right now, what a subtly wrong output would look like. For most teams the honest answer is a short list, and the workflows that don't make the list are exactly the ones running unsupervised tonight.
The usual answer is cultural: get teams to take internal tooling as seriously as anything customer-facing. That's not wrong, but it isn't the actual lever. The lever is sequencing. An eval built before the tool touches anything real forces someone to ask what could go wrong, in the abstract, before it has. That's a harder question than "what already went wrong," and it's the only version that catches something nobody has seen yet.
Writing the anticipatory version is slower than patching after an incident, and it doesn't produce the same relief a quick post-incident fix does, which is exactly why teams default to the retrofit under pressure. Nobody feels finished after writing a check for a failure that hasn't happened.
At CircleCI, the gate every agent change cleared before merging to main was the entire CI validation suite: lint, coding style checks, security scans, end-to-end tests, a check that the change carries rollback instrumentation, and a check that it declares its release strategy, canary, feature flag, or blue-green. The last two don't describe any single failure. They describe a class of them, what a bad change looks like before it has happened, encoded as a gate every change clears regardless of whether a person or an agent wrote it. None of it works, either, if the suite it plugs into has quietly stopped telling the truth, which is the prior question and the one worth checking first.
Ask, of any check currently in your suite, whether it was written because of an incident or in anticipation of one. If most of the answers are "because of," you have a regression suite wearing an eval's name, which is a fine thing to have and a bad thing to mistake for coverage.
The fix isn't guilt. It's an inventory. For the workflows nobody's watching, the internal ones, ask what a plausible bad outcome looks like before one happens, and write the check now, while it still counts as prevention instead of documentation.
A postmortem closes the loop on one incident. An eval built before the fact closes the loop on a category, which is a different and larger claim, and it's the reason the two shouldn't be filed under the same word.
The next incident in that category, if the eval was any good, doesn't happen. Nobody writes a postmortem for it, nobody notices the near miss, and the only evidence the check ever did anything is the absence of a bad Tuesday that would otherwise have arrived. That's a strange thing to have to argue for in a budget review, an outcome you can only prove by pointing at nothing, but it's the actual return on writing the check before the failure instead of after it.