Twenty real tasks from your backlog, a pass definition, cost recorded beside correctness, and a threshold gating rollout, built in two weeks, before adoption rather than after the fact.
By JP LeBlanc
Building evals before rollout means assembling fifteen to twenty real tasks from your own backlog, defining what a pass means for each one, recording cost alongside correctness, and setting a threshold a candidate workflow has to clear before it reaches production. Two weeks of work, done before adoption, not retrofitted after finance asks what the tool has actually done for delivery.
The usual rollout goes the other way. A tool goes out broadly because the case feels obvious. No baseline, no held-back comparison group, just a demo that looked good and a sense that waiting would cost more than it saved. Six months later somebody sits in a budget review with nothing but anecdotes to defend the renewal, arguing a feeling instead of a number. When we measured the agentic rollout at CircleCI, the unit was production deploys, and the number roughly doubled. Measuring didn't change the plan. It confirmed it. A confirmation you can show beats any argument in that room, and it only exists if the measurement came first.
Don't invent test cases. Pull them from work you already generated.
Your incident history is the first source, and it's the strongest one, because every incident is a free description of a way things go wrong that you didn't think to test for in advance. Your own review comments are the second: the specific moments a reviewer caught something and said so, which is a labeled example of the exact judgment call you're asking a workflow to make correctly. The third is the quiet one nobody thinks to track. The tasks people redo by hand after the tool's version wasn't good enough, which is a pass or fail signal sitting in your team's actual behavior rather than in anyone's opinion about the tool.
All three sources exist already, for free, inside a team that's been shipping for more than a quarter. An eval suite doesn't require generating new evidence. It requires noticing the evidence you're already producing and writing it down instead of letting it evaporate into the next sprint.
A pass condition has to be falsifiable by someone who wasn't in the room when the task was written. Not "the output feels right." A specific, checkable claim: this test suite passes, this function returns this structure, this migration leaves this invariant intact.
Where the honest answer is "we'd know it when we saw it," that task doesn't belong in the eval yet. It belongs in the category of ambiguous work that still gets full human review, because an eval built on an ambiguous pass condition doesn't measure the workflow. It measures whoever happened to judge it that day, and that number moves for reasons that have nothing to do with the tool.
Almost nobody does this, and it's the gap that lets a correct-but-absurdly-expensive workflow pass every check you have. A workflow that produces the right answer at ten times the token cost of a reasonable approach has a defect. It just doesn't look like one, because correctness alone was the only thing anybody was checking.
Put a cost figure on every task result, next to pass or fail, from the first run. Cost belongs in the eval, not in a spreadsheet finance builds two months later, because a workflow that looks free in a demo and expensive at volume is a design decision hiding as a line item, and the eval is the only place that decision gets caught before it ships everywhere.
A pass rate with no consequence attached is a chart, not a gate. Decide, before you run the suite for the first time, what score moves a candidate from evaluation into a pilot, and what score sends it back. Write that number down before you see the result, because a threshold set after you already like the outcome isn't a threshold. It's a decision you'd already made, wearing a number for cover.
Then actually hold the line the first time a workflow misses it and everyone in the room wants to ship it anyway because the demo was good and the team is tired of waiting. That's the moment the eval either does its job or becomes theater, and it's usually the only moment that tests it.
Add a task every time something breaks in a way the current suite wouldn't have caught. That's the correct way to grow it. What isn't correct is writing an eval task that only describes an incident you already fixed, run once, then forgotten, because an eval built after the fact isn't testing for anything you haven't already seen. It's a regression test wearing the eval's name, and the workflows that actually need eval coverage, the internal ones nobody outside engineering is watching, are exactly the ones most likely to accumulate this kind of test and call it done.
Prune as often as you add. A suite that only grows gets slow, and a slow suite gets skipped under deadline pressure, which puts you back where you started with better intentions and worse habits.
Watch the gap between eval pass rate and production escape rate. If the suite says ninety percent and production keeps turning up failures the suite would have passed, the suite isn't measuring your risk. It's measuring a version of your risk from whenever the tasks were written.
That gap is more valuable than either number alone, and it's the cheapest early warning available that your twenty tasks stopped looking like your actual work sometime in the last quarter, without anyone deciding that on purpose.