Evals, not vibes

The pattern is common enough to be predictable.

An org buys seats after a fifteen-minute demo where the tool builds a working CRUD app from a one-paragraph prompt. Six months in, somebody in finance asks what it has actually done for delivery, and the honest answer is that nobody knows.

Nobody wrote down cycle time, defect rate, deploy frequency or anything else before the rollout. So there's no baseline, and the only evidence available is a general sense in the room that things feel quicker.

That feeling is real. It's also not a number, and it doesn't survive a budget conversation.

Easy trap, because the demo genuinely is impressive, and impressive demos generate a certainty that substitutes for measurement.

That certainty is misplaced. A demo shows what a tool does under ideal conditions: clean prompt, no legacy code, no test debt, no half-documented internal API to misread. It tells you almost nothing about what happens inside your actual repository.

Why "faster" isn't the finding

Generation being faster is trivially true, and it answers a question nobody needed answered. Of course a model produces text faster than a person types.

The question that decides whether you got anything for your money is whether delivery got faster. Whole tickets, opened to shipped, review included. And whether quality held while that happened.

Different questions from "did the tool generate code quickly." And the ones a demo can't answer.

They depend on your review process, your test coverage, your team's judgment about when to trust the output. All three are properties of your organisation rather than of the model. Which is exactly why a vendor's published numbers and your actual result come apart.

What to actually track

Start with the one that matters most and gets skipped: deployments to production.

Everything else comes downstream of it. PRs. Time to first review. Cycle time on the coding step.

If your deploy count stays flat, you have not gone faster, and something in the pipeline is bottlenecked no matter what any other chart on the slide happens to say.

Lines of code was always a bad metric, rewarding verbosity over judgment, and it turns actively misleading once generation is cheap. A tool producing more code isn't producing more value. It might be producing more to review. Drop it.

Then cycle time on a consistent, well-scoped slice of tickets. Comparable work, measured the same way before and after.

Defect and revert rate too, split between agent-touched and human-only changes, because an aggregate hides a problem concentrated in one category.

Review time per PR and queue depth matter for a specific reason. If generation sped up and review didn't, the queue metric shows it clearly while everything else still looks fine on paper.

None of this needs sophisticated tooling.

A spreadsheet. Four weeks of consistent logging before the rollout.

That tells you more than any vendor case study, because it measures your team on your codebase rather than someone else's on theirs.

Benchmarks don't transfer

Public leaderboards get measured against clean, well-tested, well-documented repositories.

Fair enough, on its own terms. Those are the conditions under which a model can be scored fairly and reproducibly.

Most real codebases aren't that.

Coverage is uneven. Documentation lags the code by a year, and internal conventions live in people's heads rather than in any style guide anybody could hand to a tool.

Agent quality is unusually sensitive to exactly those conditions. More sensitive than most engineers expect going in.

A tool that looks excellent on a scored repo can look mediocre on your service.

Not because the model got worse. Because the ground truth it checks itself against, your tests and types and conventions, is thinner than what it was scored on.

The fix isn't distrust. Build a small evaluation set from ten or fifteen real tickets out of your own backlog. Run any candidate against those. Judge it there.

Run it like an experiment, not a rollout

The version that produces an actual answer looks less like "turn it on for everyone."

More like a controlled comparison. One team using the tool for a quarter, a comparable team not, both measured the same way on the metrics above. More friction than a company-wide announcement, and it's the difference between having an answer in ninety days and having an opinion.

I skipped this the first time. Rolled a tool out broadly because the case felt obvious, then spent an unpleasant afternoon six months later in a budget review with nothing but anecdotes to defend the renewal.

The renewal survived. The next tool got a baseline first.

The standard to hold

Before expanding any agentic tool past a pilot, finish this sentence with a number rather than an adjective.

Deploys to production went from X to Y, cycle time on comparable tickets went from A to B, and defect rate on agent-touched code was Z.

Can't fill the blanks? Then you don't have a result. You have a demo you liked.