5 min readcode reviewagentic engineeringdelivery

Turn your review queue back into flow

Pickup time and review time come back under control without hiring reviewers or slowing generation, once you treat the queue as a measurement problem instead of a discipline problem.

By JP LeBlanc

A dense queue of small squares backed up before a narrow gate with a person glyph beside it, and a second, lighter path where fewer squares reach the gate at all, one highlighted in rose.

Turn your review queue back into flow by measuring it first, because the fix always turns out to be different from the one everyone reaches for. Pickup time, review time, batch size, and the share of changes that need a human at all: four numbers, and almost nobody has all four written down. Once you have them, four levers move the queue, in order: shrink what has to enter it, let agents review agent output before a human ever sees it, review the spec instead of the diff wherever the work is well-specified, and put a WIP limit on agent work the way you'd put one on a person. None of the four is hiring, and none of them slows generation down.

Measure before you touch anything

A team with a review problem almost always has an opinion about what's wrong before it has a number. Too few senior engineers. Not enough discipline about PR size. The new hire slows everything down. Usually all three opinions are wrong, or at least beside the point, because nobody's looked at where the time actually goes.

Four numbers, pulled from your existing tooling in an afternoon. Pickup time: the gap between a change being ready and a human starting to look at it. Review time: how long the look itself takes once it starts. Batch size: lines or files per change, because a queue full of four-hundred-line diffs behaves nothing like one full of forty-line diffs at the same volume. And the share of changes that skip human review entirely, which tells you whether your policy is still real or just still written down somewhere.

A Series B team went from forty pull requests a week to two hundred with the same eight engineers, and deployments slowed. The instinct reads that as a review capacity problem, because two hundred is a lot more than forty. The number that tells you where the time went is pickup time, and it's the one nobody watches, because nobody watches a queue the way they watch a burn-down chart. A queue stays invisible until somebody measures it, which is the normal condition for queues.

Shrink what has to enter it

The lever with the best return is the one most teams try last: reduce the fraction of changes that require a human at all.

Autonomy should be set per task, by blast radius, not once for the whole tool. A dependency bump, a well-covered refactor, a change behind a flag that rolls back in seconds: none of that needs the same gate as a change to the billing schema. Most teams set one autonomy level for every agent-authored change because it's easier to write down in a policy doc, then wonder why the queue fills with low-risk work sitting next to the one change that actually deserved twenty careful minutes.

Get this right and the queue doesn't just get shorter. It gets more honest, because what's left in it is disproportionately the work that was worth a human's time in the first place.

Let agents review agents first

Reviewer capacity is roughly fixed. It doesn't get more elastic because leadership wants it to, and hiring reviewers is slow, expensive, and arrives well after the queue has already done its damage. What can scale immediately is a first pass that catches the obvious problems before a person opens the diff at all: an agent checking an agent's output against the tests, the invariants, and the patterns the codebase actually uses, flagging what looks wrong before a human's attention gets spent on it.

That's not the same claim as nobody needing to approve the merge. That piece is about who signs at the end. This is about what happens before a human's eyes are on the thing at all, and the two work together: a cheap automated pass first, a human where the machinery says the risk actually lives.

Review the spec, not the diff

For anything genuinely well-specified, the spec is what gets approved now, not the diff underneath it. If the invariants are named, the tests defined, and the evals passing, re-reading two hundred lines by hand is redundant motion dressed up as rigor. You're re-verifying something a machine already checked, more slowly and with less consistency than the check that already ran.

This one is uncomfortable to adopt, and it's worth saying why honestly: it requires trusting a spec you didn't personally write, on work you'd previously have inspected line by line, and that trust has to be earned by the spec quality and the eval suite behind it, not assumed on day one. Do it before the trust is earned and you've just moved the risk instead of removing it.

A WIP limit for the thing you never limited

Nobody puts a work-in-progress limit on agents, because a limit was always a human courtesy, a way of saying don't start a fourth thing while three are already waiting. Agents don't get tired starting a fourth thing. They'll happily open twelve pull requests before lunch, and every one lands in the same queue a human has to clear.

Small batches are the master variable of delivery, and a WIP limit on in-flight agent work is the mechanical way of enforcing that when the thing producing the batches doesn't get bored or tired the way a person does. Cap how many agent-authored changes can be open and unreviewed at once. When the cap is hit, the next task waits, the same as it would for a person who already has three PRs open.

At CircleCI the signal wasn't "review is taking longer." PRs in flight went up, and stacked PRs appeared: engineers building on unmerged work because they'd stopped believing the queue would clear. We didn't answer that by reviewing faster. Review agents came in from several vendors, trialled against each other, and people were assigned to fix it. The trial went yes and no. The agents were too verbose, and they didn't follow house coding standards, until trial and error found the right ones. The constraint got staffed, not absorbed.

What it looks like once it's working

The queue doesn't disappear. It gets honest: shorter, made mostly of the changes that genuinely warranted a person's time, moving at a rate a human reviewer can sustain without stacking PRs on top of PRs they've quietly stopped believing will ever clear. That's the whole target. Not zero review. Review that means something again.

Questions this answers

Why is our AI code review backlog growing even though we hired more reviewers?
Reviewer capacity is roughly fixed and generation isn't, so adding reviewers loses on arithmetic. The lever with real slope is the share of changes that require a human at all, set by blast radius, not by hiring.
Should agents review other agents' code before a human sees it?
Yes, as a first pass. An automated check against tests, invariants, and known patterns catches the obvious problems before a person's attention gets spent, and it can scale with generation in a way human reviewer headcount cannot.
Do WIP limits apply to AI agents, or just to human engineers?
They apply more, not less. Agents don't get tired starting a fourth task while three are already open for review, so a cap on in-flight, unreviewed agent work does the job a human's own sense of overload used to do automatically.