Diagnose an AI-forward org in a week

You get the tour on day two. Somebody walks you through the tooling, the rollout, the internal wiki page with the prompt library, the percentage of engineers active weekly, the Slack channel where people post wins. It's a good tour. Everyone giving it is proud of it, and they should be, because most of it was real work.

It's also the wrong tour, and it will take three weeks of your life if you let it.

The tour is about generation. Generation is the part that got cheap, the part with a dashboard, and the part almost nobody's constraint actually lives in. What you're looking for is downstream, in the machinery that's supposed to tell the difference between a good change and a plausible one, and in the queue where everything now waits. None of that is on the tour, and much of it isn't written down anywhere, so you have to go and ask.

A week is enough. Not because thoroughness doesn't matter, but because the second week is where you start negotiating instead of observing. People adapt to a diagnosis in progress. The flaky test gets quietly fixed before you see it, the queue gets worked down, someone writes the ownership doc that didn't exist on Monday. You want the org as it runs when nobody's watching, and your window for that closes fast.

What you're actually looking for

One constraint, named specifically enough that a team could start on it Monday.

Not a maturity score. Not a list of eleven improvement areas ranked by effort and impact, which is the deliverable that gets praised in the room and then never opened again. Two named failures at most, per the rule of waiting for a repeated, named problem before building anything. If you come out with eleven, you didn't diagnose, you inventoried.

The six probes below run roughly in order of what would invalidate the rest. If the first one comes back bad, you can stop. You've found it, and everything else you might have written down is downstream of that same answer.

Probe one: does a green build mean anything

Ask three engineers, separately, the same question. When a test fails in CI, what's your first thought?

You're listening for one of two answers. "Something's broken" is the good one. "Which one is it this time" is the other one, and if you hear it twice from three people, the diagnosis is over and you can spend the rest of the week on how to fix it rather than on what's wrong.

Ask for the artefacts anyway, because you'll need them to make the case: the last fifty CI runs on the main branch with their outcomes, the list of quarantined or skipped tests with the date each was skipped, and the count of re-run-until-green events in the last month. That last number is the one nobody tracks and everybody could produce in an afternoon.

What makes this probe first is that it decides whether anything else you find is trustworthy. Agentic development doesn't create a dishonest suite, it just finds out immediately whether you have one, and at volume the suite is the only check still running at the speed the work arrives.

The red flag isn't a low coverage number. It's a skip list with dates older than a quarter on it, and an engineer who can recite which failures don't count.

Time: half a day.

Probe two: where does change wait

Get the distribution of pickup time, which is the gap between a change being ready and a human starting to look at it. The distribution, not the average, because the average is where this problem hides. Then the same for review time, then the count of open changes by age.

The reference points are public now and they're stark. LinearB, across 8.1 million pull requests, has AI-assisted changes waiting 16-plus hours for a reviewer where unassisted ones wait around 200 minutes. Faros AI's telemetry across roughly 22,000 developers has median review time up 441.5% against a 33.7% rise in throughput.

You're not benchmarking against those. You're checking whether the same pattern is present, because if it is, the org's real problem is a queue and every conversation you've had this week about model choice was a distraction.

Two things to look for that don't show up in the numbers. Ask who decides what gets reviewed first, and listen for whether the answer is a person, a rule, or nothing. Then ask what happens to a change that's been waiting four days, and listen for whether anyone knows.

The red flag: a growing share of changes merging with no review at all. Faros has that up 31.3%, and it usually means a mandatory-review policy that stopped holding without anyone deciding to stop holding it. A policy nobody has repealed and nobody follows is worse than no policy, because it's still on the slide that says you're in control.

Time: one day, most of it waiting for someone to pull the data.

Probe three: who carries the pager

Pick an agent-authored change that shipped in the last month. Ask who owns it.

Watch what happens in the pause. In a healthy org you get a name in about four seconds, and it's the name of a person who owns the workstream, not the person who happened to click approve. In an unhealthy one you get a discussion, and the discussion is the finding.

Then ask the question underneath it: at 2am, when this breaks, who gets paged and what do they have to work with? Agents don't produce technical debt so much as unowned code, and the difference matters because the fix is different. Debt gets repaid with a refactor. Unowned code gets fixed by assigning ownership and putting a name on the rota, which is cheaper and takes a week.

Artefacts: the on-call rota next to the service list, and the list of agent workflows running on a schedule. That second list frequently doesn't exist, which is itself an answer.

Time: half a day.

Probe four: what does a merged change cost

Ask for AI spend broken down by team. Then by workflow.

Most orgs can't produce either one. That's fine, and it's the finding rather than a failure of the diagnosis: spend that nobody can attribute is spend that nobody owns, and unowned cost defaults to being finance's problem, which is where it stops being fixable.

If they can produce it, compute cost per merged change and look at the spread across change classes. A dependency bump and a customer-facing feature costing the same means a workflow is doing far more work than its task requires, and that's a specific, fixable defect rather than a budget conversation.

The reason this can't wait: billing moved. GitHub put every Copilot plan on usage-based billing on 1 June 2026, with completions included and agentic features metered by tokens. Any budget built on a seat count was built for a different product, and the org usually finds out in month three of the fiscal year.

Time: two hours if the data exists, two hours if it doesn't.

Probe five: what's in the context files, and who owns them

Find the AGENTS.md, CLAUDE.md, cursor rules, skill definitions, whatever this org calls them. Run git log against each one.

You're looking at dates and authors, not content. A file with fourteen contributors and no changes in five months is a file that's drifted, and everything it says is being applied at volume to work that's moved on without it.

Then the sharper test. Pick a rule at random and ask an engineer why it's there. If nobody knows, that rule is executing on every task, unexamined, and it's been doing so since whoever wrote it left the team.

Context is maintenance rather than a build, and the maintenance almost never has an owner, because it doesn't look like code and it doesn't sit in anyone's roadmap. It executes like code, though. That's the whole problem.

Time: two hours.

Probe six: what happens when an agent is wrong

Ask for the last three incidents that involved agent-authored change. Then read what changed afterwards.

Three possible answers, and they tell you almost everything about the org. The fix changed the machinery: tests, flags, blast radius, rollback. Good. That org learns at the level where learning holds. Or the fix added a rule to a document, which means it'll happen again, because a document isn't a control. Or there was no postmortem at all, because the change came from a tool rather than a person and nobody was sure it counted.

That third one is more common than you'd expect and it's the one to name loudly. Postmortems are blameless and mandatory including for near-misses, and an incident caused by an agent your org deployed is your incident.

Time: half a day.

The hour that beats all six

Sit next to two engineers for an hour each while they work. Say almost nothing.

You'll see things no artefact contains. Where they stop trusting the output and start checking by hand. How many times they re-run something. Which parts of the loop they've quietly worked around. Whether they read the diff before shipping it or scroll to the bottom. Whether the thing they're fighting is the model, the tests, or a piece of internal tooling that everyone has learned to route around and nobody has mentioned to you.

Every leader I know says this is a good idea. Very few do it, because it feels like a poor use of an expensive week and because watching people work is socially awkward for both parties. Do it anyway, on day three, once you know enough to understand what you're seeing and before you've formed the conclusion you're going to defend.

The write-up

One page. Not a deck.

What you saw, in five sentences, with the numbers you actually pulled. Then the two failures you're naming, in order, with the mechanism spelled out for each: not "test suite quality is inconsistent" but "eleven tests have been on the skip list for over a quarter, engineers no longer treat red as a stop signal, and the review queue is the only remaining check on a volume of change it can't absorb."

Then the sequence, which is usually shorter than people expect. Fix the suite before anything else, because everything downstream is measured through it. Shrink the queue by scoping blast radius rather than by adding reviewers. Name owners. Attribute spend.

And then the section most people leave out, which is what you're deliberately not doing this quarter, with the reason. A diagnosis that doesn't say no to anything hasn't made a choice, and the org will quietly fund all eleven items at ten percent each.

What this won't tell you

Whether the tooling choice was right. You can't answer that in a week and it's rarely the constraint anyway, so leave it alone unless somebody's paying a licensing cost that would fund an engineer.

Whether the org is good at the parts of AI work that aren't code. Customer-facing model behaviour, evals for a product feature, data handling: different diagnosis, different questions, and mixing the two is how a week becomes a month.

And whether any of this survives you. A diagnosis holds for about a quarter. What makes it hold longer is that somebody in the org owns each of the two failures by name before you leave, which is a conversation to have on Thursday, not in the write-up.

Where to start

Monday morning, ask three engineers what they think when a test fails. You'll know by lunchtime whether the rest of the week is a diagnosis or a rescue.