Rebuild your interview loop for agentic engineering

The take-home came back in forty minutes. It's clean, it's tested, the commit messages are better than yours. On the call, you ask why they picked the queue over a cron job and the answer is a paragraph of general reasoning that never touches the specifics of the problem.

Half your panel wants to ban the tools. The other half says banning is unenforceable and faintly ridiculous, since the job involves using them daily. That argument will eat two meetings and resolve nothing, because both sides are answering a question that stopped mattering.

The exercise didn't fail because a candidate used an agent. It failed because it was measuring implementation speed, and implementation speed is no longer scarce, and hasn't been for a while now.

Test the job, and say what the job is

Write it in one sentence before you touch the loop, because every design decision below falls out of it.

The job is to decide what should exist, specify it well enough that something else can build it, judge what comes back, and own it when it breaks at 2am.

Read that and the old loop's problems are obvious. Almost every stage you're running tests the middle of the old job (produce a correct implementation from a clear description) and almost none of it tests the four things in that sentence. The spec is what you're reviewing now, and most loops still don't ask a candidate to write one.

What stopped predicting

Unassisted take-homes, first. Not because candidates cheat, but because you can no longer tell what you measured, and a signal you can't interpret is worse than no signal because you'll act on it anyway.

Live syntax recall and algorithm puzzles were always weak predictors, and they've now gone from weak to actively misleading, since the thing they proxied for (can you get from problem to working code quickly) is the part that's cheapest to buy.

Tool-familiarity questions too. Which assistant do you use, how many hours a day, what's in your setup. You'll get an answer that reflects what their last employer paid for. It predicts nothing, and it selects hard for people who follow tooling discourse, which is not the same population as people who ship.

And speed. A candidate who produces a working implementation in fifteen minutes and a candidate who takes ninety are no longer telling you much about their ability, and might be telling you about the model they had open.

One rule before the exercises

Run the loop the way the job runs. Give them the tools, tell them so in writing beforehand, and make the exercises ones that agents make easier rather than ones agents solve outright.

The signal was never whether they use an agent. It's what they do with what it hands back: whether they check it, where they stop trusting it, what they throw away, and whether they can tell you which part of the output they're least confident in. That last question is the single most useful one in this entire guide, and it costs you nothing to add to a loop you already run.

Exercise one: the change that looks right

Sixty minutes. This is the one to build first if you only build one.

Give them a real agent-authored change against a repo they've had access to for a day. Two hundred lines or so, with a spec and a passing test run attached, on something ordinary: adding a retry to an integration, a caching layer, a new endpoint on an existing service.

Plant exactly two defects. The first is one your tests would catch if the tests were better, so a careful reader finds it by reading. The second is a wrong assumption inherited from the spec: the change does precisely what it was told and what it was told was subtly wrong. Retrying a request that isn't idempotent, caching something that's user-specific, treating a timeout as a failure when the operation completed. That second defect is the whole exercise. No test catches it. The agent had no way to know. It's what the job is.

Watch for four things. Do they read the spec at all, or go straight to the diff? Do they check what the tests actually assert rather than that they're green? When they find the second defect, do they describe the failure or the fix (strong candidates describe the failure first)? And do they say what they'd do about it, which for a good engineer is often "reject this and ship a smaller version that doesn't need the assumption"?

Scoring, and write it down before the debrief. Found the shallow defect: table stakes. Found the assumption defect: strong. Found the assumption defect and traced it back to the spec rather than blaming the tool: that's your hire signal. Fixed both without mentioning the spec: mid, and the kind of mid that generates a lot of confident, wrong work at volume.

Exercise two: the request with holes in it

Forty-five minutes. Cheapest to build, and it tests the part of the job nobody interviews for.

Hand them a one-paragraph feature request of the kind a real stakeholder sends. Something like: customers want to be notified when their usage is close to the limit. Then ask for the spec an agent could execute without asking a follow-up question.

Build it with five ambiguities in it, and know what they are before you run it. What counts as close. Whether notification means email, in-product, or both. What happens on the second breach. Whether admins get it or every user. What happens to accounts already over the limit when this ships.

You're scoring three separate things and they're worth keeping separate. How many ambiguities they noticed. Which ones they resolved themselves versus escalated, and whether that split was sensible (resolving the product-strategy question alone is a flag, escalating all five is a different flag). And whether the acceptance criteria they wrote can actually be checked by something that isn't a person with context.

The strongest answer I've seen to this kind of exercise wasn't the most thorough spec. It was the one that named the two questions the candidate couldn't answer, said who they'd ask, and wrote the spec so the answer could change later without a rewrite. That's intent written down properly, and it's rare enough that you should hire it when you find it.

Exercise three: blast radius triage

Thirty minutes, and it works well as a pairing exercise with the hiring manager.

Five changes on cards. A dependency bump. A copy change behind a flag. A migration that drops a column. A new retry policy on a payment call. A refactor of an internal utility with forty callers.

For each one: what autonomy would you give an agent here, what's the gate, and what's your rollback? Then have them defend two of the five while you push back.

You're listening for whether they reason from how expensive it is to be wrong or from how complicated the task looks. Those two produce different answers on the dropped column, which is easy work and nearly impossible to undo. Candidates who classify by difficulty rather than by reversibility will make exactly that mistake in your codebase in month two.

The other tell is the rollback answer. "Revert the commit" for a migration that dropped a column is the answer of someone who's never had that particular bad morning.

Exercise four: debug a system you didn't build

Sixty to ninety minutes, expensive to run, and the best predictor in the set. Use it as the final stage rather than a screen.

Give them a service they haven't seen, a failing behaviour, and an agent. Then sit quietly and watch how they work.

What you're watching for isn't the fix. It's the loop. Do they form a hypothesis before asking the agent anything, or do they narrate the symptom and hope? Do they verify what comes back against the actual system, or accept it because it's fluent? When the agent confidently produces a wrong explanation, and it will, how long does it take them to notice, and what do they do then? Do they ever close the laptop lid and go read the code?

The failure mode to look for is the candidate who is very fast and never checks. They'll look impressive for forty minutes. Then the agent will hand them a plausible wrong answer and they'll build on it, and you'll get to watch, in miniature, exactly what they'll do to your production system.

You don't run all four

Two, plus a conversation. Any more and you've built a loop candidates drop out of, which is a real cost that gets ignored in these discussions because it's invisible from the inside.

For most roles: exercise one and exercise two. Add exercise four for senior and staff hires, where judgment under uncertainty is most of the job. Exercise three fits inside the hiring manager's existing conversation and doesn't need a slot of its own.

Then keep the stage that was always the best one, which is the structured values-and-judgment conversation run by the people who'll actually work with this person. Keep the mechanism and drop the word "fit", because "fit" is the word that quietly produces a team of people who agree with each other, and the conversation underneath it is genuinely predictive when it's run against written criteria.

Scoring, or the part that decides whether any of this works

Every interviewer writes their score before the debrief, against the rubric, with no discussion. This matters more than the exercise design does, and it's the step most panels skip because it feels bureaucratic when everyone in the room is smart and busy.

Rubrics get written in advance, per exercise, with the specific behaviours listed. Not "strong problem solving." Something a stranger could apply: found the assumption defect, traced it to the spec, proposed a smaller change.

And calibrate on people you already employ. Run exercise one past three engineers whose judgment you trust, including one you'd consider a solid mid-level rather than a star. If your rubric doesn't separate them the way you'd separate them, the rubric is wrong and you'd have found that out on candidates instead.

What changes for early career

Don't run a smaller version of the senior loop. Test different things.

Question quality over answer quality. Give them something under-specified and see what they ask before starting, because that's the behaviour that makes a junior useful in an agentic team rather than a generator of plausible work.

Learning rate inside the session. Teach them something small at minute ten, then see whether it shows up at minute forty.

And the honest test of whether they can say "I don't know," which sounds soft and isn't. The failure mode for a junior with an agent is confident output they can't evaluate. Someone who says "I'm not sure this is right, here's how I'd find out" is telling you the most important thing about how they'll work.

What this costs, honestly

Each exercise takes three or four hours to build properly, and exercise one takes longer because planting a good assumption defect is genuinely hard. They also go stale: the repo drifts, and candidates talk to each other, so plan on rotating the defects every six months. That's a real ongoing tax and pretending otherwise is how these programmes die in month eight.

You need a real repo, ideally a slightly gnarly internal one with the secrets stripped, because a toy codebase makes exercise one trivial and exercise four meaningless.

And exercise four is expensive in senior-engineer time, which is exactly the resource you're short of. Run it last, only on candidates you're already inclined to hire, and treat it as the thing that stops a bad hire rather than the thing that finds a good one.

What to measure

Loop length in calendar days and candidate drop-out rate, because a better loop that nobody finishes is a worse loop.

Interviewer hours per hire, which is the number that tells you whether this survives a busy quarter.

And the one that actually matters and takes a year to read: agreement between loop score and performance at six months. Write the loop scores down somewhere you'll find them later. Almost nobody does, which is why almost nobody knows whether their interview process works.

Where to start

Take one agent-authored change that shipped last month, one where you know what the assumption problem was, and turn it into exercise one. That's an afternoon.

Run it on two engineers you already employ before you run it on anybody you might hire.