AI in Code Review: What to Automate, What to Never Automate

TL;DR — Automate the mechanical half of review: convention drift against your own codebase, missing error paths, public API impact, test gaps, and a map of the diff for the human reviewer. Never automate the verdict: approvals, merge blocks, quality scores, architectural judgement, or being the loudest voice on a colleague's PR. The metric that decides survival is precision, not recall — a bot at 60% precision gets muted by the whole team in two weeks and takes its real findings with it. Roll out in shadow mode, set the posting bar at "would a senior have written this comment", and put the highest-value use on your own diff before you push.

Two jobs wearing one name

"Code review" bundles things that have almost nothing in common:

1. Mechanical checking — does this follow our conventions, handle the error, have a test, break a caller? Objective. Verifiable in seconds. Nobody's judgement is on the line.
2. Judgement — is this the right change? Does the design hold up? Is the person learning? Should this exist at all?

Job 1 is drudgery, and reviewers do it badly precisely because it's drudgery — attention is finite and mostly gets spent on the first two files. Job 2 is why senior engineers review at all.

AI is good at job 1 and structurally unable to do job 2, because job 2 requires knowing things that aren't in the diff: the deadline, the migration half-finished in another repo, the fact that this pattern was tried in 2024 and abandoned, and the person's growth this quarter.

So the line isn't "how smart is the model". It's which job you're asking it to do.

Automate: the mechanical list

These earn their place, roughly in order of value:

Convention drift, measured against your repo — not a style guide. The generic advice is worthless; the codebase-specific version is genuinely good.

Every repository in :data returns Result<T>.
This new function throws. Flag the mismatch and
name the two nearest files that do it correctly.

Missing error paths by comparison. "Nine other call sites of this endpoint handle HTTP 429 with a backoff; this one doesn't." That's a real bug found by pattern-matching across a codebase — exactly the task where a model beats a tired human.

Public API impact, as a fact rather than an opinion. "This changes a signature used by :feature:tv, :feature:player and :sync." Delivered to the human, who decides what it means.

Test-gap flagging. New conditional branch, no test that reaches it. Stated once, not argued.

A map of the diff. The one I'd build first if I only got one:

4 modules, 620 lines. 480 of those are generated protobuf. The substance is SyncScheduler.kt:88–140 (new retry policy) and one migration in AppDatabase.kt. Suggested reading order: migration, scheduler, then tests.

That doesn't review anything. It makes a human reviewer better in ninety seconds, and it's the highest value-per-risk item on this list because it can't be wrong in a damaging way.

Deterministic things belong in deterministic tools. Formatting, lint, import order, detekt rules — these are not AI problems. If your AI reviewer is telling people about formatting, you have a CI gap and you're paying tokens to paper over it.

Never automate: the verdict

Approving or blocking a merge. The moment a bot can block, its false positives become everyone's emergency, and the entire team's incentive shifts to learning how to phrase code so the bot shuts up. You will have automated Goodhart's law into your merge queue.

Quality scores. "This PR: 6.4/10." Nobody can act on it, everybody resents it, and it will end up in a performance conversation within a year no matter what you promised at rollout.

Architectural judgement. The model sees a diff. It cannot see that the abstraction it wants was deliberately removed in March because it cost two weeks of debugging.

Being the first and loudest voice on someone else's PR. This is the one senior engineers underrate.

The team dynamic nobody costs in

Code review is one of the last reliable places mentoring happens. A junior opens a PR, a senior explains why the second approach is better, and the person gets a bit better permanently.

Now put a bot in front of that. Fourteen comments, instantly, publicly, mostly nits, in a tone of total confidence. Three things happen:

1. The human reviewer arrives to a resolved thread and adds "LGTM" — the teaching moment is gone, spent on things that didn't matter.
2. The author learns to pre-empt the bot, not to write good code. Different skill.
3. A junior reads fourteen public criticisms on their first PR and quietly concludes they're bad at this. Nobody intended that; it's just what the tool does by default.

The fixes are cheap and worth insisting on:

• The bot comments in a collapsed summary, once, not as fourteen inline threads.
• It speaks tentatively. "Possible: this differs from the pattern in X" — not "You must".
• It never comments on a first-time contributor's PR. Route that output to the assigned reviewer privately and let a human choose what to raise.
• It's explicitly non-blocking, and everyone knows dismissing it costs nothing.

Precision beats recall, and it isn't close

The temptation is to tune for coverage — find everything. That's the wrong objective for a tool competing for human attention.

Do the arithmetic. Ten comments per PR at 60% precision means four pieces of noise, every time. Within two weeks the team is skimming and someone has muted the channel. Now the six real findings are gone too, and your recall is effectively zero. A muted tool has no recall.

At 90% precision and three comments per PR, people read all three. Fewer findings surfaced, vastly more findings acted on — which is the only number that matters.

So the bar for posting is: would a senior engineer actually have left this comment? If it's a maybe, don't post it.

Rolling one out without burning the team's patience

1. Shadow mode, two weeks. Output goes to a private channel that two senior engineers read. Nothing is posted to PRs.
2. Score every comment — real / noise / wrong. Keep the tally in a doc.
3. Cut the categories that scored badly. Usually this kills "suggests refactors" and "comments on naming", and keeps error paths and API impact.
4. Then enable, non-blocking, collapsed, one summary comment.
5. Re-score quarterly and publish the precision number to the team. A tool whose accuracy is public gets trusted at the right level, which is neither zero nor total.

You'll cut most of what it can do. That's the point: you're spending a fixed, small attention budget, and you should spend it on the categories that earned it.

The honest limit

It doesn't know your product. It'll flag a missing null check on a field your backend contractually guarantees, and it will not notice that the feature is wrong — that this screen should never have been reachable from checkout in the first place. That second class of finding is most of the value of a real review, and it isn't on the table.

It also has no memory of your team's history unless you give it one. Feeding it your ADRs and conventions docs helps a lot, and is the single highest-leverage tuning step available.

Where it's unambiguously good: your own diff

The best placement isn't other people's PRs. It's yours, before you push.

No social cost, no shared attention budget to spend, no morale risk, and the person verifying the output is the person who wrote the code and knows which suggestions are nonsense. Noise costs you thirty seconds instead of costing the team a channel.

Reviewing your own diff with a model, on your machine, before anyone sees it — that's the version of this idea with all the upside and none of the organisational cost. Everything you'd want the team bot to catch, you catch first, and the human review gets to be about the thing it should be about.

The rule

Automate the mechanical. Keep the verdict human. A bot that maps the diff, flags a missing error path and stays quiet otherwise makes reviews better. A bot that scores your colleague's work makes reviews into something people avoid — and the thing you'll have automated away is the mentoring, which was the part worth protecting.