Building an Internal Tool With AI in an Afternoon

TL;DR — Internal tools are the highest-leverage AI use available to a senior engineer because their risk profile is different: few users, all colleagues, no adversaries, no SLA, disposable. The workflow: write the fake output first (it's the fastest spec you'll ever write), demand a boring stack out loud ("one file, no framework, no config layer" — otherwise you get a plugin system), reach real data inside twenty minutes because fake data hides every interesting problem, then iterate against reality. Rules: read-only by default, secrets from the environment, a README that says delete me when. The real change isn't the code — it's that "a day of work, not worth it" became "an afternoon, obviously worth it."

Why internal tools are the right target

Most cautious advice about AI-generated code is about production: verification cost, ownership, the long-term maintenance burden. All correct — there's a whole post in this series about when not to reach for a model, and its argument is that value collapses when verification is the expensive half.

Internal tools invert that.

• Three users, all colleagues. A bug produces a Slack message, not an incident.
• No adversaries. The threat model is "someone typo'd a flag".
• No SLA. If it breaks on a Tuesday, it gets fixed on a Wednesday, or it doesn't.
• Disposable. Half of them should be deleted in three months, and that's a success.
• Verification is trivial. You run it and look at the output. That's the whole review.

That last point is the one that matters. When checking the result takes ten seconds, generating it fast is pure gain. It's the cleanest case for using a model that exists in our work.

The example

A flaky-test leaderboard: parse the last 200 CI runs, find tests that both passed and failed on the same commit, rank by frequency, print a table.

Every Android team over about eight engineers has this problem. Nobody has a free week for it, so everybody lives with "just re-run the build" as a culture — which is how a test suite stops meaning anything. It's exactly the shape of problem that sits in the backlog forever because it's mildly annoying rather than urgent.

It took an afternoon.

Step 1: write the output first

Before any code, paste the output you want:

FLAKIEST TESTS — last 200 runs (14 days)

 #   flakes  test                                          owner
 1     31    CheckoutViewModelTest.appliesDiscount          @payments
 2     22    OrderListScreenTest.showsEmptyState            @orders
 3     19    SyncWorkerTest.retriesOnTransientFailure       @platform

 Total re-runs caused: 412   ·   CI hours burned: 6.9

That fake table is the specification, and it's the fastest one you'll ever write. Producing it forces every real decision: what counts as a flake, what window, how you attribute an owner, what the summary line should be. Those decisions are the actual design work — writing them as a prompt paragraph would have been vaguer and taken longer.

It also gives you an unambiguous done condition. The tool is finished when it prints that.

Step 2: demand a boring stack, explicitly and repeatedly

Left alone, a model will give a 200-line script a web framework, a config layer, a plugin architecture, a --verbose flag with three levels and a setup.py. Not because it's wrong in general — because that's what the corpus of "tools" looks like.

Say the constraint out loud, and restate it when it drifts:

One file. No framework, no config layer, no plugin system.
Python 3 standard library plus sqlite3 only. Under 300 lines.
If you're about to add an abstraction for a second use case,
don't — there is no second use case.

Boring stacks are also where models are strongest: enormous, stable, well-represented training surface. A stdlib-only script is the most reliable code you'll get out of one.

And a one-file tool with no dependencies still runs in two years, which is more than can be said for anything with a lockfile.

Step 3: real data within twenty minutes

This is the step that separates an afternoon from a week.

Synthetic data hides everything interesting. On this tool, real data immediately produced: three different JUnit XML schemas across job types, parameterised test names with a [0] suffix that broke grouping, jobs that emit no XML at all when they time out (the flakiest ones, naturally), and timestamps in two timezones.

Every one of those is a five-minute fix when you hit it in minute twenty and a redesign when you hit it in hour four, because by then the shape of the code has assumed clean input.

gh run download --name test-results --dir ./runs   # 200 real runs, one command
python3 flaky.py ./runs

Step 4: iterate against output, not against code

The loop is: run it, read the table, say what's wrong in one sentence, run it again.

"The counts are inflated — a test that failed twice in one run is one flake, not two." "Group parameterised variants under the base test name." "Sort by CI minutes burned rather than count, and show both."

Notice what you're not doing: reading the diff line by line. For this class of tool you're verifying the behaviour, and the behaviour is a table in front of you. That's what makes the loop fast, and it's why this works here and doesn't work for a payments change.

Skim the code for the things you can't see in output — a hard-coded path, a swallowed exception, a rm -rf you didn't ask for. Then move on.

The rules that stop it becoming a liability

1. Read-only by default. The tool reads CI artefacts and prints. It does not close flaky tests, quarantine them, or open PRs. A tool that writes to a shared system is how a Saturday gets ruined, and it's a different review with a different bar. Start read-only; graduate deliberately.

2. Secrets from the environment, never in the file.

token = os.environ["GH_TOKEN"]   # never a literal, never a default

This matters more than usual precisely because internal tools get pasted into Slack threads and forked by colleagues.

3. A README with an expiry condition.

# flaky.py
Prints the flakiest tests from the last N CI runs.
Owner: @hessam · Built: 2026-10-10 · ~250 lines, stdlib only.

DELETE ME WHEN: the CI platform ships native flake detection,
or when nobody has run it in two months.

The real cost of internal tools isn't building them — it's that they never die. They accumulate, someone becomes their reluctant owner, and a script written in an afternoon gets treated as infrastructure five years later.

4. Five users means it's production now. If a tool starts appearing in other people's workflows, that's a promotion and it needs an owner, a test, and a home in a repo. Notice the moment consciously instead of discovering it when it breaks and three teams are blocked.

The honest limit

The last 20% costs as much as the first 80%, and it's always the same 20%: your corporate SSO, your internal API's non-standard pagination, the service that requires a VPN and a rotating token.

So scope the tool to avoid it. Read the artefact off disk instead of calling the authenticated endpoint. Take a CSV export instead of a live query. Accept a path argument instead of discovering the data. The tool gets 90% of the value and stays an afternoon.

The other honest limit: this only works because you know what the tool should do. A model can build the flaky-test leaderboard; it cannot notice that your team's real problem is that nobody trusts the test suite.

What actually changed

Not the code. The threshold.

A pile of small, painful, unglamorous problems used to be filed under "that's a day of work, not worth it" — which for a senior engineer with a roadmap means never. That category is now "an afternoon, obviously worth it", and it's large: the flake leaderboard, the build-time regression report, the script that diffs two release manifests, the dashboard that shows which modules take longest to compile.

Which means the bottleneck moved. It's no longer capacity to build; it's judgement about what's worth building — noticing which recurring annoyance is actually costing the team a day a week.

That was always the senior skill. It's just become the whole job.

The rule

Use AI hardest where verification is cheapest, and internal tools are the cheapest verification you have. Write the output first, keep the stack boring, hit real data in twenty minutes, stay read-only, and put an expiry date in the README. Then spend the judgement you saved on choosing the right thing to build next.