AI and Flaky Tests: Root-Causing Instead of Retrying

TL;DR — Retrying a flaky test converts a real defect into noise you pay for forever. Flakes are unusually well suited to AI assistance because they come with both passing and failing runs of identical code — a differential most bugs don't offer. The workflow: feed the set of runs, ask for an enumeration of what the outcome could depend on besides the code under test, then ask for a reproduction that fails 100% of the time, and only then a fix. Hard policy: any suggested fix that adds a wait is a retry with extra steps. Cap the effort at 45 minutes, then quarantine with an owner and a date.

Why retrying is worse than it looks

retry(3) in your Gradle test config, or @FlakyTest, or a CI step that reruns failures. It feels pragmatic — the build goes green, the team ships.

What actually happened: a test detected non-determinism in your production code, and you configured the detector off. Roughly half of Android test flakes are the test's fault (a Thread.sleep, a leaked singleton). The other half are real concurrency or lifecycle bugs that also happen in production, where there is no retry. That's the half retrying costs you.

The second cost is slower and worse: a suite with retries stops being trusted, and an untrusted suite stops being read. Nobody investigates a red build that's red twice a week for "the usual reasons."

The property that makes this a good AI task

Almost every debugging session starts with one failure and no counterexample. A flaky test hands you something rarer:

The same code, the same test, 14 green runs and 6 red ones.

That's a differential, and finding what varies across a labelled set is precisely the kind of work models do well — it's pattern recognition over structured text, not reasoning about a system they can't observe.

Which means the single biggest mistake is throwing that advantage away by pasting one stack trace and asking "why is this flaky?" With one data point there's nothing to compare, so you get a confident, plausible race condition that isn't there, and you spend an afternoon on it.

The workflow

1. Collect the set, not the symptom

From your CI, for the last ~20 runs of that test:

• pass/fail
• duration (the most underrated signal — flaky failures are often 10× faster or slower)
• the failure message and stack
• which other tests ran in the same JVM/fork, and in what order
• the runner/emulator image, and the commit

Then:

Same test, same commit, 20 CI runs: 14 pass, 6 fail.
Attached: durations, failure traces, test execution order per run.

What is different about the failing runs? Rank by how well each
hypothesis explains ALL six failures and none of the passes.

"Explains all six and none of the passes" is the load-bearing phrase. Without it you get a list of generic flakiness causes copied from a blog post. With it, the model has to actually fit the data, and weak hypotheses get visibly discarded.

2. Ask for enumeration, not diagnosis

The prompt that consistently produces value:

Here is the test and the code under test.

List EVERY way this test's outcome could depend on something other
than the logic being tested: execution order, shared/static state,
real clock, dispatchers, thread pools, locale, timezone, animation
scale, file system, network, emulator state, JVM fork reuse.

For each: is it plausible here, and what one-line check confirms it?

Two reasons this works. Enumeration over a known space is a genuine strength, and completeness is the thing tired humans lose at 4pm — you know about timezone flakes, you just weren't thinking about them today. And "what one-line check confirms it" turns a list of possibilities into a list of experiments.

3. Ask for a reproduction, not a fix

This is the step that separates real root-causing from guessing:

Your top hypothesis is that the ViewModel's init collects on
Dispatchers.Main.immediate while the test sets the dispatcher after
construction. Write a test that fails 100% of the time if that's true
— force the ordering, don't hope for it.

Now verification is cheap and binary. It either fails every single run — hypothesis confirmed, and you have a regression test — or it doesn't, and you've discarded a wrong theory in ninety seconds instead of building on it.

A fix without a 100%-failing reproduction is a guess with a commit message. You cannot tell a fixed flake from a flake that didn't happen today.

4. Fix, then prove it

./gradlew :feature:test --tests "*.SyncViewModelTest" -Prepeat=200

Or @RepeatedTest(200), or a shell loop. Two hundred green runs after two hundred red ones is proof. "It's been fine this week" is not.

The flake families it finds quickly

These are pattern-matchable, which is why the enumeration step pays off:

Dispatcher injection. A hardcoded Dispatchers.IO or Main.immediate inside a class under test, combined with runTest's virtual time. Sometimes the coroutine wins the race, sometimes it doesn't.

Leaked state between tests. A Hilt @Singleton surviving into the next test, an object with a companion object cache, an in-memory Room database shared across a test class, a MockWebServer queue that wasn't drained. Tell-tale sign: it passes alone and fails in the suite. Confirm it in one command by shuffling test order.

Time. System.currentTimeMillis() reached directly instead of an injected Clock; a test that computes "yesterday" and fails when CI crosses midnight UTC; a token expiry compared against a real clock.

Locale and timezone. Your machine is Asia/Tehran, the runner is UTC. Date formatting, String.format with a decimal separator, toLowerCase() in Turkish locale. Deterministic per environment, which reads as flaky across a fleet.

Synchronisation faked with sleeping. Thread.sleep(500) in an instrumentation test is a bet on a machine's load. It passes on your desktop and loses on a busy CI runner.

Emulator conditions. Animations not disabled, a cold-start dialog, a system update prompt stealing focus.

Where it's actively dangerous

Every AI post in this series names the failure mode, and this one has a specific, seductive one.

1. It will suggest a wait. Asked to fix a flaky UI test, a model will very often propose an IdlingResource, a longer timeout, or composeTestRule.waitUntil(5_000). Sometimes that's legitimate. Usually it is a retry wearing a costume — it makes the failure rarer without removing the non-determinism, which is strictly worse because now it fails only under load, in a quarter, once.

Policy: reject any fix whose mechanism is "wait longer." Ask instead: what made this test depend on real time, and how do I inject that? The answer is nearly always a dispatcher, a clock, or a test rule that isn't installed.

2. It hallucinates races when starved of data. One trace in, one confident race out. Give it the passing runs too, or don't ask.

3. It cannot see your CI. maxParallelForks, whether forks are reused, the emulator image, the runner's core count, whether animationScale is zeroed. It'll hypothesise about parallelism; you have to go and read the config. Paste your build.gradle.kts test block in — it's twenty lines and it removes a whole class of wrong guesses.

4. It won't tell you to delete the test. Some flaky tests are testing something not worth testing — an animation's exact timing, a third-party SDK's callback ordering. Deleting is a legitimate outcome and you'll never hear it suggested.

When to stop

Cap it at 45 minutes. Some flakes are genuinely infrastructural — a runner with a noisy neighbour, a network blip, an emulator that occasionally boots wrong — and no amount of reading code finds those.

If the clock runs out: quarantine, and be strict about the difference.

• Retry = the test stays in the signal and lies. Green build, defect still live.
• Quarantine = the test is removed from the blocking suite, still runs, still reports, and has a named owner and a date. Untouched by that date, it gets deleted.

Track the count on a dashboard. A quarantine list that only grows is a suite being abandoned in slow motion, and putting the number where people see it is what stops that.

The rule

Never fix a flake you can't reproduce on demand. The model's job is to widen your hypothesis space and write the reproduction that collapses it — not to hand you a patch. And the moment a suggestion involves waiting longer, you've stopped root-causing and started retrying with extra steps.