Merge Queues and the Flaky-Test Tax

TL;DR — A merge queue tests the merged result in order, which kills the "green on my PR, red on main" class of failure. It also turns flakiness from a linear cost into a superlinear one: one spurious failure kicks a whole batch and re-runs every PR behind it. The bar is higher than people think — 2,000 tests at 99.99% reliability each still produces an 18% spurious failure rate per run. So: measure the tax, detect flakes mechanically (passed and failed on the same commit), quarantine with an owner and an expiry, never blanket-retry, and split the suite so only fast deterministic checks gate the queue.

What a merge queue actually solves

Not "flaky CI". One specific thing: the semantic merge conflict.

PR A renames Order.total to Order.totalMinor. PR B adds a new call site using Order.total. Both are green against main. Git merges both without a textual conflict. main is now broken, and it's broken for everybody until someone notices and fixes forward.

A merge queue tests the result of merging, in order, before anything lands:

main + A            → green → land A
main + A + B        → RED   → eject B, tell its author
main + A + C        → green → land C

This is genuinely valuable on a large Android codebase where a shared module is touched by five teams a day. It's also the only thing a queue does. Everything else is a consequence.

Why the queue amplifies flakiness

Without a queue, a flaky failure costs one engineer one re-run. Annoying, contained, and the cost falls on the person who can decide whether to care.

With a queue batching five PRs, one flaky failure means the batch fails. Depending on your configuration, either the queue bisects to find the culprit (extra runs) or it ejects and re-runs everything behind (more extra runs). Either way, one flake now costs several PRs their place in line, and the failure is attributed to whoever's PR was unlucky rather than to the flaky test.

That's the trap: the queue makes flakiness everyone's problem at the same moment it makes it nobody's fault. Throughput drops, people blame the queue, and the queue gets turned off — with the flaky tests still there.

The arithmetic nobody does

Per-test reliability has to be extreme, and the intuition is badly wrong:

2,000 tests, each 99.99% reliable
P(clean run) = 0.9999 ^ 2000 = 0.819
→ 18% of runs fail for no reason

Four nines per test, and nearly one run in five is spuriously red. At three nines per test it's 86%.

Two consequences:

• "Our tests are basically fine" is not a measurement, and it's almost certainly wrong at scale.
• Suite size is a reliability liability. Ten thousand tests need an order of magnitude better per-test reliability than two thousand to hit the same run-level number. That's a real argument for fewer, better integration tests and against reflexively adding one for every bug.

Do this in order

1. Measure the tax before you argue about it

You will not get a week to fix flakiness by describing it as annoying. You'll get it by costing it.

Last 30 days:
  CI runs                       4,180
  Runs failed, no code defect     512  (12.2%)
  Wasted CI minutes            18,400  (~$430)
  Engineer wait, at 9 min/rerun  76 h
  → roughly two engineer-weeks per month

That last line is the sentence that gets funded. "Two engineer-weeks a month, recurring" is a business case; "CI is flaky" is a complaint.

2. Detect flakes mechanically, not by opinion

The definition needs no judgement: a test that both passed and failed on the same commit is flaky. Make your CI record it.

# On failure: re-run the failing tests once, on the SAME sha.
# If they pass, that's a recorded flake — not a fixed problem.
- name: Re-run failures once
  if: failure()
  run: ./gradlew testDebugUnitTest --tests "$FAILED" || true
- name: Record
  run: ./ci/record-flake.sh "$FAILED" "$GITHUB_SHA"

Then keep a leaderboard, sorted by how many merges each test has blocked — not by raw failure count. A test that fails often on an unimportant branch matters less than one that has ejected six PRs from the queue.

Publishing the leaderboard weekly does most of the cultural work for you.

3. Quarantine with an owner and an expiry

Quarantine ≠ disable. A quarantined test still runs and still reports; it just doesn't block a merge.

@Quarantined(
    since = "2026-10-19",
    owner = "@payments",
    expires = "2026-11-16",
    reason = "Races with the retry scheduler; see APP-4412",
)
@Test fun retriesOnTransientFailure() { … }

Both fields are load-bearing. Without an owner, quarantine is where tests go to die. Without an expiry, your quarantine list is a graveyard that grows monotonically, and a year later nobody can say what coverage you actually have. Fail the build if anything is past its expiry — that's the only enforcement that works.

Cap the list, too. Something like "no more than 15 quarantined tests; adding a 16th requires deleting one" keeps the pressure where it belongs.

4. Never blanket-retry

The tempting one-liner:

// Please don't.
test { retry { maxRetries = 3 } }

It doesn't remove flakiness; it hides it, and it hides it in the worst way — the flake rate is now invisible while the underlying defect is still there. Some of what it conceals is a genuine race condition in your product, and a race that fails 1-in-20 in CI fails for real users too, on slower devices, in numbers you'll meet in Crashlytics rather than in a test report.

Retry is acceptable only when it's recorded and budgeted: every retry emits an event, the flake leaderboard counts it, and a test exceeding a threshold is auto-quarantined and assigned. Retry as telemetry, not as a cure.

5. Split the suite so only fast, deterministic checks gate the queue

Queue throughput is roughly batch size ÷ run time. A 40-minute required suite is a hard ceiling on how many PRs a day can land, regardless of team size.

Stage	What runs	Gates the queue?
Required	Unit tests, lint, compile, API dump check	Yes — target under 10 min
Post-merge	Instrumentation, screenshot, E2E	No — alerts + revert path
Nightly	Full matrix, older API levels, benchmarks	No

Moving your instrumentation suite out of the required set is usually the single biggest throughput win available, and it's safe only if you have a fast, practised revert path. That's the trade: post-merge testing buys speed with the ability to undo. Without one-click revert, don't make the trade.

6. Tune the queue itself

• Batch size — bigger batches are faster when green and much more expensive when red. If your flake rate is above a couple of percent, use a small batch or none at all until you've fixed it.
• Bisection on failure rather than serial re-runs. Log n extra runs instead of n.
• Never auto-retry a queue failure. The one place retry is unambiguously wrong: it re-runs the whole batch and hides which PR was at fault.

The cultural half

"Just re-run it" spreading through a team is the observable symptom of a test suite that has stopped meaning anything. Once re-running is the reflex, a real failure gets re-run too — twice, three times — and then merged, because it looked like the usual noise.

That's the actual danger, and it's worse than the throughput cost: you didn't lose CI time, you lost the signal. A merge queue on top of that culture doesn't fix it; it just makes the noise louder and more expensive.

So the order matters. Fix the flake rate, then turn the queue on. A queue is a multiplier — and it multiplies whatever you already have.

The rule

A merge queue is a lever on your existing test reliability, not a substitute for it. Cost the tax so you can fund the fix, define flakiness mechanically, quarantine with an owner and a date, keep retries as telemetry rather than treatment, and gate the queue on the fast deterministic subset. Then the queue does what it's for: making "green on my PR" mean "green on main".