Staged Rollouts and the Halt Criteria You Set in Advance

TL;DR — Staged rollout only protects you if you decide the stopping conditions before you start. Fix four numbers in the release ticket — crash-free sessions, ANR rate, one product metric, a per-device threshold — plus a minimum soak time per stage. Then design for the Android reality that halting is not a rollback: users who already updated keep the build, so additive migrations and server-side kill switches are what actually save you.

Why in-flight judgement fails

At 40% rollout, someone notices crash-free sessions have moved from 99.6% to 99.4%. The conversation that follows is predictable: is that the release, or a bad-data day? It's only 0.2%. Marketing has booked the announcement. Halting means a new build, another review, another week.

Every incentive in that room points at continuing, and the person best placed to judge the data is the person who built the thing. This isn't a failure of character — it's a decision made under time pressure, with sunk cost, by an interested party. The fix isn't better judgement in the moment. It's making the decision earlier, when none of those forces apply.

The four numbers, written in the release ticket

1. Crash-free sessions. An absolute threshold, e.g. "halt below 99.5%". Two refinements that make it usable:

• Compare against the same stage of the previous release, not the all-time average. New releases always look slightly worse early, because early adopters skew towards devices that update fast.
• Use crash-free sessions, not users, for rollout decisions — it responds faster.

2. ANR rate. Routinely forgotten, and Play penalises it directly through vitals thresholds. A release that adds a synchronous disk read to onCreate won't move your crash rate at all.

3. One product metric. The class of bug this catches doesn't crash anything: a checkout button that no longer submits, a login flow that fails for one auth provider. Pick the one metric that means the app is doing its job — completed checkouts per session, successful logins — and set a band.

4. A per-device or per-OS threshold. "Halt if any single device model exceeds 1% crash rate, even if the global number is within tolerance." A global average of 99.6% is entirely compatible with one OEM's flagship crashing at 15%, and if that model is popular in a market you don't check daily, aggregates will hide it for a week.

Stage timing beats stage percentage

Percentages get all the attention; duration is what determines whether you learn anything.

• 1% for 24 hours. Not four. A four-hour soak covers one region's morning and misses your evening peak, your batch jobs, and the midnight rollover bugs.
• 5–10% for 24 hours. Enough volume for device-level breakdowns to be meaningful.
• 20% → 50% → 100%, each with a full day, unless the release is trivial.

The exception is genuinely urgent: a critical fix goes out faster on purpose, and that's a decision made knowingly rather than by skipping the process.

One statistical note worth internalising: at 1% of a million users, a crash affecting 0.1% of sessions produces a handful of events. You will not distinguish it from noise. Don't pretend the 1% stage validates rare bugs — it validates catastrophic ones, which is still worth the day.

Halting is not a rollback

The most important Android-specific fact, and the one teams discover during their first bad release:

Google Play has no recall. Halting a rollout stops new users receiving the update. Everyone who already got it keeps it. If 20% of your users are on a broken build, halting leaves 20% of your users on a broken build — and shipping the fix means a new version code, another review, and another rollout that starts at 1%.

Which makes the following architectural, not procedural:

• Additive database migrations only. If v4.12 rewrites the schema destructively, users on it cannot be moved back to v4.11 in any way. Add columns and tables; don't drop or repurpose them in the same release that uses them.
• Server-side kill switches for anything risky. A feature flag you control remotely turns a release incident into a config change. This is the single highest-leverage investment in release safety, and it's the reason a bad feature is survivable while a bad migration isn't.
• Backward-compatible API contracts. A client that can only talk to the new API turns a service rollback into a client outage.
• Prepare the hotfix branch before you need it. Cut it at release time, not during the incident.

Who can halt

Write this down too. The on-call engineer must be able to halt the rollout without waking a director. If halting requires an escalation, the criteria you agreed in advance are decorative — the delay is the decision.

And halt on the criteria even when you're fairly sure the cause is unrelated. The threshold exists precisely to remove that judgement call from the moment where it can't be made cleanly. Investigate after halting; the cost of a halted rollout is a few days, and the cost of the alternative is measured in users.

What to write in the ticket

Rollout plan — v4.12.0
  Stages: 1% (24h) → 10% (24h) → 50% (24h) → 100%
  Halt if:
    · crash-free sessions < 99.5%, or 0.15% below v4.11.0 at same stage
    · ANR rate > 0.47%
    · checkout completion drops > 3% vs 7-day baseline
    · any device model > 1% crash rate with n > 500 sessions
  Owner: on-call. No escalation required to halt.
  Kill switches: new-checkout-flow, image-pipeline-v2

Five minutes to write. It's the difference between a rollout that protects you and one that just makes you slower.

The rule

A rollout percentage is not a safety mechanism; a pre-agreed threshold is. Decide what "bad" looks like while you still have no stake in the answer — because in the moment, you won't.