99.9% Crash-Free Is a Process, Not a Number

TL;DR — Crash-free rate is a lagging average, useful for reporting and nearly useless for deciding anything. 99.9% of a million daily users is a thousand people a day. The number hides which denominator you used (users vs sessions), device- and version-specific regressions, ANRs — which Play weighs more harshly than crashes — and the difference between a background crash and a checkout crash. Chased as a target it invites the quiet Goodhart failure of swallowing exceptions. What works instead is a process: a halt criterion written before the rollout, segmented dashboards, a named owner on a rota, and a weekly read of the top non-fatals.

What the number actually is

Two metrics wear the same name.

• Crash-free users — the share of unique users who saw no crash in the window. Forgiving: a user who crashes twenty times counts once.
• Crash-free sessions — the share of sessions that ended without a crash. Harsher, and closer to what a user experiences.

They're different by a meaningful margin, and teams routinely quote one against a target set for the other. Pick one, put the word "sessions" or "users" in the dashboard title, and hold the whole org to it. This is a five-minute fix that prevents a recurring, entirely avoidable argument.

Then do the multiplication once, out loud, in front of the team:

1,000,000 DAU × 0.1%  = 1,000 people/day
5,000,000 DAU × 0.05% = 2,500 people/day

"Three nines" stops sounding like an achievement and starts sounding like a queue.

Why the aggregate hides the thing you need to see

An average over a heterogeneous population tells you about the population, not about anyone in it.

A crash affecting 0.05% of users moves the headline by half a tenth of a point — noise. That same crash can be every user on one OEM's Android 11 build, or every user in one locale where a date parses differently, or every user who upgraded rather than installed fresh.

So the aggregate is structurally the last place a targeted regression appears. The dashboards that actually catch things are segmented:

• App version — always. Compare the rollout against the previous stable, not against the blended average, which is dominated by the old version during a staged rollout and will look reassuring right up until it doesn't.
• OS version and manufacturer — where vendor-specific breakage lives.
• New installs vs existing — the one nobody watches. A first-run crash silently destroys acquisition while the aggregate, dominated by happy long-term users, stays green.
• Country/locale — RTL, date and number formats, and one CDN region.

ANRs are not crashes, and they are worse

An ANR doesn't kill the app; it freezes it, and the system shows the user a dialog with your app's name on it. From the user's side that's worse than a crash — a crash at least restarts into something that works.

Google's Android vitals bad-behaviour thresholds make the relative weighting explicit: user-perceived ANR rate 0.47%, user-perceived crash rate 1.09%. Exceed them and you risk reduced Play Store visibility. An ANR budget less than half the size of the crash budget is the platform telling you which it considers more damaging.

Most teams' crash dashboard is mature and their ANR dashboard doesn't exist. Fix that asymmetry first — it's usually the highest-value hour in this whole area.

The Goodhart failure

Make crash-free rate a target on someone's performance review and you will get it. Here is how:

// The rate goes up. The product gets worse.
try {
    renderCheckout(state)
} catch (e: Exception) {
    Firebase.recordException(e)
    showGenericError()   // a blank screen with a shrug
}

The crash became a non-fatal. The graph turned green. The user still can't check out — they just can't tell you why any more, and neither can you, because the stack trace is now in a bucket nobody reviews.

There's a legitimate version of that code — a genuinely optional feature, degraded deliberately, with the failure surfaced to the user honestly. The tell is whether the catch block preserves the user's ability to continue. If it only preserves the metric, it's a lie with a try in front of it.

The same pressure produces its cousin: quietly excluding a crashy OEM from the dashboard, or filtering out "unactionable" OOMs. Any metric change that makes the number better without making the app better is a change to your reporting, and it should get the same scrutiny as a change to your product.

The process that replaces the target

1. A halt criterion, written before the rollout

Decide it while nobody is panicking, because during the panic somebody senior will say "let's give it a few more hours".

# release/3.14.0.yml
halt_if:
  crash_free_sessions_below: 99.80   # vs 99.94 baseline
  anr_rate_above: 0.40
  checkout_success_drop: 2%          # the business metric
  min_sessions_before_judging: 25000 # no decisions on noise
owner: "@hessam (primary), @sara (backup)"
review_at: ["10%", "50%"]

Note the last two lines. A halt criterion without a minimum sample size gets triggered by a single unlucky device at 1% rollout, and after that nobody trusts it. And a criterion without a named human is a document, not a control.

2. Segment by default, aggregate on request

The default view of your stability dashboard should be this version vs previous, by OS and OEM. The blended number goes on the monthly report, not on the wall.

3. A named owner on a rota

"The team owns stability" means nobody looks on the Thursday before a long weekend. A rota — one person, one week, explicitly responsible for triaging new issues within a day — survives holidays and turnover in a way that team-wide goodwill does not. Their job is triage and routing, not fixing everything themselves.

4. Read the top five non-fatals weekly

This is the leading indicator, and it costs fifteen minutes. Non-fatals are the crashes you caught. They include the swallowed exceptions above, the NullPointerException in a branch that's currently rare, the serialization failure that will become universal when the backend ships its new field.

Sort by affected users, not count — one user in a retry loop generates ten thousand events and buries the issue affecting two thousand people.

What "good" looks like

Not a number. A set of answerable questions:

• Can you tell, within thirty minutes of a release going to 10%, whether it's worse than the previous one?
• Does your dashboard distinguish new installs from upgrades?
• Do you have an ANR dashboard at all, with a threshold on it?
• Is there a specific person responsible for stability triage this week, by name?
• When you last halted a rollout, was the decision made by the criterion or by argument?

A team that answers those cleanly at 99.7% is in far better shape than a team at 99.95% that answers none of them — because the second team's number is a description of their luck.

The rule

Crash-free rate tells you whether last week was fine. It has never told anyone what to do on a Tuesday. Report the number, decide with the process: a halt criterion agreed in advance, segments that can see a regression the average can't, a named owner, and fifteen minutes a week reading the crashes that haven't happened yet.