AI-Assisted Incident Triage: From Crashlytics Stack to Hypothesis

TL;DR — Triage is where AI helps most and where it's most dangerous, because the pressure to accept a confident answer peaks exactly when verification matters most. Ask what the stack trace rules out, scope the question to the release diff, and demand a reproduction recipe for every hypothesis — that one question filters most confident nonsense. Keep the roll-back-or-forward decision human.

The situation

Crash-free rate drops from 99.7% to 96.2%. Thousands of users affected. You have an obfuscated stack trace, a release that went out four hours ago, and a decision to make with incomplete information.

Two failure modes: acting on the first plausible theory, or spending forty minutes reading code while the crash rate climbs. AI can compress the search — if you constrain it hard.

1. Ask what the trace rules out

The inversion that saves the most time. Instead of "what caused this":

Here's the deobfuscated stack trace. I have three theories: (a) the new caching layer, (b) the permission change, (c) the Compose upgrade. Given the frames in this trace, which of these are impossible, and why?

Elimination is faster than deduction, and much harder to fake. A model can wave vaguely at a cause; it can't claim a frame is absent when it's right there in the text you pasted. Ruling out two of three theories in thirty seconds is worth more than a guess at the third.

2. Scope to the release diff

An unbounded question invites speculation about your entire codebase. Bound it:

This started with release 4.12.0. Here are the six commits in it. Which could produce this stack trace, and at which specific line? For each, say what evidence in the trace supports it.

This turns speculation into a search over a small, known space. It's also the shape of the question you should be asking yourself — most incidents are caused by what just changed, and the discipline of listing the diff is worth the two minutes even without a model.

3. Demand a reproduction recipe

The filter that matters most:

What device state, user action, and data would I need to reproduce this? Be specific: OS version, whether the app was backgrounded, what the user had done first.

If it can't specify that, it hasn't understood the crash — it has pattern-matched the exception type and produced the generic explanation for IllegalStateException. This question exposes that immediately.

And when it can answer, you have something enormously valuable at 3am: a way to confirm the theory in five minutes instead of arguing about it.

4. Confirm against the actual code — non-negotiable

Hypothesis: "The ViewModel is accessed after onCleared() when the user
             backgrounds during the network call."

Verify: open the file. Is there a path where that happens?
        Does the trace's line number match?
        Does the affected-device breakdown fit?

Every hypothesis gets confirmed before you act. The reason this rule has to be absolute rather than situational is uncomfortable but true: at 3am, a confident wrong answer looks exactly like a right one, and you are least equipped to tell the difference. Fatigue plus urgency plus a fluent explanation is precisely the condition under which people skip verification.

5. Keep the decision human

Roll back or roll forward is not a technical question. It's a judgement about blast radius, user impact, confidence in the fix, and how long a store review takes. Inputs a model doesn't have.

My default, and I'd argue for it in most cases: roll back first. Halt the staged rollout, revert to the previous release. The bleeding stops, and you diagnose calmly in daylight with a reproduction and a real test. The instinct to fix forward is usually ego and sunk cost rather than analysis.

Rolling forward makes sense when you're certain of the cause, the fix is one line, and rollback is genuinely unavailable — for instance because the release included a database migration, which is exactly the scenario that argues for additive migrations in the first place.

What it's good at, honestly

Good: reading a long trace faster than you can, spotting that a frame comes from a library rather than your code, recognising a known crash signature, ruling out theories, generating the reproduction recipe.

Bad: knowing that this device model is 40% of your users in one market, that the affected screen launched yesterday, that the "obviously wrong" code is a deliberate workaround for an OEM bug. The context that turns a plausible cause into the actual one.

Afterwards

The postmortem is where the real value lands, and it's a calmer, better use:

Here's the root cause and the fix. What else in this codebase has the same shape? What test would have caught it? What monitoring would have caught it sooner?

That third question is the one teams forget. Most incidents are detected later than they could have been, and the fix for that is usually a metric nobody had thought to watch.

The rule

Use it to think faster. Never to decide. Triage rewards narrowing the search quickly and punishes acting on unverified confidence — and those are exactly the two things AI makes easier, in opposite directions.