On-Device vs API: Choosing Where Inference Runs

TL;DR — Latency and cost dominate the debate and rarely decide it. Ask instead: can this data legally leave the device?, must it work offline?, how fast do you need to iterate on the model?, and what does it cost per user per month at real scale? The first two are usually disqualifying constraints. The third is the one teams underestimate. Most mature products end up hybrid — and on-device model size is a real install-conversion cost you should measure first.

Question 1: can this data leave the device?

If you're processing health data, biometrics, private messages, images from a camera roll, or anything covered by a regulation you'd have to name in a courtroom, this question ends the discussion.

It isn't only about the law. "Your photos never leave your phone" is a product claim with marketing value, and it's a claim you can only make if it's architecturally true. Retrofitting that promise later is close to impossible, because it means rebuilding the feature.

Answer this one first. Half the time, nothing else matters.

Question 2: does it have to work offline?

Not "should it degrade gracefully" — does the core feature have to work with no connection?

• A transit app used underground: yes.
• A camera feature used on a plane: yes.
• A document summariser: almost certainly not.

There's a middle option people forget: a small on-device model as the offline fallback, with the server model as the quality path when connected. That's more work than either alone, so it needs to be a deliberate choice rather than something you back into.

Question 3: how fast do you need to iterate?

The most underestimated question, and the one that has bitten teams I've worked with hardest.

A server-side model improves for every user the moment you deploy. Bad output found on a Tuesday can be fixed on a Wednesday.

An on-device model ships in an app update. Your fix reaches 50% of users in a week or two and 90% in six, and users on old versions keep hitting the bug you already fixed. You are now supporting several model versions simultaneously, and your analytics must be versioned to make sense of them.

So: if the model is still changing weekly, on-device is a commitment you will regret. Ship server-side until quality stabilises, then move it down if the other constraints favour that.

Question 4: what does it actually cost?

Run the number at realistic DAU, not launch DAU:

100k DAU × 3 inferences/day × 30 days = 9M inferences/month

At any per-call price, multiply it out. API inference is a recurring cost that scales with your success — the good problem that quietly ruins unit economics. On-device inference costs you app size, battery, and engineering time, but the marginal user is free.

Also cost the fallback path. A server feature needs retries, timeouts, rate limiting, and a "we couldn't do this right now" UI. That engineering is part of the price.

The hybrid pattern

Most mature products converge here:

On-device for the fast, private, always-available path. Server for the heavy, improving, occasional path.

Concretely: on-device autocomplete, classification, and simple ranking; server-side summarisation, generation, and anything needing a large model.

suspend fun suggest(text: String): List<Suggestion> =
    localModel.suggest(text)          // instant, offline, private

suspend fun summarise(doc: Document): Summary =
    if (connectivity.isOnline) remote.summarise(doc)
    else Summary.Unavailable          // an honest state, not a hidden failure

Note the explicit unavailable state. The most common failure in server-side AI features isn't the model — it's pretending the offline case doesn't exist and shipping an infinite spinner.

The constraint nobody plans for: size

On-device models are big. Adding a few hundred megabytes to your download is a measurable hit to install conversion, especially in markets where that matters most.

Options, roughly in order of preference:

• Use a platform-provided model where one exists, so you ship no weights at all.
• Download the model on first use, with a real progress UI and a graceful failure path.
• Ship it in the APK only if the feature is core to first-run value.

Measure the conversion impact rather than assuming it's negligible. This is the cost that gets discovered after launch, when it's expensive to reverse.

The order to decide in

1. Data residency — often disqualifying, decide first.
2. Offline requirement — often disqualifying, decide second.
3. Iteration speed — decides your starting point.
4. Cost at scale — decides your eventual point.
5. Size — decides whether on-device is viable at all.

Latency shows up nowhere on that list, which surprises people. It matters, but it's usually a consequence of the decision rather than an input to it — and a well-designed loading state buys you more than 200ms of model speed ever will.