UX for Latency: Designing Around a 3-Second Model

TL;DR — You will not get an inference call under a second, so design the wait instead of hiding it. Stream and optimise time-to-first-token, not total duration. Replace spinners with skeletons shaped like the answer, show honest intermediate state ("searching your notes… found 4"), and never fake a determinate progress bar. Cancellation is a feature. On Android: keep the request in the ViewModel so rotation doesn't restart it, don't recompose the list on every token, and design timeout / rate-limit / refusal / partial-output states before the happy path. And consider moving the work off the critical path entirely.

The constraint, stated honestly

A model call is 2–8 seconds, varies by 3× between identical requests, and has failure modes REST never had: a rate limit, a safety refusal, a truncation, an answer that's grammatically fine and wrong.

Meanwhile the UX vocabulary Android engineers own — a spinner, a shimmer, a full-screen loader — was built for a 300ms request where the user barely registers the wait. Stretch those patterns to four seconds and the app doesn't feel slow. It feels broken, which is a different and worse thing.

The research numbers everyone quotes still apply: under ~400ms feels instant; past ~1s attention starts to drift; past ~10s the task is abandoned. You can't move the model into the first band. You can absolutely keep a 4-second interaction out of the third.

Stream, and measure the right number

The single highest-leverage change:

Time-to-first-token is the latency your user experiences. Total duration is the latency your backend bill experiences.

A 6-second response that starts rendering at 800ms feels faster than a 3-second response that appears all at once. Same person, better experience, worse total. If you take one thing from this post, make it that your dashboard has a p50/p95 for TTFT as a distinct metric from end-to-end.

val response: Flow<String> = client.streamCompletion(prompt)

val text = response
    .runningFold("") { acc, chunk -> acc + chunk }
    .onStart { markTtftPending() }
    .onEach { if (it.isNotEmpty()) markTtftOnce() }
    .conflate()                      // UI can't use 60 updates/sec anyway
    .stateIn(viewModelScope, SharingStarted.WhileSubscribed(5_000), "")

The conflate() there matters — a fast stream can emit far more often than you can render, and dropping intermediate values is the right call when only the newest matters.

Replace the spinner with something that says more

A skeleton shaped like the answer. Three grey lines where three lines of text will be, an avatar circle where the avatar goes. Two benefits: the user's expectation is set before the content arrives, and there's no layout jump when it does. A spinner communicates exactly one thing — unknown — which is the least reassuring message available.

Honest intermediate state. If your feature does retrieval then generation, say so:

Searching your notes…
Found 4 relevant notes
Writing a summary…

Each line is a real transition, so it's information, not decoration. Users read a progressing status as a system in control of itself.

No fake determinate progress. A progress bar that animates to 90% and waits is a lie people detect in one use, and once detected it discredits every other indicator in your app. If you don't know the duration, use an indeterminate indicator and let the status text carry the meaning.

Cancellation is a feature

A Stop button changes the psychology of waiting completely: an unbounded wait you can end at any moment is far more tolerable than a bounded one you can't. It's also the cheapest cost control you'll ever ship — every cancelled generation is tokens you don't pay for.

Wire it to real cancellation, not a hidden UI:

private var job: Job? = null

fun send(prompt: String) {
    job?.cancel()
    job = viewModelScope.launch { … }
}

fun stop() = job?.cancel()

And keep the partial output on screen. A half-written answer the user stopped is still theirs; wiping it punishes them for pressing your button.

Four Android traps

1. Rotation restarting the call. If the request is launched from a composable's LaunchedEffect keyed on something transient, a config change kills a 6-second call and starts it again. The user watches it restart; you paid twice. Requests belong in the ViewModel, exposed as state:

val ui = repository.streamAnswer(query)
    .stateIn(viewModelScope, SharingStarted.WhileSubscribed(5_000), Loading)

WhileSubscribed(5_000) is what survives the config-change gap without keeping the stream alive when the user has genuinely left.

2. Recomposing the world on every token. At 30 tokens/second, a naive implementation recomposes the whole LazyColumn thirty times a second. Keep the streaming text in its own leaf composable, keyed properly, and read the value as late and as low as you can. The irony of a jank-producing AI feature is avoidable and extremely common.

3. Process death mid-request. The user backgrounds the app during a 6-second generation and Android reclaims the process. On return, either resume by request id or fail cleanly — what you must not do is show a spinner for a request nobody is running any more. Persist the request id, not the in-flight state.

4. Background continuation. If the generation genuinely must finish while the user is elsewhere, it needs WorkManager or a foreground service, with the notification that implies. Most features don't need that — the right answer is usually to cancel on leave and let them retry, because silently burning battery and tokens for a screen nobody is looking at is worse than restarting.

Design the failure states first

Write these screens before the happy path, because they're where a slow feature becomes an unusable one:

• Timeout — with a hard budget you chose (say, 20s), and a specific decision about what happens: a cheaper/faster model, a cached answer, or a non-AI fallback path.
• Rate limited — a real message with a real wait, never "something went wrong".
• Refusal or empty answer — the model declined or produced nothing usable. This will happen; a generic error is the wrong shape for it.
• Partial output — the stream died at 60%. Keep what arrived, offer Retry. Throwing away four seconds of visible output because the last chunk failed is the most infuriating possible outcome.

A latency budget is a product decision, so write it down: p95 TTFT under 1.5s, hard timeout 20s, on timeout fall back to the smaller model. Now it's testable and someone owns it.

The option nobody considers

Take it off the critical path. Not everything needs to be inline and synchronous. A summary generated in the background and delivered as a notification, a draft prepared while the user scrolls, an enrichment applied after the list already rendered — a 30-second job nobody is watching costs nothing in perceived latency.

The question to ask in design review: does the user need this before they can do the next thing? If not, the entire latency problem dissolves, and that's a far better outcome than any amount of shimmer.

The rule

Latency isn't a number to reduce — it's a state to design. The model is not getting under a second, and no amount of engineering will change that this quarter. What you control is whether four seconds reads as a system working, or a system stuck.