Screenshot Tests That Don't Cry Wolf

TL;DR — Screenshot tests fail for two reasons: a real visual regression, and everything else. If "everything else" happens often, the team learns to regenerate baselines without reading the diff, and the suite becomes an expensive PNG store. Fix it by removing nondeterminism (clock, locale, fonts, animations, network images), rendering on the JVM instead of a device, scoping tests to components rather than screens, choosing a tolerance on purpose, and surfacing the diff image in code review.

The failure mode is social, and it's rational

The suite starts useful. Then a Material library bump shifts every button by one pixel. Then a CI machine with a different GPU produces different antialiasing. Then a test rendering "2 minutes ago" fails at midnight.

Each time, the fix is to regenerate the baselines. After the fourth time, --update-screenshots becomes the reflex — and it is a correct reflex given the evidence: historically, these failures have not been bugs. From that point the suite runs, stays green, and detects nothing.

So the goal isn't more discipline. It's a suite where a red build is genuinely surprising.

1. Kill nondeterminism at the source

Anything that can change between two runs will:

@get:Rule val paparazzi = Paparazzi(
    deviceConfig = PIXEL_5,          // fixed size, density, orientation
    theme = "android:Theme.Material.Light.NoActionBar",
    renderingMode = SHRINK,          // fit content, not the whole device
)

@Test fun priceTag_onSale() {
    paparazzi.snapshot {
        AppTheme {                    // fixed colours, fixed typography
            PriceTag(
                price = Money(1299, EUR),
                now = FIXED_INSTANT,  // never Clock.System.now()
            )
        }
    }
}

The checklist, in the order these actually bite:

• Time. Inject a clock. Relative timestamps are the single most common source of overnight failures.
• Locale and formatting. Pin them. Number and date formats differ per machine.
• Animations. Disable them, or assert a specific animation frame. A running animation samples at whatever moment the capture happens.
• Remote images. Never load them. Use a fake image loader that returns a solid colour.
• Fonts. Bundle them. A system font substitution changes every glyph in the diff.
• Randomness and IDs. Any generated id that reaches the UI must be seeded.

2. Render on the JVM

Paparazzi and Roborazzi render Compose without a device or emulator. That matters for one reason: identical pixels on your laptop and on CI.

Device-based screenshot tests bring in GPU drivers, emulator versions, and OS-level rendering changes. The result is a diff that reproduces only on CI, which is another way of saying a diff nobody investigates. JVM rendering also runs in seconds, which is what lets you keep hundreds of these.

The trade-off, stated honestly: JVM rendering is not byte-identical to on-device rendering. You are testing your layout and styling, not the platform's compositor. For most regressions — wrong padding, truncated text, a state that renders the wrong colour — that's exactly the right scope. If you need to catch device-specific rendering bugs, that's a separate, much smaller device suite.

3. Scope to components, not screens

A full-screen snapshot changes whenever anything on that screen changes. Most of those changes are intended, so most of its failures are noise — and its diff is a wall of pixels in which the actual regression is hard to see.

// Five snapshots, each with one reason to fail
@Test fun priceTag_default()
@Test fun priceTag_onSale()
@Test fun priceTag_freeItem()
@Test fun priceTag_longCurrencyCode()
@Test fun priceTag_rtl()

That last one earns its place. Screenshot tests are the cheapest RTL and large-font regression check available — configurations nobody manually re-tests, which is precisely why they break.

Keep one or two full-screen snapshots for layout composition. Not thirty.

4. Choose the tolerance on purpose

Paparazzi(maxPercentDifference = 0.1)

Zero tolerance means subpixel antialiasing differences fail the build. Loose tolerance means a missing icon passes. There's no universal right answer, so pick a number, write the reason next to it, and change it deliberately rather than each time something goes red.

5. Make the diff cheaper to read than to accept

This is the part teams skip, and it's the part that decides whether the suite survives.

On failure, upload the expected/actual/diff triple as a CI artifact and post it as a comment on the PR. If a reviewer can see the regression in the PR without checking out the branch, they'll catch it. If seeing it requires downloading a zip, they'll approve the baseline update.

Treat baseline changes as reviewable content: they belong in the diff, and "regenerated snapshots" is not an acceptable commit message on its own.

Where they don't belong

Screenshot tests answer "does this render as intended". They do not answer "does this work". Behaviour — clicks, state transitions, navigation — belongs in Compose UI tests or unit tests, which fail with a sentence instead of a picture.

And skip them for anything genuinely fluid: a chart of live data, a video frame, a randomised layout. Pinning those to a pixel is fighting the feature.

The rule

A screenshot test's output is the diff image, not the pass/fail. Every design decision above serves one goal — making that image worth looking at. A suite whose diffs nobody reads isn't testing anything; it's storing PNGs at CI prices.