Playback Analytics That Explain a Bad Night

TL;DR — Streaming quality is not an uptime problem, and averages actively hide it. Measure five things per session: video start failure rate, time to first frame (P50 and P95), rebuffer ratio, exits before video start, and ABR downshift rate. Then make each one sliceable by CDN, ASN, device model, DRM security level, title and app version — a bad night is almost always one cell in that grid, not a global degradation. Classify failures using PlaybackException.errorCode rather than one "playback_error" counter, and sample the happy path, never the errors.

Why the dashboard says everything is fine

Two things are simultaneously true on a bad night: your service was up, and a meaningful share of viewers had a bad time. Uptime is a property of your infrastructure. Quality of experience is a property of a session — one viewer, one device, one network, one CDN edge, one title.

Averaging across sessions destroys the signal by construction. Ten thousand people watching happily plus four hundred people unable to start averages to "slightly elevated start time". The four hundred are the entire story, and the mean is designed to hide them.

That's the whole shape of the problem: you're aggregating over a population that isn't homogeneous.

The five metrics that explain a night

1. Video start failure rate (VSF)

The share of playback attempts that never produced a first frame. Manifest 404, DRM licence denied, decoder init failure, token expired.

This is the most important metric in streaming and the one most commonly missing, because a failed start generates no watch time, no rebuffer events, and often no support ticket. The user taps play, sees a spinner, and leaves. Your engagement metrics register it as "didn't watch anything today", which looks like a content problem.

2. Time to first frame (TTFF), P50 and P95

From tap to first rendered frame. Report both percentiles — the mean is meaningless here because the distribution has a heavy tail, and the tail is the complaint.

Break it into phases if you can: manifest fetch, DRM licence acquisition, first segment download, decoder init. On DRM-protected content, licence acquisition is frequently the dominant term and almost nobody measures it separately.

3. Rebuffer ratio

rebuffer ratio = total rebuffer time / total play time

A ratio, not a count, which is what makes a three-hour film comparable to a four-minute clip. Under 0.5% is a healthy target for VOD; live is tighter.

Exclude seek-induced buffering — that's the user's doing and it will mask the involuntary stalls you actually care about.

4. Exits before video start (EBVS)

Of the people who tapped play and didn't hit a failure, how many left during the spinner? This is TTFF's consequence, and it converts a latency number into a business number. It's also the metric that makes the case for spending a sprint on start time.

5. ABR downshift rate

How often the adaptive bitrate ladder steps down. Quality degrades before it stalls, so downshifts are your leading indicator: a CDN going bad shows up here fifteen minutes before rebuffer ratio moves.

The metrics aren't the point — the dimensions are

Any of those five, aggregated globally, tells you that last night was bad. None of them tells you why.

Every session event needs to carry:

Dimension	Catches
CDN / edge	One CDN region degraded — the single most common cause
ASN / ISP	Peering problem, one carrier, one country
Device model + SoC	Decoder bugs, one OEM's firmware update
DRM security level (Widevine L1/L3)	Licence policy rejecting L3, or L1 unavailable after an update
Title / encoding profile	One bad transcode, one broken manifest
App version	You shipped it
Live vs VOD	Different failure modes entirely
Geography	Regional routing

A bad night is one cell in that grid. With the grid, "Widevine L3 devices on one ISP failing licence acquisition" is a ten-minute query. Without it, it's a week of guessing and a postmortem that says "transient network issues", which is what teams write when they never found out.

Include the CDN edge identifier in the session record. Most CDNs expose it in a response header. It's what lets you join your client-side data to their logs, and that join is the difference between "our players saw errors" and "your lhr-04 edge was returning 502s between 21:10 and 21:40".

Instrumenting Media3

PlaybackStatsListener does most of the work and is underused:

private val statsListener = PlaybackStatsListener(
    /* keepHistory = */ true
) { eventTime, stats ->
    analytics.playbackSession(
        // Quality of experience
        ttffMs         = stats.meanTimeToFirstFrameMs,
        rebufferRatio  = stats.totalRebufferTimeMs /
                            stats.totalPlayTimeMs.coerceAtLeast(1).toDouble(),
        rebufferCount  = stats.totalRebufferCount,
        meanBitrate    = stats.meanVideoFormatBitrate,
        joinTimeMs     = stats.totalJoinTimeMs,
        // Outcome
        fatalError     = stats.fatalErrorCount > 0,
        // Dimensions — the half that makes it diagnostic
        cdnEdge        = session.cdnEdge,
        asn            = session.asn,
        drmLevel       = session.widevineSecurityLevel,
        contentId      = session.contentId,
        profile        = session.encodingProfile,
        isLive         = session.isLive,
    )
}

player.addAnalyticsListener(statsListener)

And classify errors instead of counting them:

override fun onPlayerError(error: PlaybackException) {
    val category = when (error.errorCode) {
        in DRM_ERROR_RANGE     -> "drm"
        in IO_ERROR_RANGE      -> "network"
        in DECODER_ERROR_RANGE -> "decoder"
        else                   -> "other"
    }
    analytics.playbackFailure(
        category = category,
        code     = error.errorCodeName,   // human-readable, keep it
        cdnEdge  = session.cdnEdge,
        drmLevel = session.widevineSecurityLevel,
    )
}

Media3 already separates DRM from I/O from decoder failures. Collapsing all of it into one playback_error counter throws away the only field that would have told you which team to wake up.

Sampling: invert the usual rule

The instinct is to sample everything at 10% to control cost. For playback that's backwards.

• Sample the happy path. 10% of successful sessions is plenty for percentiles and trends.
• Never sample failures. They're rare — that's the point — and they're the entire diagnostic dataset. Sampling them at 10% means a regression affecting 200 users produces 20 events, which looks like noise.

This one change usually costs nothing and makes the difference between a diagnosable incident and a plausible story.

The bad-night playbook

When it happens, in order:

1. Rebuffer ratio and VSF by CDN, for the window, compared to the same window a week earlier. Weekly comparison matters — traffic patterns are strongly diurnal and weekly.
2. If one CDN — split by region and ASN. You now have something to send them, with edge IDs.
3. If all CDNs — split by app version and title. You shipped something, or one transcode is bad.
4. If it's one device family — check DRM level and decoder. An OEM firmware rollout is a common cause and invisible from every other angle.
5. If nothing separates — check what changed on the backend: licence server latency, manifest generation, token TTL. A licence server at P95 4s produces exactly this pattern with no client-side error at all.

Most nights end at step 2.

What to put on the wall

One graph: rebuffer ratio and VSF, this week vs last week, split by CDN. Everything else is a drill-down you reach for when that graph moves. A wall of forty charts gets ignored; two lines that overlay a known-good week get read every morning.

The rule

An average across sessions can't explain a bad night — a distribution across dimensions can. Measure per session, carry the CDN edge and the ASN, classify the error codes you're already being handed, and never sample a failure. Then the postmortem names a cause instead of naming the weather.