How do you design an A/B test for a mobile feature, and what pitfalls do mobile release cycles add?

A senior mobile engineer is rarely asked to run an A/B test — the growth or data science team usually owns the statistics platform. What you get asked to design is the part that platform can't see: how the experiment is wired into a binary that Apple and Google control the release of, and how that changes what "random assignment" and "clean read" mean in practice.

#Start from the same place web does

The core experiment design doesn't change just because the client is a mobile app. You still need a single, falsifiable hypothesis ("moving the CTA above the fold increases day-1 activation by at least 2 points"), a primary metric that maps directly to that hypothesis, a small set of guardrail metrics that must not regress (crash-free rate, ANR rate, session length, uninstall rate), and a unit of randomization — almost always the installed user, identified by a stable pseudonymous id, not the device and not the account, because a user can reinstall or hold multiple accounts.

Before anything ships you run a power analysis: given your baseline conversion rate and daily active users, how long do you need to run the test to detect the minimum effect size you actually care about, at the significance and power thresholds the team has agreed on? On mobile this number is almost always worse than the team expects, for reasons that have nothing to do with statistics and everything to do with how binaries move.

#Where the mobile release train breaks the playbook

A web A/B test can ship a new variant to 100% of traffic in minutes and roll it back just as fast if something looks wrong. Mobile can't do either of those things through the normal release channel. Once a build is approved and rolling out, you are not editing it — you are waiting for App Store and Play Store review on the next one, which can take anywhere from a few hours to a few days, and even after approval, adoption of a new binary crawls: some users update within the hour, plenty don't touch it for weeks, and a meaningful slice never update until they're forced to by an OS requirement or a "please update" gate.

That single fact reshapes the whole experiment:

• The experiment can't live only in native code. If the variant and the assignment logic are compiled into the binary, you can't kill a bad variant without another store submission. Real mobile experiments assign and gate behavior through a remote config or feature-flagging SDK (Firebase Remote Config, LaunchDarkly, Statsig, a homegrown flag service) that the app fetches at launch or on a timer. The binary ships both branches of the code; the flag decides which one runs, and you can flip that flag server-side in minutes.
• Version fragmentation is a confound, not an inconvenience. At any moment your population is running four or five app versions simultaneously. If the feature you're testing depends on something that only shipped in the latest build (a new SDK, a bug fix, a permission prompt), your treatment group is silently contaminated by users who can't even render the new experience correctly. You gate the experiment by minimum supported app version, and you treat "hasn't updated" as its own segment to watch, not noise to average away.
• Staged/phased rollouts collide with your randomization. Both stores let you ship a new binary to a percentage of the population first (Play Console's staged rollout, TestFlight/App Store phased release). If that percentage-based rollout is also how you're assigning treatment vs. control, you've coupled two independent randomizations and probably broken both — the population getting the new binary and the population getting the new feature need to be sampled independently, or your "control" group turns out to be, say, disproportionately non-updaters.
• Store review adds a fixed lag you cannot buy your way out of. If your treatment ships a bug, you cannot hotfix a native crash the way a web team ships a revert in ten minutes. Anything genuinely risky in an experiment — new native code paths, permission changes, payment flows — needs a server-side kill switch as a first-class part of the design, decided before launch, not improvised during an incident.

#Metrics that only make sense once you're on-device

A few pitfalls show up specifically because you're measuring an installed app rather than a page load. Attrition is asymmetric: a bad web variant costs you a session; a bad mobile variant can cost you the install entirely, and uninstalls are notoriously under-observed — you generally only infer them from a user going silent, days later, with no clean signal of when or why. Attribution gets murkier post-ATT (App Tracking Transparency) and with Android's privacy sandbox changes, so cross-session identity for the same logical user is less reliable than it used to be, which quietly inflates your variance. And connectivity matters in a way it rarely does for web: users on flaky networks may fail to fetch the remote flag at all and silently fall back to a default — decide up front what that default is, because "flag fetch failed" is itself a non-random subpopulation (poor connectivity correlates with geography, device tier, and behavior).

#Analysis-time pitfalls worth naming out loud in an interview

Sample ratio mismatch (SRM) — checking that the control/treatment split you actually observed matches the split you configured — matters more on mobile because there are more places assignment can silently leak: a caching layer serving a stale flag value, a crash that only affects one arm and removes those users from your funnel before they're ever logged, or an app-store staged rollout confounding who is even eligible to be bucketed. Novelty effects are also sharper on mobile, because a redesigned screen is one of a handful of screens a user sees every day, so an early lift is often just attention, not durable behavior change — the standard mitigation is a longer minimum run plus a look at metric trend over time rather than a single readout at "day 7." Finally, treat this as an org-design problem as much as a stats problem: someone has to own the decision to roll a winning variant out to 100% through the next release train, and that rollout itself needs the same phased/guardrail thinking as the original experiment, because "the A/B test said ship it" and "shipping it safely to a billion devices" are two different engineering problems.

🧠 A/B testing on mobile
0/3 · 0/3 answered
1. Why is a remote feature-flag SDK considered a prerequisite for most mobile A/B tests, rather than a nice-to-have?
It automatically calculates statistical significanceApp stores require all experiments to use a specific SDKIt makes the app bundle smallerIt lets you change which variant runs, or kill a bad one, without waiting for app store review on a new binary
2. What's the risk of using a staged/phased store rollout percentage as your experiment's treatment/control split?
It's not allowed by Apple's guidelinesIt couples app-version exposure with feature exposure, confounding the two effectsIt's slower than a normal rolloutIt costs extra in App Store Connect
3. Why are uninstalls a harder guardrail metric on mobile than, say, bounce rate is on web?
Uninstalls only happen on AndroidThere's no reliable, immediate signal that an uninstall happened or why; it's usually inferred later from silenceUninstalls always happen within the first sessionUninstalls are illegal to track under GDPR