Design a feature-flag / remote-config system for a mobile app with millions of users

#Clarify the requirements first

Before drawing boxes, a senior candidate pins down why the flag system exists, because that answer decides the whole architecture. There are at least three distinct use cases hiding under "feature flags": operational kill switches (turn off a broken payment flow in seconds, no app-store review), gradual rollouts (ship a feature to 1% of users, watch crash-free rate, ramp to 100%), and experimentation (A/B test two checkout flows and measure conversion). Each has a different latency requirement — a kill switch needs to propagate in seconds, a rollout percentage can tolerate minutes, and an experiment just needs consistency per user, not speed. I'd also ask about scale (tens of millions of daily active devices), targeting granularity (app version, OS, locale, custom user attributes), and — critically — what happens when the device has no network at launch, because that last case is the one most designs get wrong.

#High-level architecture

The system splits cleanly into a control plane and a data plane. The control plane is where product and engineering define flags, targeting rules, and rollout percentages through a dashboard; it writes an immutable, versioned configuration document. The data plane is what actually reaches millions of devices, and it must never be "call an API server directly" — at that scale you want a CDN or edge cache in front of a small number of static, versioned JSON payloads, not a database absorbing millions of reads per minute.

#The client SDK: local-first by design

The SDK's contract with the app is simple: reads must be synchronous, non-blocking, and always return something, even at cold start with no network. That means the last successfully fetched config is persisted to disk, and app launch reads from that persisted snapshot before any network call resolves. A background refresh then updates the snapshot for next time. This is the same "last-known-good" pattern you'd use for any cache that sits between the app and a flaky network — never block the UI thread on a config fetch.

Wrap long lines
interface FeatureFlagClient {
    fun isEnabled(key: String, default: Boolean = false): Boolean
    fun <T> getVariant(key: String, default: T): T
    fun refresh(force: Boolean = false)
}

class RemoteConfigClient(
    private val store: FlagStore,          // persisted last-known-good payload
    private val fetcher: ConfigFetcher,    // CDN-backed HTTP client, supports ETag
    private val context: EvaluationContext // stable deviceId, userId, appVersion, locale
) : FeatureFlagClient {

    @Volatile private var snapshot: ConfigSnapshot = store.loadCached() ?: ConfigSnapshot.empty()

    override fun isEnabled(key: String, default: Boolean): Boolean {
        val rule = snapshot.flags[key] ?: return default
        return evaluate(rule) as? Boolean ?: default
    }

    override fun <T> getVariant(key: String, default: T): T {
        val rule = snapshot.flags[key] ?: return default
        @Suppress("UNCHECKED_CAST")
        return evaluate(rule) as? T ?: default
    }

    private fun evaluate(rule: FlagRule): Any? {
        if (!rule.matchesTargeting(context)) return rule.defaultValue
        val bucket = stableBucket(context.stableId, rule.key) // 0..9999
        val variant = rule.rollout.firstOrNull { bucket < it.cumulativeUpperBound }
        return variant?.value ?: rule.defaultValue
    }

    override fun refresh(force: Boolean) {
        fetcher.fetch(etag = if (force) null else snapshot.etag) { result ->
            when (result) {
                is FetchResult.NotModified -> {} // TTL renewed, nothing changed
                is FetchResult.Updated -> { snapshot = result.snapshot; store.persist(snapshot) }
                is FetchResult.Failed -> {} // keep serving last-known-good, retry later
            }
        }
    }
}

// Deterministic: same device + same flag key always lands in the same bucket
fun stableBucket(stableId: String, flagKey: String): Int {
    val hash = murmur3_32("$stableId:$flagKey".toByteArray())
    return (hash.toUInt() % 10_000u).toInt()
}

#The kill-switch problem: CDN latency is a feature, until it isn't

CDN caching with a 1-5 minute TTL is exactly right for a rollout percentage — it trades a little staleness for enormous scalability. It is wrong for an emergency kill switch, where "wait up to five minutes" can mean five minutes of a broken checkout flow charging cards twice. The fix is to decouple a small set of high-priority flags from the bulk config's TTL: a lightweight push channel (a silent APNs/FCM notification) tells the app "something changed, refetch now," bypassing the cache wait entirely. This is the one part of the design interviewers probe hardest, because it's the part that's easy to hand-wave as "just lower the TTL," which doesn't scale to millions of clients polling every few seconds.

#Data model, versioning, and rollback

Each flag definition carries a key, a type, a default value, an ordered list of targeting rules, and a rollout waterfall (cumulative percentage bands mapping to variants). Because the published payload is immutable and versioned, rollback is not "redeploy" — it's re-pointing the CDN alias at the previous version, which is fast and safe precisely because nothing mutates in place. I'd also keep environments (staging/prod) fully separate and log an audit trail of who changed what, since a misconfigured flag is effectively a production incident.

#Security, exposure logging, and the fast path for QA

Flags evaluated on-device are visible to anyone who inspects the app's traffic, so the config payload must never carry secrets — only behavior toggles and non-sensitive values. For experimentation to produce valid statistics, the SDK must log an exposure event at the moment a flag is actually evaluated in a user's session, not merely when the config was fetched; conflating "fetched" with "exposed" is a subtle bug that quietly corrupts every A/B test built on top of it. Finally, give engineers and QA a local override panel so they can force any variant without waiting on a real rollout — most production incidents I've seen with flag systems come from nobody being able to test the "off" state before it shipped.