How do you handle fan-out event processing at scale — one event triggering many downstream consumers?

#The shape of the problem

A single business event rarely has a single consequence. When an "OrderPlaced" event fires, a payments-adjacent system wants to reserve inventory, a notifications system wants to send a confirmation email, a fraud system wants to score the order in near-real-time, a data platform wants to land it in the warehouse, and a partner integration might need a webhook fired. That's five to ten independent downstream reactions to one upstream fact — and the number only grows as the business grows. The interview question "how do you handle fan-out at scale" is really asking: how do you keep adding consumers without any of them becoming coupled to each other, or to the producer?

#The core mistake: one shared queue

#Principle 1: decouple producer from consumer count

The order service that publishes OrderPlaced should never know or care how many things react to it. It publishes once, to one topic/bus. Adding a sixth consumer next quarter should require zero changes to the producer — that's the entire point of an event-driven architecture over direct service-to-service calls. Concretely: a Kafka topic that any number of consumer groups can attach to, or an SNS topic fanning out to any number of subscribed SQS queues, or an EventBridge bus with independent rules routing to independent targets.

#Principle 2: isolate failure domains

Each consumer needs its own delivery mechanism — its own Kafka consumer group with its own offsets, or its own SQS queue subscribed to the fan-out topic — specifically so that one consumer's outage cannot block another's. If the partner webhook endpoint is down for an hour, that should stall only the webhook consumer's queue (which keeps buffering and retrying), not the inventory or email consumers, which have nothing to do with it. This is the single most common fan-out mistake: teams share one queue or one consumer group across unrelated concerns to save infrastructure, and then a slow or broken downstream integration takes the whole pipeline down with it.

#Principle 3: isolate throughput and scaling

Different consumers have wildly different load and latency profiles. Fraud scoring might need sub-second reaction and scale with request volume; the analytics loader might batch every five minutes regardless of volume; the email sender is rate-limited by the email provider, not by your infrastructure. Because each consumer owns its own queue/consumer group, each can scale its own worker pool independently — the fraud consumer group can run twenty parallel instances while the email consumer runs two, without any coordination between them.

#Principle 4: per-consumer retry and dead-lettering

At-least-once delivery plus N independent consumers means each one needs its own retry policy and its own dead-letter queue. A malformed event that breaks the loyalty-points consumer's deserializer should dead-letter only on that consumer's queue after N retries — it must not block that queue's later messages, and it absolutely must not affect the inventory consumer, which may process that same event successfully. Treat each consumer's DLQ as its own alerting surface, owned by the team that owns that consumer.

#Principle 5: ordering is per-consumer, not global

Don't try to guarantee one global order across every consumer of a fanned-out event — that's rarely achievable and rarely needed. What you can and often must guarantee is order within a consumer, for a given key: the inventory consumer needs OrderPlaced processed before OrderCancelled for the same order ID, which you get by partitioning that consumer's stream by order ID (Kafka partition key, or a FIFO SQS message group ID scoped to that consumer's queue) — independent of how any other consumer is ordered or partitioned.

#Principle 6: idempotency, always

Fan-out combined with retries and consumer-group rebalancing guarantees at-least-once, not exactly-once, delivery to each consumer. Every consumer must be idempotent? against redelivery of the same event — typically by tracking processed event IDs (or a natural idempotency key) before applying an effect, not by hoping the infrastructure deduplicates for you.

Each branch below the bus is independent: a stalled or overwhelmed consumer only ever backs up its own queue/offsets, throttles its own workers, and fills its own DLQ. Nothing about that failure is visible to, or capable of blocking, any sibling branch.

🧠 Check your understanding
0/3 · 0/3 answered
1. A team fans out OrderPlaced events by having five different services all read from one shared SQS queue, each service supposed to just ignore messages it doesn't care about. What is the main problem with this design?
SQS cannot hold more than one message type in a queueThis design has no problem — it is a valid fan-out patternA message pulled by one service is not delivered to the other four, since a queue hands each message to exactly one consumerSQS queues cannot scale past five concurrent readers
2. In a well-isolated fan-out design, what should happen to the Notifications consumer's processing if the Fraud Scoring consumer's downstream service goes down for two hours?
Notifications should take over Fraud Scoring's messages to avoid data lossNothing — Notifications keeps consuming and processing normally, since each consumer group has its own offset/queue and retry policyNotifications should also pause, to keep all consumers in syncThe event bus should stop accepting new events until Fraud Scoring recovers
3. Why is per-consumer idempotency necessary even when the messaging system guarantees at-least-once delivery correctly?
At-least-once delivery means a consumer may receive and process the same event more than once, so effects must be safe to apply twiceAt-least-once delivery already guarantees each consumer processes an event exactly onceIdempotency replaces the need for a dead-letter queueIdempotency is only needed for consumers using SQS, not Kafka