Design a notification system that fans out a single event to push, email, and SMS
One "order shipped" event needs to become three very different downstream calls — each with its own provider, its own failure mode, and its own idea of an acceptable retry.
LEARN BY DOINGPut your intuition to work3 steps
Design a notification system that fans out a single event to push, email, and SMS
Use the lesson’s examples and explanation to check your idea.
Personal reflection. Nothing is sent to an AI service.#One event, three completely different delivery problems
"Fan out a notification to push, email, and SMS" sounds like a routing problem, but the interesting part is that push, email, and SMS aren't three flavors of the same thing — they're three systems with different cost structures, different failure modes, and different regulatory obligations. Push is nearly free and can be retried aggressively. SMS costs real money per message and some providers rate-limit hard. Email sits in between, but a mistake there can get your sending domain blacklisted. Treating all three as "just send a message to a queue" is where naive designs fall apart.
Requirements worth pinning down first:
- A single business event (order shipped, password reset, promotional blast) can fan out to zero, one, or several channels depending on user preferences (opted into push but not SMS, for example).
- Idempotency — the same event must never be delivered twice on the same channel, even if the producer retries or the pipeline redelivers.
- Each channel needs independent retry and failure handling — a stuck SMS provider shouldn't back up email delivery.
- Delivery tracking — product and support teams need to know, per user per channel, whether a notification was sent, delivered, bounced, or opted out.
- Compliance — email carries CAN-SPAM obligations (unsubscribe links, sender identification), SMS carries carrier and TCPA-style consent obligations; opt-outs must be enforced before send, not logged after the fact.
#Architecture
The notification service is the single entry point: it receives a business event, looks up the target user's preferences and quiet-hours window, checks the idempotency store, and hands off to a fan-out router that publishes one message per eligible channel onto that channel's own queue. From there, each channel is an independent pipeline with its own worker pool, its own provider integration, and its own dead-letter queue — a completely isolated blast radius per channel.
#Idempotency and deduplication
Every message carries a deterministic idempotency key — typically hash(event_id, user_id, channel) — checked against the idempotency store (a Redis or DynamoDB table with a TTL a little longer than your max possible redelivery window) before a worker calls its provider. This matters because the whole pipeline is built on at-least-once delivery: the event bus can redeliver, a worker can crash after sending but before acking, and a retry from a DLQ replay must not become a second email in the user's inbox. The idempotency check, not "we don't expect duplicates," is what makes at-least-once safe to build on.
#Per-channel workers and retry policy
Each channel warrants a genuinely different retry policy, because the cost and urgency of failure differ:
| Channel | Typical retry behavior | Why |
|---|---|---|
| Push | Aggressive — short backoff, several attempts over minutes | Essentially free; a missed push is low-stakes and often superseded by a fresher one anyway |
| Moderate — exponential backoff over hours, then DLQ | Transient SMTP/provider errors are common; a delayed email is usually fine, a duplicate is embarrassing | |
| SMS | Conservative — few attempts, short window, then DLQ | Costs real money per attempt and per-number rate limits are strict; a stale SMS ("your ride arrived" an hour late) can be actively wrong |
A message that exhausts its retries lands in that channel's dead-letter queue rather than blocking the queue behind it — a poison message for one user must never stall delivery to everyone else. DLQ entries feed alerting and, for high-value notifications (password resets, security alerts), a manual or automated re-drive path.
#Delivery tracking and the feedback loop
Each worker writes a status record (queued -> sent -> delivered/bounced/failed) to a shared delivery status store, keyed by event, user, and channel. This does double duty: it powers customer-support-facing "did my user get this?" queries, and it closes a feedback loop back into the preference store — an email provider's bounce webhook, for instance, should be able to automatically suppress future sends to a permanently-invalid address, and an SMS carrier's "STOP" reply must immediately flip that user's SMS preference to opted-out before the next send is even attempted, not just logged for later cleanup.