Design an event-driven order processing pipeline from order placement to fulfillment

#Why this is the hardest question in the module

Every earlier lesson in this module dealt with one piece — a broker, a schema, a fan-out, a capture pipeline. This one asks you to assemble those pieces into a system that spans four services — Order, Inventory, Payment, Shipping — each with its own database, its own deploy cycle, its own team, and no shared transaction across any of them. You cannot wrap "reserve inventory, charge the card, schedule a shipment" in a single ACID transaction the way you could with four tables in one database. The interview is really testing whether you reach for the right pattern here: a saga — a sequence of local transactions, each in one service, coordinated by events, with an explicit compensating transaction defined for every step that might need to be undone.

#Choreography vs. orchestration

There are two ways to coordinate a saga, and a strong answer names both and picks one with a reason:

• Choreography — each service reacts to events from the others and emits its own events in turn; there is no central coordinator. Order publishes OrderPlaced; Inventory listens, reserves stock, publishes InventoryReserved; Payment listens to that, charges the card, publishes PaymentCharged; and so on. Every service only needs to know the shape of the events it produces and consumes — nothing else. This scales well organizationally (services stay decoupled) but the overall flow only exists implicitly, spread across N services' code — reconstructing "what state is order #4521 in right now" means correlating events across every service's logs.

• Orchestration — a dedicated saga orchestrator (a state machine, often literally implemented with something like AWS Step Functions, Temporal, or a hand-rolled state table) explicitly calls each service in sequence, tracks the saga's current state, and decides what compensating action to trigger on failure. This makes the flow's state explicit and queryable in one place, at the cost of a new component every service must integrate with, and a central piece of coordination logic that can itself become a bottleneck for new steps.

This lesson builds the pipeline choreography-style, since it's the clearer way to show the event flow itself, and calls out where an orchestrator would change the picture.

#The happy path, event by event

1. OrderPlaced — the Order service validates the request (items, addresses, prices) and, within the same local transaction that inserts the order row, writes an outbox event (see the CDC/outbox lesson in this module) so publication is guaranteed to match what was actually committed. Order status: PENDING.
2. InventoryReserved — the Inventory service consumes OrderPlaced, attempts to reserve stock for every line item in one local transaction, and on success publishes InventoryReserved. Reservation, not decrement — the stock is held, not yet permanently deducted, until the order is confirmed.
3. PaymentCharged — the Payment service consumes InventoryReserved, charges the customer's payment method (via a payment gateway, itself idempotent? per attempt), and publishes PaymentCharged on success.
4. ShippingScheduled — the Shipping service consumes PaymentCharged, books a carrier pickup/label, and publishes ShippingScheduled.
5. OrderFulfilled — the Order service consumes ShippingScheduled and flips the order to its terminal success state, which is what finally makes the order visible to the customer as confirmed end-to-end.

#Where it goes wrong — and the compensations that undo it

This is the part most candidates skip, and the part that actually demonstrates saga understanding: every step that changed state needs an explicit, opposite action, because there is no database engine that will roll it back for you.

Notice the compensations run in reverse order of the forward steps that already succeeded — this is the general saga rule: to undo step N, you compensate N, then N-1, then N-2, walking backward only through steps that actually committed.

#One compensation path in detail

#The full flow, happy path and compensations together

#Design details that separate a working saga from a fragile one

Correlation and ordering. Every event in the saga carries the order ID, which is used as the partition key (Kafka) or message group ID (SQS FIFO) for every topic involved. That guarantees all events for one order are processed in order within each service, without needing any global ordering across services — Inventory processing order #4521's events in sequence is all that matters, independent of order #9987 running through the same pipeline concurrently on a different partition.

Idempotency at every step. At-least-once delivery plus retries means Payment might see InventoryReserved for the same order twice. Every consumer keys its side effect (charge, reservation, refund) on an idempotency key derived from the order ID and step name, so a redelivered event is a safe no-op rather than a double charge.

The outbox pattern at every service, not just Order. Each service that both mutates its own state and publishes the next event in the chain has the same dual-write risk Order does. Inventory reserving stock and publishing InventoryReserved needs the same local-transaction-plus-outbox discipline, or you reintroduce the exact inconsistency the saga pattern exists to avoid, just one hop downstream.

Timeouts are a first-class failure mode. A saga step that never emits its expected next event — because a service crashed mid-processing, or a downstream dependency is wedged — needs an explicit timeout that triggers the same compensating chain as a failure event. Without it, an order can sit in PENDING forever with no automatic path to resolution.

Dead-letter queues per consumer, with alerting. A malformed or perpetually-failing event on any one step's queue should dead-letter after bounded retries rather than blocking that queue indefinitely — and someone needs to be paged when it does, because a stuck saga step is a stuck order, and stuck orders are refund requests waiting to happen.

Exposing eventual consistency honestly. The order is not "confirmed" the instant OrderPlaced is written — it passes through real intermediate states (PENDING, potentially REJECTED/CANCELLED) that can last anywhere from milliseconds to (during an incident) minutes. The UI needs to represent that honestly — a spinner or a "processing" state — rather than optimistically showing success before the saga has actually completed.

#If you'd used an orchestrator instead

Swapping to orchestration doesn't change any of the events above — it changes who decides what happens next. A saga orchestrator would consume OrderPlaced itself, explicitly call (or emit a command to) Inventory, wait for InventoryReserved or a timeout, then explicitly proceed to Payment, and so on — holding the saga's current step and history in its own persisted state. The compensating chain becomes orchestrator logic ("on payment failure, call Inventory's release endpoint, then mark order cancelled") rather than something you have to reconstruct by reading five services' consumers. The trade you make: every service now needs to expose the specific commands/APIs the orchestrator calls, and the orchestrator itself needs to be highly available, since it's now on the critical path for every order.