How do distributed transactions work across microservices — walk through the saga pattern, orchestration vs. choreography

#Why "Just Use a Transaction" Stops Working

A relational transaction gives you atomicity because one engine controls the log, the locks, and the commit point. The moment "the order" and "the payment" and "the inventory count" live in three different databases owned by three different services, there is no single log to write a commit record into. You can't BEGIN/COMMIT across a network boundary between services you don't jointly control — and even where you technically could (two-phase commit), it doesn't scale: it holds locks across a network round trip and blocks indefinitely if the coordinator dies mid-protocol (the companion lesson on 2PC covers exactly why). So distributed systems trade atomicity for eventual consistency achieved through compensation, and the pattern that formalizes this is the saga.

#What a saga actually is

A saga is a sequence of local transactions, T1 through Tn, where each Ti commits fully inside a single service (using that service's normal ACID transaction against its own database) and then triggers the next step. If every Ti succeeds, the saga completes and the system ends up consistent. If some Tk fails, the saga does not roll back like a database transaction — instead it runs compensating transactions C(k-1), C(k-2), ... C1 for every step that already committed, in reverse order. A compensating transaction is a new, forward-moving business operation that semantically undoes the effect of an earlier one — "issue a refund," not "delete the payment row" — because the original transaction is already durably committed and other transactions may have observed it.

#Orchestration vs. choreography

Orchestration puts a single coordinator in charge: it knows the whole saga's steps up front, calls each participant's local transaction explicitly (usually via a command, e.g., an HTTP call or a command message), waits for the result, and — on failure — calls the compensating action on every already-completed participant, in reverse order. The workflow logic lives in exactly one place, which makes it easy to read, test, and monitor ("where is saga #4471 stuck?" has one obvious place to look), at the cost of that orchestrator being a new piece of infrastructure that has to know about every participant's API.

Choreography has no coordinator. Each service reacts to events published by others: it does its local transaction, then publishes a domain event describing what happened; other services subscribe to the events they care about and decide, independently, whether to run their own next step or their own compensation. This avoids the central coupling of orchestration and lets services be added without modifying a central workflow, but the overall business process is no longer written down anywhere — it's the emergent result of N services' independent subscriptions, which makes it materially harder to answer "what happens, in order, when a payment fails?" without drawing the diagram yourself, and makes cyclic event dependencies a real risk if two services end up implicitly depending on each other's reactions.

#The concrete example we'll implement

An order-placement saga across four services:

1. OrderService creates the order in PENDING.
2. PaymentService charges the customer's card.
3. InventoryService reserves stock for the line items.
4. ShippingService schedules the shipment and marks the order CONFIRMED.

If inventory reservation fails (say, a race condition sold the last unit), the saga must compensate: refund the payment and cancel the order. Shipping is never reached, so it needs no compensation.

#An orchestrator, in code

The orchestrator below models each step as a pair of async functions — execute and compensate — and runs them in order, unwinding on the first failure. This is the shape underlying real saga frameworks (Temporal workflows, AWS Step Functions, Axon Framework's saga manager, or a hand-rolled version like this one).

Wrap long lines
class SagaStep {
  constructor(name, execute, compensate) {
    this.name = name;
    this.execute = execute;
    this.compensate = compensate;
  }
}

class SagaOrchestrator {
  constructor(steps) {
    this.steps = steps;
  }

  async run(context) {
    const completed = [];

    for (const step of this.steps) {
      try {
        context[step.name] = await step.execute(context);
        completed.push(step);
      } catch (err) {
        console.error('Saga step failed: ' + step.name + ' -> ' + err.message);
        await this.compensate(completed, context);
        throw new Error('Saga aborted at step ' + step.name);
      }
    }

    return context;
  }

  async compensate(completedSteps, context) {
    for (let i = completedSteps.length - 1; i >= 0; i--) {
      const step = completedSteps[i];
      try {
        await step.compensate(context);
      } catch (compErr) {
        // A failed compensation needs its own alerting/retry path --
        // silently swallowing it would leave the system inconsistent.
        console.error('Compensation FAILED for step ' + step.name + ': ' + compErr.message);
      }
    }
  }
}

const placeOrderSaga = new SagaOrchestrator([
  new SagaStep(
    'order',
    async (ctx) => orderService.create(ctx.customerId, ctx.items),
    async (ctx) => orderService.cancel(ctx.order.orderId)
  ),
  new SagaStep(
    'payment',
    async (ctx) => paymentService.charge(ctx.order.orderId, ctx.amount),
    async (ctx) => paymentService.refund(ctx.payment.paymentId)
  ),
  new SagaStep(
    'inventory',
    async (ctx) => inventoryService.reserve(ctx.order.orderId, ctx.items),
    async (ctx) => inventoryService.release(ctx.order.orderId, ctx.items)
  ),
  new SagaStep(
    'shipment',
    async (ctx) => shippingService.schedule(ctx.order.orderId),
    async (ctx) => shippingService.cancelSchedule(ctx.order.orderId)
  ),
]);

// placeOrderSaga.run({ customerId: 'c-1', items: [...], amount: 4999 })
//   throws and leaves the system consistent if inventory reservation fails,
//   because payment and order are compensated in reverse order.

#The same failure, choreographed

Without a coordinator, InventoryService listens for the event that means "it's my turn" and publishes an event describing its own outcome; PaymentService listens for the failure event to compensate itself:

// InventoryService: reacts to PaymentCompleted, publishes its own outcome
fun onPaymentCompleted(event: PaymentCompletedEvent) {
    val reserved = inventoryRepository.tryReserve(event.orderId, event.items)
    if (reserved) {
        eventBus.publish(InventoryReservedEvent(event.orderId))
    } else {
        eventBus.publish(InventoryReservationFailedEvent(event.orderId, event.paymentId))
    }
}

// PaymentService: reacts to the failure by compensating itself --
// nobody told it to; it decided to, based on an event it subscribed to.
fun onInventoryReservationFailed(event: InventoryReservationFailedEvent) {
    paymentRepository.refund(event.paymentId)
    eventBus.publish(PaymentRefundedEvent(event.orderId))
}

Notice what's missing compared to the orchestrator: nothing in this file knows the saga has four steps, or what order they run in, or that OrderService also needs to hear about the failure to cancel itself (that's a third handler, elsewhere, subscribed to the same event). That knowledge is now implicit in the sum of all the event subscriptions across the codebase — which is exactly the tradeoff choreography makes.

#Idempotency is not optional

Message brokers backing sagas generally guarantee at-least-once delivery, so every handler above can be called twice for the same event (a redelivered message after a slow ack, a retried HTTP command after a timeout that actually succeeded server-side). If chargeCard isn't idempotent?, a retried command double-charges the customer. The standard fix is an idempotency key: the caller attaches a unique id (often the saga instance id + step name) and the handler stores processed ids, short-circuiting a repeat before it touches money or inventory again.

:::key Compensating transaction, precisely Not a rollback. A compensating transaction is a new, independently committed operation that semantically reverses an already-committed step's business effect — issuing a refund, releasing a reservation, cancelling a shipment — because the original write is durable and may already be visible to other transactions. :::

:::warning At-least-once delivery means your handlers WILL see duplicates Every saga participant must be idempotent under retried commands and redelivered events, or the saga's failure-recovery path becomes a second source of bugs on top of the failure it was meant to handle. :::

:::analogy Think "undo stack," not "rollback" A database rollback erases what never should have counted. A saga's compensation is closer to an undo stack in an editor: the original action really happened and was visible, so undoing it is itself a new, recorded action — which is why compensations must be designed per business operation, not generated automatically. :::

🧠 Check your understanding
0/3 · 0/3 answered
1. In the order-placement saga, InventoryService fails to reserve stock after PaymentService already charged the customer. What should happen next in an orchestrated saga?
ShippingService is compensated even though it was never calledThe saga retries InventoryService.reserve indefinitely until it succeedsThe orchestrator calls a compensating action (e.g., refund) on PaymentService, then compensates OrderService, in reverse completion orderThe orchestrator rolls back the payment database transaction directly
2. What is the main practical downside of choreography compared to orchestration?
It cannot handle failures at allIt requires all services to share one databaseThe overall business process isn't written down in one place, making it harder to trace and reason aboutIt can only be used with synchronous HTTP calls
3. Why must every saga participant's command/event handler be idempotent?
It's required by the SQL standard for distributed transactionsIdempotent handlers run faster than non-idempotent onesMessage delivery is typically at-least-once, so retries and redeliveries can invoke the same handler more than once for the same logical stepIdempotency improves compression of event payloads