Design a payment processing system — walk through the architecture and where idempotency and consistency matter most

#The One Rule Everything Else Serves

A payment system's primary correctness property is not "fast" or "scalable" — it's this: never charge twice for one intent, and never lose track of money that moved. Every architectural decision below — the idempotency? keys, the state machine, the outbox pattern, the nightly reconciliation — exists to serve that one rule under conditions where networks are unreliable and a "timeout" from an external gateway is fundamentally ambiguous: did the charge happen or not?

#High-Level Architecture

The core pieces: a Payment Service that orchestrates the flow and owns the payment's lifecycle; a Ledger — the durable, append-only, double-entry record of every movement of money, and the actual source of truth (the payment service's cached "status" is a projection of the ledger, not the other way around); an external Payment Gateway (Stripe, Adyen, a card network processor) that you don't control and must treat as unreliable; a webhook receiver for the gateway's asynchronous confirmations; an event bus / outbox for publishing internal events (order confirmed, refund issued) without a dual-write race; and a reconciliation job that periodically diffs your ledger against the gateway's settlement reports and flags anything that doesn't match.

#The Payment State Machine

A payment is not a boolean. Model it explicitly:

Every transition is a row appended to the ledger, not an in-place update — you want the full history, not just the current status, because disputes and audits ask "what happened, in what order, and why."

#The Flagship Flow: Idempotency, Timeout Handling, Webhooks, Reconciliation

#Where Idempotency Actually Has to Live

Idempotency isn't one mechanism — it has to exist at every hop that can be retried:

• Client → Payment Service: the client generates the idempotency key once (typically per checkout attempt) and resends the exact same key on every retry. A UNIQUE constraint on idempotency_key in the database is what actually enforces this under concurrency — two simultaneous requests with the same key will have one succeed the INSERT and one fail it, and the failing one simply looks up and returns the winner's result instead of erroring.
• Payment Service → Gateway: your own retries of a timed-out gateway call must reuse the same idempotency key, and the gateway is contractually obligated to dedup on it (Stripe, for example, holds idempotency keys for 24 hours). Without this, a network timeout on your side — where the charge actually succeeded but the response never arrived — turns into a double charge the moment you retry naively.
• Gateway → Payment Service (webhooks): webhook delivery is at-least-once, never exactly-once. The same charge.captured event can and will arrive twice. Handling it means checking current state before applying the transition (already CAPTURED? → 200 OK, no-op) rather than assuming every webhook delivery is novel.

#The Outbox Pattern: Why You Can't Just "Write to the DB, Then Publish an Event"

If the payment service updates its own database and then separately publishes an event to a queue, there's a window where the DB commit succeeds and the process crashes before the publish — the event is lost, and nothing downstream (order confirmation, inventory release, notifications) ever fires, even though the money moved. The outbox pattern fixes this by writing the event into an outbox table in the same database transaction as the state change, and a separate relay process tails that table and publishes to the event bus, marking rows as sent. The publish can be retried freely because it's now decoupled from the original transaction — you've traded a dual-write race for an at-least-once relay, which is a problem event consumers can handle by being idempotent themselves.

#Reconciliation Is the Safety Net, Not the Plan

Even with idempotency keys everywhere, treat a nightly (or hourly, for high volume) reconciliation job as mandatory, not optional. It fetches the gateway's settlement report and diffs it line-by-line against the internal ledger: a charge the gateway shows as captured that your ledger has stuck in PENDING means a webhook was missed or a retry logic bug exists; a charge in your ledger with no matching gateway settlement means the opposite. Reconciliation doesn't prevent inconsistency — the real-time idempotency machinery is what does that — it's the mechanism that guarantees inconsistency, when it inevitably slips through some edge case, gets caught within hours instead of surfacing as a customer complaint weeks later.