What makes a message consumer idempotent, and why does at-least-once delivery force you to design for it?

#Why 'at-least-once' delivery is the default

Every message broker has to pick a trade-off around delivery guarantees, and the honest answer is that 'exactly-once delivery' barely exists in practice. Brokers like Kafka, RabbitMQ, and SQS all default to at-least-once: a message is redelivered until the consumer acknowledges it, because the alternative -- acknowledging before you're sure the work happened -- risks silently losing messages. The moment you accept at-least-once delivery, duplicate processing stops being an edge case and becomes a certainty you have to design around.

Redelivery happens for mundane, unavoidable reasons: a consumer crashes after committing its database write but before sending the ack; a network blip drops the ack on its way back to the broker; a consumer group rebalances mid-batch and hands the same partition offset to a different instance; a visibility timeout expires on a slow SQS consumer even though the work actually finished. None of these require a bug -- they are the normal cost of running distributed systems over an unreliable network.

#What idempotency actually means for a consumer

An idempotent? consumer produces the same end state whether a message is processed once or ten times. That is a property of the operation, not of the message format -- some operations are idempotent for free, and others need deliberate engineering to become safe under redelivery.

#The dedup table pattern

When the operation itself is not naturally idempotent -- the common case, since most interesting business logic mutates state relative to what came before -- you need an explicit deduplication boundary. The most reliable version of this is a table that records which message ids have already been processed, guarded by a unique constraint, written in the same database transaction as the business effect.

import psycopg2
from psycopg2 import errors

def handle_message(conn, message):
    message_id = message['id']
    cur = conn.cursor()
    try:
        # The unique constraint on message_id is the entire mechanism:
        # a duplicate delivery fails this insert instead of running twice.
        cur.execute(
            'INSERT INTO processed_messages (message_id, processed_at) '
            'VALUES (%s, now())',
            (message_id,)
        )
    except errors.UniqueViolation:
        conn.rollback()
        return  # duplicate delivery: already handled, safe no-op

    # Business effect and the dedup marker commit as one atomic unit.
    cur.execute(
        'UPDATE accounts SET balance = balance + %s WHERE id = %s',
        (message['amount'], message['account_id'])
    )
    conn.commit()
    cur.close()

The unique constraint is what actually enforces the guarantee -- not an application-level 'have I seen this id before' check, which races against itself under concurrent consumers. Two instances of this consumer can pull the same redelivered message at the same time; only one of them wins the insert, and the other rolls back cleanly.

#Where to put the idempotency boundary

The dedup table is the strongest option but not the only one, and picking the right boundary depends on what you're protecting.

🧠 Check your understanding
0/3 · 0/3 answered
1. A consumer processes a message and updates the database, but crashes before acknowledging the broker. What happens next under at-least-once delivery?
The broker blocks until the original consumer restartsThe message moves to a dead-letter queue immediatelyThe broker redelivers the message to a consumerThe broker discards the message since it was already processed
2. Why should the dedup-table insert and the business update happen in the same database transaction?
It makes the SQL shorterIt lets the database run them in parallel for speedIf either fails, both roll back together, so you never end up with a recorded message and no effect, or an effect with no recordIt is required by the SQL standard for INSERT statements
3. Which operation is naturally idempotent without any extra bookkeeping?
send a confirmation emailappend a line to a log fileset a status field to a fixed valueincrement balance by 10