What is a dead letter queue, and how do you handle a 'poison message' that keeps failing?

#The dead letter queue is where messages go to be looked at, not to disappear

A dead letter queue (DLQ) is a separate queue that holds messages a consumer could not successfully process after a bounded number of attempts, instead of retrying them forever or silently dropping them. It exists because two bad outcomes are both unacceptable: retrying a message infinitely wastes consumer capacity and, in a queue where processing order matters, can block every message behind it (head-of-line blocking) while the broken one keeps getting redelivered; dropping it after one failure risks losing something that mattered, with zero trace of why it failed.

A poison message is the specific case that makes a DLQ necessary: a message that will never succeed no matter how many times you retry it, because the failure is deterministic — malformed JSON, a required field missing, a foreign key referencing an order that got deleted, a bug that a particular payload shape triggers every time. Retrying a poison message is different from retrying a transient failure (a downstream service timeout, a momentary DB connection blip) where a retry has a real chance of succeeding on attempt two or three. The DLQ pattern exists to give transient failures their retries while cutting poison messages off before they do damage.

#Tracking retry count is the load-bearing detail

None of this works without a way to know, for a given message, how many times it's already been attempted — that's what decides whether the next failure gets a retry or a one-way trip to the DLQ. Brokers handle this differently:

• SQS tracks it for you automatically via the ApproximateReceiveCount message attribute, and a redrive policy on the queue moves a message to its configured DLQ once that count crosses a threshold — no application code required, though you can still read the count yourself for custom logic.
• RabbitMQ has no built-in retry counter, but a dead-letter-exchange (DLX) combined with a per-message TTL gives you a delayed-retry queue, and you track the attempt count yourself in a message header (x-retry-count), incrementing it on every republish.
• Kafka has no native DLQ concept at all — there's no per-message ack/nack, only offset commits — so the pattern is built entirely in application code: a retry topic (or several, often with increasing delay) and a dead-letter topic, with the retry count carried as a record header.

Here's the RabbitMQ version end to end, including the retry-count header and the routing decision:

Wrap long lines
// consumer.js — RabbitMQ consumer with retry-count tracking and DLQ routing
const amqp = require("amqplib");

const MAIN_QUEUE = "orders.process";
const RETRY_QUEUE = "orders.retry"; // messages sit here for a TTL, then bounce back to MAIN_QUEUE
const DLQ = "orders.dlq";
const MAX_RETRIES = 3;
const RETRY_DELAY_MS = 5000;

async function start() {
  const connection = await amqp.connect("amqp://localhost");
  const channel = await connection.createChannel();

  // Nothing ever consumes from RETRY_QUEUE directly — messages just wait out
  // their TTL, then RabbitMQ dead-letters them straight back to MAIN_QUEUE.
  await channel.assertQueue(RETRY_QUEUE, {
    durable: true,
    arguments: {
      "x-dead-letter-exchange": "",
      "x-dead-letter-routing-key": MAIN_QUEUE,
      "x-message-ttl": RETRY_DELAY_MS,
    },
  });
  await channel.assertQueue(MAIN_QUEUE, { durable: true });
  await channel.assertQueue(DLQ, { durable: true });
  channel.prefetch(1);

  channel.consume(MAIN_QUEUE, async (msg) => {
    if (!msg) return;
    const retryCount = msg.properties.headers["x-retry-count"] || 0;

    try {
      await processOrder(JSON.parse(msg.content.toString()));
      channel.ack(msg);
    } catch (err) {
      if (isPoisonMessage(err) || retryCount >= MAX_RETRIES) {
        channel.sendToQueue(DLQ, msg.content, {
          persistent: true,
          headers: {
            "x-retry-count": retryCount,
            "x-failure-reason": err.message,
            "x-original-queue": MAIN_QUEUE,
          },
        });
        channel.ack(msg); // done with it on the main queue — it's been handled
        console.error(`order moved to DLQ after ${retryCount} attempts: ${err.message}`);
      } else {
        channel.sendToQueue(RETRY_QUEUE, msg.content, {
          persistent: true,
          headers: { "x-retry-count": retryCount + 1 },
        });
        channel.ack(msg); // remove from main queue; the retry queue owns it now
      }
    }
  });
}

function isPoisonMessage(err) {
  // Deterministic failures will never succeed on retry — skip straight to the DLQ.
  return err instanceof SyntaxError || err.code === "INVALID_PAYLOAD";
}

async function processOrder(order) {
  // real processing — can throw on bad input or a downstream outage
}

start();

Two details matter more than they look: the code always acks the original message whether it retries or dead-letters it — the ack removes it from MAIN_QUEUE, and a new copy takes over in RETRY_QUEUE or DLQ. Forgetting that ack leaves the original stuck redelivering forever alongside the copy you just created. And isPoisonMessage short-circuits straight to the DLQ for errors that are clearly deterministic (bad JSON, a validation failure) rather than burning three retry cycles on something no retry could ever fix — classifying errors as transient vs. permanent is what keeps the retry queue from becoming a slow, expensive way to reach the same DLQ.

#A DLQ is only useful if someone is watching it

The most common way teams get this wrong isn't the routing logic — it's what happens after a message lands in the DLQ. A queue with no alerting on its depth silently accumulates failed orders, failed payments, failed webhook deliveries for weeks until someone notices revenue doesn't add up. Treat DLQ depth like an error-rate metric: alert on it, review it on a cadence, and build (or buy) a replay tool that lets someone inspect a dead-lettered message, fix the underlying bug or bad data, and safely republish it to the main queue — "safely" meaning your consumer's processing is idempotent?, because a replayed message is functionally a redelivery, and you're back to the at-least-once duplicate-handling problem regardless of how it got redelivered.

🧠 Check your understanding
0/2 · 0/2 answered
1. A consumer keeps failing to process a message because of a permanent bug in parsing malformed JSON. What's the best design response?
Drop the message silently to keep the queue movingRetry it indefinitely so it's eventually processed once the bug is fixedDetect that the failure is deterministic and route it to the DLQ immediately, without burning the normal retry budgetIncrease the consumer's prefetch count
2. Why must a RabbitMQ consumer ack the original message even when it's routing that message to a retry queue or DLQ instead of processing it successfully?
RabbitMQ automatically discards unacked messages after 60 secondsAcking is only required after successful processingAcking increases the message's priorityWithout the ack, the original message stays on the main queue and gets redelivered, duplicating the copy just sent to the retry queue/DLQ