What causes consumer lag in a message queue, and how do you handle backpressure when consumers can't keep up?

Consumer lag is the distance, measured in offsets, between the latest message written to a partition (the log end offset) and the last offset a consumer group has actually committed as processed. A lag of zero means the consumer is caught up in real time; a lag of 50,000 means there are 50,000 unprocessed messages sitting in that partition waiting for this consumer group. Lag is tracked per partition, and a healthy-looking average can hide one badly stuck partition — which is exactly the failure mode interviewers like to probe.

#What actually causes lag

Lag grows whenever a consumer group's processing rate falls below the topic's production rate, for any of several distinct reasons:

A skewed partition key makes several of these worse: if one key (say, one very active customerId) accounts for a disproportionate share of traffic, its partition can lag badly even while every other partition looks perfectly healthy.

#Monitoring lag

Kafka exposes committed offsets and log-end offsets directly, so lag is computable without any extra infrastructure. The quickest check is the CLI:

kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --describe --group billing-service

GROUP            TOPIC          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
billing-service  order-events   0          128934          129500          566
billing-service  order-events   1          130220          130221          1
billing-service  order-events   2          125001          131890          6889

Partition 2 here is the one worth paging someone about — its lag is orders of magnitude higher than its siblings, which points at a stuck consumer thread, a poison-pill message, or a hot key rather than an overall capacity problem. In production this is usually wired into a Prometheus exporter or a tool like Burrow and alerted on the rate of change of lag, not just its absolute value — a steady lag of 10,000 that isn't growing is a capacity question you can plan for; a lag climbing by 1,000/minute is an incident.

#Applying backpressure

Backpressure means slowing down (or explicitly pausing) intake once your ability to process falls behind your ability to receive — the alternative is an unbounded internal buffer that eventually runs the consumer out of memory. A few concrete techniques, usually combined:

• Bounded prefetch. Configure max.poll.records (Kafka) or a channel's prefetch count (RabbitMQ) so a single poll never pulls more than you can hold and process safely — rather than accepting an unbounded batch and building an in-memory backlog yourself.
• Explicit pause/resume. When an internal queue in front of a worker pool fills up, call consumer.pause() on the affected partitions so the client stops fetching more records for them (while still sending heartbeats to stay in the group), and consumer.resume() once the backlog drains below a threshold.
• Bounded internal queues. Size the queue between the poll loop and the worker pool explicitly, and make offering to it block or fail fast rather than growing unbounded.
• Scale consumers up to partition count. Adding a consumer beyond the partition count does nothing (see the Kafka architecture question) — if you need more throughput than the partition count allows, you have to grow partitions, accepting the ordering trade-off that comes with it.
• Batch downstream calls. Batching database writes or HTTP calls inside the handler (rather than one round trip per message) often raises effective throughput enough to eliminate lag without adding any infrastructure.
• Circuit-break the downstream dependency. If lag is caused by a slow or failing external system, keep hammering it and you make things worse; a circuit breaker that fails fast and retries later protects both the consumer and the downstream service.

Wrap long lines
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "billing-service");
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 50);   // bounded prefetch per poll
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("order-events"));

BlockingQueue<ConsumerRecord<String, String>> inFlight = new ArrayBlockingQueue<>(500);
Set<TopicPartition> paused = ConcurrentHashMap.newKeySet();

while (running) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));

    for (ConsumerRecord<String, String> record : records) {
        // Backpressure: if the downstream worker pool can't keep up, pause the
        // partition instead of piling records into unbounded memory.
        boolean accepted = inFlight.offer(record, 200, TimeUnit.MILLISECONDS);
        if (!accepted) {
            TopicPartition tp = new TopicPartition(record.topic(), record.partition());
            if (paused.add(tp)) {
                consumer.pause(List.of(tp));
                log.warn("Pausing {} - internal queue full, falling behind", tp);
            }
        }
    }

    if (!paused.isEmpty() && inFlight.remainingCapacity() > 250) {
        consumer.resume(paused);
        log.info("Resuming {} partitions - backlog drained", paused.size());
        paused.clear();
    }

    consumer.commitSync(currentOffsets(records)); // only commit what workers confirmed
}

Note that pause()/resume() operate on partitions the consumer already owns — they stop the client from fetching new records for that partition without leaving the consumer group, which keeps rebalances (and the disruption they cause every other partition) out of the picture entirely.

🧠 Check your understanding
0/3 · 0/3 answered
1. A consumer group looks healthy overall (low average lag), but one partition out of eight has a lag of 500,000 and climbing while the rest sit near zero. What's the most likely cause?
A poison-pill message or a hot key is stuck on that one partition, stalling its consumer while the others process normallyKafka is dropping messages from that partitionThe topic needs more replicasThe whole cluster is under-provisioned
2. Why is calling consumer.pause() on a partition preferable to blocking inside the poll loop while waiting for a slow downstream call?
Blocking the loop risks missing max.poll.interval.ms, which gets the consumer kicked from the group and triggers a rebalance affecting every partitionpause() deletes the backlog so it never has to be processedThere's no difference between the two approachespause() increases the partition count automatically
3. You need more consumer throughput on a topic that already has one consumer per partition. What's the correct next step?
Lower session.timeout.msIncrease the partition count (accepting the ordering trade-off), since you're already at the partition-count ceiling for parallelismAdd more consumer instances to the same groupIncrease max.poll.records so each poll returns more data