Walk through Kafka's architecture — brokers, partitions, and consumer groups — and how they achieve both scale and ordering

Apache Kafka is not a traditional message queue — it's a distributed, append-only commit log that happens to support publish/subscribe semantics. That distinction drives almost every architectural decision underneath it, and it's usually where a senior-level answer separates itself from a beginner one.

#Topics, partitions, and brokers

A topic is a named stream of events, e.g. order-events. Kafka splits every topic into one or more partitions, and partitions — not topics — are the real unit of storage, parallelism, and ordering. Each partition is an ordered, append-only sequence of records identified by a monotonically increasing offset. Partitions are spread across brokers (the servers that make up the cluster); for fault tolerance, each partition is replicated to N brokers, one of which is elected leader. All reads and writes for that partition go through the leader; the rest? are followers that replicate the log and can take over if the leader fails.

The number of in-sync replicas (ISR) that have fully caught up to the leader determines how durable a write can be. A producer's acks setting controls the trade-off directly:

#Ordering is per-partition, not per-topic

This is the detail interviewers probe hardest: Kafka guarantees ordering only within a single partition. If a topic has 6 partitions, there is no global order across them — messages produced roughly at the same time can be read back in different relative orders on different partitions. If you need every event for a given entity (a user, an order, an account) processed in the order it happened, you route them to the same partition by giving them the same partition key. Kafka hashes the key to deterministically pick a partition, so orderId=123 always lands on the same partition, and a single consumer thread reads it back in write order.

#Scaling reads with consumer groups

A consumer group is Kafka's mechanism for scaling out consumption while keeping each partition owned by exactly one consumer at a time. When several consumer processes share a groupId, Kafka's group coordinator assigns each partition to exactly one consumer in that group — so a topic with 6 partitions can be consumed in parallel by up to 6 consumers in the same group; a 7th consumer would sit idle. This is also why partition count is your real scaling ceiling, not consumer count.

Different consumer groups are fully independent: each group gets its own copy of the entire stream and tracks its own offsets. That's how one topic can feed a billing service and an analytics service at the same time without either affecting the other's read position.

When a consumer joins, leaves, or crashes, Kafka triggers a rebalance — partitions are reassigned across the group's remaining members. Rebalances briefly pause consumption for the group, which is why frequent restarts or overly aggressive session.timeout.ms settings can visibly hurt throughput in production.

#Producer and consumer in practice

const { Kafka, logLevel } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'orders-service',
  brokers: ['broker1:9092', 'broker2:9092', 'broker3:9092'],
  logLevel: logLevel.INFO,
});

// --- Producer: a keyed message guarantees per-entity ordering ---
async function publishOrderEvent(order) {
  const producer = kafka.producer({ allowAutoTopicCreation: false });
  await producer.connect();

  await producer.send({
    topic: 'order-events',
    messages: [
      {
        key: order.orderId,        // same key -> same partition -> in-order delivery
        value: JSON.stringify({ type: 'ORDER_CREATED', ...order }),
        headers: { 'event-version': '1' },
      },
    ],
    acks: -1,   // wait for every in-sync replica before considering the write durable
    timeout: 30000,
  });

  await producer.disconnect();
}

// --- Consumer: one member of a consumer group, commits offsets manually ---
async function startBillingConsumer() {
  const consumer = kafka.consumer({ groupId: 'billing-service' });
  await consumer.connect();
  await consumer.subscribe({ topic: 'order-events', fromBeginning: false });

  await consumer.run({
    autoCommit: false, // commit only after the side effect is safely applied
    eachMessage: async ({ topic, partition, message }) => {
      const event = JSON.parse(message.value.toString());
      await chargeCustomer(event); // must be idempotent — redelivery can happen

      await consumer.commitOffsets([
        { topic, partition, offset: (Number(message.offset) + 1).toString() },
      ]);
    },
  });
}

At-least-once delivery is Kafka's default posture: if a consumer crashes after processing a message but before committing its offset, that message is redelivered and reprocessed. Exactly-once semantics exist (idempotent? producers plus transactional writes across topics), but they add real overhead and only cover Kafka-to-Kafka pipelines — the moment your consumer does something external (charges a card, sends an email), you're back to needing an idempotent handler regardless of what Kafka guarantees on its own side.

🧠 Check your understanding
0/3 · 0/3 answered
1. A topic has 4 partitions. A single consumer group has 6 consumer instances subscribed to it. What happens?
All 6 consumers share every partition evenly using round-robin per messageKafka automatically creates 2 more partitions to use the idle consumersThe group refuses the 5th and 6th consumersOnly 4 consumers get a partition each; the other 2 sit idle
2. Why does Kafka only guarantee ordering within a single partition, not across a whole topic?
Because brokers process partitions in a random order by designBecause ordering across partitions is guaranteed but only when acks=allBecause consumers read partitions out of order on purpose to improve throughputBecause each partition is an independent, sequentially-written log with its own offsets, and different partitions can live on different brokers with no coordination between them
3. What does setting acks=-1 (or 'all') on the producer buy you?
The message is delivered to all consumer groups at onceLower latency writes since the leader responds immediatelyThe write is confirmed only after every in-sync replica has it, surviving a leader failureIt disables replication to speed up throughput