Design a distributed message queue (Kafka)

Design a distributed message queue like Kafka: producers send messages to topics, consumer groups read them in order and can replay them, messages are kept for two weeks, and nothing is lost when a broker dies.

1. Clarify the requirements

Before drawing anything, ask questions. These are the answers an interviewer would typically give:

Throughput?

200,000 messages a second in, 400,000 out (two consumer groups).

Message size?

Small: about 1 KB.

Ordering?

Messages must be consumed in the order they were produced, within a partition.

How long are messages kept?

Two weeks; consumers can re-read old messages.

Delivery guarantees?

At-least-once by default; at-most-once and exactly-once should be possible.

2. Estimate the scale

Assumptions: 200K messages/s produced, 400K/s consumed, 1 KB each, kept 14 days, 3 copies.

• Write bandwidth: about 200 MB/s200K × 1 KB = 200 MB/s in.
• Storage for 14 days (×3 copies): about 726 TB200 MB/s × 86,400 × 14 ≈ 242 TB, × 3 replicas ≈ 726 TB.
• Brokers needed (≈ 50 MB/s each): about 12 brokers200 MB/s × 3 copies = 600 MB/s of writes ÷ 50 MB/s ≈ 12 brokers (before headroom).

3. The high-level design

1. Producers write to brokersProducers batch messages and send each to the leader of its partition. The key decides the partition, so related messages stay in order.
2. A controller runs the clusterThe controller tracks which brokers are alive, elects partition leaders and stores topic metadata.
3. Consumer groups read at their own paceEach group gets every message; inside a group, each partition is read by one consumer. Consumers commit offsets after processing (at-least-once) and stay idempotent to avoid duplicates.
4. Watch it in productionMetrics and alerts close the loop: consumer lag, under-replicated partitions and produce latency.

4. The components and what each one does

Producers Client (web)

Batch messages; the key picks the partition.

Brokers (partitioned logs) Event stream

Topics split into partitions: append-only logs written sequentially to disk (page cache, batching). Each partition has a leader and followers; in-sync replicas (ISR) take over if the leader dies. acks=all for durability. Kept 14 days.

Controller (KRaft) Scheduler / cron · KRaft / ZooKeeper

Tracks brokers, elects partition leaders, stores topic metadata (ZooKeeper in older versions).

Consumer group A Workers

One consumer per partition; commit offsets after processing (at-least-once); idempotent.

Consumer group B Workers

Reads the same messages independently, at its own pace.

Metrics & alerts Metrics & alerts

Dashboards and alerts on p99 latency, error rate, queue depth and cache hit ratio, so a bottleneck shows up before it becomes an outage.

5. Deep dive: partitions, replication and delivery

How is a topic split so it scales, how are partitions replicated, and how do producers and consumers get at-least-once (or exactly-once) delivery?

• A topic is split into partitions: append-only logs, each on one leader broker. Order is guaranteed only within a partition (pick the key accordingly).
• Each partition has followers; the in-sync replicas (ISR) take over if the leader dies. acks=all waits for the ISR, acks=1 only for the leader.
• Consumers in a group split the partitions and commit their offsets. Commit after processing for at-least-once; idempotent producers + transactions for exactly-once.

6. What interviewers listen for

• Topics split into partitions; order within a partitionPartitions are how a topic scales; order holds only inside one.
• Leader/follower replication with in-sync replicasEvery partition has a leader and followers; the ISR takes over when the leader dies.
• Producer acks trade latency for durabilityacks=0/1/all: faster or safer.
• Consumer offsets and consumer groupsConsumers commit offsets, so they can resume and replay.
• At-most-once, at-least-once, exactly-onceSay which you give by default, and how a consumer avoids duplicates.
• Sequential disk writes, batching, page cacheAppend-only logs and batching are what make a disk-based queue fast.

7. Follow-up questions

1. The broker leading partition 7 crashes. What happens to producers and consumers of that partition?
2. A consumer charges customers. It crashed after processing a message but before committing the offset. What happens on restart, and how do you avoid double charges?
3. One consumer group falls hours behind. Does it slow anyone else down? How do you catch up?

The design in a nutshell

• Producers (batching, keyed by message key) → partition leaders on brokers; each partition is an append-only log on disk, replicated to followers (ISR).
• A controller (ZooKeeper or KRaft) tracks brokers, elects leaders and stores topic metadata.
• Consumer groups pull from partitions, each partition read by one consumer in the group, and commit offsets: at-least-once by default, exactly-once with idempotent producers + transactions.
• Retention by time (14 days) or size; monitor under-replicated partitions, consumer lag and produce latency.

Related lessons: Delivery guarantees and idempotent consumers · Message queue · Partitioning strategies