20. Distributed Message Queue (Build Kafka) Advanced + Distributed Systems Capstone

A. Realistic Interview Prompt

Every prior problem that used a queue got to assume the answer to the question this problem exists to actually answer: what does "durable, ordered, fan-out-capable, at-least-once" concretely require underneath the API call. The central technical decision this design is built around is a topic modeled not as a single traditional queue but as a set of independent, ordered, append-only, replicated logs called partitions; producers choose a partition per message (by key hash or round-robin) to get real write parallelism while preserving order for anything that needs it; consumers are organized into independent consumer groups that each track their own position (offset) in the log rather than the broker deleting a message once it's been read; each partition is served by one leader broker plus follower replicas kept in sync via a replication protocol structurally similar to the leader-election and consensus machinery in Prerequisites E and Prerequisites F; and the system is honest about defaulting to at-least-once delivery, explicitly pairing that default with the idempotency pattern (Prerequisites O, generalized in Patterns Playbook E) as the only real way to get effectively-once processing, rather than pretending the broker alone can hand you exactly-once for free. A candidate who reaches for "a table with a processed boolean column" here isn't missing an optimization — they're rebuilding, badly, the one structure every other problem in this document has been quietly relying on.

B. Difficulty

Advanced, and deliberately the hardest, most synthesis-heavy problem in this document. Nothing in this problem is conceptually novel in isolation — partitioning is Prerequisites C's sharding applied to a log instead of a database; replication is Prerequisites C's leader-follower model applied per-partition; leader failover is Prerequisites F's leader election; the delivery-semantics honesty is Prerequisites O's idempotency argument stated at its origin instead of at one of its many destinations. What makes this the capstone is that it requires holding all of those previously-separate ideas in the same design simultaneously, and recognizing that this is the one problem in the document where "the message queue" isn't a dependency to wave at — it is the entire system.

C. What This Problem Tests

• Whether a candidate recognizes that "queue" is the wrong mental model the moment multiple independent consumers need to read the same stream — and reaches for a retained, partitioned log instead of a traditional delete-on-consume broker, the single most consequential decision in this problem (J, K, P)
• Real understanding of why sequential, append-only disk writes are fast in a way random-access, indexed writes are not — the same reasoning P16-M applied when choosing to land clicks in Kafka rather than a database, here examined from the broker's own side (N)
• Comfort naming replication, quorum-style acknowledgment, and leader failover as the same general shape as Prerequisites E and Prerequisites F, applied at partition granularity instead of whole-database or whole-service granularity (O)
• The single most commonly-missed piece of judgment on this problem: recognizing that "exactly-once" is not a property a broker can hand a consumer for free, and being able to say precisely where that honesty has already been assumed elsewhere in this document — Notification System's worker-layer idempotency check against "the queue's at-least-once delivery" (P12, Deep Dive P) and Ad Click Aggregation's click-generation-time dedup key (P16, Deep Dive S) both exist because of exactly the guarantee this problem builds (Q)
• Whether a candidate can distinguish this problem from a generic "build a scalable service" exercise — the interesting failure modes here are almost entirely about ordering, replication safety, and offset bookkeeping, not about caching or load balancing

D. Requirements

Functional

1. createTopic(name, partition_count, replication_factor) — provision a new topic with a fixed number of partitions and a target replication factor
2. produce(topic, key, value, headers) → {partition, offset} — append one record to a topic, letting the producer or the broker choose which partition it lands on (R)
3. subscribe(consumer_group_id, topics) + poll() → a batch of unconsumed records from whichever partitions this consumer instance has been assigned within its group (P)
4. commitOffset(consumer_group_id, topic, partition, offset) — durably record how far a given consumer group has progressed through a given partition (P)
5. Support many independent consumer groups reading the same topic concurrently, each entirely at its own pace and its own committed offset, with no coordination required between groups (P)

Non-Functional

• High, sustained write throughput — the defining load-bearing requirement; illustratively hundreds of thousands of produced messages/sec cluster-wide (G), absorbing bursty producer traffic without the ingest path itself becoming the bottleneck
• Durability — once a write is acknowledged to a producer, it must survive a broker crash, not just live in one machine's memory (O)
• Ordering within a partition, and only within a partition — no global ordering guarantee across an entire topic (M, V), a deliberate scope limitation, not an oversight
• Horizontal scalability — adding brokers must increase the cluster's total throughput and storage ceiling, without a hard limit tied to any single machine (M, N)
• Fault tolerance without extended unavailability or silent data loss — a single broker failure, including the failure of a partition's current leader, must not take that partition permanently offline or quietly lose already-acknowledged data (O)
• Honest, configurable delivery semantics — the system defaults to at-least-once delivery and says so explicitly, rather than implying a false exactly-once guarantee it cannot actually keep (Q)
• Bounded recovery for a lagging or crashed consumer — a consumer group that falls behind or restarts must resume from its last committed offset, not from the beginning of the topic's entire retained history (P)

Out of scope: cross-datacenter/geo-replication between independent clusters (a real, separate problem — mirroring an entire cluster's topics to another region, briefly named in AA); a schema registry or schema-evolution enforcement for record payloads; end-to-end exactly-once processing of a side effect that reaches outside the message queue itself (deliberately addressed at the consumer via idempotency instead of promised by the broker, Q); tiered/cold-storage offload of expired segments to object storage (a real production feature, structurally similar to P16-K's cold-storage archival, but not this problem's central concern).

E. Clarifying Questions

F. Assumptions

• Confirmed: ordering is only ever required relative to a natural key, never globally across an entire topic (E's clarifying question 1) — this is the assumption that makes key-based partitioning (M) a correct design, not a compromise.
• Assumption: the durability/latency trade defaults to requiring acknowledgment from every in-sync replica (acks=all) for anything that must not be lost, with a faster, leader-only acknowledgment available as an explicit, opt-in trade for less critical traffic (E's clarifying question 2, O).
• Confirmed: multiple independent consumer groups genuinely need to read the same topics — this problem exists precisely because P12's channel workers and P16's aggregation, fraud-detection, and archival consumers all needed exactly this, and simply assumed it worked (E's clarifying question 3, P).
• Assumption: true exactly-once processing of an arbitrary external side effect is not something the broker is asked to guarantee on its own — the design commits to at-least-once delivery plus consumer-side idempotency as the correct, honest answer (E's clarifying question 4, Q).
• Assumption: most topics are transient event streams under time/size-based retention by default; a small number of topics (most notably the internal offset-tracking topic itself, P) are compacted changelogs instead, chosen per-topic rather than uniformly (E's clarifying question 5, S).

G. Capacity Estimation

H. Core Entities

I. API Design

POST /topics                          { name, partition_count, replication_factor }  -> { topic_name }         -- provisioning (K)
POST /topics/{topic}/records          { key, value, headers }  -> { partition, offset }                       -- append to a partition's leader (M, N)
(client library) subscribe(group_id, topics); poll(timeout_ms)  -> [ { partition, offset, key, value } ]      -- pull-based fetch from assigned partitions (P)
(client library) commitOffset(group_id, { topic, partition, offset })  -> ack                                 -- durably advances the group's position (P)
GET  /topics/{topic}/partitions/{p}/watermarks  -> { low, high }                                              -- internal/admin: retention boundaries (S)

Why produce returns {partition, offset} directly, inline with the write: a producer that cares about ordering or replay position needs to know exactly where its record landed — returning it as part of the same acknowledgment (computed off the same append the write just performed, O) avoids a separate round trip for information the producer almost always wants immediately, the same reasoning P19-I applies to returning a leaderboard rank inline with a score update.

Why commitOffset is a separate call from poll, rather than the broker automatically advancing the offset the instant a batch is fetched: automatically committing on fetch would mean a consumer that crashes mid-processing has already told the broker it's done with records it never actually finished — silently losing them (at-most-once, by accident). Making the commit an explicit, separate call lets the consumer control exactly when its progress becomes durable relative to when its own processing actually completes, which is the entire lever Deep Dive 5 (Q) turns to get at-least-once instead of an accidental, weaker guarantee.

J. Basic / Naive High-Level Design (V1)

The simplest thing that satisfies the functional requirement in isolation: a single-node, in-memory queue, using the traditional broker model described in Prerequisites G — a producer appends a message to an in-memory list; a consumer pulls the next message off the front of that list and, once it acknowledges having processed it, the message is deleted from the list entirely.

V1 — one process, one in-memory list, one consumer removing each message the moment it's handled. Correct for a single producer talking to a single consumer, and every one of its simplifying assumptions is exactly what the rest of this problem exists to remove.

Why this breaks on multiple, independent, structural fronts at once, not just "under load": (1) it has no persistence at all — a crash of the one app server loses every message still sitting in that in-memory list, acknowledged-to-nobody or not; (2) it has no parallelism — one list, appended and drained by single-threaded access, cannot exceed one machine's throughput no matter how it's tuned; but most importantly, (3) the delete-on-consume step itself is the actual design flaw, not an implementation detail — the instant a second, entirely independent consumer (a second team's analytics pipeline, say) also wants to read the same stream of events, there is nothing left for it to read, because the first consumer's acknowledgment already erased it. This is precisely the traditional-queue model Prerequisites G contrasts with Kafka's log-based one, and it's the single assumption that has to be discarded before anything else in this problem can be solved — not sharded, not replicated, not scaled: replaced, with a model where reading a message and deleting it are no longer the same event.

K. Data Model & Store Choice

The core structure: a partition is not a queue in the traditional sense at all — it's a durable, append-only log, physically backed by a sequence of segment files on disk. A new record is always appended to the current active segment's tail; once a segment reaches a configured size or age, it's sealed and a new active segment is opened. A separate, sparse index file per segment maps a subset of offsets to their exact byte position in the segment, so locating any given offset for a read is a cheap index lookup plus a short scan forward, never a full scan of the segment from the beginning.

Why this specific structure, and not a general-purpose database table keyed by message id: every operation this design actually needs — append a record, read forward sequentially from a given offset — is naturally served by a structure whose entire layout is the append order, the same "let the data structure's native shape answer the actual access pattern" reasoning P19-K applies to a sorted set for ranked reads. A relational table with a monotonic id column can technically express the same thing, but every insert pays for index maintenance and every read pays for an index traversal; a log's "next record" is always physically adjacent to the one before it, so both operations are close to the cheapest thing a disk can do at all — this is the identical reasoning P16-M gives for landing raw ad clicks in Kafka instead of a relational database, examined here from the broker's own implementation rather than from a consumer's choice to use it.

Sequential I/O and the OS page cache, stated concretely: a sequential append is, mechanically, one contiguous write to the end of a file — no index to update, no row lock to acquire, no risk of a page split. Reads benefit the same way: because a partition's recent segments are also its most recently written ones, they tend to already be sitting in the operating system's page cache, and many broker implementations can serve a consumer fetch via a zero-copy transfer straight from that page cache to the network socket, without ever copying the bytes through user-space memory at all. Both the write path and the read path are, deliberately, doing close to the least amount of work a general-purpose database could ever do for the same operation.

L. End-to-End Flow — A Producer Publishes a Message, It's Replicated to Followers, and a Consumer Group Reads and Commits It

The producer's write is acknowledged only once the record has actually reached the in-sync replica set (O), not merely the leader's own disk. The consumer's fetch and offset commit happen on an entirely independent schedule — nothing about the produce path waits for any consumer, and nothing about the consumer path deletes the record once read (P), which is exactly what lets a second, unrelated consumer group replay the identical record days later.

M. Deep Dive 1 — Partitioning a Topic for Parallelism

Requirement this addresses: horizontal scalability and high sustained throughput (D) — the direct fix for V1's single-list ceiling (J), and the structural decision that makes every later deep dive in this problem possible at all.

Why a topic is a set of partitions, not one shared log: a single append-only log, however well-implemented, is still fundamentally served by one leader machine's disk and network interface — exactly the same ceiling Prerequisites C names generally for any single-machine data store. Splitting a topic into many independent partitions, each with its own leader (potentially on a different broker), spreads both the write load and the storage across the cluster, and — just as importantly — gives a consumer group real parallelism: since each partition within a group is consumed by exactly one member of that group at a time (P), the partition count is the hard upper bound on how many consumer instances in one group can usefully work in parallel at all.

Key-based partitioning, and the ordering guarantee it actually provides: a producer that supplies a key has that key hashed and mapped, deterministically, to one specific partition — partition = hash(key) % partition_count — which means every record sharing that key always lands on the same partition, in the order it was produced, and is therefore read by a consumer in that same relative order. This is a deliberately narrower guarantee than "the whole topic is ordered": two records with different keys can land on different partitions and be consumed in either relative order, or even concurrently, with no guarantee either way (E's clarifying question 1, V).

Why changing the partition count later is a genuinely disruptive operation, unlike consistent hashing's minimal-disruption property (Prerequisites J): hash(key) % partition_count is a plain modulo, not a consistent-hashing ring — increasing partition_count changes the result of that modulo for the overwhelming majority of existing keys, meaning a key's future records can start landing on a different partition than its past records did. This silently breaks the exact per-key ordering guarantee this deep dive exists to provide, for any key whose new partition assignment differs from its old one. This is precisely why partition count is chosen deliberately upfront, sized for projected growth (G), rather than treated as a value that can be casually increased the way a stateless service's instance count can — the same "the shard key and its stability matter more than convenience" lesson Prerequisites C and Patterns Playbook G draw generally, sharpened here into a concrete, avoidable failure mode.

N. Deep Dive 2 — Durable, Append-Only Log Storage per Partition

Requirement this addresses: durability and high sustained write throughput simultaneously (D) — making K's data-structure choice concrete in terms of actual disk and network cost.

The direct, quantitative reason sequential writes win at this workload: a spinning or solid-state disk's sustained sequential write throughput is dramatically higher than its random-write throughput, because a sequential write never requires a seek and rarely requires more than appending to the same open file handle; a relational insert with even a couple of secondary indexes pays for scattered, random-access index maintenance on every single write. At hundreds of thousands of writes/sec cluster-wide (G), that per-write random-access overhead is exactly what would make a general-purpose indexed database infeasible here and what makes a segment-file log tractable — the identical trade-off P16-M names for exactly the same reason, one level up the stack.

Producer-side batching, compounding the same benefit: rather than issuing one network round trip and one disk append per individual record, a well-tuned producer briefly buffers several records destined for the same partition and sends them as a single batch (a small, configurable linger delay in exchange for meaningfully larger, more efficient batches) — both the network call and the resulting disk append amortize their fixed overhead across many records instead of paying it per record, the same batching-for-efficiency trade P19-R's per-player write coalescing makes, applied here to raw throughput rather than to redundant intermediate values.

The honest cost of the default flush policy, named rather than glossed over: a broker does not fsync every single append to physical disk before acknowledging it by default — doing so would collapse throughput to whatever the disk's raw fsync latency allows. Instead, the operating system's own page cache absorbs writes and flushes them to disk on its own schedule, while durability against a process crash (not a full machine power-loss event) is covered by the fact that the data has already left the broker process and is sitting in the OS's memory, and durability against a full machine failure is covered separately, by replication (O) rather than by the local disk's flush timing at all. This is a deliberate division of labor: replication is what actually protects a record from loss; local disk flush timing is tuned purely for throughput.

O. Deep Dive 3 — Replication of Partitions

Requirement this addresses: fault tolerance without silent data loss or extended unavailability (D) — Prerequisites C's leader-follower replication model, applied at the granularity of one partition rather than one whole database.

The model: every partition has exactly one leader replica, hosted on one broker, and some number of follower replicas, each hosted on a different broker. Followers replicate by continuously pulling new records from the leader (a pull-based design, not the leader pushing to followers) and appending them to their own local copy of the same log, in the same order. The in-sync replica set (ISR) is the subset of a partition's replicas — the leader plus whichever followers have not fallen behind the leader by more than a configured lag bound — that are considered genuinely caught up right now.

Acknowledgment levels, and the direct durability/latency trade each one makes:

The high watermark, and why a consumer never sees an under-replicated record: the leader tracks the highest offset that has been confirmed replicated to every current ISR member — the high watermark — and only ever exposes records up to that offset to consumers, even though the leader's own local log may already physically contain slightly newer, not-yet-fully-replicated records. This is the same "don't expose a write until it's actually safe" discipline a consensus protocol's commit index enforces (Prerequisites E's ZAB/Raft mention) — it guarantees a consumer can never read a record that could still vanish if the leader fails one instant later.

What happens on leader failure, and the one unsafe shortcut it must never take: a cluster controller — itself a singleton role elected the same lease-based way Prerequisites F describes generally, backed by the same kind of consensus quorum Prerequisites E names for ZooKeeper/etcd — detects a leader's failure via a missed heartbeat, and promotes a new leader exclusively from the current ISR. Promoting a replica that was not in the ISR (one that had fallen meaningfully behind) would silently truncate the partition's history to whatever that lagging replica happened to have — a real, quiet data-loss event dressed up as a successful failover. Allowing exactly this ("unclean leader election") is a real, explicit, and dangerous configuration knob in production systems, named here deliberately as a bad-design candidate (Z) rather than a reasonable default.

P. Deep Dive 4 — Consumer Groups & Offset Tracking

Requirement this addresses: the exact requirement V1's delete-on-consume model failed outright (D, J) — multiple independent consumer groups reading the same topic, each at its own pace, without coordinating with each other at all.

Why the broker never deletes a record on consumption: a record, once committed (O), simply stays in the log until it's removed by retention or compaction (S) — entirely independent of whether, or how many times, or by whom, it's been read. "Reading" a record is nothing more than a consumer advancing its own bookmark forward through an otherwise untouched log. This single decision is what let P12's per-channel worker pools and P16's aggregation, fraud-detection, and cold-storage-archival consumers all independently read the identical stream of events without any of them needing to know the others existed — the exact fan-out those two problems assumed already worked, built here for real.

Offsets, concretely: each ConsumerGroup (H) tracks, for every partition it consumes, the offset of the next record it hasn't yet processed. That committed offset isn't kept in application memory alone — it's durably persisted as a keyed record (group_id, topic, partition → committed_offset) in an internal topic that is itself just an ordinary Kafka topic, using log compaction (S) so only the latest committed offset per key is ever retained. A consumer that crashes and restarts simply reads its own last committed offset back out of that internal topic and resumes exactly where it left off — no re-scan of the entire topic history required.

Partition assignment, and the parallelism ceiling it implies: within one consumer group, each partition is assigned to exactly one member at a time — two members of the same group never process the same partition concurrently, which is precisely what keeps per-partition ordering (M) meaningful to a consumer at all. This also means a group's real, useful parallelism is capped at the topic's partition count (M) — adding an 11th consumer to a group consuming a 10-partition topic simply leaves that 11th consumer idle.

Commit strategy, and the direct trade Q inherits from it: committing an offset after a record has been fully processed is the safe default — a crash between processing and committing simply causes that record to be redelivered and reprocessed on restart (at-least-once, the honest default, Q). Committing before processing starts inverts the risk: a crash mid-processing leaves the offset already advanced past a record that was never actually finished, silently skipping it (at-most-once, chosen only when losing an occasional record is genuinely more acceptable than ever reprocessing one).

Rebalancing, when group membership changes: when a consumer joins, leaves, or is judged dead (a missed heartbeat), the group coordinator reassigns partitions among the group's remaining members. A naive ("eager") rebalance revokes every partition from every member before reassigning anything, pausing the entire group's consumption briefly even for members whose own assignment doesn't actually change; incremental/cooperative rebalancing narrows that pause to only the specific partitions actually being reassigned, letting unaffected members keep consuming throughout.

Q. Deep Dive 5 — Delivery Semantics: At-Least-Once by Default

Requirement this addresses: honest, configurable delivery semantics (D) — the single most consequential piece of intellectual honesty this entire design has to get right, and the reason every prior problem in this document that consumed from a queue was already quietly building on top of it.

Why end-to-end exactly-once is, in the most literal sense, not achievable for an arbitrary external side effect: there are at least two independent places duplication can be introduced. On the produce side, a network timeout after a successful append but before the producer receives its acknowledgment leaves the producer unable to distinguish "it worked but the ack was lost" from "it never happened" (Prerequisites O's exact framing) — a naive retry can produce a duplicate record. On the consume side, a consumer that crashes after fully processing a record's side effect but before committing its offset (P) will, on restart, be redelivered that same already-processed record, because its last committed offset still points to before it. Kafka's own idempotent-producer feature (deduplicating retried produce attempts via a per-partition producer id and sequence number) and its transactional API (genuinely exactly-once for a read-process-write pipeline that stays entirely within Kafka) both close real gaps — but only up to Kafka's own boundary. The instant a consumer's side effect leaves that boundary — charging a card, sending a push notification, writing to a completely different external database — no broker-side mechanism, however sophisticated, can make that external side effect exactly-once by itself, because the broker has no visibility into, or control over, what happens after the message leaves it.

The honest, achievable substitute, and where this document has already been relying on it: pair at-least-once delivery (this problem's actual, honestly-stated guarantee) with an idempotent consumer (Prerequisites O, generalized as its own named pattern in Patterns Playbook E) — a stable key attached to each unit of work, checked before the side effect runs, so a redelivered record produces a duplicate delivery but never a duplicate side effect. This is not a new idea introduced here for the first time; it's the exact mechanism P12's Deep Dive P already named explicitly — "the queue itself only guarantees at-least-once delivery to a worker... a worker can crash after calling a provider but before acknowledging the queue message, causing the same enqueued message to be redelivered" — and the exact reason P16 deduplicates click events using a key established at click-generation time, before they ever reach the aggregation layer. Both of those designs assumed this problem's guarantee already existed and built the correct defense against it. Effectively-once, in practice, is at-least-once delivery plus an idempotent consumer — it is not, and cannot be, a property the broker hands anyone for free, on its own.

R. Deep Dive 6 — Producer Partitioning Strategy

Requirement this addresses: giving a producer real control over the ordering-versus-load-balance trade (D, M) — the concrete mechanics behind M's "producers choose a partition" decision.

S. Deep Dive 7 — Retention & Compaction

Requirement this addresses: bounding storage growth (G) without discarding data a topic's actual use case still needs (D) — the design has to answer "when does a record actually get deleted," and the honest answer is that it depends entirely on what the topic represents.

Why these two strategies aren't interchangeable, and picking the wrong one for a given topic is a real correctness gap, not just a storage-efficiency one: applying time-based retention to a changelog topic would eventually delete the only record of a key's current value once it aged out, even though nothing superseded it — a silent, wrong answer to "what is this key's value right now." Applying compaction to a genuine event stream would collapse every historical event for a given key down to just its latest one, destroying exactly the full history an analytics or audit consumer (P16) is reading the topic for in the first place. The two strategies protect two genuinely different promises, and a topic gets to choose, deliberately, which one it's making.

T. Full Solution at Scale — Final Architecture

Every partition's leader and followers, the controller's failure-detection path, and each independent consumer group's own offset bookkeeping all operate on separate, decoupled schedules — a slow or crashed consumer group never blocks a producer, and a follower's replication never blocks a consumer reading from an already-committed high watermark.

U. Design Evolution

What stays stable throughout: the minimal produce/poll/commitOffset contract from I — every refinement underneath changes how many partitions a topic has, how many copies of each record exist, and how a consumer's progress is tracked, never the basic shape of what a producer or consumer actually calls.

V. Consistency Model

This design makes one deliberately narrow ordering promise and is explicit about everything it does not promise on top of it. Ordering is guaranteed strictly within a single partition — a consumer reading one partition sees records in exactly the order the leader appended them, never reordered, never skipped ahead of an unreplicated record (O's high-watermark discipline). There is no ordering guarantee across partitions, and by extension no global ordering guarantee across an entire topic — two records with different keys, on different partitions, carry no meaningful relative order at all, by design (M), not by omission.

The delivery-semantics half of the model is equally explicit, and is the part most designs get wrong by implying more than they deliver: this system is an at-least-once system, full stop — a produce retry can duplicate a record at the edge of Kafka's own boundary (mitigated, not eliminated, by the idempotent producer), and a consumer crash between processing and committing an offset will redelivery-and-reprocess a record it already handled (P). "Exactly-once" is not a property this broker, or any broker, hands a consumer for free — it is a cooperative illusion, correctly achieved only by pairing this system's honest at-least-once guarantee with an idempotent consumer (Q, Prerequisites O, Patterns Playbook E). A design that claims "exactly-once" as a broker-level property, without naming the consumer-side idempotency work that guarantee actually depends on, has misdescribed its own system.

The direct contrast worth naming explicitly: unlike Real-Time Leaderboard's read-your-own-writes-versus-approximate-global-view split (P19-V), which tunes two different guarantees for two genuinely different queries, this design doesn't offer a consistency dial at all — ordering-within-a-partition and at-least-once delivery are fixed, structural properties of the log itself, not configurable trade-offs a caller opts into per request.

W. Failure Scenarios

• A partition's leader broker crashes: the controller detects the missed heartbeat and promotes a replica from the current in-sync replica set (never a lagging, out-of-ISR replica) to leader; producers and consumers are briefly redirected to the new leader, and no already-committed (high-watermark-cleared) record is lost (O).
• A follower falls too far behind to stay in the ISR: it's simply dropped from the in-sync set until it catches back up — acks=all writes now only wait on the remaining, genuinely in-sync members, so a slow follower degrades that partition's effective replication factor temporarily rather than blocking every write on the cluster's slowest machine indefinitely (O).
• A consumer crashes mid-processing, before committing its offset: on restart (or after the group's rebalance reassigns its partitions), whichever consumer next owns that partition resumes from the last committed offset, redelivering the record that was in flight — the expected, honest at-least-once behavior (P, Q), correctly handled downstream only if that consumer's side effect is idempotent.
• A producer retries after a timeout whose original request actually succeeded: without an idempotent producer, this creates a genuine duplicate record on the partition; with it, the broker recognizes the retried producer-id/sequence-number pair and discards the duplicate before it's ever appended (Q).
• A synchronized burst of produce traffic exceeds one partition's throughput momentarily: because partitioning (M) already spreads a topic's total load across many leaders, a burst concentrated on one key's partition is a localized hot-partition problem, addressed the same way P16-R addresses a viral ad's click volume — salting the key to spread that one key's traffic across several partitions temporarily.
• The cluster controller itself fails: a new controller is elected from the surviving brokers via the same consensus/leader-election machinery (Prerequisites E, Prerequisites F) that elects it in the first place — existing partition leaders keep serving reads and writes uninterrupted throughout, since the controller's role is failure detection and reassignment, not sitting on the live read/write path.

X. Bottlenecks & Scaling Boundary

Y. Design Trade-Offs

Z. Common Bad Designs

• A traditional delete-on-consume queue presented as if it were "basically Kafka." Works fine for exactly one consumer, and permanently forecloses replay and every additional independent consumer group the moment a second one shows up — the defining mistake on this problem (J).
• One partition per topic, "for simplicity." Trivially preserves global ordering, but caps the topic's entire throughput and every consumer group's parallelism at what one broker and one consumer instance can handle — the exact ceiling M's partitioning exists to remove.
• Allowing unclean leader election as a default, not an explicit, rare opt-in. Technically keeps the partition available during a leader failure, but at the real risk of silently promoting a replica that was never actually caught up, truncating history that had already been acknowledged (O).
• Advertising "exactly-once" as something the broker alone provides, with no mention of consumer-side idempotency. A confident-sounding claim that quietly stops being true the instant a consumer's side effect reaches outside the message queue — and a claim this document's own prior problems (P12, P16) never actually made, because they already knew better (Q).
• Using log compaction on a genuine event stream, or time-based retention on a genuine changelog. Both compile and run without error, and both quietly produce the wrong answer to "what does this topic actually contain" for the use case it's really serving (S).

AA. Interviewer Follow-Ups

AB. Interviewer Evaluation Rubric

• Weak candidate: designs a traditional delete-on-consume queue and calls it done, with no answer for what happens when a second independent consumer needs to read the same events, or what happens when the one broker crashes.
• Mid-level: correctly reaches for partitioning and replication by name, but can't precisely explain why sequential log appends beat random-access database writes, and hasn't considered what an in-sync replica set actually protects against.
• Senior: explains the high-watermark/ISR mechanism correctly, proactively distinguishes ordering-within-a-partition from any topic-wide guarantee, and identifies the consumer-group offset model as the direct fix for delete-on-consume's fundamental flaw.
• Staff+: is explicit, unprompted, that exactly-once is not a property the broker alone can provide for an external side effect, correctly frames at-least-once-plus-idempotency as the honest achievable substitute, and can name exactly where this document's own earlier problems (P12, P16) already depended on that same fact without spelling it out at the time.

AC. Interview Simulation

AD. Online Solution Comparison

AE. Interview Cheat Sheet