12. Notification System Intermediate → Advanced

A. Realistic Interview Prompt

Nearly every problem in this document has, at some point, waved at "send the user a notification" as an unexamined side effect — News Feed's new-post event (P3), Chat System's offline-delivery fallback (P4), Ticketmaster's booking confirmation (P8), and Job Scheduler's own worked example of "a job that sends a best-effort notification email" (Problem 10, Deep Dive P). This problem is the system all of those were quietly assuming already exists. The honest, load-bearing answer underneath "send a notification" turns out to be one integrated decision: an async fan-out pipeline — ingest, dedupe, queue, and hand off to per-channel worker pools behind a pluggable provider adapter for push, SMS, email, and in-app — with a client-supplied idempotency key on every request, and per-user rate limiting plus digesting layered on top so one busy upstream service never turns into a wall of pings on someone's phone.

B. Difficulty

Intermediate, growing distributed-systems-heavy in the deep dives — the "accept a request, put it on a queue" shape is already familiar from half this document's async-offload examples (Patterns Playbook C); what makes this problem genuinely hard is the fan-out surface itself: one logical event can need to become zero, one, or several physical delivery attempts across entirely different external systems, each with its own delivery semantics, rate limits, and failure modes, all while still guaranteeing that no single event turns into a flood on any one user's device.

C. What This Problem Tests

• Recognizing "send a notification" as async, off-the-hot-path work (Patterns Playbook C) rather than something a caller waits on synchronously — the same instinct behind click analytics in URL Shortener (P1) and view-count aggregation in Video Streaming (P7), applied here as the central design decision instead of a supporting one
• Idempotent retries via a client-generated key (Prerequisites O; Patterns Playbook E) as the load-bearing defense against duplicate sends — the same mechanism as Chat System's client_msg_id and Ticketmaster's payment-confirmation key, here protecting a user from receiving the same push twice
• The adapter pattern applied to genuinely heterogeneous downstream systems — push, SMS, and email are not interchangeable, and a design that treats them as one uniform "send" call is missing the actual substance of the problem
• Distinguishing "this exact event was already delivered" (deduplication — a correctness problem) from "this user is being sent too much, too fast" (rate limiting and digesting — a UX and abuse-mitigation problem) — two problems that look superficially similar and need two genuinely different mechanisms
• Reusing rate limiting as a cross-cutting concern (Patterns Playbook K) rather than reinventing throttling logic from scratch — the same token-bucket mechanism from Rate Limiter (P2), now applied per-user instead of per-API-client

D. Requirements

Functional

1. Any internal service can submit a notification request specifying a target user, a notification type, and payload data for the message content
2. The system resolves which channel(s) — push, SMS, email, in-app — a notification is delivered through, honoring an explicit override from the caller when one is supplied
3. Every request carries a client-supplied idempotency key; resubmitting the same key within a bounded window results in exactly one delivery attempt, not two
4. Users can view and update, per notification type and per channel, which channels they're opted into or out of
5. Per-channel delivery status for a given notification (queued, sent, delivered, failed) is queryable after the fact

Non-Functional

• High, bursty throughput — a single triggering event (a flash sale, an outage alert, a viral post) can legitimately need to fan out to millions of recipients within seconds
• At-least-once delivery of the underlying send attempt, with the dedup guarantee above absorbing the user-visible duplicates that at-least-once mechanically produces
• Low latency specifically for critical/transactional notifications (an OTP code, a security alert, a payment confirmation) — low seconds, not "eventually"
• Best-effort, delay-tolerant delivery for low-priority notifications (marketing, engagement nudges), which can be batched, deferred, or dropped under load without violating any correctness guarantee
• Resilience to a single downstream provider being degraded or fully down — one SMS gateway or push provider having an outage should not take down that entire channel
• Users are never buried under a burst of near-duplicate low-priority notifications generated by the same underlying activity spike (ten people liking the same post inside a minute)

Out of scope: this system is not itself a real-time messaging/chat product — Chat System (P4) already owns person-to-person messaging; rich template-authoring/CMS tooling; a full engagement-analytics dashboard (acknowledged as a real extension in AA); owning the business logic behind what "an order shipped" means — this system only owns getting a notification about that event delivered.

E. Clarifying Questions

F. Assumptions

• Confirmed: submission via an API call carrying an idempotency key, channel resolution with an explicit-override escape hatch, and queryable per-channel delivery status.
• Assumption: traffic is a genuine mix of transactional and best-effort notifications (E's clarifying question 1) — the assumption that makes priority-differentiated queues (S) a first-class part of the design rather than a footnote.
• Assumption: the system resolves channels from stored user preference by default, honoring an explicit override for the minority of cases (security alerts) where a caller legitimately needs to force multi-channel delivery regardless of preference (E's clarifying questions 2 and 5).
• Assumption: exact deduplication of a repeated idempotency key and volume-based rate limiting/digesting are both required, and are treated throughout this design as two distinct mechanisms rather than one (E's clarifying question 3).
• Assumption: "delivered" means "successfully handed off to the channel provider" (a push accepted by APNs/FCM, an SMS accepted by the carrier gateway) — true open/read receipts are a real, named extension (AA) but not required for the core design (E's clarifying question 4).

G. Capacity Estimation

H. Core Entities

I. API Design

-- caller-facing (any internal service):
POST /notifications
  { idempotency_key, user_id, type, payload, priority, channels? (optional override) }
  -> { notification_id, status: "queued" }
GET  /notifications/{id}/status          -> per-channel DeliveryAttempt status list (H)
GET  /users/{id}/preferences             -> current per-type, per-channel opt-in state (O)
PUT  /users/{id}/preferences             -> update opt-in state
-- internal, cluster-facing, not caller-facing:
(ingest) validate + dedup (P) + persist request -> enqueue onto priority-appropriate queue (S)
(channel worker pools) consume queue -> resolve provider adapter (N) -> call provider -> record DeliveryAttempt
(digest scheduler) periodically flush an open DigestBatch (Q) into one consolidated send

Why POST /notifications returns "queued" immediately rather than waiting for the actual send: the caller only needs confirmation that the request was accepted and will be handled — waiting for an actual push/SMS/email round trip before responding would tie the caller's request latency to a downstream provider's latency, directly undermining the fast, non-blocking ingest path this system exists to provide (Patterns Playbook C). A caller that genuinely needs to know the outcome polls GET /notifications/{id}/status, or subscribes to a webhook/callback — never by blocking the original submission.

J. Basic / Naive High-Level Design (V1)

The simplest thing that satisfies the functional requirement in isolation: an API endpoint that, in the same request, directly calls a single hardcoded provider for whichever channel is requested.

V1 — one synchronous call, straight from the request path, to one hardcoded provider per channel. No queue, no dedup.

Why this breaks immediately, not eventually: two separate, independent flaws, both fatal on their own. First, calling the channel provider synchronously inside the request path ties the calling service's own latency to the provider's — a slow or degraded push provider now makes every unrelated feature that happens to send a notification slow too, and a burst of triggering events (G) has nowhere to go but directly at the provider's own rate limit, with no buffer in between. Second, there is no deduplication at all: if the calling service's request times out and it retries — the standard, correct behavior for a client facing an ambiguous failure (Prerequisites O) — the notification is sent a second time, because nothing recorded that the first attempt might already have gone through. A user ends up with two identical push notifications for one event, and the system has no way to know or prevent it. Both flaws point at the same fix: move the send off the request path entirely, and give every request an idempotency key before it ever reaches a provider.

K. Data Model & Store Choice

Preferences and templates live in a relational database: both are comparatively low in volume relative to raw notification traffic, need simple, consistent key-based lookups ("what are this user's preferences," "what's the current template for this type"), and benefit from the referential structure a relational store provides — this is the same "match the store to the access pattern" reasoning from Prerequisites B and Distributed Cache's storage choice (P11-K), just landing on the opposite side of that decision here.

Delivery attempt logs live in a high-write-throughput, append-only store (a wide-column store in the Cassandra/DynamoDB mold), partitioned by notification_id or by user_id plus a time bucket — this is write-heavy, rarely-updated, mostly-append data at a volume (G) that would overwhelm a relational store's indexing and replication overhead if forced through the same engine as preferences.

Dedup keys and rate-limit counters live in Redis, keyed with a TTL matching their respective window: both need an atomic check-and-set at very high throughput (thousands of operations per second per hot user during a burst) and both are inherently time-bounded — a dedup window or a rate-limit window that never expires would leak memory forever. Redis's atomic SET ... NX EX primitive is exactly the "collapse read-decide-write into one operation" move named generally in Patterns Playbook B.

Digest batches are similarly short-lived, held in Redis (or an equivalently fast, TTL-capable store) rather than the durable delivery-log store — a batch's entire lifecycle is "accumulate for a bounded window, then flush and cease to exist," which is a poor fit for a store optimized for permanent, queryable history.

L. End-to-End Flow — A Backend Service Triggers a Notification

Deduplication happens once, at ingest, before anything is queued — everything after that point is free to retry at the message-delivery layer without re-litigating "have we seen this before."

M. Deep Dive 1 — Async Fan-Out via Queue

Requirement this addresses: high, bursty throughput (D), and directly fixes V1's fatal synchronous-send flaw (J).

The ingest API's only job is to validate the request, run the dedup check (P), persist the NotificationRequest, and enqueue it — then return immediately. Everything downstream of that queue happens on its own schedule, at whatever rate the channel worker pools and the providers behind them can sustainably handle. This is the exact "keep expensive work off the hot path" move named generally in Patterns Playbook C — the fast, essential part (accepting and recording the request) is synchronous; the slow, non-essential-to-the-caller part (actually contacting a push/SMS/email provider) is deferred entirely.

Why a Kafka-style log, not just a traditional delete-on-consume queue (Prerequisites G), is the right default here: a single triggering event frequently needs to feed more than one independent downstream consumer at once — the push worker pool needs to send it, and an analytics/engagement pipeline may independently need to record that it happened, at its own pace, without either one blocking the other. This is precisely the scenario Prerequisites G names as the reason to reach for a log-based broker: "multiple entirely independent downstream systems can each read the full stream of events at their own pace." A traditional single-delivery queue would force an artificial choice about which one consumer gets the message.

What the queue actually buys under a burst (G): queue depth absorbs the gap between a 30,000–50,000/sec ingest spike and whatever steady rate the channel workers and their downstream providers can sustain — the burst is smoothed into a longer, steadier drain instead of being thrown directly at a provider's own rate limits, which would simply produce a wave of provider-side throttling errors instead of a smooth, if slightly delayed, delivery.

N. Deep Dive 2 — Multi-Channel Provider Abstraction

Requirement this addresses: the multi-channel functional requirement (D) — push, SMS, email, and in-app are not interchangeable; each has its own addressing scheme, payload format, and failure vocabulary.

The adapter pattern, concretely: each channel worker pool talks to one internal interface — send(notification) -> DeliveryResult — and a separate adapter implementation behind that interface handles the actual vendor-specific request/response shape. The worker pool itself never branches on "which vendor," only on "which channel," keeping the fan-out logic (M) and the provider-integration logic (N) cleanly separated — the same separation of concerns that lets Distributed Cache's storage engine stay ignorant of what a value actually contains (P11-K).

Provider failover within a single channel: a channel can be configured with a ranked list of provider adapters (a primary and one or more secondaries) rather than exactly one; when the primary's error rate or latency crosses a threshold, new sends for that channel route to the next provider in the list automatically, and traffic shifts back once the primary recovers. This bounds the blast radius of a single vendor outage to "degraded, temporarily slower delivery on one channel," rather than "that entire channel is down for every user."

O. Deep Dive 3 — User Preferences & Opt-Outs

Requirement this addresses: per-type, per-channel opt-in/opt-out (D) — and, for several channels, a genuine legal compliance obligation, not just a UX nicety.

The preference check happens before channel resolution, not after: a channel worker resolves which channel(s) a notification should go out on by first checking UserPreference (H) for that user and type; a channel the user has explicitly opted out of is simply removed from the candidate set, the same way it would be if the calling service had never requested it.

Compliance is a first-class part of this deep dive, not an afterthought: email delivery must include a functioning, one-click unsubscribe mechanism (the practical requirement behind CAN-SPAM compliance); SMS delivery must honor carrier-mandated opt-out keywords (a user texting STOP must actually stop receiving messages, a TCPA-driven requirement); both are enforced at the preference layer, not left to each calling service to reimplement independently.

Transactional notifications are the deliberate exception to marketing opt-out, not a bug: a payment confirmation, a security alert, or an OTP code is not subject to the same marketing-opt-out rules — a user who has opted out of promotional email has not opted out of being told their account was just accessed from a new device. The system distinguishes types that are legally/functionally transactional from those that are marketing at the template/type level (H), and the preference check for a transactional type only ever restricts channel choice, never whether the user is notified at all.

P. Deep Dive 4 — Deduplication & Idempotency

Requirement this addresses: exactly-once-feeling delivery for a repeated request (D), and directly fixes V1's second fatal flaw (J) — this is the identical mechanism generalized in Prerequisites O and named as its own pattern in Patterns Playbook E, here protecting a user from a duplicate push instead of protecting a customer from a duplicate charge.

The mechanism: the calling service attaches a stable, unique idempotency_key to every request (D). At ingest, the API performs an atomic SET key NX EX dedup_window against Redis (K) — if the key doesn't yet exist, this call both claims it and sets it in one atomic step, exactly the "collapse the check and the write into one operation" move from Patterns Playbook B; if the key already exists, the request is a duplicate, and the API simply returns the original notification_id and status instead of enqueueing anything new.

Why this single check is not the whole story: the queue itself (M) only guarantees at-least-once delivery of a message to a worker (Prerequisites G) — a worker can crash after calling a provider but before acknowledging the queue message, causing the same enqueued message to be redelivered and reprocessed. The ingest-time dedup check above already prevented a second request from becoming a second queued message, but it says nothing about a single queued message being processed twice. The fix is a second, narrower idempotency check at the worker layer, keyed by notification_id + channel, immediately before calling the provider adapter (N) — the same principle (a stable key, checked atomically before the side effect) applied a second time, against a different source of at-least-once delivery.

Choosing the dedup window: long enough to cover any realistic retry interval a calling service might use (minutes to low hours), short enough that the Redis key set doesn't grow unbounded — this is a tunable, not a fundamental constraint, and is called out explicitly as a configuration decision rather than a fixed constant.

Q. Deep Dive 5 — Rate Limiting & Digesting

Requirement this addresses: protecting a user from notification volume, not duplication (D) — a genuinely different problem from P, per E's clarifying question 3.

Per-user, per-channel rate limiting: the same token-bucket mechanism designed in full for Rate Limiter (P2) and named as a reusable cross-cutting concern in Patterns Playbook K, applied here per (user_id, channel) pair instead of per API client — each pair gets a bucket that refills at a configured rate, capping how many raw sends a single user can receive on a single channel in a short window, protecting them from a buggy or unexpectedly bursty upstream service (ten retried webhook deliveries for one shipment update, say).

Digesting — the better answer than simply dropping overflow: rather than silently discarding notifications a rate limit would otherwise reject, low-priority notifications in excess of the limit accumulate into an open DigestBatch (H) keyed by (user_id, type) over a bounded window (fifteen minutes, or hourly, depending on type); a background digest scheduler flushes the batch as a single consolidated notification once the window closes or a size threshold is hit — "5 people liked your post" instead of five separate, near-simultaneous pushes.

Critical/transactional notifications are exempt from both mechanisms entirely, by design: an OTP code delayed by a digest window, or dropped by a rate limit tuned for marketing traffic, is a functional failure, not a UX rough edge — priority (S) is checked before rate limiting or digesting is ever applied, and a critical notification bypasses both unconditionally.

The trade-off digesting makes explicitly: a digested notification's individual member is, by construction, visible to the user later than it would have been sent standalone — a deliberate latency cost, traded for a materially better experience than either a flood of near-duplicate pings or silent, unexplained drops.

R. Deep Dive 6 — Retry & Failure Handling

Requirement this addresses: resilience to a degraded provider (D) — a delivery attempt can fail for reasons ranging from a transient network blip to a permanently invalid destination, and the two need genuinely different handling.

Dead-lettering after exhausting retries: a retryable attempt that fails every allowed attempt moves to a distinct "failed permanently" state in the delivery log (H), available for investigation, rather than being retried forever or silently dropped — the same dead-letter pattern common to message-queue systems generally.

Permanent failures feed a token-cleanup loop, not just a log entry: a push provider reporting a device token as invalid/unregistered is fed back to remove that token from the user's device record — otherwise every future notification to that user repeats the identical, guaranteed-to-fail attempt indefinitely, quietly wasting capacity on a channel that will never again succeed for that device until the app re-registers.

S. Deep Dive 7 — Priority & Delivery Guarantees

Requirement this addresses: low-latency delivery for critical notifications alongside best-effort delivery for everything else (D) — a single shared queue and worker pool cannot honor both at once, because a large marketing blast sitting ahead of an OTP in the same queue delays the OTP by however long the blast takes to drain.

Physically separate queues/topics per priority tier, each with its own worker pool: critical/transactional traffic and best-effort/marketing traffic are never colocated in the same queue, so a backlog on one tier has no mechanical way to delay the other. Each tier's worker pool is sized and alerted on independently — the critical tier's alerting fires on seconds of consumer lag; the best-effort tier's tolerates minutes.

Why this belongs in its own deep dive rather than folded into M: M established that queuing decouples ingest from delivery generally; this decides that "one queue" is the wrong shape once traffic has genuinely different latency requirements — the same "not every pattern applies uniformly" lesson the Patterns Playbook draws explicitly about hot-path offloading (Patterns Playbook C's payment-flow exception), applied here to fan-out instead.

T. Full Solution at Scale — Final Architecture

Everything above is the composition of M through S — no new component introduced here that wasn't already justified by a specific deep dive.

U. Design Evolution

What stays stable throughout: the small POST /notifications contract from I — every later refinement changes how a request is routed, throttled, or delivered, never the basic shape of what a calling service submits.

V. Consistency Model

This system is, by deliberate design, eventually consistent almost everywhere, with one narrow strong-consistency island — the same shape named generally in Patterns Playbook J. The dedup check (P) is that island: it must be a single atomic operation against a store that guarantees a check-and-claim never race, because the entire correctness guarantee of "no duplicate delivery for one idempotency key" collapses the moment two concurrent requests can both observe "not yet claimed" before either writes its claim — exactly Pattern B's check-then-act race, resolved the same way. Everything downstream of that one check is eventually consistent by default and deliberately so: preference updates (O) may take a short, bounded amount of time to propagate to a worker's cached view; delivery-status visibility (H) lags slightly behind the actual provider call; a digest batch's contents (Q) are, by definition, not "final" until the window closes. None of this looseness threatens correctness, because the one guarantee users and calling services actually depend on — "you won't get this exact notification twice" — lives entirely inside the one strongly-consistent check, not smeared across the rest of the pipeline.

W. Failure Scenarios

• A calling service retries a request after a timeout: the ingest-time dedup check (P) recognizes the repeated idempotency key and returns the original result without enqueueing a second delivery — the exact scenario V1 (J) had no defense against.
• A channel worker crashes mid-send: the queue's at-least-once delivery (Prerequisites G) redelivers the message to another worker; the worker-layer idempotency check (P) prevents that redelivery from becoming a second provider call if the first one had already gone through.
• A push provider (APNs/FCM) is fully degraded: provider failover (N) routes new sends to a configured secondary provider for that channel; if none is configured, that channel's queue backs up while other channels continue delivering normally, isolated by the priority/channel separation in place.
• A single user's activity spikes (many likes, comments, or events in a short window): per-user rate limiting caps raw send volume, and digesting (Q) consolidates the overflow into one notification instead of dropping it silently.
• A marketing campaign generates a massive best-effort burst: the priority-separated queues (S) contain the resulting backlog to the best-effort worker pool alone — the critical queue's latency is entirely unaffected.
• A device's push token has gone stale (app uninstalled, token rotated): the provider reports it as invalid on the next send attempt; the permanent-failure path (R) marks the attempt failed without retrying and triggers cleanup of the stale token, rather than repeating a guaranteed-failing call indefinitely.

X. Bottlenecks & Scaling Boundary

Y. Design Trade-Offs

Z. Common Bad Designs

• Sending synchronously, in the request path, directly to the provider. Couples the caller's latency to a third-party vendor's latency and leaves no buffer for any real burst — the single most important mistake to avoid on this problem.
• No idempotency key, or an idempotency key that isn't checked atomically. A caller's correct, standard retry behavior under an ambiguous failure silently becomes a duplicate notification to the user.
• Treating deduplication and rate limiting as the same mechanism. They solve two genuinely different problems — "did this already happen" versus "is this happening too much" — and conflating them tends to produce a design that does neither one correctly.
• One shared queue for every notification, regardless of priority. Guarantees that a large best-effort campaign will, sooner or later, delay a time-sensitive transactional notification behind it.
• Retrying every provider failure identically, with no classification. Wastes worker and provider capacity retrying destinations (a dead push token, an unsubscribed number) that will never succeed, no matter how many times they're tried.

AA. Interviewer Follow-Ups

AB. Interviewer Evaluation Rubric

• Weak candidate: sends notifications synchronously from the request path with no queue at all, and has no answer for what happens when a provider is slow or a caller retries; treats "call the push API" as the entire design.
• Mid-level: correctly reaches for an async queue and a basic idempotency key, but doesn't distinguish deduplication from rate limiting, and hasn't considered that different notification types have genuinely different latency requirements.
• Senior: proactively separates critical from best-effort traffic into different queues/SLAs; treats the provider integration as an adapter behind one internal interface rather than a hardcoded call; is explicit about where the one strongly-consistent check in the system lives and why it needs to be there.
• Staff+: raises digesting unprompted as a better answer than dropping overflow traffic; reasons concretely about a second idempotency check at the worker layer, not just at ingest, because at-least-once delivery shows up twice in the pipeline; discusses compliance (unsubscribe, opt-out keywords) as a real design constraint rather than an implementation detail to hand-wave past.

AC. Interview Simulation

AD. Online Solution Comparison

AE. Interview Cheat Sheet