SYSTEM DESIGN #12 · INTERVIEW GUIDE
Design a Messenger
Building a messenger sounds simple until you face the hard questions: how do you deliver a message to a user who is offline on three devices? How do you guarantee ordering without a global lock? How do you encrypt messages end-to-end without the server ever seeing the plaintext? This system covers the full architecture: persistent WebSocket connections for online users, Kafka per-conversation partitioning for ordering guarantees, a Cassandra message store optimised for time-range reads, and FCM/APNs push notification fallback for offline delivery. The Signal Protocol double-ratchet encryption ensures forward secrecy — even if a server is compromised, past messages cannot be decrypted. After this guide you will know how to build a WhatsApp-scale messenger that handles 100B messages/day with sub-100ms delivery latency.
WebSocketSignal ProtocolCassandraFCM / APNsKafka
💡
The Gist — What Problem Are We Solving?
A real-time postal service that never loses a letter
When you send a message, it travels to a server that holds it safely until the recipient is online, then delivers it instantly. The engineering challenges: doing this for a billion people simultaneously across unreliable mobile networks, ensuring messages are delivered in order exactly once, and making sure nobody — not even the server operator — can read the messages.
💬Think of it as a postal service that delivers letters in milliseconds, never loses one, and can’t open them even if it wanted to.
These are the capabilities the system must deliver — what users and operators can actually do with it.
💬
1:1 Messaging
💬Send and receive text, images, files; delivery and read receipts
👥
Group Chat
👥Groups up to 1,000 members; fan-out delivery; member management
🔔
Push Notifications
🔔Notify offline users via FCM/APNs; no message content in payload
📱
Multi-Device
📱Same account on phone, tablet, desktop — all in sync
🔐
E2E Encryption
🔐End-to-end encryption via Signal Protocol; server never has plaintext
⚡
Non-Functional Requirements
These define how well the system must perform — the quality attributes that separate a toy from a production system.
⚡ Delivery Latency
⚡<100ms p99 for online recipients
🛡️ Durability
🛡️Zero message loss; at-least-once delivery with client dedup
🌍 Scale
🌍1B+ DAU; 100B+ messages/day
🔒 Encryption
🔒E2E: server sees only ciphertext
📱 Consistency
📱Same message history on all devices
📊
Key Metrics — The Numbers That Define This System
The headline numbers to know cold — and be ready to explain how each one is achieved.
30s TTL
presence heartbeat
🏗️
System Architecture Diagram
Full data flow from source to serving. Each layer scales independently.
Ingestion
Messenger Flow
Client App
→
→
→
↓
Processing
→
WebSocket Gateway
recipient
→
→
↓
🗺️
End-to-End User Journey
Trace a single request end-to-end — the story interviewers want you to tell fluently.
1
Alice sends message
— Client encrypts message with Bob’s public key (Double Ratchet); sends to WebSocket gateway
2
Gateway routes
— Message Service validates; assigns monotonic message_id; publishes to Kafka topic keyed by conversation_id
3
Bob is online
— Delivery Service looks up Bob’s gateway via presence service; pushes ciphertext to Bob’s WebSocket connection
4
Bob is offline
— Push notification sent via FCM/APNs — payload contains only ‘you have a new message’, no content
5
Bob comes online
— Client connects; requests missed messages since last_seen_id; gap-fill from Cassandra
🔭
High-Level Design — Component Breakdown
Core components — each with a single, well-defined responsibility. The key architectural insight: each layer scales independently, and failure in one component is isolated from the rest.
1 — Client App
Handles responsibilities for the Client App layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
2 — WS Gateway
Terminates WebSocket connections. Maintains connection registry in Redis. Routes incoming messages to target connection via gRPC. Scales horizontally with sticky routing via consistent hash on user_id.
3 — Msg Service
Handles responsibilities for the Msg Service layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
4 — Kafka
Distributed event bus with RF=3 for durability. Partitioned by user_id_hash for per-user ordering. LZ4 compression reduces storage cost by 60%. Exactly-once semantics via idempotent producers and transactional consumers.
5 — Delivery
Handles responsibilities for the Delivery layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
6 — Cassandra
Wide-column store optimised for time-range reads. Partition key = entity ID ensures co-location. Clustering order = timestamp DESC for efficient recent-N queries. Replication factor=3, consistency QUORUM writes / LOCAL_ONE reads.
🔬
Low-Level Design — Deep Dives
Deep dives worth explaining in detail in any senior engineering interview. For each: know the data structure, the algorithm, the why, and the trade-off you made.
1 — WebSocket Gateway
50K connections/server
WebSocket gateway manages persistent connections. Connection registry stored in Redis hash: user_id → {gateway_id, connection_id, last_seen}. On message send: lookup recipient’s gateway in Redis → forward via internal gRPC call to target gateway → gateway pushes to WebSocket. Gateway instances are stateless except for in-memory connection map (Redis as source of truth). Connection heartbeat every 30s; stale connections (no heartbeat 60s) evicted and registry updated.
class WSGateway:
connections: Dict[str, WebSocket] = {}
async def on_connect(self, user_id, ws):
self.connections[user_id] = ws
redis.hset(‘registry’, user_id, f'{GATEWAY_ID}:{id(ws)}’)
async def deliver(self, user_id, message):
if ws := self.connections.get(user_id):
await ws.send(message)
2 — Cassandra Message Store
Time-ordered · Per-conversation
Cassandra table: PRIMARY KEY ((conversation_id), sent_at DESC, message_id). Partition key = conversation_id ensures all messages in a conversation are co-located on the same node. Clustering order DESC means most recent messages retrieved first. Read pattern: SELECT * WHERE conversation_id = ? LIMIT 50 — single partition read, O(1) after index. Write: INSERT with TTL=7776000 (90 days) for media, unlimited for text. Replication factor=3 for durability; consistency level QUORUM for writes, LOCAL_ONE for reads.
CREATE TABLE messages (
conversation_id uuid,
sent_at timeuuid,
message_id uuid,
sender_id uuid,
ciphertext blob,
PRIMARY KEY ((conversation_id), sent_at, message_id)
) WITH CLUSTERING ORDER BY (sent_at DESC)
3 — Push Notification Fallback
FCM · APNs · Offline Queue
Offline delivery pipeline: WebSocket delivery attempt fails → check Redis registry (user offline) → write message to Mnesia offline queue (TTL=30 days) → publish push notification via FCM (Android) or APNs (iOS). Push payload: {message_count: N, conversation_id} — never message content (E2E encryption means server has no plaintext). App on foreground: fetches queued messages from API. Queue delivery in order: timestamp-ordered batch delivery on reconnect.
async def deliver_or_queue(recipient_id, message):
gateway = registry.get(recipient_id)
if gateway and await gateway.deliver(recipient_id, message):
return # online delivery
offline_queue.enqueue(recipient_id, message)
await push.notify(recipient_id, badge_count=offline_queue.count(recipient_id))
4 — Signal Protocol E2E Encryption
Double Ratchet · Forward Secrecy
Initial session: X3DH key agreement using recipient’s identity key, signed prekey, and one-time prekey from key server. Establishes shared secret without server involvement. Double Ratchet: each message derives a new message key from the ratchet chain — compromise of one key does not expose past or future keys. Group messaging via Sender Keys: one encrypt per message, O(N) delivery fan-out. Key server stores only public keys; private keys never leave devices.
# Session initialisation (X3DH)
shared_secret = x3dh(
sender_identity_key,
sender_ephemeral_key,
recipient_identity_key,
recipient_signed_prekey,
recipient_one_time_prekey
)
session = DoubleRatchet(shared_secret)
⚖️
Trade-offs & Decision Log
Every senior interview comes down to these decisions. Know the exact trade-off, the reasoning, and the specific numbers that justify each choice.
⚖️ WebSocket vs Long-Polling for Real-Time Delivery
✓
WebSocket (persistent TCP) ✅ Chosen
- True bidirectional — typing indicators, read receipts, presence
- Single connection per device — 50K connections/server at 2KB each
- <100ms message delivery for online users
- Requires sticky sessions or connection registry in Redis
→
Long-Polling (HTTP)
- Works behind restrictive firewalls and proxies
- Higher server CPU — new HTTP connection every 20-30s
- 3-5× higher latency than WebSocket
- Not suitable for typing indicators
💡Decision: WebSocket for all online delivery; FCM/APNs push for offline users; long-polling fallback only for enterprise firewall environments
⚖️ Fan-out on Write vs Fan-out on Read for Group Messages
✓
Fan-out on Write ✅ For groups <500
- Message pre-delivered to each member’s inbox
- O(1) read — no fan-out at query time
- Write amplification: 1 message → N inbox writes
- Bounded by group size limit of 1,024 in WhatsApp model
→
Fan-out on Read (for large groups)
- Single write per message regardless of group size
- Read requires resolving all recipients — O(N) at read time
- Better for channels with 10K+ subscribers
- Slightly higher read latency
💡Decision: Fan-out on write for groups <500 members; fan-out on read for broadcast channels >500 — avoids write amplification at scale
🎯Interview Questions — Answered
The exact questions interviewers ask — with production-grade answers
Q1
How are messages guaranteed to be delivered exactly once?
Exactly-once delivery uses a three-phase approach: (1) Producer idempotency: Kafka producer with enable.idempotence=true and transactions; each message has a unique message_id (ULID). (2) Consumer deduplication: Redis SET message_id with 24h TTL; if seen before, ack and discard. (3) Delivery acknowledgement: recipient device sends ACK with message_id; server marks message as delivered in Cassandra. If ACK not received within 60 seconds, server retries delivery via WebSocket (if online) or push notification (if offline). Messages are never deleted from Cassandra — soft-deleted with delivered_at timestamp.
Q2
How does end-to-end encryption work for group messages?
Signal Protocol’s Sender Keys mechanism: (1) When a user joins a group, they generate a SenderKeyDistributionMessage (SKDM) encrypted for each group member using their individual Signal sessions. (2) Subsequent messages are encrypted once with the group’s SenderKey — O(1) encryption regardless of group size. (3) When a member leaves, all remaining members rotate their SenderKeys (forward secrecy: departed member cannot decrypt future messages). (4) The server never sees message content — only encrypted blobs and routing metadata (sender_id, group_id, timestamp). Key server stores only public keys; private keys never leave devices.
Q3
How does the system handle a user who is offline for 30 days?
Messages are retained in the offline queue (Mnesia) for 30 days by default. After 30 days, messages are deleted and the sender is notified with a ‘message not delivered’ status. For media (images, video), content is stored in S3 with a 30-day expiry — a pre-signed URL is included in the message. If the recipient comes online within 30 days, all queued messages are delivered in a single bulk transfer ordered by timestamp. For large backlogs (>1000 messages), messages are delivered in pages of 100 with read-receipts confirming each page before the next is sent.
System Design Series · Every Tuesday & Thursday
Level up your system design interviews
Each post covers Gist, Functional & Non-Functional Requirements, Key Metrics, System Diagram, User Journey, HLD, LLD, and Trade-offs & FAQs.
Subscribe to never miss a post →
Previous Articles
Categories: System Design
Tags: cassandra, end-to-end encryption, interview prep, messenger, push notifications, system design, websocket
Leave a Reply