SYSTEM DESIGN #20 · INTERVIEW GUIDE
Design WhatsApp
WhatsApp reached 2 billion users with a backend team of 50 engineers — one of the greatest feats of infrastructure efficiency in software history. The secret is Erlang: a language built for fault-tolerant, massively concurrent systems where each connection is a lightweight actor consuming just 2KB of memory. This system covers the full WhatsApp architecture: persistent TCP connections via XMPP, the Signal Protocol for end-to-end encryption, consistent-hash routing to Erlang servers, Mnesia for offline message queuing, and FCM/APNs for push notification delivery when users are offline. The group messaging architecture shows how fan-out is handled for groups of up to 1,024 members without blowing out the message bus. After this guide you will know how to build a 2B-user messaging system with a fraction of the infrastructure most companies would assume is needed.
Erlang / OTPSignal ProtocolMnesiaFCM / APNsXMPP
💡
The Gist — What Problem Are We Solving?
100 billion messages per day — delivered, encrypted, then deleted
WhatsApp is famously lean: 900 million users on 50 engineers and ~100 servers at its peak. The secret is Erlang — a language built for telephone exchanges that handles millions of simultaneous connections as lightweight 2KB processes. Add Signal Protocol end-to-end encryption (server never sees message content), and a philosophy of deleting messages from the server as soon as they’re delivered.
💬Think of it as a telephone exchange that whispers — it connects you to your contacts instantly, never eavesdrops, and forgets the conversation the moment it’s delivered.
These are the capabilities the system must deliver — what users and operators can actually do with it.
💬
1:1 Messaging
💬Text, images, video, voice messages; delivery and read receipts
👥
Group Chat
👥Groups up to 1,024 members; efficient group encryption
📞
Voice/Video Calls
📞End-to-end encrypted calls; WebRTC-based
🔐
E2E Encryption
🔐Signal Protocol; server sees only: sender, recipient, timestamp, size
📱
Multi-Device
📱Up to 4 linked devices; all receive messages independently
⚡
Non-Functional Requirements
These define how well the system must perform — the quality attributes that separate a toy from a production system.
⚡ Throughput
⚡100B+ messages/day (~1.16M messages/sec)
🔒 Privacy
🔒Server never has message plaintext; minimal metadata
💾 Storage
💾Messages deleted from server after delivery; no server-side history
🌍 Scale
🌍2B+ monthly active users; global
⏱️ Delivery
⏱️<1s for online recipients; queued up to 30 days for offline
📊
Key Metrics — The Numbers That Define This System
The headline numbers to know cold — and be ready to explain how each one is achieved.
🏗️
System Architecture Diagram
Full data flow from source to serving. Each layer scales independently.
Ingestion
WhatsApp Architecture
Client
Erlang-based protocol
→
Connection Server
Erlang OTP, 2M conns/node
→
→
Mnesia/Cassandra
offline queue
↓
Processing
Delivery; E2E Encryption: Key Server
public keys
→
→
→
🗺️
End-to-End User Journey
Trace a single request end-to-end — the story interviewers want you to tell fluently.
1
Alice sends message to Bob
— Client encrypts message with Bob’s public key via Double Ratchet. Sends ciphertext to connection server over persistent TCP.
2
Message routed
— Message Router looks up Bob’s connection server via consistent hash. Bob is online → push ciphertext to Bob’s connection.
3
Bob online — delivery
— Bob’s client receives ciphertext; Double Ratchet decrypts; display message. Bob’s client sends delivery ACK.
4
Delivery ACK received
— Server receives ACK → delete message from server → send delivered receipt to Alice (double grey tick).
5
Bob reads message
— Bob opens conversation → client sends read ACK → Alice sees double blue tick.
🔭
High-Level Design — Component Breakdown
Core components — each with a single, well-defined responsibility. The key architectural insight: each layer scales independently, and failure in one component is isolated from the rest.
1 — Client
Handles responsibilities for the Client layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
2 — Connection Srv
Handles responsibilities for the Connection Srv layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
3 — Msg Router
Handles responsibilities for the Msg Router layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
4 — Mnesia Queue
Handles responsibilities for the Mnesia Queue layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
5 — Key Server
Handles responsibilities for the Key Server layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
🔬
Low-Level Design — Deep Dives
Deep dives worth explaining in detail in any senior engineering interview. For each: know the data structure, the algorithm, the why, and the trade-off you made.
1 — Erlang Connection Server
2KB/process · 1M conns/node
Each client connection is a separate Erlang process (gen_server behaviour). Process memory: ~2KB (stack + heap for idle connection). 1 million connections = 2GB RAM per node — highly efficient. Erlang scheduler multiplexes all processes over OS threads using cooperative preemption. Message passing between processes is zero-copy for immutable binaries (XMPP/Noise packets). Connection supervisor tree: supervisor restarts crashed connection processes automatically.
-module(connection_server).
-behaviour(gen_server).
init({Socket, UserId}) ->
gproc:reg({n, l, UserId}), % register for lookup
{ok, #state{socket=Socket, user_id=UserId}}.
handle_info({tcp, _, Data}, State) ->
Msg = noise:decrypt(Data, State#state.session),
router:route(Msg),
{noreply, State}.
2 — Signal Protocol Key Server
X3DH · One-Time Prekeys
Key server stores per-user: identity_key (long-term), signed_prekey (rotated weekly), one_time_prekeys[] (batch of 100, consumed one per new session). API: GET /keys/{user_id} → returns identity_key + signed_prekey + one one_time_prekey (removed from server). If one_time_prekeys exhausted: returns signed_prekey only (slightly lower security — fallback mode). Users notified when prekey count drops below 10 to re-upload.
def get_prekey_bundle(user_id):
identity_key = db.get(f’identity:{user_id}’)
signed_prekey = db.get(f’signed_prekey:{user_id}’)
otk = db.lpop(f’one_time_prekeys:{user_id}’) # consume one
return {
‘identity_key’: identity_key,
‘signed_prekey’: signed_prekey,
‘one_time_prekey’: otk # None if exhausted
}
3 — Message Router
Consistent Hash · Mnesia Queue
Message routing: (1) Sender Erlang process looks up recipient’s connection server via gproc (global process registry, O(1)). (2) If online: direct Erlang message passing (zero network hop for same-node, one inter-node hop for cross-node). (3) If offline: write to Mnesia offline queue (ETS-backed, RAM-resident). (4) On reconnect: Mnesia queue drained and delivered in order. Consistent hash ring (libketama) maps user_id → Erlang node for cross-node routing.
route_message(To, Msg) ->
case gproc:lookup_pid({n, l, To}) of
undefined ->
mnesia:write(#offline_msg{to=To, msg=Msg, ts=os:timestamp()});
Pid ->
Pid ! {deliver, Msg}
end.
4 — Group Message Fan-out
Sender Keys · O(1) Encrypt
Sender Keys protocol: group creator generates SenderKey (AES-256 + HMAC-SHA256). SenderKey encrypted individually for each member using their Signal session → distributed via key server. Group message: encrypt once with SenderKey → single ciphertext. Server routes single ciphertext to all member offline queues (fan-out at delivery, O(N) delivery writes). Member leaves: remaining members rotate SenderKey (batched if multiple leaves simultaneously) to ensure forward secrecy.
def send_group_message(group_id, plaintext, sender_key):
# Single encryption
ciphertext = sender_key.encrypt(plaintext)
# Fan-out delivery to all members
members = db.get_group_members(group_id)
for member_id in members:
queue_message(member_id, ciphertext) # O(N) writes, O(1) encrypts
⚖️
Trade-offs & Decision Log
Every senior interview comes down to these decisions. Know the exact trade-off, the reasoning, and the specific numbers that justify each choice.
⚖️ XMPP vs Custom Binary Protocol
✓
Custom Binary Protocol (Noise) ✅ Chosen
- 50% smaller message frames vs XMPP XML
- Noise Protocol Framework provides E2E encryption + authentication
- Optimised for mobile — minimal battery impact
- Requires custom client implementation on all platforms
→
XMPP (standard)
- Open standard — interoperable with other clients
- XML overhead: 10× larger than binary for simple messages
- No built-in E2E encryption in base XMPP
- Extensible — OMEMO extension adds E2E but at complexity cost
💡Decision: Custom Noise-based binary protocol for efficiency and E2E encryption; XMPP bridge available for business API integrations only
⚖️ Mnesia vs Cassandra for Message Queue
✓
Mnesia (Erlang native) ✅ For offline queue
- Co-located with Erlang process — zero network latency for reads
- Ideal for transient offline queues (retain until delivered)
- RAM-resident with disk persistence — microsecond access
- Limited to ~50GB per node — not for archive storage
→
Cassandra for message archive
- Linear scalability to petabytes
- Optimised for time-range scans (chat history)
- 100× higher latency than Mnesia for single-key lookups
- Requires separate cluster and ops team
💡Decision: Mnesia for online/offline delivery queue (TTL=30 days); Cassandra for persistent chat history; S3 for media older than 90 days
🎯Interview Questions — Answered
The exact questions interviewers ask — with production-grade answers
Q1
How does WhatsApp achieve 2 billion users with so few servers?
Erlang’s actor model is the key: each connection is a lightweight Erlang process consuming ~2KB of memory (vs ~1MB for a thread-per-connection model). 1 million connections per server × 2KB = 2GB RAM — easily achievable on a single commodity server. Erlang’s scheduler multiplexes millions of lightweight processes over OS threads using cooperative preemption — no context-switching overhead. Message passing between processes is zero-copy for immutable data. The result: 10 million concurrent connections per server cluster (WhatsApp runs ~few hundred servers for 2B users). This is 200× more efficient than a Java thread-per-connection model.
Q2
How does the Signal Protocol provide forward secrecy?
Double Ratchet Algorithm provides perfect forward secrecy: (1) Diffie-Hellman Ratchet: each message exchange generates a new DH key pair. Even if one session key is compromised, past messages (encrypted with previous keys) cannot be decrypted. (2) Symmetric-key Ratchet: derived keys advance a KDF chain — each message key is derived from the previous chain key using HMAC-SHA256. Once used, chain keys are deleted from device. (3) Prekey Bundle: initial session established via X3DH (Extended Triple Diffie-Hellman) using one-time prekeys from the key server. One-time prekeys ensure each new session has a unique shared secret — even if the long-term identity key is later compromised, sessions established with different one-time prekeys cannot be retroactively decrypted.
Q3
How is group message delivery handled at scale for 1,024-member groups?
Sender Keys protocol for group messaging: (1) Group creator generates a SenderKey (AES-256 + HMAC-SHA256 key pair). (2) SenderKey is encrypted individually for each group member using their Signal session and distributed via the key server. (3) Group messages are encrypted once with the SenderKey — O(1) encryption regardless of group size. (4) Server routes the single ciphertext to all members’ message queues — O(N) delivery but O(1) encryption. (5) Key rotation: when a member leaves, the remaining members generate a new SenderKey (forward secrecy — departed member cannot decrypt future messages). Rotation is batched if multiple members leave simultaneously.
System Design Series · Every Tuesday & Thursday
Level up your system design interviews
Each post covers Gist, Functional & Non-Functional Requirements, Key Metrics, System Diagram, User Journey, HLD, LLD, and Trade-offs & FAQs.
Subscribe to never miss a post →
Previous Articles
Categories: System Design
Tags: erlang, interview prep, messaging at scale, signal protocol, system design, whatsapp
Leave a Reply