SYSTEM DESIGN #06 · INTERVIEW GUIDE
Scalable A/B Testing Platform
Most companies run A/B tests wrong: they peek at results early, ship winners that regress in production, and run experiments on overlapping user segments that invalidate each other’s results. A truly scalable experimentation platform eliminates all of these failure modes — it assigns users to variants in under 1ms using MurmurHash3, detects Sample Ratio Mismatch with a Chi-square test before any metric is trusted, and applies Sequential Probability Ratio Testing (SPRT) so you can stop experiments early without inflating false positive rates. This system is designed to run 10,000 simultaneous experiments across web, mobile, and server-side surfaces — the architecture used by Netflix, Airbnb, and LinkedIn. You will learn how to build the assignment service, the event pipeline, the stats engine, and the guardrail metrics that prevent a ‘winning’ feature from silently breaking revenue.
MurmurHash3SPRTKafkaClickHouseChi-Square
💡
The Gist — What Problem Are We Solving?
Running thousands of experiments simultaneously without them interfering
Want to test whether a red button converts better than a blue one — while running 9,999 other tests simultaneously? This system assigns each user to the right variant in under 5ms, ensures the same user always sees the same variant, and tells you with mathematical proof when a result is reliable enough to act on.
💬Think of it as a giant science lab that runs thousands of controlled experiments at once and tells you which ideas actually work.
These are the capabilities the system must deliver — what users and operators can actually do with it.
🔬
Experiment Management
🔬Create, configure, start, pause, stop experiments; define variants and metrics
⚡
Assignment Service
⚡Deterministic variant assignment in <5ms; same user always gets same variant
📊
Metrics Collection
📊Track assignment and conversion events per variant per experiment
📈
Stats Engine
📈Running t-test and SPRT for early stopping; p-value, CIs, lift calculation
🚨
Auto-stop
🚨Pause automatically on statistical significance or guardrail metric violation
⚡
Non-Functional Requirements
These define how well the system must perform — the quality attributes that separate a toy from a production system.
⚡ Assignment Latency
⚡<5ms p99 — must not slow down ad serving
📈 Throughput
📈100K assignment decisions/sec
🔒 Consistency
🔒Same user always sees same variant across sessions
🔀 Mutual Exclusion
🔀Users in at most 1 experiment per namespace
⏱️ Config Propagation
⏱️<30s from experiment change to all servers
📊
Key Metrics — The Numbers That Define This System
The headline numbers to know cold — and be ready to explain how each one is achieved.
10K
concurrent experiments
🏗️
System Architecture Diagram
Full data flow from source to serving. Each layer scales independently.
Ingestion
→
Config Store
Redis pub/sub
→
Assignment Service
MurmurHash3
→
↓
🗺️
End-to-End User Journey
Trace a single request end-to-end — the story interviewers want you to tell fluently.
1
Marketer creates experiment
— Defines: variants, traffic split 50/50, primary metric CTR, guardrail metric page-load time
2
User hits ad serving
— hash(user_id + experiment_id) % 10000; user gets variant based on bucket range
3
Consistent assignment
— Same user returns: same hash → same bucket → same variant always
4
Events tracked
— Impression + click tagged with exp_id and variant_id; published to Kafka
5
Stats engine runs
— Flink computes running z-test every 5 min; SPRT checks early stopping every 15 min
🔭
High-Level Design — Component Breakdown
Core components — each with a single, well-defined responsibility. The key architectural insight: each layer scales independently, and failure in one component is isolated from the rest.
1 — Experiment UI
Handles responsibilities for the Experiment UI layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
2 — Config Store
Handles responsibilities for the Config Store layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
3 — Assignment
Handles responsibilities for the Assignment layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
4 — Kafka
Distributed event bus with RF=3 for durability. Partitioned by user_id_hash for per-user ordering. LZ4 compression reduces storage cost by 60%. Exactly-once semantics via idempotent producers and transactional consumers.
5 — Stats Engine
Handles responsibilities for the Stats Engine layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
6 — Results DB
Handles responsibilities for the Results DB layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
🔬
Low-Level Design — Deep Dives
Deep dives worth explaining in detail in any senior engineering interview. For each: know the data structure, the algorithm, the why, and the trade-off you made.
1 — Assignment Service
MurmurHash3 · <1ms
Stateless assignment service. Given (user_id, experiment_id), computes variant assignment as: bucket = MurmurHash3(user_id + experiment_id) % 10000. Variant thresholds stored in Redis (e.g., control=0-4999, treatment=5000-9999 for 50/50 split). Redis read <1ms including network. Assignment logged to Kafka experiment_assignments topic. Sticky assignment: bucket is deterministic — same user always gets same variant for the same experiment.
import mmh3
def assign(user_id, experiment):
bucket = mmh3.hash(f'{user_id}{experiment.id}’) % 10000
for variant, threshold in experiment.thresholds:
if bucket < threshold:
return variant
2 — Metric Event Pipeline
Kafka · Flink · ClickHouse
User actions (page views, clicks, conversions, revenue) consumed from Kafka. Flink job joins events with assignment log by user_id and experiment_id. Joined events written to ClickHouse experiment_metrics table partitioned by experiment_id. SPRT computation runs as a Flink aggregate: counts and sums per variant updated in real time. Dashboard polls SPRT state API every 60 seconds.
events
.connect(assignments)
.keyBy(e->e.user_id, a->a.user_id)
.process(new ExperimentJoiner())
.sinkTo(clickhouseSink(‘experiment_metrics’))
3 — SPRT Stats Engine
Sequential Probability Ratio Test
SPRT computes likelihood ratio Λ = L(H1)/L(H0) after each new observation. Stop condition: Λ ≥ B=(1-β)/α → reject H0 (significant effect). Λ ≤ A=β/(1-α) → accept H0 (no effect). Otherwise continue. Parameters: α=0.05 (Type I error), β=0.2 (Type II error). B=19, A=0.0526. For conversion rate experiments: Λ updated as Bernoulli likelihood ratio. For continuous metrics (revenue): normal SPRT with known variance from pilot data.
def sprt_update(lambda_prev, x, p0, p1):
lr = (p1**x * (1-p1)**(1-x)) / (p0**x * (1-p0)**(1-x))
lambda_new = lambda_prev * lr
if lambda_new >= B: return ‘REJECT_H0’
if lambda_new <= A: return 'ACCEPT_H0'
return 'CONTINUE'
4 — SRM Detector
Chi-Square · Auto-Pause
Sample Ratio Mismatch detection runs every hour. Chi-square test: expected_control = total × split_control; observed_control = actual_assignments_control. χ² = Σ (observed-expected)²/expected. If χ² > critical value (p<0.001 threshold), experiment is auto-paused and an alert sent. Common SRM causes: caching serving stale variant, bot filtering applied to one variant only, SDK version differences. SRM is checked before any metric results are shared to prevent acting on contaminated data.
from scipy import stats
def detect_srm(observed, expected_ratio, alpha=0.001):
expected = [sum(observed) * r for r in expected_ratio]
chi2, p = stats.chisquare(observed, expected)
return p < alpha # True = SRM detected
⚖️
Trade-offs & Decision Log
Every senior interview comes down to these decisions. Know the exact trade-off, the reasoning, and the specific numbers that justify each choice.
⚖️ SPRT (Sequential) vs Fixed-Horizon Testing
✓
SPRT — Sequential Testing ✅ Chosen
- Can stop experiments early (2-3× faster average runtime)
- Controls false positive rate at any stopping point
- Handles optional stopping without inflating Type I error
- More complex to implement and explain to stakeholders
→
Fixed-Horizon (classical)
- Simpler — set sample size upfront, run to completion
- Well-understood by data scientists
- Peeking inflates false positive rate to 26%+ (vs 5% nominal)
- Slow — must wait for full sample even when result is obvious
💡Decision: SPRT with α=0.05, β=0.2 for most experiments; fixed-horizon for pre-registered regulatory/finance experiments
⚖️ Client-Side vs Server-Side Assignment
✓
Server-Side Assignment ✅ Chosen
- No flicker — user sees correct variant from first render
- Works for API and backend experiments
- Assignment persists across devices with same user_id
- Adds 10-20ms to server response time
→
Client-Side (JS SDK)
- Zero server-side latency impact
- Flicker on first load — user briefly sees control before variant
- Easy to implement with third-party tools
- Assignment cannot be trusted for server-computed metrics
💡Decision: Server-side for revenue/checkout/API experiments; client-side JS SDK only for pure UI cosmetic tests
🎯Interview Questions — Answered
The exact questions interviewers ask — with production-grade answers
Q1
How do you prevent experiments from interfering with each other (interaction effects)?
Three-layer isolation: (1) Orthogonal assignment — users hashed with experiment_id salt (MurmurHash3(user_id + experiment_id)) ensures independent uniform distribution per experiment. (2) Mutex groups — experiments that share the same UI surface are placed in a mutual exclusion group; a user can only be in one experiment per mutex group. (3) CUPED variance reduction — uses pre-experiment metric values as covariates to reduce noise, allowing detection of smaller effects with the same sample size. Interaction effects between experiments in separate mutex groups are measured post-hoc using regression analysis on the metric event log.
Q2
What is Sample Ratio Mismatch (SRM) and why does it matter?
SRM occurs when the observed split between control and treatment differs from the intended split (e.g., 51% / 49% instead of 50% / 50%). This indicates a systematic bug in assignment, logging, or filtering that makes the groups non-comparable — any metric difference is confounded by the selection bias. Detection: Chi-square test with p-value threshold of 0.001 (very conservative to avoid false SRM flags). Common causes: bots filtered from one variant only, page caching serving stale variant assignments, or logging failures in one variant. Any experiment with SRM detected is immediately paused and flagged for investigation before results are shared.
Q3
How long does an experiment need to run before results are trustworthy?
Minimum runtime is determined by three constraints: (1) Statistical power: sample size must reach the pre-calculated minimum based on MDE (minimum detectable effect), baseline conversion rate, α, and β. (2) Novelty effect: minimum 1 full week to avoid users behaving differently just because the UI changed. (3) Business cycles: minimum 1 full week to capture Mon-Sun variation (e.g., weekend shoppers behave differently). SPRT can signal a winner before the minimum runtime if the effect size is large — but the experiment continues to the minimum runtime anyway to confirm novelty effect has dissipated.
System Design Series · Every Tuesday & Thursday
Level up your system design interviews
Each post covers Gist, Functional & Non-Functional Requirements, Key Metrics, System Diagram, User Journey, HLD, LLD, and Trade-offs & FAQs.
Subscribe to never miss a post →
Previous Articles
Categories: System Design
Tags: a/b testing, experimentation, feature flags, interview prep, sprt, statistics, system design
Leave a Reply