SYSTEM DESIGN #08 · INTERVIEW GUIDE
Real-Time Anomaly Detection System
A metric can be broken for hours before anyone notices — and by then, a misconfigured campaign has spent $50,000 targeting the wrong audience. This Real-Time Anomaly Detection System watches every marketing KPI across every channel and fires an alert within 2 minutes of a statistically significant deviation. It combines EWMA smoothing (to handle natural variance) with Z-score thresholds (to catch real anomalies), applies 1,000 detection rules in a single Flink job, and deduplicates alerts in Redis so on-call engineers aren’t spammed every 30 seconds. The architecture covers seasonal decomposition for time-series with weekly cycles, auto-resolution when metrics recover, and multi-channel notification routing. After this guide you will know how to build an alerting system that wakes you up when it matters and stays silent when it doesn’t.
Apache FlinkEWMAZ-ScoreRedisPagerDuty
💡
The Gist — What Problem Are We Solving?
A smoke detector for your data pipelines — silent, always watching
The scariest failures in measurement pipelines are silent ones: no error message, just wrong numbers quietly flowing into reports. A tracking tag breaks on a Friday, nobody notices until Monday, and two days of data and decisions are corrupted. This system watches every stream 24/7 using statistical baselines to know that Sunday night dips are normal but Tuesday ones are not.
💬Think of it as a smoke detector for your data pipelines — silent, always watching, and very loud the moment something goes wrong.
These are the capabilities the system must deliver — what users and operators can actually do with it.
📉
Volume Anomaly
📉Detect event volume drops and spikes vs 7-day same-time baseline
📊
Rate Anomaly
📊Detect error rate, fraud rate, conversion rate deviations
🏷️
Tag Failure
🏷️Detect specific tag going silent or firing with wrong payload
⚙️
Rule Engine
⚙️1000 configurable rules with custom thresholds and conditions
🔔
Alert Routing
🔔Slack, PagerDuty, email; severity levels: INFO / WARNING / CRITICAL
⚡
Non-Functional Requirements
These define how well the system must perform — the quality attributes that separate a toy from a production system.
⏱️ Detection Latency
⏱️<5 minutes from anomaly occurrence to alert fired
🎯 False Positive Rate
🎯<1% — alert fatigue destroys effectiveness
📈 Throughput
📈500K events/sec scored in real time
📋 Rule Scale
📋1000 concurrent custom rules without performance degradation
🔕 Alert Dedup
🔕30-minute dedup window — same alert does not fire repeatedly
📊
Key Metrics — The Numbers That Define This System
The headline numbers to know cold — and be ready to explain how each one is achieved.
🏗️
System Architecture Diagram
Full data flow from source to serving. Each layer scales independently.
Ingestion
→
Flink
EWMA Z-score + Rule Engine
→
→
↓
Processing
Notification Router
Slack/PagerDuty) + Alert Store
→
🗺️
End-to-End User Journey
Trace a single request end-to-end — the story interviewers want you to tell fluently.
1
Tag breaks silently
— Deploy at 14:00 removes GA4 tag; checkout events drop from 12K/min to zero
2
Flink detects in 2 min
— Rule: event_count_1min < 10% of 7-day same-time average; Z-score = -4.2
3
Alert deduped and fired
— SHA256 fingerprint checked against Redis; new alert → CRITICAL to Slack + PagerDuty
4
Engineer reverts deploy
— Events resume at 14:18; volume returns to baseline
🔭
High-Level Design — Component Breakdown
Core components — each with a single, well-defined responsibility. The key architectural insight: each layer scales independently, and failure in one component is isolated from the rest.
1 — Kafka
Distributed event bus with RF=3 for durability. Partitioned by user_id_hash for per-user ordering. LZ4 compression reduces storage cost by 60%. Exactly-once semantics via idempotent producers and transactional consumers.
2 — Flink
Stateful stream processor with 30-second checkpointing to S3. KeyedProcessFunction provides per-key state (Union-Find, EWMA baseline, experiment assignment). Exactly-once processing via two-phase commit sink.
3 — Rule Engine
Handles responsibilities for the Rule Engine layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
4 — Dedup
Handles responsibilities for the Dedup layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
5 — Notifier
Handles responsibilities for the Notifier layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
🔬
Low-Level Design — Deep Dives
Deep dives worth explaining in detail in any senior engineering interview. For each: know the data structure, the algorithm, the why, and the trade-off you made.
1 — EWMA Anomaly Detector
Flink · Stateful · Z-Score
Flink KeyedProcessFunction keyed on metric_id. State: EWMA mean (μ) and variance (σ²) with α=0.1. On each metric data point: μ_new = α×value + (1-α)×μ_old; σ²_new = α×(value-μ_new)² + (1-α)×σ²_old. Z-score = (value – μ) / √σ². Threshold: Z>3 → alert. 14-day warm-up period required for reliable baseline. State checkpointed to S3 every 60 seconds for recovery.
class EWMADetector(KeyedProcessFunction):
def process(self, metric, ctx):
mu, var = self.state.value() or (metric.value, 0)
mu = ALPHA * metric.value + (1-ALPHA) * mu
var = ALPHA * (metric.value-mu)**2 + (1-ALPHA) * var
z = (metric.value – mu) / (var**0.5 + 1e-9)
self.state.update((mu, var))
if abs(z) > Z_THRESHOLD:
ctx.output(ALERT_TAG, Alert(metric, z))
2 — Deduplication Layer
Redis · 30min TTL
Alert deduplication prevents spam: Redis SETNX alert:{metric_id} with 30-minute TTL. If key exists, suppress alert. If key absent, set it and emit alert. Additionally, correlated metric grouping: pre-built correlation map (Pearson r>0.9 over 90 days) suppresses alerts for metrics correlated with an already-firing alert. Correlation map recomputed weekly. A grouped alert listing all correlated metrics is emitted instead of N separate alerts.
def should_alert(metric_id, correlated_ids):
# Check primary metric
if redis.setnx(f’alert:{metric_id}’, 1):
redis.expire(f’alert:{metric_id}’, 1800)
# Check correlated
for cid in correlated_ids:
redis.setex(f’alert:{cid}’, 900, 1)
return True
return False
3 — Notification Router
Slack · PagerDuty · Auto-Resolve
Severity routing: Z>5 or absolute threshold breach → PagerDuty P1. Z 3-5 → Slack #alerts. Recovery (Z<2 for 3 consecutive windows) → auto-resolve PagerDuty incident + Slack notification. Notification payload: metric name, current value, expected range, Z-score, chart link to Grafana, suggested runbook link. On-call rotation stored in PagerDuty — no hardcoded escalation paths.
def route_alert(alert):
msg = format_alert(alert)
if alert.severity == ‘CRITICAL’:
pagerduty.create_incident(msg)
slack.post(‘#alerts’, msg)
if alert.type == ‘RESOLVED’:
pagerduty.resolve_incident(alert.incident_id)
4 — Auto-Resolution Engine
Recovery Detection
After an alert fires, the auto-resolver monitors the metric for recovery. Recovery condition: Z-score below 2.0 (within 1σ of baseline) for 3 consecutive observation windows. On recovery: PagerDuty incident resolved, Slack thread updated, alert suppression key deleted from Redis. If metric remains anomalous for >4 hours, auto-escalates to P1 and pages secondary on-call. Recovery rate tracked in metrics to tune threshold sensitivity.
def check_recovery(alert, current_z):
recovery_count = redis.incr(f’recovery:{alert.id}’)
if current_z < 2.0:
if recovery_count >= 3:
resolve_incident(alert)
else:
redis.delete(f’recovery:{alert.id}’)
⚖️
Trade-offs & Decision Log
Every senior interview comes down to these decisions. Know the exact trade-off, the reasoning, and the specific numbers that justify each choice.
⚖️ EWMA vs Static Threshold Alerting
✓
EWMA Z-Score ✅ Chosen
- Adapts to metric baseline automatically — no manual threshold tuning
- Handles day-of-week seasonality with 7-day EWMA
- False positive rate <5% on well-tuned α parameter
- Requires warm-up period (14 days) before reliable alerts
→
Static absolute thresholds
- Simple to configure and explain
- Extremely high false positive rate on seasonal metrics
- Must be manually updated when traffic patterns change
- Cannot distinguish 10% drop on Black Friday from a real outage
💡Decision: EWMA Z-score (α=0.1, window=168h/7d) as primary detector; static min/max hard limits as safety net for critical metrics
⚖️ Streaming Detection vs Batch Anomaly Jobs
✓
Flink Streaming ✅ Chosen
- Detects anomalies within 2 minutes of occurrence
- Processes 1M metric data points/sec
- Stateful windowing for multi-dimensional correlations
- Higher infrastructure cost vs batch
→
Hourly batch jobs
- Simple to implement with standard SQL
- Detection lag: up to 60 minutes
- Budget wasted for up to an hour before alert fires
- Cannot correlate metrics across dimensions in real time
💡Decision: Flink streaming for all marketing KPIs; nightly batch for trend and seasonality model retraining
🎯Interview Questions — Answered
The exact questions interviewers ask — with production-grade answers
Q1
How is the Z-score threshold calibrated to minimise false positives?
Threshold calibration uses 90 days of historical metric data: (1) Compute EWMA residuals (actual – predicted) for each metric. (2) Fit a normal distribution to the residual series. (3) Set threshold at 3σ for informational alerts and 4σ for PagerDuty pages. This gives a false positive rate of 0.27% per metric per day at 3σ. With 1,000 metrics monitored, expected false alerts per day = 2.7 — acceptable. Metrics with fat-tailed distributions (e.g., revenue per user) use 4σ threshold. Thresholds auto-recalibrate weekly using the prior 90 days of residuals.
Q2
How does the system handle planned traffic changes (product launches, campaigns)?
Two mechanisms: (1) Scheduled suppression windows — marketing team submits a ‘suppress anomaly detection for metric X from 09:00-18:00 on launch day’ via internal API. Flink Anomaly job skips alert emission for suppressed windows. (2) Expected uplift registration — for campaigns expected to drive a 30% revenue spike, the expected value is registered. Anomaly detection then alerts if actual is <20% of expected uplift (under-performance) or >150% (over-performance indicating a potential tracking error). Both mechanisms are time-bounded and self-expire.
Q3
How are correlated metrics deduped to avoid alert storms?
Deduplication operates at two levels: (1) Per-metric deduplication — Redis key alert:{metric_id} with 30-minute TTL. If the same metric fires again within 30 minutes, alert is suppressed. (2) Correlated metric grouping — a pre-built correlation map (Pearson r>0.9 over 90 days) groups related metrics (e.g., impressions, clicks, and conversions for the same campaign). When one metric in a group fires, a 15-minute suppression window is applied to all correlated metrics. The notifier sends a single grouped alert listing all affected metrics. Groups are recalculated weekly.
System Design Series · Every Tuesday & Thursday
Level up your system design interviews
Each post covers Gist, Functional & Non-Functional Requirements, Key Metrics, System Diagram, User Journey, HLD, LLD, and Trade-offs & FAQs.
Subscribe to never miss a post →
Previous Articles
Categories: System Design
Tags: alerting, anomaly detection, ewma, interview prep, kafka, monitoring, real-time, system design
Leave a Reply