SYSTEM DESIGN #07 · INTERVIEW GUIDE
Vendor Integration Hub
Every ad platform — Meta CAPI, Google Ads, TikTok Events API, Trade Desk — has its own schema, rate limits, retry semantics, and authentication protocol. Without a dedicated integration hub, each new vendor connection becomes a bespoke engineering project that breaks in production and takes months to debug. This system centralises all vendor connectivity behind a single internal API: it translates canonical events into vendor schemas using JSONata transformations, enforces per-vendor rate limits with token buckets, protects downstream services with Circuit Breakers, and routes failures to a Dead Letter Queue for reliable retry. You will learn how to build an integration layer that can onboard a new ad platform in under a day, handle 100K events/sec per vendor, and survive a Meta API outage without losing a single conversion signal.
KafkaCircuit BreakerJSONataDLQToken Bucket
💡
The Gist — What Problem Are We Solving?
One universal plug for sending data to 50+ ad platforms
Your team works with Meta, Google, TikTok, The Trade Desk, and dozens of others. Each has a different API, different data format, different rate limits, and breaks at different times. This system is the universal adapter: you send one canonical event and it transforms, routes, retries, and logs everything for every vendor — without any one vendor outage affecting the others.
💬Think of it as a universal translator and diplomat — it speaks every vendor’s language and never loses a message even if a vendor goes silent.
These are the capabilities the system must deliver — what users and operators can actually do with it.
🔌
Connector Registry
🔌Register, configure, enable/disable vendor connectors without code deploy
🔄
Schema Mapping
🔄Transform canonical internal event to each vendor’s required format
🔁
Retry and DLQ
🔁Exponential backoff retry (3 attempts); dead-letter queue for manual replay
🔒
Credential Management
🔒OAuth token rotation and API key management via Vault; automated refresh
⚡
Circuit Breaker
⚡Open circuit on 5 consecutive failures; half-open probe after 30s
⚡
Non-Functional Requirements
These define how well the system must perform — the quality attributes that separate a toy from a production system.
🛡️ Hub Uptime
🛡️99.9% independent of any vendor uptime
⏱️ Vendor Timeout
⏱️500ms timeout per vendor API call
🔁 Retry Policy
🔁3 attempts with exponential backoff: 1s, 2s, 4s
📈 Throughput
📈Handle 100K events/min across all vendors combined
🔄 Schema Hot-Reload
🔄Mapping rule changes deployed without service restart
📊
Key Metrics — The Numbers That Define This System
The headline numbers to know cold — and be ready to explain how each one is achieved.
🏗️
System Architecture Diagram
Full data flow from source to serving. Each layer scales independently.
↓
↓
Storage
Vendor API
after recovery); All calls
→
🗺️
End-to-End User Journey
Trace a single request end-to-end — the story interviewers want you to tell fluently.
1
Internal system fires conversion event
— Canonical event published to Kafka: {event_type, user_id_hash, value, currency, timestamp}
2
Hub worker picks up event
— Schema Mapper applies vendor-specific JSONata rule; OAuth token fetched from Vault
3
Meta API call attempted
— Circuit CLOSED: proceed. HTTP POST to Meta CAPI. 200 → success, logged to audit.
4
Meta returns 503
— Failure counter increments. After 5 failures: circuit OPENS. Event published to Meta DLQ.
🔭
High-Level Design — Component Breakdown
Core components — each with a single, well-defined responsibility. The key architectural insight: each layer scales independently, and failure in one component is isolated from the rest.
1 — Kafka
Distributed event bus with RF=3 for durability. Partitioned by user_id_hash for per-user ordering. LZ4 compression reduces storage cost by 60%. Exactly-once semantics via idempotent producers and transactional consumers.
2 — Schema Mapper
Handles responsibilities for the Schema Mapper layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
3 — Circuit Breaker
Handles responsibilities for the Circuit Breaker layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
4 — Vendor APIs
Handles responsibilities for the Vendor APIs layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
5 — Audit Log
Handles responsibilities for the Audit Log layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
🔬
Low-Level Design — Deep Dives
Deep dives worth explaining in detail in any senior engineering interview. For each: know the data structure, the algorithm, the why, and the trade-off you made.
1 — Canonical Event Schema
JSONata · Schema Registry
All internal events use a canonical schema: {event_id, event_type, canonical_user_id, campaign_id, timestamp, properties{}}. Vendor-specific fields mapped from canonical via JSONata expressions stored in schema registry. Schema versioning follows SemVer — breaking changes require new major version. Producers specify schema version in message header; consumer can handle multiple versions during migration.
// JSONata mapping for Meta Pixel
{
“data”: [{
“event_name”: event_type,
“event_time”: $floor(timestamp / 1000),
“user_data”: {“hashed_email”: canonical_user_id},
“custom_data”: properties
}]
}
2 — Rate Limiter
Token Bucket per Vendor
Per-vendor token bucket implemented in Redis Lua: each vendor has a bucket with capacity=max_burst and refill_rate=api_limit/second. Lua script atomically checks and deducts tokens. If bucket empty, event is held in Kafka consumer (paused partition) until tokens refill. Token bucket parameters loaded from vendor config: Meta CAPI = 1000/sec, Google Ads = 200/sec, TikTok = 500/sec.
— Redis Lua: token bucket
local tokens = tonumber(redis.call(‘GET’, key) or capacity)
if tokens >= cost then
redis.call(‘SET’, key, tokens – cost)
redis.call(‘EXPIRE’, key, 3600)
return 1
else
return 0
end
3 — Circuit Breaker
3-State · Auto-Recovery
Circuit breaker per vendor: states = {CLOSED, OPEN, HALF_OPEN}. CLOSED → OPEN: error rate >50% in 60s sliding window or 10 consecutive failures. OPEN → HALF_OPEN: after 30s cooldown, allow single test request. HALF_OPEN → CLOSED: test succeeds. HALF_OPEN → OPEN: test fails (reset cooldown). State stored in Redis for distributed consistency across multiple consumer instances. Events received while OPEN are immediately routed to DLQ.
class CircuitBreaker:
def call(self, vendor_fn, *args):
state = redis.get(f’cb:{self.vendor}’)
if state == ‘OPEN’:
raise CircuitOpenError()
try:
result = vendor_fn(*args)
self.record_success()
return result
except Exception as e:
self.record_failure()
raise
4 — Dead Letter Queue Processor
Retry · Alerting
DLQ Kafka topic receives events that failed after max retries. DLQ consumer retries with exponential backoff schedule: 1min, 5min, 15min, 1h, 4h, 24h. After 7 days without success, event is written to S3 cold storage and an alert fires. Daily reconciliation job compares DLQ event count against vendor-side receipt API — systematic gaps above 0.1% trigger investigation. DLQ dashboard shows per-vendor failure rates and error categories.
def process_dlq(event):
attempt = event.retry_count
delay = min(2**attempt * 60, 86400) # cap at 24h
if attempt > 7:
archive_to_s3(event)
alert(‘dlq_max_retries’, event)
return
schedule_retry(event, delay_seconds=delay)
⚖️
Trade-offs & Decision Log
Every senior interview comes down to these decisions. Know the exact trade-off, the reasoning, and the specific numbers that justify each choice.
⚖️ Synchronous API Calls vs Async Queue per Vendor
✓
Async Queue (Kafka + DLQ) ✅ Chosen
- Vendor outages are fully absorbed — events buffer in Kafka
- Rate limits enforced without blocking ingestion pipeline
- Independent retry with exponential backoff per vendor
- Slight delivery delay (seconds) vs synchronous
→
Synchronous HTTP per event
- Immediate delivery confirmation
- Single vendor outage causes backpressure into ingestion
- Rate limit violations cause dropped events
- Retry logic complex to implement correctly at scale
💡Decision: Async Kafka queue with per-vendor consumer groups; DLQ for events that fail after 5 retries
⚖️ Single Schema vs Canonical Event Model
✓
Canonical Internal Schema ✅ Chosen
- One event format for all upstream systems
- Vendor translation isolated to the integration hub
- Adding a new vendor requires only a new JSONata mapping
- Translation overhead: 1-3ms per event
→
Vendor-native schemas throughout
- Zero translation overhead
- Every upstream system must know every vendor’s schema
- Adding a vendor requires changes across all producers
- Schema drift causes silent mapping errors
💡Decision: Canonical schema with JSONata transformations per vendor; schema registry enforces contract versioning
🎯Interview Questions — Answered
The exact questions interviewers ask — with production-grade answers
Q1
How does the circuit breaker know when to open?
The circuit breaker monitors three metrics per vendor: (1) Error rate — opens if >50% of requests fail in a 60-second sliding window. (2) Response time — opens if p95 latency exceeds 5 seconds (vendor SLA). (3) Consecutive failures — opens after 10 consecutive failures regardless of window rate. State machine: Closed → Open (on threshold breach) → Half-Open (after 30s cooldown, allow 1 request) → Closed (if test request succeeds) or back to Open (if test fails). Events that arrive while circuit is open are written to Kafka DLQ immediately — no retry attempted until circuit closes. DLQ consumer retries with exponential backoff (1s, 2s, 4s, 8s, up to 256s).
Q2
How do you handle vendor API schema changes without downtime?
The JSONata transformation layer uses a schema registry (Confluent Schema Registry or internal equivalent) with semantic versioning. Each vendor has a versioned transformation config: {vendor: ‘meta’, event_type: ‘purchase’, schema_version: ‘2024-01’, mapping: {…}}. When Meta updates their API, a new schema_version is published to the registry. The transformation layer reads the config at runtime — no code deployment needed for schema changes. A/B testing of schema versions is supported: 10% of traffic routed to new schema version for validation before full cutover. Old schema versions retained for 90 days for rollback.
Q3
What is the retry strategy for failed vendor API calls?
Tiered retry strategy based on HTTP response code: (1) 429 Too Many Requests — respect Retry-After header; back off exponentially with jitter. (2) 5xx Server Error — retry up to 5 times with exponential backoff (base 2s, max 256s); after 5 failures, route to DLQ. (3) 4xx Client Error — do NOT retry (schema mismatch, invalid credentials); route directly to DLQ with alert. (4) Network timeout — retry immediately once, then exponential backoff. DLQ events are retained for 7 days. A daily reconciliation job compares DLQ event counts against vendor-side receipt APIs to detect systematic gaps.
System Design Series · Every Tuesday & Thursday
Level up your system design interviews
Each post covers Gist, Functional & Non-Functional Requirements, Key Metrics, System Diagram, User Journey, HLD, LLD, and Trade-offs & FAQs.
Subscribe to never miss a post →
Previous Articles
Categories: System Design
Tags: adtech, api gateway, circuit breaker, interview prep, resilience, system design, vendor integration
Leave a Reply