SYSTEM DESIGN #16 · INTERVIEW GUIDE
Design Instagram
Instagram serves 500 million daily active users, processes 100 million photo uploads per day, and delivers a personalised feed in under 500ms — all while running Stories that expire in 24 hours, Reels that need real-time transcoding, and a Explore page that surfaces content from accounts you’ve never followed. The architecture that makes this possible is a hybrid push-pull feed system: celebrities with 10M+ followers use pull-on-read (fan-in), while regular users use push-on-write (fan-out) — capped at 1,000 followers to bound write amplification. This guide covers the full picture: media upload and processing, CDN delivery, feed generation, Stories with guaranteed deletion, and the social graph that powers the Follow model. Everything you need to answer ‘Design Instagram’ end to end.
RedisCassandraKafkalibvipsCDN
💡
The Gist — What Problem Are We Solving?
A global photo album where 2 billion people share moments in real time
Instagram is one of the most demanding distributed systems ever built: 2 billion users, celebrities with 400 million followers each, photo and video delivery at CDN scale, news feeds that must load in under a second, and Stories that automatically disappear after 24 hours. The fan-out problem alone — delivering a Selena Gomez post to 400 million feeds — requires a completely different architecture than delivering a normal user’s post.
💬Think of it as a global bulletin board — every photo posted instantly appears on millions of personalised boards, then Stories clean themselves up after 24 hours.
These are the capabilities the system must deliver — what users and operators can actually do with it.
📸
Post Creation
📸Upload photo/video; apply filters; set caption, tags, location; post to feed
📰
News Feed
📰Personalised feed of posts from followed accounts; infinite scroll
📖
Stories
📖24-hour ephemeral stories; view receipts; reaction support
❤️
Engagement
❤️Likes, comments, shares, saves; notification on each
🔍
Search & Explore
🔍Hashtag and username search; Explore feed for discovery
⚡
Non-Functional Requirements
These define how well the system must perform — the quality attributes that separate a toy from a production system.
📤 Upload Scale
📤Millions of photos and videos posted per day
⏱️ Feed Load
⏱️<1s for feed to appear on app open
🌐 CDN
🌐All media served from CDN edge; zero origin hits for popular content
⏰ Stories TTL
⏰Exact 24-hour expiry; no stories served after expiry
🌍 Scale
🌍2B MAU; 400M+ DAU; global
📊
Key Metrics — The Numbers That Define This System
The headline numbers to know cold — and be ready to explain how each one is achieved.
Content-addressed
immutable URLs
🏗️
System Architecture Diagram
Full data flow from source to serving. Each layer scales independently.
Ingestion
Instagram Architecture
Upload
→
Media Processing
resize + transcode
→
→
↓
Processing
→
→
Redis feed cache
push for <1M followers) / Pull-on-read
→
↓
🗺️
End-to-End User Journey
Trace a single request end-to-end — the story interviewers want you to tell fluently.
1
User posts photo
— Client uploads to presigned S3 URL. Post metadata saved to Postgres. Kafka event published.
2
Media processing
— Resize to multiple resolutions (320, 640, 1080px). Strip EXIF. Generate thumbnail. Video transcoded to H.264 multiple bitrates. Content-addressed URL (SHA256 of content).
3
Feed fan-out
— For user with 1K followers: delivery service fans out post_id to all followers’ feed caches in Redis. For celebrity (400M followers): skip fan-out, merge at read time.
4
Follower opens feed
— Feed service: fetch pre-computed feed from Redis (normal users) + merge celebrity posts fetched on-demand. Rank by ML model. Return top 20.
5
Story posted
— Story object stored with 24h expiry timestamp in Redis sorted set. View receipts tracked per viewer in Cassandra.
🔭
High-Level Design — Component Breakdown
Core components — each with a single, well-defined responsibility. The key architectural insight: each layer scales independently, and failure in one component is isolated from the rest.
1 — Upload API
Handles responsibilities for the Upload API layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
2 — Media Proc
Handles responsibilities for the Media Proc layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
3 — CDN
Content Delivery Network serving static assets and media from 200+ PoPs. Cache-Control: immutable for media segments. Origin shield reduces origin load by 99%. Multi-CDN failover for resilience.
4 — Feed Service
Handles responsibilities for the Feed Service layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
5 — Redis
In-memory data structure server handling hot-path lookups in <1ms. Used for: canonical ID cache, rate limiting (token buckets), session state, feature store, and leaderboards. Cluster mode with 6 shards for horizontal scale.
6 — Stories
Handles responsibilities for the Stories layer. Designed for independent horizontal scaling — additional instances added without architectural changes. Communicates asynchronously with adjacent components to maximise throughput and fault isolation.
🔬
Low-Level Design — Deep Dives
Deep dives worth explaining in detail in any senior engineering interview. For each: know the data structure, the algorithm, the why, and the trade-off you made.
1 — Media Upload & Processing
S3 Presign · libvips Pipeline
Client receives pre-signed S3 PUT URL (15-minute TTL) via API. Client uploads directly to S3 — no bytes flow through application servers. S3 event notification triggers processing Lambda: resize to 5 variants (150px thumb, 480px, 1080px, 2160px, original), convert to WebP (Android) + HEIC (iOS) + JPEG (fallback), extract EXIF metadata, run NSFW classifier. Processing pipeline uses libvips (8× faster than ImageMagick). All variants written back to S3 with immutable CDN headers.
# S3 event → Lambda → libvips
def process_upload(s3_key):
original = s3.get_object(Key=s3_key)
img = pyvips.Image.new_from_buffer(original, ”)
for size, suffix in VARIANTS:
thumb = img.thumbnail_image(size)
s3.put_object(Key=f'{s3_key}_{suffix}.webp’,
Body=thumb.write_to_buffer(‘.webp’))
2 — Hybrid Feed Generator
Push <10K · Pull >1M followers
Push path: on post creation, Flink consumer reads follower list (Redis Set). For each follower: ZADD feed:{follower_id} timestamp post_id. Feed capped at 300 entries (ZREMRANGEBYRANK on overflow). Pull path: celebrities excluded from fan-out. At feed read time, last 10 posts from each followed celebrity fetched from post timeline (Redis ZREVRANGE) and merged with push feed. Final feed = merge_sort(push_feed, pull_results)[:50].
def generate_feed(user_id):
# Push feed from Redis
push = redis.zrevrange(f’feed:{user_id}’, 0, 49, withscores=True)
# Pull from celebrities
celebrities = get_celebrity_followings(user_id)
pull = merge_celebrity_posts(celebrities, limit=10)
return merge_sort(push, pull)[:50]
3 — Stories TTL Pipeline
Redis TTL · S3 Hard Delete
Story creation: metadata stored in Redis hash story:{id} with TTL=86400 (24h). S3 key stored in DynamoDB for deletion reference. On Redis keyspace expiry notification: Lambda triggered → S3 DeleteObject → DynamoDB record updated with deleted_at. CDN cache purged for story URLs. Viewer access gated on Redis TTL — pre-signed URL cannot be generated after TTL expires. Daily reconciliation: S3 inventory vs DynamoDB to find orphaned objects.
# Redis keyspace notification handler
def on_story_expiry(story_id):
meta = dynamodb.get_item(Key={‘story_id’: story_id})
s3.delete_object(Bucket=’stories’, Key=meta[‘s3_key’])
dynamodb.update_item(Key={‘story_id’: story_id},
UpdateExpression=’SET deleted_at = :now’,
ExpressionAttributeValues={‘:now’: int(time.time())})
4 — Social Graph Service
Redis Sets · Follow/Unfollow
Follow graph stored in Redis: following:{user_id} = Sorted Set of {followed_id: follow_timestamp}. followers:{user_id} = Sorted Set of {follower_id: follow_timestamp}. Follow: ZADD both sets. Unfollow: ZREM both sets. Follower count: ZCARD followers:{id}. Mutual follows (friends): ZINTERSTORE mutual:{a}:{b} following:{a} followers:{a}. Large accounts (>1M followers) persist follower list to Cassandra; Redis stores only the hot subset (last 100K followers).
def follow(follower_id, followed_id):
ts = time.time()
pipe = redis.pipeline()
pipe.zadd(f’following:{follower_id}’, {followed_id: ts})
pipe.zadd(f’followers:{followed_id}’, {follower_id: ts})
pipe.execute()
if is_celebrity(followed_id):
fanout_queue.skip(follower_id) # pull-based for celebrities
⚖️
Trade-offs & Decision Log
Every senior interview comes down to these decisions. Know the exact trade-off, the reasoning, and the specific numbers that justify each choice.
⚖️ Push vs Pull Feed Generation (Celebrity Problem)
✓
Hybrid: Push <1K followers, Pull >1M ✅ Chosen
- Regular users: fan-out on write — O(1) read, pre-cached feed
- Celebrities: fan-out on read — no write amplification (Beyoncé has 300M followers)
- Threshold at 10K followers separates 99.9% of accounts
- Feed generation at read time adds 50ms for pull users
→
Pure push (fan-out all writes)
- Uniform <10ms feed reads for all users
- 1 Beyoncé post = 300M Redis writes — kills write throughput
- Unacceptable write amplification for power users
- Cache invalidation at 300M entries is operationally infeasible
💡Decision: Push for accounts <10K followers; pull for >10K; Redis sorted set stores feed with score=timestamp for merge
⚖️ CDN Strategy for Media Delivery
✓
Multi-CDN with Smart Routing ✅ Chosen
- Anycast routes each request to lowest-latency PoP
- Automatic failover if CDN PoP degrades (TTFB >200ms)
- Different CDNs for different media types (images vs video)
- Higher cost than single CDN contract
→
Single CDN provider
- Simpler contract and operations
- Single CDN outage takes down all media globally
- No competitive pricing leverage
- Regional performance gaps in emerging markets
💡Decision: Primary CDN (Fastly/Akamai) for images; secondary CDN for video with adaptive bitrate; origin shield reduces origin load by 99%
🎯Interview Questions — Answered
The exact questions interviewers ask — with production-grade answers
Q1
How is the feed kept fresh for users who follow thousands of accounts?
Hybrid fan-out with staleness budget: (1) Push: for accounts <10K followers, new posts are written to each follower's Redis feed cache (sorted set, score=timestamp) at post time. (2) Pull: for accounts >10K followers, the post is not pushed — instead the feed service pulls the last 10 posts from each celebrity account at read time and merges with the pre-cached feed. (3) Freshness: Redis feed cache has a soft TTL of 5 minutes for online users — if the user requests feed and cache is >5min old, a background refresh is triggered. (4) Cache size limit: 300 posts per user in Redis feed — older posts are dropped (they appear in paginated history via Cassandra).
Q2
How do Stories disappear after 24 hours without a background job sweeping millions of records?
Stories expiry uses Redis TTL rather than a sweep job: each Story’s metadata is stored in Redis with TTL=86400 (24h). When the TTL expires, Redis keyspace notification fires a Pub/Sub event consumed by the expiry service, which deletes the S3 media object and removes the story from all viewer feeds. This is O(1) per expiry — no batch scan needed. Edge case: if the expiry service is down when TTL fires, the Redis key is gone but the S3 object remains. A daily reconciliation job (S3 inventory vs Story metadata DB) finds and deletes orphaned objects. Viewer access is gated on Redis TTL — once expired, the S3 pre-signed URL can no longer be generated, making media inaccessible even if the object temporarily persists.
Q3
How are photos compressed without visible quality loss?
Multi-format pipeline: (1) Upload: original stored in S3 at full resolution. (2) Processing: ImageMagick / libvips generates 5 variants: thumbnail (150px), low-res (480px), medium-res (1080px), high-res (2160px for Retina displays), and original. (3) Format selection: WebP for Android/Chrome (40% smaller than JPEG at same quality); HEIC for iOS Safari; JPEG fallback for older browsers. (4) Quality setting: SSIM-guided compression — quality parameter auto-tuned per-image to maintain SSIM ≥0.95 vs original, rather than a fixed quality=85. (5) CDN serve: Accept header determines format; Vary: Accept header ensures correct variant is cached per user-agent.
System Design Series · Every Tuesday & Thursday
Level up your system design interviews
Each post covers Gist, Functional & Non-Functional Requirements, Key Metrics, System Diagram, User Journey, HLD, LLD, and Trade-offs & FAQs.
Subscribe to never miss a post →
Previous Articles
Categories: System Design
Tags: feed fanout, instagram, interview prep, redis, social media, stories, system design
Leave a Reply