Real-time Chat System Design: WhatsApp-like Architecture

This document presents a scalable backend architecture for real-time chat systems that handles millions of concurrent users with low latency and high reliability requirements typical of messaging platforms like WhatsApp.

1. Executive Summary

Clarifying questions

1. What is the expected read/write ratio for messages (e.g., 10:1, 50:1)?
2. What are the consistency requirements for message delivery (e.g., strong consistency vs. eventual consistency)?
3. Is this a global system or focused on a single region?
4. How sensitive is the system to latency (e.g., p99 under 200ms)?
5. What is the expected user base and message volume (e.g., 1M DAU, 100M messages/day)?
6. Are there any external integrations (e.g., SMS gateways, push notification services)?
7. What is the expected growth rate and timeline for scaling?

Problem being solved

Design a real-time chat system that supports instant messaging between users with features like message delivery, read receipts, typing indicators, group chats, and media sharing. The system must handle millions of concurrent users while maintaining low latency and high availability.

Back-of-envelope calculations

• DAU → Peak QPS: 1M DAU × 10 (peak-to-average ratio) × 86,400 seconds/day = ~8.6M peak QPS
• Storage per user/event: 500 bytes/user + 1KB/message × 100 messages/day = ~150KB/user/year
• Bandwidth:
  • Read: 8.6M QPS × 1KB = 8.6 MB/s
  • Write: 8.6M QPS × 1KB = 8.6 MB/s
• Cache hit rate: 90% → DB QPS = 8.6M × (1 - 0.9) = 860K QPS
• Server count: 860K QPS ÷ 10K QPS/server = ~86 servers at peak

SLA targets

• Availability: 99.9% (3.6s downtime/month)
• Latency: p50 < 100ms, p95 < 200ms, p99 < 500ms
• Durability: RPO < 1 minute, RTO < 5 minutes

Key architectural decisions summary

• Multi-region active-active deployment for global availability
• Event-driven architecture with Kafka for message streaming
• CQRS pattern for separate read/write models to optimize performance
• Redis-based caching with LRU eviction and cache warming strategy
• Service mesh (Istio) for secure inter-service communication
• PostgreSQL with read replicas and sharding for scalable data storage

2. System Goals & Non-Functional Requirements

Latency Targets

• p50: 200ms for message send/receive (critical path)
• p95: 500ms for message send/receive (critical path)
• p99: 1000ms for message send/receive (critical path)
• End-to-end chat session establishment: < 2s

Throughput Requirements

• Average QPS: 100,000 messages/sec (assumes 10M DAU with 100 messages/day per user)
• Peak QPS: 1M messages/sec (10x average, 86,400 seconds/day)
• Peak concurrent connections: 5M (assuming 500K concurrent users with 10 connections each)
• Message size: 1KB average (with 5KB max for media)

Availability Target

• 99.9% (3 9s) - Critical for real-time communication, user expectation
• RTO: < 5 minutes for any failure scenario (including regional outages)
• RPO: < 1 minute for message durability
• Justification: Real-time chat systems require high availability to maintain user engagement and prevent message loss

Durability and Consistency Requirements

• Consistency Model: Eventual consistency for message delivery (stronger for read-after-write)
• Durability: Messages must survive system failures and maintenance windows
• CAP Trade-off Decision: Prioritize Availability and Partition Tolerance over Consistency (AP in CAP)
• Message Ordering: Per-conversation ordering guarantees (per-partition in distributed systems)
• Conflict Resolution: Last-write-wins or vector clock-based conflict resolution for concurrent edits

Data Retention and Compliance Requirements

• Message retention: 2 years (compliance with financial services, healthcare)
• User data deletion: 30 days after account deactivation (GDPR compliance)
• Compliance: GDPR, HIPAA (if healthcare chat), SOC2 Type II
• Audit logging: Full audit trail of message operations (send, receive, delete)
• Data sovereignty: Messages stored in geographically compliant regions

Security and Regulatory Requirements

• End-to-end encryption: Required for message content (optional in HLD scope)
• Authentication: OAuth2/OIDC with MFA support for high-security channels
• Authorization: RBAC with role-based access control (admin, user, guest)
• Data in transit: TLS 1.3+ for all communication channels
• Data at rest: AES-256 encryption with key rotation every 90 days
• Compliance monitoring: Regular audits for GDPR, HIPAA, SOC2 compliance
• Vulnerability scanning: Automated security scanning in CI/CD pipeline (Snyk, Trivy)
• Access control: Rate limiting per user and IP address to prevent abuse

3. High-Level Architecture Diagram

C4Context
    title Real-time Chat System - Context Diagram

    Person(user, "End User")
    Person(admin, "Admin/User")
    
    System_Boundary(b1, "Chat Service") {
        System(chat_svc, "Chat Service", "Handles chat logic and message routing")
        System(auth_svc, "Auth Service", "Handles authentication and authorization")
        System(msg_svc, "Message Service", "Processes and stores messages")
        System(user_svc, "User Service", "Manages user profiles and status")
        System(notif_svc, "Notification Service", "Handles push notifications")
    }
    
    System_Boundary(b2, "Infrastructure") {
        System(nginx, "API Gateway", "Routes requests and handles auth")
        System(kafka, "Message Broker", "Handles async messaging")
        System(redis, "Redis Cache", "Caches user sessions and metadata")
        System(db, "Database Cluster", "Stores messages, users, and chat metadata")
        System(s3, "Object Storage", "Stores media files")
    }

    Rel(user, nginx, "1. Send messages")
    Rel(nginx, chat_svc, "2. Route to chat service")
    Rel(chat_svc, msg_svc, "3. Store messages")
    Rel(chat_svc, auth_svc, "4. Verify user")
    Rel(chat_svc, user_svc, "5. Get user info")
    Rel(msg_svc, kafka, "6. Publish to Kafka")
    Rel(kafka, chat_svc, "7. Process messages")
    Rel(chat_svc, notif_svc, "8. Send notifications")
    Rel(notif_svc, s3, "9. Store media")
    Rel(chat_svc, redis, "10. Cache user data")
    Rel(msg_svc, db, "11. Store messages")
    Rel(user_svc, db, "12. Store user data")
    Rel(admin, nginx, "13. Admin dashboard access")

C4Container
    title Real-time Chat System - Container Diagram

    Container_Boundary(c1, "Chat Service") {
        Container(chat_svc, "Chat Service", "Go/Java", "Handles chat logic and message routing")
        Container(msg_svc, "Message Service", "Go/Java", "Processes and stores messages")
        Container(user_svc, "User Service", "Go/Java", "Manages user profiles and status")
        Container(notif_svc, "Notification Service", "Go/Java", "Handles push notifications")
    }

    Container_Boundary(c2, "Infrastructure") {
        Container(nginx, "API Gateway", "Nginx", "Routes requests and handles auth")
        Container(kafka, "Message Broker", "Kafka", "Handles async messaging")
        Container(redis, "Redis Cache", "Redis", "Caches user sessions and metadata")
        Container(db, "Database Cluster", "PostgreSQL", "Stores messages, users, and chat metadata")
        Container(s3, "Object Storage", "S3", "Stores media files")
    }

    Rel(nginx, chat_svc, "REST/gRPC")
    Rel(chat_svc, msg_svc, "gRPC")
    Rel(chat_svc, user_svc, "gRPC")
    Rel(chat_svc, notif_svc, "gRPC")
    Rel(msg_svc, kafka, "Kafka Producer")
    Rel(kafka, chat_svc, "Kafka Consumer")
    Rel(chat_svc, redis, "Redis Cache")
    Rel(msg_svc, db, "PostgreSQL")
    Rel(user_svc, db, "PostgreSQL")
    Rel(notif_svc, s3, "S3 Upload")

    UpdateRelStyle(nginx, chat_svc, $offsetX=50, $offsetY=-20)
    UpdateRelStyle(chat_svc, msg_svc, $offsetX=30, $offsetY=20)
    UpdateRelStyle(chat_svc, user_svc, $offsetX=30, $offsetY=20)
    UpdateRelStyle(chat_svc, notif_svc, $offsetX=30, $offsetY=20)
    UpdateRelStyle(msg_svc, kafka, $offsetX=30, $offsetY=20)
    UpdateRelStyle(kafka, chat_svc, $offsetX=30, $offsetY=20)
    UpdateRelStyle(chat_svc, redis, $offsetX=30, $offsetY=20)
    UpdateRelStyle(msg_svc, db, $offsetX=30, $offsetY=20)
    UpdateRelStyle(user_svc, db, $offsetX=30, $offsetY=20)
    UpdateRelStyle(notif_svc, s3, $offsetX=30, $offsetY=20)

Major Components and Services

Core Services:

• Chat Service: Main service handling chat logic, message routing, and real-time communication
• Message Service: Processes and stores messages with persistence layer
• User Service: Manages user profiles, status information, and contact lists
• Notification Service: Handles push notifications for mobile devices

Infrastructure Components:

• API Gateway (Nginx): Routes requests, handles authentication, and rate limiting
• Message Broker (Kafka): Asynchronous messaging for message distribution and event processing
• Cache Layer (Redis): Session caching, user metadata, and real-time presence information
• Database Cluster: PostgreSQL for structured data storage (messages, users, chat metadata)
• Object Storage (S3): Media file storage (images, videos, documents)

Traffic Flow and Data Flow

1. User Authentication: End users authenticate through API Gateway to Auth Service
2. Message Processing: Chat messages flow through API Gateway → Chat Service → Message Service → Kafka
3. Real-time Delivery: Kafka consumers process messages and deliver to connected clients
4. User Data: User profiles and status information accessed through User Service
5. Notifications: Push notifications sent via Notification Service to mobile devices
6. Media Handling: Large files stored in S3 with metadata managed in database

External Integrations

• Mobile Push Notification Services: FCM (Firebase Cloud Messaging) or APNs
• Third-party Media Processing: Optional integration with media transcoding services
• Analytics Services: Integration points for usage analytics and metrics collection

Data Flow Patterns

The system follows a publish-subscribe model for real-time message delivery through Kafka, enabling horizontal scaling and loose coupling between services. The architecture supports both synchronous API calls for user operations and asynchronous event-driven processing for message distribution.

4. Service Architecture

Microservices vs. Monolith vs. Modular Monolith Decision

For a real-time chat system like WhatsApp, we'll adopt a modular monolith architecture as the initial approach with clear service boundaries that can evolve into independent microservices. This decision balances:

• Development speed: Faster iteration and deployment cycles
• Operational simplicity: Single deployment unit with shared infrastructure
• Scalability flexibility: Clear service boundaries for future horizontal scaling
• Complexity management: Avoids distributed system complexity during early stages

The modular monolith approach allows us to start with a single codebase but clearly delineated services that can be extracted later without breaking existing functionality.

Service Decomposition Strategy

We'll decompose the system into the following core services:

1. User Service - User management, profile information, contact list
2. Message Service - Message handling, delivery, read receipts, message history
3. Chat Service - Group chat management, chat metadata, chat lists
4. Presence Service - Online status, last seen information, typing indicators
5. Media Service - File storage, media processing, CDN integration
6. Notification Service - Push notifications, in-app alerts, webhooks

Each service will own its data and handle specific domain concerns while maintaining loose coupling through well-defined APIs.

Service Boundaries and Contracts

Each service will expose:

• REST/gRPC APIs for internal communication
• Event-driven interfaces for cross-service coordination
• Idempotent operations where possible to handle retries gracefully

Service contracts will follow semantic versioning principles with clear breaking change policies.

Inter-Service Communication Pattern

Synchronous (REST/gRPC):

• User ↔ Message service for message validation and user context
• Chat ↔ Message service for chat metadata operations

Asynchronous (Event-driven):

• All other inter-service communication will use event streams
• Message service publishes "message delivered" events to presence service
• Media service emits "media uploaded" events to chat service

Service Mesh Consideration

We'll implement a service mesh (Istio) for:

• Traffic management and routing
• Security policy enforcement (mTLS)
• Distributed tracing and observability integration
• Circuit breaker and retry policies

Service Registry and Discovery Strategy

Service Registry: Consul or Eureka for service registration and discovery Dynamic Service Discovery: Client-side load balancing with health checks

Services will register with the registry on startup and heartbeat every 30 seconds. Health checks will use liveness/readiness probes to detect service failures.

Service Design Principles

1. Stateless services where possible to enable horizontal scaling
2. Idempotent operations for message delivery and user actions
3. Async processing for non-critical operations (notifications, media processing)
4. Event-driven architecture for real-time updates and coordination
5. Service isolation to prevent cascading failures

Cross-Service Communication Patterns

• Request-Response: For user authentication, message retrieval
• Event Streaming: For real-time presence updates, message delivery confirmations
• Pub/Sub: For broadcast notifications and system events

Future Migration Path

The modular monolith design provides a clear migration path to independent microservices:

1. Extract User Service as standalone service
2. Move Message Service to independent deployment
3. Implement dedicated database per service
4. Replace shared state with event-sourced patterns

This approach allows for gradual migration while maintaining system stability and reducing risk during the transition.

5. Data Architecture

Database Selection per Service

User Service

• Database: PostgreSQL (primary) + Redis (caching)
• Justification: Relational model required for user profiles, contacts, and relationships. PostgreSQL offers strong consistency, ACID compliance, and mature ecosystem for user data operations.

Chat Service

• Database: MongoDB (primary) + Redis (caching)
• Justification: Document-based storage for chat messages with flexible schema evolution. MongoDB's geospatial features and aggregation pipeline support chat analytics and search capabilities.

Media Service

• Database: DynamoDB (metadata) + S3 (storage)
• Justification: DynamoDB for fast metadata lookup with consistent low-latency reads. S3 for actual media storage with cost-effective tiering and global distribution.

Notification Service

• Database: Redis (primary) + PostgreSQL (audit trail)
• Justification: Redis for high-throughput, low-latency notification queue with TTL support. PostgreSQL for durable audit logging and compliance.

Data Partitioning Strategy

Horizontal Sharding

• Strategy: Hash-based sharding on chat_id for chat messages and user conversations
• Justification: Ensures even distribution of data across shards while maintaining referential integrity for chat threads. Hashing on chat_id provides consistent routing and predictable performance.

Range Partitioning

• Strategy: Time-based partitioning for media metadata (by month/year)
• Justification: Efficient for time-series queries and allows for easy data lifecycle management (retention policies).

Data Replication Strategy

Primary-Replica Model

• Configuration: 3x replica sets for PostgreSQL and MongoDB
• Justification: Ensures high availability and read scalability. Primary writes are replicated asynchronously to maintain consistency while allowing for failover.

Cross-Region Replication

• Configuration: Active-active replication between regions (e.g., US-East, EU-West)
• Justification: Reduces latency for users in different geographic regions and provides disaster recovery capability.

Cross-Service Data Ownership

No Shared Databases Rule

• Enforcement: Each service owns its data and exposes only necessary APIs
• Implementation: Service-to-service communication via gRPC or message queues (Kafka) for cross-service operations

CQRS Pattern Application

Chat Service

• Read Model: Redis cache for real-time message delivery
• Write Model: MongoDB for message persistence and history
• Justification: Separates read and write concerns to optimize performance for real-time chat operations

Notification Service

• Read Model: Redis streams for notification delivery
• Write Model: PostgreSQL for durable storage and audit logging
• Justification: Enables high-throughput notification processing with guaranteed delivery

Event Sourcing Consideration

Use Case: Message history and audit trail

• Implementation: Events stored in Kafka topics with message IDs as primary keys
• Trade-offs:
  • Pros: Complete audit trail, system recovery capability, historical analysis
  • Cons: Increased storage requirements, complexity in querying past states

Object Storage Design

Storage Strategy: S3-compatible object storage (AWS S3/GCS)

• Access Patterns:
  • Media uploads: POST /media/upload
  • Media retrieval: GET /media/{id}
• Metadata Separation:
  • Metadata stored in DynamoDB (size, type, owner)
  • Content stored in S3 with versioning enabled

Storage Tiering Strategy

Hot/Cold Storage

• Hot Tier: Recent messages (last 30 days) stored in MongoDB with Redis caching
• Warm Tier: Messages older than 30 days moved to time-series database (InfluxDB) with reduced read frequency
• Cold Tier: Archive storage in S3 Glacier for long-term retention (1 year+)
• Justification: Balances cost with access frequency, reduces I/O costs for infrequently accessed data

Backup and Restore Strategy

Backup Configuration:

• RTO: 15 minutes (for critical services)
• RPO: 5 minutes (for user data)
• Backup Frequency:
  • Full backups: Daily
  • Incremental backups: Hourly
• Storage:
  • Cloud-native backup services (AWS Backup, GCP Backup)
  • Cross-region replication for disaster recovery
• Restore Process:
  • Automated restore testing weekly
  • Point-in-time recovery capability for last 30 days

Data Protection:

• Encryption: AES-256 encryption at rest and in transit
• Access Control: IAM policies with least privilege principle
• Audit Logs: Comprehensive logging of all data access and modification operations

6. Caching Architecture

Cache Layers Overview

The caching strategy follows a multi-tier approach to optimize performance and reduce database load:

Client → CDN → API Gateway → Application Cache → Database

1. Client-side caching - Mobile/web clients cache recent messages and user metadata
2. CDN layer - Global content delivery for static assets and pre-computed message metadata
3. API Gateway cache - Request-level caching for common read operations (user profiles, room metadata)
4. Application cache - In-memory (Redis) caching for session data and frequently accessed message threads
5. Database cache - Query result caching (PostgreSQL query planner cache)

Cache Strategy Implementation

L1: Client-side Caching

• Storage: Local storage (mobile) / browser cache (web)
• Eviction Policy: Time-based (30s-2min) + size-based (500MB limit)
• Use Cases: Recent messages, user avatars, read receipts
• Benefits: Reduces network overhead and improves perceived latency

L2: Distributed Cache (Redis)

• Primary Use: Session storage, message thread caching, user presence
• Cache Keys:
  • user:{userId}:presence → online/offline status
  • room:{roomId}:messages → recent messages (last 50)
  • user:{userId}:rooms → room membership list
• Eviction Policy: LRU with TTL (30s-24h)
• Replication: Redis Cluster with 3 master nodes, 3 replicas per shard
• Memory Management: Memory eviction policies (allkeys-lru) with cache warming

L3: Application-Level Caching

• Implementation: In-memory caches in each service instance (Ehcache/ConcurrentHashMap)
• Cache Scope: Per-service, not shared across instances
• Use Cases:
  • User session data (user metadata, permissions)
  • Room metadata (names, member lists, read status)
  • Message content for active threads
• Eviction: Time-based (5-10 minutes) + size limits

Cache Invalidation Strategy

Event-Driven Invalidation

• Primary Mechanism: Publish cache invalidation events via message broker (Kafka)
• Invalidation Triggers:
  • Message sent → invalidate room message cache
  • User status change → invalidate user presence cache
  • Room metadata update → invalidate room cache
• Consistency Model: Eventual consistency with cache TTLs as safety net

Cache Warming Strategy

• Cold Start Handling:
  • Pre-warm cache for popular rooms (top 1000 by activity)
  • Warm user session caches during login flow
• Background Refresh:
  • Scheduled cache refresh for frequently accessed rooms
  • Async warm-up on cache miss with fallback to DB read

Cache Performance Targets

• Hit Rate: 85%+ for application cache, 95%+ for CDN
• Latency:
  • L1 (client): <50ms
  • L2 (Redis): <10ms p95
  • L3 (App): <5ms p95
• QPS:
  • Redis cache: 100K+ QPS (peak)
  • CDN cache: 500K+ QPS (peak)

Cache Consistency Trade-offs

Read-Through vs Write-Through

• Read-Through: Cache miss → DB read → cache populate → return data
• Write-Through: Write operations → DB + cache update (sync)
• Decision: Read-through for most operations, write-through for critical user presence updates

Cache Stampede Prevention

• Cache Locking: Distributed locks using Redis SET with NX/EX pattern
• Stale-While-Revalidate: Return stale cache while background refresh occurs
• Cache TTL Strategy: Randomized TTLs to prevent simultaneous cache expiration

Cache Size and Scaling Considerations

Memory Allocation

• Per Instance: 2GB Redis cache per instance (300M messages * 8 bytes avg)
• Cluster Size: 12 Redis instances (4 shards × 3 replicas) for peak load
• Memory Optimization:
  • String compression (Zstandard)
  • Object serialization (Protocol Buffers vs JSON)
  • TTL-based automatic cleanup

Horizontal Scaling

• Sharding Strategy: Hash-based sharding by room ID or user ID
• Cross-Service Coordination: Cache invalidation events across services via Kafka
• Cache Partitioning:
  • Room-level partitioning for message caches
  • User-level partitioning for session caches

Cache Monitoring and Alerting

Key Metrics

• Cache hit ratio (per layer)
• Cache latency distribution (p50, p95, p99)
• Cache miss rate and reasons
• Memory usage and eviction rates

Alerting Strategy

• Critical: Cache hit ratio drops below 80%
• Warning: Cache latency exceeds 50ms p95
• Info: Cache warming operations > 10% of total requests

Cache Performance Optimization Techniques

Batch Operations

• Message Fetching: Batch room message fetches to reduce round trips
• User Presence: Batch presence updates for multiple users in same room

Prefetching

• Room Preload: Fetch recent messages for rooms user is likely to access next
• User Metadata: Pre-fetch user profiles for room members when entering chat

Compression and Serialization

• Protocol Buffers: 30% smaller payloads than JSON for structured data
• Compression: GZIP compression for large message batches (50+ messages)
• TTL Management: Dynamic TTL adjustment based on usage patterns

This caching architecture balances performance, consistency, and cost while supporting the real-time nature of chat communications. The multi-layer approach ensures optimal performance for both frequent and infrequent operations while maintaining system reliability under high load conditions.

7. Messaging & Event Streaming Architecture

Message Broker Selection

For a real-time chat system, we'll use Apache Kafka as our primary message broker with the following rationale:

• High throughput: Kafka's design handles millions of messages per second efficiently
• Durability: Messages are persisted to disk and can be retained for configurable periods
• Scalability: Horizontal scaling capabilities meet our expected growth patterns
• Stream processing: Kafka Streams and KSQL support complex message transformations
• Ecosystem maturity: Strong tooling ecosystem (Confluent Control Center, kafkactl, etc.)
• Producer/consumer model: Excellent fit for chat system's publish-subscribe pattern

Event Schema Registry and Versioning

We'll implement the Confluent Schema Registry for:

• Schema evolution: Version control of event schemas with backward/forward compatibility rules
• Type safety: Ensures message format consistency across services
• Performance: Reduced payload size through schema ID references instead of full schemas
• Governance: Centralized management of message contracts

Topic Partitioning and Consumer Group Design

Partitioning Strategy:

• User-based sharding: Messages are partitioned by user_id to ensure all messages for a user are in the same partition
• Partition count: 128 partitions per topic to support high-concurrency scenarios with adequate parallelism
• Replication factor: 3 for fault tolerance and availability

Consumer Groups:

• Chat service consumers: Handle message delivery to connected clients
• Notification service consumers: Process read receipts, typing indicators, and presence events
• Analytics consumers: Aggregate chat metrics for business intelligence

Event Ordering Guarantees

Per-partition ordering:

• Messages within a partition maintain strict ordering
• This ensures chat message sequence integrity for each user
• Critical for maintaining conversation flow and read receipts

Global ordering:

• Not required for chat messages (per-user ordering is sufficient)
• Avoids unnecessary complexity and performance overhead

Exactly-once vs. At-least-once Processing Trade-offs

Decision: At-least-once processing with idempotent consumers

• Why not exactly-once: Kafka's exactly-once semantics (EOS) add significant complexity and performance overhead that isn't justified for chat systems where duplicate messages are tolerable
• At-least-once benefits:
  • Simpler implementation and maintenance
  • Better performance characteristics
  • Easier to reason about failure scenarios
• Idempotent consumers:
  • Implemented at the consumer level to handle duplicates gracefully
  • Use message IDs to deduplicate incoming events

Event Replay and Reprocessing Strategy

Replay capability:

• Kafka's retention policy (7 days) enables event replay for debugging and reprocessing
• Use cases:
  • Debugging message processing issues
  • Recomputing analytics from a specific point in time
  • Recovering from data loss scenarios

Reprocessing pipeline:

• Tooling: Kafka Connect with custom connectors for historical data reprocessing
• Governance:
  • Versioned event schemas to support backward compatibility during reprocessing
  • Audit logs for tracking reprocessing activities

Dead Letter Queue and Poison Pill Handling

DLQ Strategy:

• Dead letter topics: Dedicated Kafka topics for messages that fail processing after retries
• Retry mechanism:
  • Max 3 retries with exponential backoff (1s, 2s, 4s) before routing to DLQ
  • Retry logic implemented in consumer services with circuit breaker pattern

Poison pill handling:

• Detection: Failed messages that exceed retry limits are routed to DLQ
• Monitoring:
  • Alerting on DLQ queue size and processing delays
  • Manual intervention capability for problematic messages
• Recovery:
  • DLQ consumers can process failed messages with updated logic or manual overrides

Message Format and Serialization

Serialization format:

• Avro with Schema Registry: Provides schema evolution, efficient binary encoding, and strong typing
• Compression: Snappy compression to reduce bandwidth usage while maintaining performance

Message payload structure:

{
  "message_id": "uuid",
  "user_id": "string",
  "conversation_id": "string",
  "timestamp": "iso8601",
  "type": "message|read_receipt|typing_indicator|presence",
  "content": "json_payload",
  "source": "client|server"
}

Cross-service Communication Patterns

Event-driven architecture:

• Internal events: Chat messages, read receipts, typing indicators are published as Kafka events
• External integrations:
  • Webhooks for external chat system integrations (e.g., Slack, WhatsApp Business API)
  • Event-driven microservices communicate through Kafka topics

Service interaction flow:

1. Message creation → Chat service publishes to messages topic
2. Message delivery → Chat service consumes from messages topic and forwards to connected clients
3. Read receipts → Notification service consumes read_receipts topic and updates message status

Performance Considerations

Throughput targets:

• Peak QPS: 50,000 messages/sec (assuming 10:1 peak-to-average ratio)
• Partition distribution: 128 partitions to support concurrent processing
• Consumer parallelism: 8 consumers per topic for optimal throughput

Latency targets:

• End-to-end: < 200ms for message delivery to connected clients
• Processing: < 50ms per message in consumer services

Resource allocation:

• Kafka cluster: 3 brokers in a multi-AZ configuration for high availability
• Topic retention: 7-day window to balance storage costs with replay capability

8. Reliability Engineering

Failure Mode Analysis

Single Points of Failure:

• Message Broker: Kafka cluster failure could halt message processing
• Primary Database: Single point for all chat data writes
• Authentication Service: Could block all user sessions and message sending
• API Gateway: Centralized request routing point that could cause cascading failures
• Load Balancer/Ingress Controller: Could prevent traffic from reaching services

Service-Level Failures:

• User Session Service: Could cause message delivery failures for users with stale sessions
• Push Notification Service: Could lead to missed messages in real-time scenarios
• File Storage Service: Could prevent media sharing capability

Fault Tolerance Patterns

Circuit Breaker Strategy:

• API Gateway: Implements circuit breaker for downstream services (e.g., message store, push service)
• Service-to-Service: Each microservice uses circuit breaker for external dependencies (e.g., auth, storage)
• Timeout Configuration: 200ms for internal calls, 1s for external services

Bulkhead Isolation:

• Message Processing: Separate thread pools per chat room or user group to prevent queue blocking
• Database Connections: Connection pooling with limits per service (max 50 concurrent DB connections)
• External Service Calls: Isolated thread pools for file storage, push notifications, and auth

Retry Strategy:

• Idempotent Operations: All message writes are idempotent to allow safe retries
• Exponential Backoff: 100ms → 500ms → 2s → 5s with jitter for retry attempts
• Max Retry Limit: 3 retries before sending to dead letter queue

Multi-Region / Multi-AZ Strategy

Active-Active Design:

• Primary Region: Handle 100% of traffic with read replicas for low-latency reads
• Secondary Region: Handle 50% of traffic with read-only replicas for failover capability
• Cross-Region Replication: Use Kafka for cross-region message replication with eventual consistency

Traffic Splitting:

• DNS-Based Routing: Route traffic to primary region with health checks (50% primary, 50% secondary)
• Service Mesh: Use Istio to implement cross-region traffic management and failover
• User Affinity: Maintain user session affinity across regions using consistent hashing

Chaos Engineering Strategy

GameDay Exercises:

• Database Failures: Simulate primary DB outages to test read replicas and failover
• Network Partitioning: Isolate services to test circuit breaker behavior and data consistency
• Service Degradation: Simulate slow database or auth service to test timeout and retry logic

Steady-State Hypotheses:

• Message Delivery: Ensure all messages are delivered within 2 seconds of sending
• User Availability: Maintain 99.9% uptime for active users during regional outages
• Data Consistency: Ensure no message loss or duplication across region failovers

Disaster Recovery Plan

RTO/RPO Targets:

• RTO: 5 minutes for primary region failover (user impact minimal)
• RPO: 10 seconds for message data consistency (message loss < 10s)
• Recovery Time: 30 minutes for complete regional recovery (including data sync)

Recovery Procedures:

1. Automatic Failover: When primary region becomes unavailable, redirect traffic to secondary
2. Data Sync: Use Kafka replication to sync data between regions
3. User Session Migration: Maintain session state through Redis cluster with cross-region replication

Runbook Design for Top-5 Failure Scenarios

Scenario	Detection	Mitigation Steps	Recovery Time
Database Outage	DB connection timeout alerts	1. Switch to read replicas 2. Enable write failover 3. Alert DB team	5 mins
Push Service Failure	Message delivery latency > 30s	1. Route to fallback notification service 2. Retry with exponential backoff 3. Alert push team	10 mins
Authentication Service Down	Auth token validation failures	1. Use cached auth tokens 2. Implement fallback to local session cache 3. Alert auth team	15 mins
Kafka Cluster Failure	Message processing lag > 1 min	1. Use local message queue 2. Retry with backoff 3. Alert messaging team	20 mins
Network Partition	Service communication timeout	1. Enable circuit breaker 2. Use local cache for reads 3. Implement retry with timeout	5 mins

Monitoring & Alerting:

• Critical Alerts: Immediate PagerDuty page for database, auth, and messaging failures
• Warning Alerts: Slack notifications for latency spikes or retry thresholds exceeded
• Health Checks: Every 30 seconds for all critical services with automatic failover triggers

9. Observability Architecture

Logging Stack

• Structured JSON logging with consistent log format across all services using structured fields (timestamp, service, level, trace_id, span_id, message)
• Loki as the centralized log store with Promtail agents for log collection and forwarding
• Grafana dashboard for log visualization and search with multi-service query capabilities
• Log retention policy: 30 days for operational logs, 90 days for audit logs
• Log filtering: Exclude noisy debug logs in production but include them in staging

Distributed Tracing

• OpenTelemetry as the tracing framework with automatic instrumentation for HTTP/gRPC services and manual instrumentation for business logic
• Jaeger as the tracing backend with sampling rate of 1% for production and 100% in staging
• Trace context propagation through HTTP headers (W3C TraceContext) and gRPC metadata
• Service map visualization to identify performance bottlenecks across service boundaries
• Error correlation: Traces linked to error logs for root cause analysis

Metrics Stack

• Prometheus as the primary metrics store with 15s scrape interval for critical services
• Grafana for dashboard creation and alerting configuration with pre-built chat system dashboards
• Service-level metrics: QPS, latency (p50/p95/p99), error rate, throughput, cache hit ratio
• Infrastructure metrics: CPU, memory, disk I/O, network utilization per service container
• Business metrics: Active users, message volume, session duration, delivery success rate

Alerting Strategy

• Severity levels:
  • P0: Critical (system down, data loss) - PagerDuty with SMS alerts
  • P1: High (performance degradation, high error rate) - Slack + PagerDuty
  • P2: Medium (warning thresholds, maintenance windows) - Slack only
• Alerting rules:
  • Message delivery latency > 500ms (P1)
  • Service availability < 99.9% (P0)
  • Cache hit rate < 80% (P2)
• Alert routing: Service owner teams get direct alerts, cross-service issues go to platform team
• Silence rules: Auto-silence during scheduled maintenance windows

SLI/SLO Definition and Error Budget Tracking

• SLIs:
  • Message delivery success rate (99.9% SLO)
  • Message delivery latency (p50 < 200ms, p95 < 1s)
  • Service availability (99.9% SLO)
  • User session duration (95th percentile > 30 minutes)
• SLOs:
  • 99.9% availability for chat service (5m downtime/month max)
  • 99.9% message delivery success rate (0.1% error budget)
• Error budget burn tracking: Dashboard showing remaining error budget with auto-alerts when budget is >80% consumed
• Steady-state hypothesis: System maintains message delivery consistency under load and failure scenarios

Capacity Planning and Load Forecasting

• Historical metrics analysis using Prometheus to identify usage patterns and seasonal trends
• Load forecasting model: Machine learning model trained on historical traffic data to predict peak loads (e.g., 30% increase during business hours)
• Capacity planning dashboard: Real-time view of resource utilization vs. forecasted demand with auto-scaling recommendations
• Proactive scaling triggers:
  • CPU utilization > 80% for 5 minutes → auto-scale up
  • Message queue depth > 10k → alert and potential horizontal scaling
• Resource allocation:
  • CPU: 100m for idle services, 500m-2000m for high-load services
  • Memory: 128Mi-512Mi for idle, 1Gi+ for message processing services

10. API Gateway & Edge Layer

API Gateway Selection

For a real-time chat system like WhatsApp, I recommend AWS API Gateway as the primary API gateway with custom edge services for specific high-throughput scenarios.

Justification:

• Managed service benefits: AWS API Gateway provides built-in rate limiting, authentication, and request/response transformation with minimal operational overhead
• Integration with AWS ecosystem: Seamless integration with Lambda, ECS, and other AWS services
• Real-time capabilities: Supports WebSocket connections for chat message delivery
• Global distribution: Built-in support for multi-region deployments with CloudFront integration
• Cost efficiency: Pay-per-request model with automatic scaling

Request Routing Strategy

The API gateway will implement a multi-tier routing strategy:

1. WebSocket connections: Direct routing to chat service via WebSocket protocol
2. REST APIs: Route to appropriate microservices (user, message, group) based on endpoint patterns
3. Static content: Direct routing to S3 or CloudFront for media files and assets

graph LR
    A[Mobile/Web Clients] --> B(API Gateway)
    B --> C[WebSocket Connections]
    B --> D[REST Endpoints]
    D --> E[User Service]
    D --> F[Message Service]
    D --> G[Group Service]

Authentication and Authorization

Authentication Flow:

• JWT tokens for mobile/web clients (signed with RS256)
• OAuth2 integration with AWS Cognito for user identity management
• mTLS for service-to-service communication

Authorization:

• JWT claims validation (user_id, role) at API gateway level
• RBAC implementation for chat group operations
• Token refresh and revocation handling

Rate Limiting and Throttling Design

Global Rate Limits:

• 1000 requests/second per client for REST endpoints
• 5000 messages/minute per user for WebSocket connections

Per-Client Limits:

• 5000 requests/hour per mobile device
• 1000 concurrent WebSocket connections per user

Implementation:

• AWS API Gateway built-in throttling with custom quota policies
• Redis-based distributed rate limiting for advanced scenarios

Request Transformation and Protocol Translation

Protocol Translation:

• HTTP to WebSocket (for real-time chat)
• JSON to binary format for media uploads
• Format normalization across different client types (iOS, Android, Web)

Request Transformation:

• Input validation and sanitization
• Field mapping between client and service formats
• Content type negotiation (JSON, binary, multipart)

BFF (Backend for Frontend) Design

BFF Strategy:

• Implement BFF pattern for mobile clients to optimize payload size and reduce round trips
• Web client BFF will aggregate multiple service calls into single responses

Key Features:

• Client-specific API surface optimization
• Payload compression and filtering
• Caching of frequently accessed data (user profiles, contact lists)
• Error handling and client-specific response formatting

Implementation:

• Custom BFF services per client platform (iOS, Android, Web)
• Shared logic in common service layer with platform-specific adapters

Edge Layer Considerations

CDN Integration:

• CloudFront for media file delivery (images, videos, documents)
• Edge caching for frequently accessed user data
• Geo-replication for global users

Multi-Region Support:

• Active-active deployment across regions (us-east-1, eu-west-1, ap-southeast-1)
• Route 53 for DNS failover and load balancing
• Cross-region WebSocket connection management

Performance Optimization:

• HTTP/2 support for reduced latency
• Compression (gzip, brotli) for response bodies
• Pre-warming of edge services to handle traffic spikes

Security at Edge Layer

DDoS Protection:

• AWS Shield Advanced for layer 3/4 protection
• Rate limiting to prevent abuse
• IP reputation filtering

WAF Rules:

• OWASP top 10 rule sets for API protection
• Custom rules for chat-specific abuse patterns (spam, flooding)
• IP allowlists for known good clients

TLS Termination:

• TLS 1.3 termination at edge
• Certificate management via AWS ACM
• Support for custom domains with SSL certificates

Monitoring and Observability

Edge Layer Metrics:

• Request/response latency by endpoint
• Error rates and 5xx/4xx distributions
• Rate limiting violations and throttled requests
• Media delivery performance (latency, bandwidth usage)

Alerting:

• High rate limiting violations (>5% of traffic)
• WebSocket connection failures
• Media delivery degradation (latency > 2s)

This edge layer design provides the foundation for handling millions of concurrent chat users while maintaining security, performance, and operational simplicity. The combination of managed services (API Gateway) with custom edge capabilities ensures optimal performance for real-time chat while maintaining flexibility for future scaling needs.

11. Authentication & Authorization Architecture

Identity Provider Integration

For a real-time chat system like WhatsApp, we'll integrate with AWS Cognito as the primary Identity Provider. This choice provides:

• Native support for social login providers (Google, Facebook, Apple)
• Built-in user pools with federation capabilities
• Integration with AWS services for seamless deployment
• Multi-factor authentication (MFA) support
• Fine-grained access control through groups and roles

Alternative considerations included Auth0 (more mature ecosystem) and custom IdP (higher control but more operational overhead). Cognito was chosen for its AWS-native integration and reduced operational burden in cloud environments.

OAuth2 / OIDC Flow Design

sequenceDiagram
    participant Client
    participant Cognito
    participant Backend
    participant TokenService

    Client->>Cognito: Request login (OIDC flow)
    Cognito->>Client: Redirect to auth provider
    Client->>Cognito: Auth response with code
    Cognito->>TokenService: Exchange code for tokens
    TokenService->>Cognito: Validate token
    TokenService->>Client: Return JWT access token
    Client->>Backend: Send JWT in Authorization header
    Backend->>TokenService: Validate JWT
    TokenService->>Backend: Confirm valid token

The authentication flow follows the Authorization Code with PKCE (Proof Key for Code Exchange) pattern, ensuring secure token exchange even in public clients. This addresses the security concerns around mobile app token leakage.

Service-to-Service Authentication

For inter-service communication, we implement mTLS (mutual Transport Layer Security) with SPIFFE/SPIRE for identity management:

• Services authenticate to each other using mTLS certificates
• SPIFFE identities are used to enforce service-to-service access control
• Certificate rotation handled automatically through SPIRE
• No shared secrets or tokens required between services

This approach provides strong security guarantees while maintaining operational simplicity in a microservices architecture.

RBAC / ABAC Model Design

We implement RBAC (Role-Based Access Control) as the primary model for chat system authorization:

• Roles: User, Admin, Moderator, Guest
• Permissions:
  • Read/Write chat messages
  • Create/delete groups
  • Mute/unmute users
  • Admin access to user data

ABAC (Attribute-Based Access Control) is applied for granular permissions:

• Time-based access restrictions (e.g., admin access only during business hours)
• Device-specific permissions (e.g., mobile app restrictions)
• Location-based access control (geo-fencing for compliance)

RBAC handles the majority of use cases with simpler management, while ABAC provides flexibility for edge cases and compliance requirements.

Token Revocation Strategy

We implement a hybrid approach combining:

1. Short-lived JWT tokens (15-30 minutes) with refresh token rotation
2. Blacklist-based revocation for immediate logout scenarios (e.g., password change, account compromise)
3. Token validation service that checks revocation status before granting access

The refresh token rotation ensures that even if a refresh token is compromised, it becomes invalid after one use. The blacklist mechanism provides immediate revocation capability for security events.

Token Security Considerations

• JWT signing: RS256 with long-lived RSA keys (rotated quarterly)
• Token storage: Client-side secure storage (iOS Keychain, Android Keystore)
• Refresh token handling: Secure HTTP-only cookies with SameSite flags
• Token expiration: Immediate invalidation on logout, automatic refresh on valid use

Security Enhancements

• Rate limiting on authentication endpoints to prevent brute force attacks
• Device fingerprinting to detect suspicious login patterns
• Session management with automatic session timeout after inactivity
• Two-factor authentication for sensitive operations (admin actions, account recovery)

Compliance Considerations

• GDPR compliance through user consent management and data portability features
• SOC2 Type II compliance with audit trails for authentication events
• HIPAA compliance (if applicable) through encryption and access logging requirements

Future Extensibility

• Integration with OAuth2 authorization servers for enterprise SSO
• Support for OpenID Connect federation with other identity providers
• Implementation of OAuth2 scopes for granular permission control
• Support for FIDO2/WebAuthn for passwordless authentication

This architecture balances security, compliance, and operational simplicity while providing extensibility for future authentication enhancements.

12. Search Architecture

Search Use Cases and Requirements

For a real-time chat system like WhatsApp, search functionality is typically not a core feature but may be required for:

• Message search within conversations
• Contact search by name or phone number
• Media/file search (if media sharing is supported)
• Conversation search (by participant or content)

Given that chat systems primarily focus on real-time messaging, search is often an optional enhancement rather than a core capability.

Search Indexing Pipeline

If search is included:

1. Source: Messages stored in database, contact information from user profiles
2. Transform:
   • Extract text content from messages (including media metadata)
   • Normalize and tokenize text for indexing
   • Extract metadata (timestamps, sender IDs, conversation IDs)
3. Index:
   • Build inverted indices for fast text retrieval
   • Maintain separate indexes for contacts and messages

Index Schema Design

Message Search Index:

{
  "message_id": "string",
  "conversation_id": "string",
  "sender_id": "string",
  "content": "text",
  "timestamp": "datetime",
  "media_type": "string",
  "media_metadata": "json"
}

Contact Search Index:

{
  "user_id": "string",
  "name": "text",
  "phone_number": "string",
  "display_name": "text"
}

Search Query Patterns

• Text-based: Full-text search within conversations
• Metadata-based: Filter by date range, user, media type
• Fuzzy matching: Handle typos in contact names or partial matches

Real-time vs. Near-real-time Indexing Trade-offs

Real-time indexing (low latency, high cost):

• Pros: Immediate search availability
• Cons: High write overhead, potential inconsistency

Near-real-time indexing (batched updates):

• Pros: Lower write cost, better performance
• Cons: Delayed search results (1-5 minutes)

For chat systems, near-real-time indexing is preferred due to the real-time nature of messaging and the acceptable delay for search results.

Relevance Tuning Strategy

• Query boosting: Prioritize sender name matches over content matches
• Timestamp weighting: Recent messages weighted higher
• Conversation context: Messages in active conversations weighted higher
• User preferences: Personalized relevance based on user behavior

Search Scaling Design

Sharding Strategy:

• Shard by conversation ID or user ID
• Each shard maintains its own search index
• Cross-conversation search requires distributed query processing

Replication Strategy:

• Primary search index per shard
• Read replicas for high availability
• Search index consistency handled via eventual consistency model

Implementation Considerations

Since search is not a core chat capability, the system will:

1. Use external search engines (Elasticsearch/OpenSearch) for robust full-text capabilities
2. Implement search as optional feature with graceful degradation when unavailable
3. Maintain separate search service to avoid impacting core messaging performance

Search Service Design

If search is implemented:

• Search API: RESTful endpoints for query execution
• Indexing Service: Background job to update search indexes
• Search Worker: Processes indexing events from message streams

Performance Considerations

• Search queries should be fast (p95 < 200ms) to avoid user perception of lag
• Caching: Frequently searched terms can be cached at application level
• Query optimization: Use search query analyzers to handle complex queries efficiently

Monitoring and Metrics

• Search query latency (p50, p95, p99)
• Indexing throughput and lag
• Search result accuracy (precision/recall)
• Cache hit rates for search queries

Cost and Resource Implications

Search capability adds:

• Storage overhead for inverted indices
• Compute resources for indexing and query processing
• Network bandwidth for distributed search operations

For a chat system, search is typically implemented as an optional feature with lower priority than core messaging functionality. The system should be designed to support search capability without compromising the primary chat experience, allowing for easy feature enablement as business requirements evolve.

13. Scalability Deep Dive

Bottleneck Identification

Message Processing Pipeline:

• Peak QPS: 10M messages/day → ~116 messages/sec average, 1.2K peak
• Database QPS: With 80% cache hit rate, DB receives ~232 QPS (116 * 0.8 * 2)
• Message size: 5KB average → 57.5MB/sec write throughput
• Read traffic: 10x write ratio → 575MB/sec read

Database Bottlenecks:

• PostgreSQL: ~10K QPS per instance (with proper tuning)
• Vertical scaling limited to 50K QPS per node
• Horizontal sharding required beyond 100K QPS

Network Bottlenecks:

• WebSocket connections: 50K concurrent connections per server
• Message propagation latency: 100ms target for global chat systems

Horizontal vs. Vertical Scaling Decisions

Stateless Services (API, Auth, Message Processing):

• Horizontal scaling preferred: Services are stateless and can scale independently
• Auto-scaling based on QPS, CPU, and memory metrics
• Kubernetes HPA with target CPU 70% and QPS thresholds

Stateful Services (Database, Message Store):

• Hybrid approach: Vertical scaling for single-region, horizontal for multi-region
• PostgreSQL: 2x read replicas + 1 primary per shard (50K QPS limit)
• Redis: Clustered setup with 3-node primary + replica sets

Database Scaling Path

Phase 1 (0-50K QPS):

• Single PostgreSQL instance with read replicas
• Redis for caching and session storage

Phase 2 (50K-200K QPS):

• Horizontal sharding by user ID hash (range-based partitioning)
• 16 shards per region with 2 read replicas each
• Connection pooling (max 100 connections per DB instance)

Phase 3 (200K+ QPS):

• Distributed database (CockroachDB or Spanner)
• Multi-region replication with read replicas in each region
• Database sharding by user ID range

Stateless vs. Stateful Service Design Implications

Stateless Services:

• Advantages: Easy to scale, fault-tolerant, no session affinity required
• Implementation: All services use ephemeral storage with external state stores (Redis, DB)
• Deployment: Kubernetes pods with no local state persistence

Stateful Services:

• Challenges: Session management, data consistency, cross-service coordination
• Solutions:
  • Redis for session storage and real-time presence
  • Message queues for reliable event delivery between services
  • Distributed locks for coordination (etcd or Redis)

Session Affinity and Load Balancing

WebSocket Connection Handling:

• No session affinity required: Each WebSocket connection is handled by any available service instance
• Load balancing strategy: Round-robin or weighted distribution based on server capacity
• Connection state management: Redis-based session store with TTL for cleanup

Cross-Service Communication:

• Stateless inter-service calls: Use gRPC or HTTP/2 with circuit breaker and retry logic
• Service discovery: Kubernetes DNS or service mesh (Istio) for dynamic routing
• Load distribution: Distribute traffic across multiple service instances to prevent hotspots

Performance Optimization Targets

Database Query Optimization:

• Indexes on user_id, timestamp, and message_type fields
• Query caching for frequently accessed conversations (e.g., last 100 messages)
• Batch processing for message delivery to reduce DB round-trips

Memory and CPU Utilization:

• Target 70% CPU utilization for horizontal scaling decisions
• Memory usage optimized to reduce GC pressure (object pooling)
• Connection reuse and efficient WebSocket frame handling

Network Efficiency:

• Compression for message payloads (per-message compression)
• Efficient WebSocket protocol implementation with minimal overhead
• Batched message delivery to reduce network round-trips

Scaling Metrics and Triggers

Service Auto-scaling:

• QPS > 1000 → Scale up by 25%
• CPU utilization > 70% → Scale up
• Memory usage > 85% → Scale up or trigger GC

Database Scaling:

• QPS > 50K per instance → Add new shard
• Read replica lag > 500ms → Add more replicas
• Disk I/O > 80% → Consider storage tiering or migration to faster storage

Cache Layer Scaling:

• Cache hit ratio < 80% → Scale Redis cluster
• Memory usage > 90% → Add new cache nodes or increase memory allocation per node

This scalability approach ensures the system can grow from MVP to enterprise scale while maintaining performance and reliability targets. The architecture is designed with horizontal scaling as the primary growth mechanism, allowing for elastic resource allocation based on real-time demand.

14. Infrastructure & Deployment Architecture

Cloud Provider and Region Selection

Primary Provider: AWS (with multi-region support)

• Justification: AWS provides the most mature ecosystem for real-time messaging systems with extensive global infrastructure, robust networking capabilities (AWS Direct Connect), and mature services like SQS, SNS, Lambda, and Kinesis that align well with chat system requirements.

Region Strategy:

• Primary Region: us-east-1 (high availability, low latency for North American users)
• Secondary Region: eu-west-1 (for EU users, disaster recovery)
• Tertiary Region: ap-southeast-1 (for Asia-Pacific users)

Kubernetes Cluster Design

Cluster Architecture:

• Multi-Tenant Clusters: Separate clusters per environment (dev, staging, prod) with namespace isolation
• Resource Quotas: Per-namespace resource limits (CPU, memory, pods) to prevent resource contention
• Node Pools:
  • Frontend Pool: t3.medium instances for API gateway and webhooks
  • Backend Pool: c5.xlarge instances for stateful services (chat, user)
  • Worker Pool: m5.large instances for background processing and message handling
• Autoscaling: Cluster Autoscaler with HPA for pod-level scaling

Namespace Strategy:

• chat-system (core chat services)
• user-system (user management and auth)
• media-system (file storage and processing)
• monitoring (observability stack)
• infrastructure (shared infrastructure services)

Container Image Strategy

Base Images:

• Alpine Linux for minimal footprint
• Official language-specific base images (node:alpine, golang:alpine)
• Custom build base with pre-installed dependencies (Go tools, Node modules)

Multi-stage Builds:

• Build Stage: Full build environment with dev dependencies
• Runtime Stage: Minimal runtime image with only necessary binaries
• Security Scanning: Trivy integration in CI pipeline to scan for vulnerabilities

Image Registry:

• AWS ECR (Elastic Container Registry) with lifecycle policies
• Image tagging strategy: v1.2.3, latest, sha256:<digest>

Deployment Strategy

Release Pattern: Blue-Green deployment with traffic shifting

• Blue Environment: Current stable version (active)
• Green Environment: New version being validated
• Traffic Shift: 25% traffic shift every 5 minutes to green, then full switch

Rollback Strategy:

• Automatic rollback on health check failures (5xx errors, latency > 500ms)
• Manual rollback trigger via deployment dashboard
• Rollback to previous stable version with same configuration

Traffic Management:

• Ingress Controller: NGINX Ingress Controller with TLS termination
• Service Mesh Integration: Istio for traffic splitting, circuit breaking, and mTLS between services
• Canary Analysis: Gradual traffic shift with success rate monitoring

Auto-scaling Design

Horizontal Pod Autoscaler (HPA):

• Metrics: CPU utilization (80% target), custom metrics (queue depth, message processing rate)
• Scale Out: 50% CPU threshold triggers scaling up
• Scale In: 20% CPU threshold triggers scaling down

Vertical Pod Autoscaler (VPA):

• Resource Optimization: Recommends optimal CPU/memory requests based on historical usage
• Update Policy: Auto-update with proper pod restart strategy

KEDA Integration:

• Message Queue Scaling: Scale based on SQS queue depth (chat message queue)
• Database Scaling: Scale based on database connection pool utilization
• Custom Metrics: Scale based on real-time chat message rate

Infrastructure as Code Approach

IaC Tools:

• Terraform: For infrastructure provisioning (VPC, subnets, security groups, RDS instances)
• Helm Charts: For Kubernetes application deployment and configuration management
• Terragrunt: For environment-specific configurations and module organization

Deployment Pipeline:

1. Code Commit: Push to GitHub (main branch triggers CI)
2. CI Pipeline: Build, test, security scan, and container image push to ECR
3. CD Pipeline: Deploy to staging via Helm charts, run integration tests
4. Production Deployment: Blue-green deployment with manual approval gate

Configuration Management:

• Secrets Management: AWS Secrets Manager for sensitive data (DB credentials, API keys)
• Environment Variables: Per-environment configuration via Kubernetes ConfigMaps
• Feature Flags: Integration with LaunchDarkly for feature toggles

Monitoring and Alerting in IaC:

• CloudWatch Alarms for cluster health, node status, and pod readiness
• Prometheus alerting rules for Kubernetes resource utilization thresholds
• Slack integration for critical alerts with severity levels (P0, P1, P2)

Multi-Region Deployment Strategy

Active-Active Architecture:

• Primary Region: us-east-1 (handles majority of traffic)
• Secondary Region: eu-west-1 (handles regional traffic and provides redundancy)
• Cross-Region Replication: DynamoDB Global Tables for user data, Redis Multi-AZ for caching

Traffic Splitting:

• Route 53: DNS-based traffic splitting (70% primary, 30% secondary)
• Service Mesh: Istio for cross-region service communication with proper failover

Security and Compliance

Security Groups:

• Egress Rules: Allow only necessary outbound traffic (AWS services, external APIs)
• Ingress Rules: Restrict access to specific IP ranges and ports (443, 80, 9090)
• Network ACLs: Additional layer of security for subnet-level filtering

Compliance Considerations:

• SOC 2 Type II: For data center and infrastructure security
• GDPR Compliance: Data encryption at rest and in transit, user consent management
• HIPAA Compliance: For healthcare chat use cases (if applicable)

Cost Optimization

Reserved Instances:

• Database: Reserved instances for RDS (PostgreSQL) in production
• Compute: Reserved instances for EC2 (c5.xlarge) in development and staging

Spot Instances:

• Worker Pool: Spot instances for background processing tasks (file conversion, analytics)
• Testing Environments: Spot instances for non-critical testing and staging environments

Resource Optimization:

• Pod Resource Requests: Precise CPU/memory requests to avoid over-provisioning
• Cluster Autoscaling: Dynamic scaling to minimize idle capacity
• Storage Tiering: S3 Standard-IA for chat history older than 30 days, Glacier for archival

Observability and Monitoring in Infrastructure

Infrastructure Metrics:

• Node-Level Metrics: CPU, memory, disk I/O, network throughput
• Cluster-Level Metrics: Pod count, resource utilization, error rates
• Network Metrics: Latency, packet loss, bandwidth usage

Alerting Strategy:

• Critical Alerts (P0): Cluster unavailability, node failures, database connection issues
• High Alerts (P1): High latency, queue depth exceeding thresholds, cache miss rate > 50%
• Medium Alerts (P2): Resource utilization > 80%, service degradation, authentication failures

This infrastructure design provides a scalable, secure, and cost-effective foundation for a real-time chat system with multi-region support, robust deployment capabilities, and comprehensive observability.

15. Security Architecture

VPC Design and Network Segmentation

The system will operate within a multi-AZ VPC with a public/private subnet model. Public subnets will host load balancers, API gateways, and edge services, while private subnets will contain application services, databases, and cache clusters. Each service will reside in its own isolated subnet with security groups enforcing strict ingress/egress rules. Cross-service communication will be restricted to internal VPC traffic only, with all external access routed through the API gateway layer.

Zero-Trust Network Model

All network traffic will follow a zero-trust model where services authenticate and authorize every request regardless of origin. Network policies will enforce strict segmentation between service tiers, with no direct communication between services except through designated API endpoints or message brokers. All inter-service communication will use mTLS for encryption and mutual authentication.

TLS Termination Strategy

TLS termination will occur at the load balancer level for external traffic, with services handling internal communication through mTLS. All external endpoints will support TLS 1.3 with HSTS headers and certificate pinning for client-side validation. Internal service-to-service communication will use mutual TLS with certificate-based authentication.

DDoS Protection Strategy

AWS Shield Advanced will be enabled for DDoS protection at the network and application layers. Rate limiting will be implemented at the API gateway level with automatic scaling to handle traffic spikes. CloudFront integration will provide additional DDoS protection for static content and API endpoints.

Secrets Management Architecture

Secrets will be managed using AWS Secrets Manager with automatic rotation for database credentials and API keys. Service accounts will use IAM roles with least-privilege permissions rather than hardcoding credentials. Secrets will be rotated every 90 days with automated failover mechanisms to prevent service disruption.

Vulnerability Scanning in CI/CD

Container images will be scanned for vulnerabilities using Trivy during the build process. Security policies will be enforced through automated checks in the CI/CD pipeline, blocking deployments with critical vulnerabilities. Dependency scanning will be integrated to detect known vulnerable libraries.

WAF Rules for Critical Endpoints

AWS WAF will be configured with custom rules to protect against common attack patterns including SQL injection, XSS, and rate limiting for sensitive endpoints. Rules will be updated regularly based on threat intelligence feeds and security audits.

Access Control and Identity Management

All services will authenticate using IAM roles with service-specific policies. API gateway will enforce authentication through AWS Cognito for user sessions and JWT validation. Service-to-service communication will use mTLS with certificate-based authentication rather than shared secrets.

Data Protection and Encryption

All data at rest will be encrypted using AWS KMS with customer-managed keys. Data in transit will be encrypted using TLS 1.3 for external communications and mTLS for internal services. Database connections will use encrypted channels with key rotation policies.

Audit and Compliance

All security-relevant events will be logged to CloudWatch with audit trails for compliance reporting. Access logs will capture authentication attempts, authorization decisions, and data access patterns. Regular security audits will be conducted to ensure compliance with SOC2 and GDPR requirements.

Threat Modeling and Risk Assessment

Regular threat modeling sessions will be conducted to identify potential attack vectors and security gaps. Security controls will be evaluated against OWASP Top 10 and NIST Cybersecurity Framework to ensure comprehensive coverage of security risks.

16. Deployment & Release Strategy

Deployment Strategy and Traffic Management

The chat system will use a blue-green deployment strategy for production releases to ensure zero-downtime deployments and fast rollback capability. This approach involves maintaining two identical production environments where only one is active at a time. During a release, the inactive environment is updated with new code and configuration, then traffic is switched from the old to the new environment.

For canary deployments, we'll implement a progressive rollout pattern where traffic is gradually shifted to the new version (e.g., 10% → 50% → 100%) based on success metrics and error rates. This is particularly important for services like message processing where reliability is paramount.

Traffic management will be handled through Kubernetes Ingress controllers with custom routing rules that support weighted traffic distribution and health-based routing. The system will implement traffic splitting using Istio or similar service mesh for fine-grained control over traffic distribution between versions.

Artifact Promotion and Release Gates

The deployment pipeline follows a trunk-based development model with feature branches merged into main after CI validation. Artifacts are promoted through the following stages:

1. Dev Environment: Continuous integration with automated unit and integration tests
2. Staging Environment: Full end-to-end testing including performance and load testing
3. Production Gates:
   • Automated smoke tests (API health checks, database connectivity)
   • Performance benchmarks (latency and throughput targets)
   • Security scanning (SAST, DAST, dependency vulnerability checks)
   • Manual approval for critical services (message processing, user auth)

Zero-Downtime Database Migration Strategy

Database schema migrations will be handled using online schema change tools like gh-ost for MySQL or Liquibase for PostgreSQL to avoid table locks during migration. For major schema changes, we'll implement:

• Dual-write strategy: New fields are written to both old and new schema during transition
• Read-through pattern: Services read from both schemas with fallback logic
• Data migration job: Background process to sync data between schema versions
• Rollback capability: Reverse migration scripts and backup restoration procedures

Feature Flag Integration and Progressive Rollouts

We'll integrate LaunchDarkly or Flagship for feature flag management to enable:

• Gradual feature rollouts with percentage-based traffic distribution
• A/B testing capabilities for UI and behavioral changes
• Emergency feature toggles for quick incident response
• Targeted rollouts to specific user segments or regions

Progressive rollout logic will be implemented as:

• Percentage-based rollouts: Start with 5% and increase to 100%
• Gradual traffic increase: 10% → 30% → 60% → 100% over 2-hour windows
• Automated rollback: If error rate exceeds threshold (e.g., >1% of requests), automatically revert to previous version

Canary Analysis and Auto-Rollback Triggers

Canary analysis will be automated using:

• Error rate monitoring: Track p99 error rates and latency spikes
• Performance regression detection: Compare median response times between versions
• Business metric validation: Monitor user engagement and message delivery success rates

Auto-rollback triggers include:

• Error rate threshold: >5% error rate in 5-minute window
• Latency degradation: >20% increase in p95 response time
• Business impact: >10% drop in message delivery success rate

Release Cadence and Branching Strategy

The system follows a trunk-based development model with:

• Daily builds: Main branch is built and tested every day
• Release windows: Weekly release cycles (Tuesday/Wednesday) for non-critical features, emergency releases as needed
• Feature branches: Short-lived branches (max 2 weeks) for experimental features

Branching strategy:

• Main branch: Production-ready code with all tests passing
• Release branches: Created for major releases with hotfixes applied
• Feature branches: For experimental features and large-scale refactors

Deployment Automation and Rollback Procedures

Deployment automation will be implemented using:

• GitOps approach: Kubernetes manifests stored in Git repositories with ArgoCD for continuous delivery
• Automated rollback: Rollback to previous stable version on failure using Kubernetes deployment rollbacks or manual intervention
• Runbook automation: Pre-defined scripts for common rollback scenarios and incident response procedures

Release Validation and Testing Strategy

Each release undergoes:

• Automated testing: Unit tests (95% coverage), integration tests, end-to-end tests
• Performance testing: Load tests simulating 100K concurrent users with 5000 messages/second
• Security testing: Vulnerability scans, penetration testing, compliance checks
• User acceptance testing: Small group of beta users for new features before full rollout

Release Coordination and Communication

Release coordination includes:

• Pre-release communication: Notify all stakeholders of upcoming releases and expected downtime windows
• Post-release validation: Monitor system health and user feedback within 24 hours of release
• Release notes: Detailed documentation of changes, breaking changes, and migration steps for each release

Rollback Process and Incident Response

In case of rollback:

1. Immediate alert: Trigger PagerDuty for on-call engineers
2. Traffic redirection: Switch traffic back to stable version within 5 minutes
3. Root cause analysis: Identify and document the failure cause
4. Post-mortem: Conduct 48-hour post-mortem meeting with all stakeholders
5. Fix implementation: Apply fix to main branch and re-deploy after validation

The rollback process will be automated for critical services but require manual approval for user-facing features to ensure proper communication with users.

17. Cost Architecture

Major Cost Drivers Analysis

Compute Costs:

• API Gateway & Application Services: Primary compute cost driver, scaled by QPS and instance count
• Message Processing Services: CPU-intensive for real-time message handling and delivery
• Background Processing Workers: Scheduled tasks like cleanup, stats aggregation, and user status updates

Storage Costs:

• Database Storage: Relational DBs (PostgreSQL) for chat metadata and user data
• Message Storage: Time-series or document stores for message history (hot/warm tiered)
• File Storage: Media files (images, videos) stored in object storage (S3/GCS)
• Cache Storage: Redis clusters for session and message caching

Network & Data Transfer Costs:

• Ingress/Egress: High-volume real-time traffic between clients and servers
• Cross-region replication: Multi-region data sync increases bandwidth costs
• CDN egress: Content delivery to global users

Data Transfer Costs:

• Real-time WebSocket traffic: High-frequency message exchange
• File upload/download: Media sharing between users

Cost Optimization Strategy per Layer

Compute:

• Spot Instances: For background workers and non-critical batch jobs (30-50% cost reduction)
• Auto-scaling: HPA with VPA to optimize instance usage and prevent over-provisioning
• Serverless Functions: For event-driven tasks like notification processing or message validation

Storage:

• Tiered Storage Strategy: Hot (SSD), Warm (HDD), Cold (Archive) storage based on access frequency
• Object Storage Lifecycle Policies: Automatically move older media to cheaper storage tiers
• Database Compression: Enable compression for message content and metadata

Network:

• CDN Integration: Reduce bandwidth costs by caching static assets at edge locations
• WebSocket Compression: Enable per-message compression to reduce bandwidth usage by 30-50%
• Multi-region Load Balancing: Distribute traffic to nearest regions to minimize cross-region egress

Reserved vs. Spot Instance Strategy

Reserved Instances:

• Primary Services: API gateway, message processing services (1-2 year commitments)
• Database Nodes: For stable read replicas and primary instances

Spot Instances:

• Batch Processing Workers: Background tasks like analytics, cleanup jobs, and report generation
• Cache Warm-up Services: Temporary services for warming caches during peak hours

Data Egress Cost Minimization

CDN Usage:

• Static Assets: Images, icons, and client libraries served via CDN to reduce origin server load
• Media Delivery: File storage with CDN integration for global distribution

Regional Proximity:

• Multi-region Deployment: Deploy services in regions closest to user base to minimize cross-region data transfer costs
• Edge Computing: Use edge services for content filtering and compression before reaching origin

FinOps Dashboards and Budget Alerts

Cost Monitoring:

• Real-time Cost Dashboard: Built with Grafana or Datadog to visualize spend by service, region, and team
• Budget Alerts: Set up alerts for 80%, 90%, and 100% of monthly budget thresholds
• Cost Allocation Tags: Tag all resources by team, service, and environment for accurate cost attribution

Optimization Metrics:

• Cost Per Active User (CAU): Target under $0.50/user/month for optimal efficiency
• Resource Utilization Rate: Maintain >60% average CPU/memory usage to avoid waste
• Storage Efficiency Ratio: Ensure <50% of total storage is unused or redundant

This cost architecture ensures efficient resource utilization while maintaining system performance and scalability for a real-time chat service.

18. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• Message persistence vs. ephemeral messaging: Should all messages be persisted or only certain types (e.g., group messages vs. one-on-one)? This affects storage costs, compliance requirements, and user expectations.
• Cross-platform consistency: How strictly should the same message content and delivery behavior be maintained across web, mobile, and desktop clients? This impacts development complexity and feature parity.
• Message threading strategy: Should messages be grouped by conversation or by thread within a conversation? This affects UI/UX design and search capabilities.
• Push notification strategy: Should all messages trigger push notifications or only specific message types (e.g., mentions, group invites)? This affects user experience and battery life on mobile devices.

Key architectural risks

Risk	Impact	Likelihood	Mitigation
Message ordering inconsistency	High (user experience degradation)	Medium	Implement strict per-conversation message ordering with sequence numbers and leader election in distributed systems
Cache inconsistency across services	High (data corruption, user confusion)	Medium	Implement cache invalidation strategies with TTLs and cache warming for critical paths
Message delivery failure in edge regions	High (user frustration, data loss)	Low-Medium	Implement message retry mechanisms with dead letter queues and cross-region replication
DDoS attacks on real-time connections	High (service unavailability)	Medium	Deploy rate limiting, connection throttling, and DDoS protection at edge layer
Data sovereignty compliance	High (regulatory penalties)	Medium	Implement region-specific data storage and processing with proper encryption and access controls
Cache warming impact on database	Medium (performance degradation)	Medium	Implement intelligent cache warming with pre-computed message lists and background warming

Trade-offs made with justification

• Consistency vs. availability: Chose eventual consistency for message delivery over strong consistency to ensure high availability and reduce latency in global deployments. This aligns with WhatsApp's user experience expectations where "seen" receipts and read receipts are eventually consistent.
• Monolithic vs. microservices: While microservices provide better scalability and team autonomy, the real-time chat system's tight coupling between services (message routing, user presence, push notifications) makes a modular monolith more appropriate for initial development and reduces inter-service communication overhead.
• Real-time vs. batch processing: Prioritized real-time message delivery over batch processing to ensure immediate user experience, despite higher infrastructure costs and complexity in maintaining consistent state across distributed systems.
• Storage vs. compute trade-off: Chose higher storage costs with better performance over compute-intensive approaches to avoid latency spikes during peak hours and ensure consistent user experience.

Alternatives considered and rejected

• Traditional pub/sub systems (e.g., RabbitMQ): Rejected due to lack of horizontal scalability and complexity in maintaining message ordering across distributed systems.
• Event sourcing pattern: Rejected for simplicity and reduced complexity in the MVP phase, though it's a potential future enhancement for audit trails and complex message state management.
• Multi-region active-active: Rejected in favor of single-region active-passive for initial launch to simplify the architecture and reduce operational complexity, with multi-region support planned as a future enhancement.
• Custom WebSocket implementation: Rejected in favor of proven open-source solutions (e.g., Socket.IO) to reduce development time and leverage community support for edge cases.

Technical debt acknowledged

• Message history truncation: Current design assumes unlimited message history storage, which will require future implementation of message retention policies and archiving strategies to manage storage costs.
• Push notification handling: Current implementation assumes all clients will receive push notifications, but future versions need to handle client-specific notification preferences and device lifecycle management.
• Cross-service data consistency: No shared databases between services, but there's potential for inconsistency in user presence and message status updates across different service boundaries that will require careful state management.
• User session management: Session handling is simplified in the current design and will need to be enhanced for better mobile app lifecycle management, including backgrounding and reconnection strategies.
• API versioning strategy: Current API design doesn't yet support versioning, which will be required as the system evolves and backward compatibility needs to be maintained for existing clients.

19. Phased Implementation Roadmap

Phase 1: MVP (Minimum Viable Product)

Timeline: 8-12 weeks Focus: Core chat functionality with basic reliability

Core Components:

• Basic message storage and retrieval (1:1 and group chats)
• Real-time messaging via WebSocket connections
• User authentication and authorization (JWT-based)
• Basic message delivery guarantees (at-least-once)
• Simple message persistence in PostgreSQL with Redis caching
• Basic API gateway and rate limiting

Key Milestones:

• User registration and login flow
• 1:1 chat creation and messaging
• Group chat creation and message broadcast
• Basic read receipts and typing indicators
• First release to internal users with limited scale

Dependencies:

• Authentication service (Phase 1)
• Message persistence layer (Phase 1)
• Basic infrastructure (K8s cluster, DB instances)

Phase 2: Scale & Reliability

Timeline: 12-16 weeks Focus: Horizontal scaling, reliability improvements, and enhanced features

Core Components:

• Multi-region deployment with active-active failover
• Horizontal sharding of users and messages (by user ID hash)
• Message persistence with multi-region replication
• Enhanced reliability patterns (circuit breaker, retry logic)
• Advanced caching strategies with L2 cache
• Message deduplication and ordering guarantees
• Push notification integration (APNs/Firebase)
• Enhanced monitoring and alerting

Key Milestones:

• Multi-region support with automatic failover
• Sharding implementation and data distribution
• Message ordering guarantees (per-user, per-group)
• Enhanced read receipts and delivery status tracking
• Production deployment with 10x scale capability

Dependencies:

• Multi-region infrastructure (Phase 1)
• Message streaming system (Phase 2)
• Enhanced monitoring stack (Phase 2)

Phase 3: Excellence & Optimization

Timeline: 16-24 weeks Focus: Advanced features, performance optimization, and operational excellence

Core Components:

• End-to-end encryption (E2EE) implementation
• Advanced search capabilities (message content search)
• Media file handling with object storage integration
• Advanced analytics and reporting dashboard
• Feature flag system for progressive rollouts
• Comprehensive audit logging and compliance features
• Optimized resource utilization with cost control

Key Milestones:

• E2EE implementation and key rotation
• Media storage and retrieval optimization
• Advanced analytics dashboard with real-time metrics
• Compliance features (GDPR, HIPAA support)
• Production-grade cost optimization and resource management

Dependencies:

• Advanced monitoring (Phase 2)
• Secure communication protocols (Phase 2)
• Media storage infrastructure (Phase 3)

Critical Path Dependencies:

1. Authentication & Authorization must be completed before any chat features
2. Message persistence layer is the core data store for all chat operations
3. Real-time communication infrastructure (WebSocket, message streaming) is critical for user experience
4. Multi-region deployment capability enables horizontal scaling and reliability

Risk Mitigation:

• Phase 1: Start with single region to reduce complexity and ensure core functionality works
• Phase 2: Implement sharding gradually with careful data migration strategy
• Phase 3: Use feature flags to enable advanced features progressively and monitor impact

Success Metrics:

• Phase 1: 95% message delivery success rate, <200ms p95 latency
• Phase 2: 99.9% availability, <100ms p95 latency, 10x scale capability
• Phase 3: End-to-end encryption compliance, <50ms p95 latency, cost optimization >20%