Skip to main content
Interview-Mapagam

Push Notification Service HLD - Scalable Real-time Messaging Architecture

Comprehensive high-level design for a scalable push notification service handling millions of concurrent users with multi-platform support and real-time delivery.

  • PublishedAugust 17, 2026
  • Length11,203 words
  • Reading time57 min read
Backend HLD

This document presents a production-ready high-level architecture for a scalable push notification service that handles real-time messaging across multiple platforms while maintaining high availability and low latency requirements.

1. Executive Summary

Clarifying questions

  1. What's the expected read/write ratio for notification operations?

    • Is this primarily a write-heavy system (e.g., push notifications generated by backend services) or does it support frequent read operations like notification history?

  2. What are the consistency and durability requirements?

    • Are notifications required to be delivered exactly once, or is at-least-once acceptable?
    • What's the SLA for delivery latency?

  3. Is this a global service or region-specific?

    • Will the system need to support multi-region replication or is it a single-region deployment?

  4. How sensitive is the system to latency?

    • Are notifications expected to be delivered in real-time (e.g., under 100ms) or can they tolerate delays?

  5. What's the expected growth in users and notifications?

    • What is the projected DAU and peak QPS for the system?

  6. Are there any external integrations or push providers (e.g., Firebase, Apple APNS)?

    • Will the system act as a proxy or directly interface with push providers?

  7. What are the data ownership and compliance requirements?

    • Is there any need to ensure data residency or support for GDPR/HIPAA?

Problem being solved

The Push Notification Service is designed to provide a scalable, reliable, and low-latency system for sending real-time push notifications to mobile devices (iOS, Android) and web clients. The service abstracts the complexity of interacting with multiple push providers (e.g., Firebase Cloud Messaging, Apple Push Notification Service) and handles routing, delivery, retry logic, and tracking of notification status.

Back-of-envelope calculations

Assuming:

  • 10 million DAU (Daily Active Users)
  • Peak-to-average ratio of 10:1
  • Average notification per user per day = 2
  • Average notification size = 500 bytes

Peak QPS:

  • DAU = 10M
  • Avg. notifications per user/day = 2
  • Total daily notifications = 20M
  • Peak QPS = (20M * 10) / 86400 ≈ 2310 QPS

Storage per user/event:

  • Notification metadata (ID, timestamp, status) = 256 bytes
  • Per user per day = 2 * 256 = 512 bytes
  • Total storage per user/year = ~187 KB
  • Total storage for 10M users = ~1.87 TB/year

Bandwidth (read + write throughput):

  • Write: 2310 QPS * 500 bytes = ~1.1 MB/s
  • Read: Assuming 10% read (notification history, status checks) = ~110 KB/s
  • Total bandwidth: ~1.2 MB/s

Cache hit rate assumption (80%):

  • DB QPS = 2310 * (1 - 0.8) = ~462 QPS

Number of servers needed at peak:

  • Assuming 10,000 QPS per server (typical for a well-optimized backend)
  • Servers needed = 2310 / 10,000 ≈ 0.23 → 1 server (but with redundancy and capacity for growth, likely 3–5 servers in production)

SLA targets

  • P50 latency: < 100ms (critical path)
  • P95 latency: < 500ms
  • P99 latency: < 2s
  • Availability: 99.9% (3.6 minutes downtime/month)
  • Delivery success rate: > 99.5%
  • RTO (Recovery Time Objective): < 5 minutes
  • RPO (Recovery Point Objective): < 1 minute

Key architectural decisions summary

  • Microservices-based architecture to isolate concerns and enable independent scaling.
  • Event-driven design using message brokers for decoupling notification generation from delivery.
  • Multi-provider push gateway to abstract and support multiple push providers (APNS, FCM).
  • Caching layer to reduce database load and improve response times.
  • Reliability-first design with circuit breakers, retry logic, and dead-letter queue handling.
  • Stateless API gateway with rate limiting and authentication.
  • Observability stack including tracing, metrics, and structured logging for operational insight.
  • Multi-region support to ensure high availability and low latency for global users.

2. System Goals & Non-Functional Requirements

Latency Targets

  • p50: 100ms for notification delivery from API request to device
  • p95: 500ms for notification delivery
  • p99: 2s for notification delivery (including retry/backoff)
  • Internal service-to-service latency: p50 < 20ms, p99 < 100ms

Throughput Requirements

  • Average QPS: 50,000 notifications/sec (assumes 100M DAU with 50% active users)
  • Peak QPS: 500,000 notifications/sec (10x average)
  • Peak burst: 2M notifications/sec for 5-second window
  • Storage write rate: 100,000 notifications/sec (for event logging and tracking)
  • Storage read rate: 200,000 reads/sec (for device token lookup and delivery status)

Availability Target

  • 99.9% (3 nines) availability for core notification delivery service
    • Justification: Critical business function with direct impact on user engagement and retention
    • Acceptable downtime: ~4.3 minutes/month for peak hours, ~8.6 minutes/month for non-peak
    • SLA: 99.9% availability with 50ms p50 latency for core delivery path

Durability and Consistency Requirements

  • Consistency Model: Eventual consistency for non-critical paths, strong consistency for critical delivery state (delivery status, read receipts)
  • Durability: All notification events and metadata must be durably stored with RPO < 5 minutes
  • CAP Trade-off: Prioritize availability and partition tolerance over consistency (CP → AP)
    • Reasoning: In case of network partitions, system should continue delivering notifications to available devices
  • Delivery guarantee: At-least-once delivery for critical notifications (e.g., user messages, alerts)
    • Non-critical notifications may use at-least-once with retry logic and deduplication

Data Retention and Compliance Requirements

  • Notification history retention: 30 days for analytics, 7 days for delivery tracking (configurable per tenant)
  • User data retention: 30 days after user deactivation or deletion (per privacy policy)
  • Compliance: GDPR, CCPA compliance for EU/CA users; HIPAA compliance if healthcare data is involved
  • Audit logging: All notification delivery events must be logged for compliance and troubleshooting

Security and Regulatory Requirements

  • Authentication: JWT-based authentication for service-to-service communication, OAuth2 for client applications
  • Authorization: RBAC model with granular permissions (e.g., "send notification", "read delivery status")
  • Encryption: TLS 1.3 for all external communications, AES-256 encryption at rest for sensitive data
  • Data classification: Tiered security based on sensitivity (public, internal, confidential)
  • SOC2 Type II compliance: Required for enterprise customers and financial services integration
  • Audit trail: All notification send operations must be logged with user, timestamp, and payload metadata
  • Rate limiting: Per-client rate limits to prevent abuse and DDoS attacks (e.g., 1000 notifications/sec per client)
  • Token validation: Device tokens must be validated and refreshed before delivery to prevent stale device handles

3. High-Level Architecture Diagram

C4Context
    title Push Notification Service - System Context

    Person(user, "End User")
    Person(admin, "Admin/Developer")
    
    System_Boundary(b1, "Push Notification Service") {
        System(push_service, "Notification Service", "Handles push notification lifecycle")
        System(device_service, "Device Management", "Manages device registrations and tokens")
        System(template_service, "Template Service", "Manages notification templates")
        System(event_service, "Event Service", "Processes user events and triggers notifications")
        System(external_provider, "Push Provider", "External service (e.g., FCM, APNs)")
    }
    
    System_Ext(sms_service, "SMS Service", "Optional SMS delivery")
    System_Ext(email_service, "Email Service", "Optional email delivery")
    
    Rel(user, push_service, "Sends notifications to")
    Rel(admin, push_service, "Manages templates and sends test notifications")
    Rel(push_service, device_service, "Registers/updates device info")
    Rel(push_service, template_service, "Retrieves notification templates")
    Rel(push_service, event_service, "Triggers notifications based on events")
    Rel(push_service, external_provider, "Sends to device via")
    Rel(push_service, sms_service, "Sends SMS via")
    Rel(push_service, email_service, "Sends Email via")

    UpdateLayoutConfig($c4ShapeInRow = 3, $c4BoundaryInRow = 1)
C4Container
    title Push Notification Service - Container Diagram

    Container_Boundary(c1, "Push Notification Service") {
        Container(push_service, "Notification Service", "Go/Java", "Handles push notification lifecycle")
        Container(device_service, "Device Management", "Go/Java", "Manages device registrations and tokens")
        Container(template_service, "Template Service", "Go/Java", "Manages notification templates")
        Container(event_service, "Event Service", "Go/Java", "Processes user events and triggers notifications")
        Container(cache, "Redis Cache", "Redis", "Caches device info and templates")
        Container(db, "PostgreSQL DB", "PostgreSQL", "Stores notification history, templates, and device info")
        Container(message_queue, "Kafka", "Apache Kafka", "Message broker for event streaming")
    }

    Container_Ext(external_provider, "Push Provider", "FCM/APNs", "External push service")
    Container_Ext(sms_service, "SMS Service", "Twilio/Plivo", "SMS delivery service")
    Container_Ext(email_service, "Email Service", "SendGrid/SES", "Email delivery service")

    Rel(push_service, device_service, "Read/Write")
    Rel(push_service, template_service, "Read")
    Rel(push_service, event_service, "Read")
    Rel(push_service, cache, "Cache reads/writes")
    Rel(push_service, db, "Read/Write")
    Rel(push_service, message_queue, "Publish events")
    Rel(push_service, external_provider, "Send push")
    Rel(push_service, sms_service, "Send SMS")
    Rel(push_service, email_service, "Send Email")

    UpdateLayoutConfig($c4ShapeInRow = 3, $c4BoundaryInRow = 1)

Major Components and Services

Core Services:

  • Notification Service: Main service handling notification lifecycle, routing, and delivery
  • Device Management Service: Manages device registrations, tokens, and device metadata
  • Template Service: Handles notification template creation, versioning, and rendering
  • Event Service: Processes user events and triggers notifications based on rules

Data Stores:

  • PostgreSQL DB: Primary storage for device info, templates, notification history, and metadata
  • Redis Cache: In-memory caching for device tokens, templates, and frequent lookups

Messaging & Integration:

  • Apache Kafka: Event streaming for notification triggers and system events
  • External Push Providers: FCM (Android), APNs (iOS) for actual device delivery
  • SMS/Email Services: Optional delivery channels for fallback or multi-channel notifications

Traffic Flow and Data Flow

  1. User Registration: Device registration flows to Device Management Service
  2. Template Creation: Admin creates templates via Template Service
  3. Event Triggering: Events from Event Service trigger notifications via Kafka
  4. Notification Processing: Notification Service reads templates, caches device info, and sends to external providers
  5. Delivery: Push notifications are sent through FCM/APNs or fallback channels (SMS/Email)
  6. Feedback Loop: Delivery status and user interactions are captured for analytics and retry logic

4. Service Architecture

Microservices vs. Monolith vs. Modular Monolith Decision

For a push notification service, I recommend a modular monolith approach as the initial architecture with clear service boundaries for future decomposition. This decision balances:

  • Development velocity - faster iteration and deployment cycles during early stages
  • Operational simplicity - single deployment unit with shared infrastructure
  • Scalability readiness - modular structure enables clean service boundaries later
  • Reduced complexity - no cross-service coordination overhead in early stages

The modular monolith will evolve into microservices once we have established clear domain boundaries and traffic patterns.

Service Decomposition Strategy

Based on the push notification service's core capabilities, I propose the following service decomposition:

  1. Notification Service - Core business logic for creating, routing, and delivering notifications
  2. User Preferences Service - Manages user notification preferences and opt-in status
  3. Delivery Metrics Service - Tracks delivery success/failure rates, engagement metrics
  4. Template Service - Manages notification templates and content rendering
  5. Event Processing Service - Handles event-driven notifications and triggers

Each service will own its data domain and have clear responsibilities:

  • Notification Service handles the core flow: create → route → deliver → report
  • User Preferences Service manages user consent and preference settings
  • Delivery Metrics Service provides analytics and monitoring capabilities
  • Template Service handles content templating and personalization
  • Event Processing Service manages event-driven triggers and workflows

Service Boundaries and Contracts

Each service will expose:

  • RESTful APIs for CRUD operations and state management
  • gRPC services for high-performance internal communication
  • Event-driven interfaces using pub/sub for cross-service coordination

Service contracts will follow:

  • Versioned APIs (v1, v2) with clear deprecation policies
  • Strong typing using Protocol Buffers for gRPC services
  • Comprehensive error handling with standardized error codes and messages

Inter-Service Communication Pattern

Communication strategy:

  1. Synchronous (gRPC/REST) - For real-time operations:

    • User preference queries
    • Template rendering
    • Immediate delivery status checks

  2. Asynchronous (Pub/Sub) - For decoupled operations:

    • Notification creation events
    • Delivery status updates
    • Metrics aggregation

  3. Batch Processing - For bulk operations:

    • Scheduled notification batches
    • User preference updates

Service Mesh Consideration

I recommend not implementing service mesh at this stage. The service mesh adds significant operational complexity and overhead that isn't justified for a notification service with moderate scale. Instead, we'll implement:

  • Direct service-to-service communication with circuit breaker patterns
  • Manual retry logic where needed
  • Basic observability through custom metrics and tracing

Service Registry and Discovery Strategy

For service discovery, I recommend:

  1. Kubernetes-based service discovery - Leverage built-in DNS and service mesh capabilities
  2. Consul integration - For advanced service discovery features and health checks
  3. Custom service registry - For specific notification service requirements

Service registration will be handled through:

  • Kubernetes Service objects for basic discovery
  • Consul Catalog for service metadata and health monitoring
  • Custom registry API for notification-specific features like preference caching

Cross-Service Data Ownership

Following the "no shared databases" principle:

  1. Notification Service owns notification records and delivery status
  2. User Preferences Service owns user consent and preference data
  3. Template Service owns template definitions and content
  4. Metrics Service owns analytics and reporting data

Cross-service data access will be handled through:

  • Event-driven synchronization using message queues
  • Read-only replicas for performance-critical queries
  • Service-to-service APIs for real-time data access

CQRS Pattern Application

CQRS will be applied selectively:

  1. Notification Service - Implements CQRS for:

    • Write side: Notification creation and routing
    • Read side: Delivery status and metrics views

  2. Metrics Service - Implements CQRS for:

    • Write side: Raw event ingestion
    • Read side: Aggregated analytics and dashboards

  3. Template Service - Implements CQRS for:

    • Write side: Template management
    • Read side: Content rendering and caching

Event Sourcing Consideration

Event sourcing will be not implemented at this stage due to:

  • Complexity overhead - Not justified for current scale
  • Operational complexity - Requires significant operational expertise
  • Business requirements - Current use cases don't require full event history

Instead, we'll use a state-based approach with proper audit logging and change tracking for compliance and debugging needs.

Service Dependencies and Data Flow

The service architecture follows this data flow:

  1. Event Processing Service receives triggers from external systems
  2. Notification Service creates notification records and publishes to queue
  3. User Preferences Service validates user consent and preferences
  4. Template Service renders content for specific platforms (iOS, Android, Web)
  5. Delivery Metrics Service tracks delivery success/failure rates
  6. Notification Service orchestrates delivery to respective push providers (APNS, FCM)

This design enables:

  • Scalable processing through queue-based workflows
  • Flexible delivery to multiple platforms
  • Comprehensive metrics for performance monitoring
  • Easy extensibility for new notification types and platforms

5. Data Architecture

Database Selection per Service

Push Notification Service (Core)

  • Primary DB: PostgreSQL (v14+) - Relational database for storing notification metadata, user preferences, and delivery status tracking
  • Justification: Strong consistency requirements for delivery confirmation, complex queries for user preferences, and ACID compliance for transactional operations. PostgreSQL's JSONB support enables flexible notification payload storage.

User Preferences & Settings Service

  • Primary DB: MongoDB (v6+) - Document database for user preference profiles, device tokens, and segmentation data
  • Justification: Flexible schema for varying user preference structures, fast geospatial queries for location-based preferences, and horizontal sharding capability for scale.

Analytics & Metrics Service

  • Primary DB: TimescaleDB (v2+) - Time-series database for notification delivery metrics, engagement analytics, and performance monitoring
  • Justification: Optimized for time-series data with efficient compression, built-in aggregation functions, and seamless integration with PostgreSQL for cross-service queries.

Event Processing Service

  • Primary DB: Redis (v7+) - In-memory key-value store for caching frequently accessed notification templates, user session data, and rate limiting state
  • Justification: Extremely low latency for read-heavy operations, built-in pub/sub capabilities for real-time updates, and support for various data structures (strings, hashes, lists).

Data Partitioning Strategy

Horizontal Sharding by User ID

  • Partition Key: User ID (hash-based sharding)
  • Justification: Even distribution of data across shards, predictable query routing, and minimal cross-shard joins required for user-centric operations.

Range-based Partitioning for Analytics

  • Partition Key: Notification timestamp (time-range partitioning)
  • Justification: Efficient time-based queries for analytics, automatic data lifecycle management, and optimized storage tiering.

Data Replication Strategy

Primary-Replica Model

  • Configuration: 1 primary + 2 read replicas per shard
  • Justification: High availability for read operations, automatic failover, and reduced load on primary database.

Cross-Region Replication

  • Configuration: Multi-region read replicas for disaster recovery
  • Justification: Geographic distribution of data for low-latency access, compliance with data residency requirements, and improved fault tolerance.

Cross-Service Data Ownership

No Shared Databases Rule

  • Enforcement: Database access controlled via service accounts with role-based permissions, strict schema versioning, and audit logging for all data access.

Data Ownership Model

  • Primary Service: Push Notification Service owns notification metadata and delivery status
  • Secondary Services: User Preferences Service owns user preference data, Analytics Service owns metrics and analytics data

CQRS Pattern Application

Push Notification Service

  • Read Side: Redis cache + PostgreSQL read replicas for fast retrieval of user preferences and notification templates
  • Write Side: PostgreSQL primary for transactional operations and state updates

Analytics Service

  • Read Side: TimescaleDB for time-series analytics queries
  • Write Side: Event-driven writes from notification service and user interaction events

Event Sourcing Consideration

Use Case: Not implemented due to complexity and cost of maintaining event streams for all notification operations.

Trade-offs:

  • Pros: Complete audit trail, ability to reconstruct state at any point in time, supports complex business logic evolution
  • Cons: Increased complexity in event management, higher storage requirements, additional operational overhead

Object Storage Design

Media/Content Storage

  • Storage Type: S3-compatible object storage (MinIO/GCS)
  • Use Case: Large notification content (images, videos, rich media attachments)
  • Access Patterns: Read-heavy with bursty write patterns during content creation
  • Metadata Strategy: Separate metadata (JSON) from content (binary) with S3 metadata tags for indexing and lifecycle management

Storage Tiering Strategy

Hot Tier: PostgreSQL primary and Redis cache (SSD-backed)

  • Access Frequency: High (95% of reads)
  • Cost: Highest, optimized for performance

Warm Tier: MongoDB read replicas and TimescaleDB analytics

  • Access Frequency: Medium (50% of reads)
  • Cost: Moderate, balanced performance and cost

Cold Tier: Archived notification logs and historical analytics data (S3 Glacier)

  • Access Frequency: Low (1% of reads)
  • Cost: Lowest, optimized for cost over performance

Backup and Restore Strategy

Backup Frequency:

  • Full backups: Daily (1 AM UTC)
  • Incremental backups: Hourly

RTO/RPO Targets:

  • RTO: 15 minutes for primary services
  • RPO: 5 minutes for critical data (notification metadata, user preferences)
  • RPO: 1 hour for analytics data

Backup Storage:

  • Primary: Cloud-native backup services (AWS Backup, GCP Backup)
  • Secondary: Cross-region storage for disaster recovery

Restore Strategy:

  • Automated restores: For minor failures and point-in-time recovery
  • Manual restores: For major incidents or compliance requirements with rollback capability

6. Caching Architecture

Cache Layers Overview

The push notification service will implement a multi-tier caching strategy to minimize database load and improve response latency:

mermaid
graph LR
    A[Client App] --> B[CDN Edge]
    B --> C[API Gateway]
    C --> D[Notification Service]
    D --> E[In-Memory Cache (L1)]
    E --> F[Distributed Cache (L2)]
    F --> G[Database]

Caching Strategy

L1 Cache (In-Memory):

  • Technology: Redis Cluster with persistence
  • Size: 2GB per instance (100K active users * 20KB avg user cache)
  • Eviction Policy: LRU with TTL of 30 minutes for user sessions
  • Use Case: Session and user preference caching

L2 Cache (Distributed):

  • Technology: Redis Cluster with replication
  • Size: 20GB per instance (1M active users * 20KB avg)
  • Eviction Policy: LFU with TTL of 1 hour for notification templates and metadata
  • Use Case: Template caching, device token validation, user segmentation

Cache Invalidation Strategy

  1. Write-through: Critical user data updated in cache before DB write
  2. Cache-Aside: Background updates for non-critical data (e.g., user preferences)
  3. Time-based expiration: 30-minute TTL for session data, 1-hour for templates
  4. Event-driven invalidation: Cache invalidation on user preference updates via pub/sub

Cache Warming Strategy

  1. Startup Warmup: Load templates and frequently accessed user data during service boot
  2. Pre-warming: Warm cache with top 1000 most active users' data every 2 hours
  3. Warm-up API: Dedicated endpoint to warm cache for specific user segments

Cache Hit Rate Targets

  • L1 Cache: 85% hit rate (session and preference data)
  • L2 Cache: 90% hit rate (templates and metadata)
  • Database QPS Reduction: Target 75% reduction in DB queries through caching

Performance Impact

With 100K concurrent users and 25% cache hit rate:

  • Database QPS: From 10,000 to 2,500 (75% reduction)
  • Latency Improvement: 60-80% reduction in response time for cached requests
  • Cache Memory Usage: ~22GB total (2GB L1 + 20GB L2)

Cache Consistency

  • Read Consistency: Eventual consistency with cache TTLs
  • Write Consistency: Strong consistency for critical user session data
  • Cross-service Cache: Shared cache cluster with consistent key naming strategy

Cache Monitoring

  • Metrics: Hit rate, miss rate, eviction rate, cache size
  • Alerts: Cache hit rate below 80% triggers alert
  • Dashboard: Grafana dashboard with cache performance trends

7. Messaging & Event Streaming Architecture

Message Broker Selection

For a push notification service, we'll select Apache Kafka as our primary message broker due to:

  • High throughput: Kafka can handle millions of messages per second with low latency
  • Durability: Messages are persisted to disk and replicated across brokers
  • Scalability: Linear horizontal scaling with partitioning model
  • Stream processing capabilities: Integration with Kafka Streams or KSQL for real-time processing
  • Ecosystem maturity: Rich ecosystem of tools (Confluent Schema Registry, Kafka Connect)
  • Producer/consumer guarantees: Exactly-once processing semantics with idempotent producers

Event Schema Registry and Versioning

We'll integrate Confluent Schema Registry to manage event schemas:

  • Schema evolution: Backward and forward compatibility rules enforced
  • Version control: Automatic versioning with schema validation
  • Type safety: Prevents malformed events from being published
  • Documentation: Schema evolution history and metadata available

Topic Partitioning and Consumer Group Design

Notification topics design:

  • notifications.user.events - User-specific events (user login, profile update)
  • notifications.system.events - System-wide notifications (maintenance, alerts)
  • notifications.broadcast.events - Broadcast messages to segments

Partitioning strategy:

  • User ID hash: For notifications.user.events - ensures all events for a user go to same partition
  • Event type + timestamp: For notifications.system.events - allows for parallel processing by event type
  • Segment ID: For notifications.broadcast.events - enables broadcast to specific user segments

Consumer groups:

  • notification-processing-group: Handles all notification processing logic
  • notification-delivery-group: Manages delivery to push providers (APNs, FCM)
  • notification-analytics-group: Processes analytics and metrics

Event Ordering Guarantees

Per-partition ordering: Kafka ensures strict ordering within partitions, which is crucial for:

  • User event sequence integrity (login → profile update → logout)
  • Delivery order consistency to push providers

Global ordering: For critical system events, we'll implement a global ordering mechanism using:

  • Sequence numbers in event headers
  • Deduplication logic in consumers

Exactly-once vs. At-least-once Processing Trade-offs

Decision: Exactly-once processing for user event streams, At-least-once for system events

Justification:

  • User events: Critical for maintaining state consistency (e.g., preventing duplicate login notifications)
  • System events: Less critical, but better to process once more than miss an alert

Implementation:

  • Idempotent consumers with proper state management
  • Transactional producers for multi-step operations
  • Kafka's transactional API for cross-service consistency

Event Replay and Reprocessing Strategy

Design approach:

  • Event retention: 30 days of event history for replay capability
  • Replay API: REST endpoint to trigger reprocessing by user ID or time range
  • Event versioning: Schema version tracking for backward compatibility
  • Audit logs: Full event history with metadata for debugging

Reprocessing triggers:

  • User data change (profile update, device token refresh)
  • Push provider failure (retry with exponential backoff)
  • System maintenance windows

Dead Letter Queue and Poison Pill Handling

Implementation:

  • DLQ topics: notifications.dlq.user.events, notifications.dlq.system.events
  • Retry logic: Max 3 retries with exponential backoff (1s → 5s → 15s)
  • Poison pill detection: Events that fail consistently after max retries
  • Alerting: Slack/PagerDuty notifications for DLQ growth above threshold

DLQ monitoring:

  • Daily DLQ size and failure rate metrics
  • Manual review process for failed events
  • Automated reprocessing of known good patterns

Recovery strategy:

  • Periodic DLQ analysis and manual reprocessing
  • Alert thresholds for DLQ queue size (500+ messages)
  • Manual intervention workflow for persistent failures

Cross-service Event Integration

Notification service events:

  • User login/logout → trigger push notifications
  • Profile updates → update device tokens or send update alerts
  • System maintenance → broadcast to all users

Integration with other services:

  • User service: User status changes, device token updates
  • Content service: Content availability, new content notifications
  • Analytics service: Engagement metrics, notification delivery tracking

Event transformation:

  • Standardized event format across services
  • Event enrichment with user context where needed
  • Consistent error handling and logging across service boundaries

Message Format and Serialization

Event format:

{
  "event_id": "uuid",
  "event_type": "user.login",
  "timestamp": "2023-12-01T10:00:00Z",
  "user_id": "user-123",
  "payload": {
    "device_token": "token-abc",
    "notification_data": {}
  },
  "schema_version": "1.0"
}

Serialization format:

  • Avro: Schema-aware, efficient binary serialization
  • JSON: Human-readable for debugging and development
  • Protocol Buffers: Alternative for high-performance scenarios

Performance Considerations

Throughput targets:

  • Peak QPS: 50,000 events/sec (based on 10M DAU with 5x average event rate)
  • Message size: ~2KB average (user events)
  • Latency targets: <50ms for event processing

Resource allocation:

  • Kafka cluster with 3 brokers (3x redundancy)
  • Partition count: 128 partitions per topic (scalable to 1000+)
  • Memory allocation: 8GB heap per Kafka broker
  • Disk storage: SSD-backed with 1TB per broker

Monitoring and Alerting

Key metrics to monitor:

  • Consumer lag (per topic/partition)
  • Producer throughput and latency
  • DLQ queue size and growth rate
  • Event processing success/failure rates
  • Partition distribution balance

Alert thresholds:

  • Consumer lag > 100,000 messages (critical)
  • DLQ growth > 500 events in 1 hour (warning)
  • Producer error rate > 1% (warning)
  • Partition imbalance > 20% (warning)

8. Reliability Engineering

Failure Mode Analysis

Single Points of Failure:

  • Notification Service API Gateway: If single point of failure, all notifications halt
  • Message Broker (Kafka): Central queue for all notification events - if down, no new notifications can be queued
  • Primary Database: If unavailable, notification state and user preferences cannot be read/written
  • Push Provider Integration (APNs/Firebase): If external provider goes down, notifications fail to send
  • Caching Layer: Redis cluster failure can cause cache misses and DB overload

Service-Level Failures:

  • User preference service: Unavailability affects notification filtering
  • Push provider adapters: Individual platform failures (iOS, Android) cause partial delivery
  • Event processing service: Failure to process events causes backlog and delay

Fault Tolerance Patterns

Circuit Breaker Pattern:

  • Applied to external push provider integrations (APNs, FCM)
  • Trip after 5 consecutive failures within 10 seconds
  • Reset after 30 seconds of successful calls
  • Prevents cascading failures when push providers are down

Bulkhead Pattern:

  • Isolated processing pools for different notification types (e.g., marketing vs. transactional)
  • Limits concurrent processing to prevent one notification type from starving others
  • Rate limiting per user/device to prevent burst overloads

Timeout & Retry Strategy:

  • 3-second timeout for push provider calls (APNs/Firebase)
  • Exponential backoff retry (1s, 2s, 4s) up to 3 attempts for transient failures
  • Retry with exponential jitter for distributed systems

Multi-Region / Multi-AZ Strategy

Active-Active Architecture:

  • Deploy in 3 regions (us-east-1, eu-west-1, ap-southeast-1)
  • Each region has independent notification service instances
  • Cross-region replication for user preferences and device tokens
  • Region failover with automatic traffic routing through DNS

Cross-Region Data Synchronization:

  • Use Kafka for cross-region event replication
  • Implement eventual consistency model with conflict resolution
  • Maintain user preference consistency across regions using distributed consensus

Chaos Engineering Strategy

GameDay Exercises:

  • Monthly chaos experiments simulating:
    • Push provider outages (iOS/Android)
    • Database connection failures
    • Network partitioning between services
    • Cache cluster unavailability

Steady-State Hypotheses:

  • Notification delivery rate stays within 95% of target
  • Error rates remain below 0.1% for critical paths
  • System recovers within 5 minutes of simulated failures

Disaster Recovery Plan

RTO/RPO Targets:

  • RTO: 5 minutes for primary notification delivery (99.9% availability)
  • RPO: 1 minute for user preferences and device token updates
  • Recovery Time: 30 minutes for complete region failover

Recovery Procedures:

  1. Automatic failover to secondary region
  2. Route traffic through healthy region via DNS failover
  3. Manual intervention for data consistency validation
  4. Gradual traffic restore once systems are stable

Runbook Design - Top 5 Failure Scenarios

1. Push Provider Outage (APNs/Firebase)

  • Alert: Critical service degradation
  • Action: Activate circuit breaker, queue notifications for retry
  • Recovery: Monitor provider status, resume after 30 minutes

2. Database Unavailability

  • Alert: Service degradation
  • Action: Failover to read replicas, enable cache-only mode
  • Recovery: Monitor DB health, restore primary after 15 minutes

3. Message Broker Failure

  • Alert: Event queue unavailable
  • Action: Switch to local in-memory queue, enable retry logic
  • Recovery: Restart Kafka cluster, replay pending events

4. Cache Cluster Failure

  • Alert: Increased latency and DB load
  • Action: Enable cache warming, reduce cache TTLs
  • Recovery: Restart Redis cluster, warm caches with user data

5. Region-wide Outage

  • Alert: Complete system unavailability
  • Action: Failover to secondary region, redirect traffic
  • Recovery: Restore primary region, validate data consistency

Resilience Testing Framework

Automated Testing:

  • Integration tests with mocked external providers
  • Load testing with failure injection (e.g., 5% of requests fail)
  • Performance regression tests to ensure no degradation under stress

Monitoring & Alerting:

  • Service health checks every 30 seconds
  • Error rate thresholds (5% for critical, 1% for non-critical)
  • Latency monitoring with p99 alerts at 200ms threshold

Service Degradation Handling:

  • Graceful degradation to email or SMS fallback when push fails
  • Priority queue handling for mission-critical notifications
  • Retry logic with exponential backoff for transient failures

This reliability architecture ensures the push notification service maintains high availability and graceful degradation under various failure conditions while providing clear recovery procedures for operational teams.

9. Observability Architecture

Logging Stack

  • Structured JSON logging with consistent log format across all services using standardized fields (timestamp, service, level, trace_id, span_id, message, error_stack)
  • Loki as the centralized logging system with multi-tenant support for different teams and environments
  • Log aggregation pipeline with log shipping via Fluentd/FluentBit to Loki, supporting log rotation and compression
  • Log retention policy of 30 days for operational logs, 90 days for audit logs
  • Log filtering and search capabilities via Grafana dashboard with regex support and log streaming for real-time debugging

Distributed Tracing

  • OpenTelemetry as the tracing framework with automatic instrumentation for HTTP and gRPC services
  • Jaeger as the distributed tracing backend with sampling rate of 1% for production traces
  • Trace context propagation through HTTP headers and gRPC metadata to maintain trace continuity across service boundaries
  • Service map visualization in Grafana to identify performance bottlenecks and dependencies between services
  • Error correlation between logs and traces to enable root cause analysis

Metrics Stack

  • Prometheus as the primary metrics backend with 15s scrape interval for critical services
  • Grafana as the dashboarding platform with pre-built dashboards for each service and system-wide views
  • Service-level metrics including QPS, latency (p50/p95/p99), error rates, and throughput
  • Infrastructure metrics including CPU, memory, disk I/O, and network utilization per service pod
  • Custom metrics for push notification delivery rates, retry counts, and queue depths

Alerting Strategy

  • Severity levels:
    • P0: Critical alerts (e.g., >5% of notifications failing, service down)
    • P1: High alerts (e.g., >20% of notifications delayed, latency >500ms)
    • P2: Medium alerts (e.g., >10% of notifications dropped, queue depth >10k)
  • Alert routing via PagerDuty for P0/P1, OpsGenie for P2
  • Alert deduplication using alert labels and grouping to prevent noise
  • Silence rules for scheduled maintenance windows and known issues

SLI/SLO Definition and Error Budget Tracking Dashboard

  • SLIs:
    • Delivery success rate (99.5% SLO)
    • Notification delivery latency (<200ms p99)
    • System availability (99.9% SLO)
  • SLO tracking dashboard in Grafana with error budget burn rate visualization
  • Automated alerting when error budget is at 20% or less for 5 minutes
  • Rolling window SLOs with 30-day windows to prevent gaming of metrics

Capacity Planning and Load Forecasting Approach

  • Historical capacity analysis using time-series data from Prometheus to identify usage patterns
  • Load forecasting model using linear regression and seasonal decomposition for peak prediction
  • Proactive scaling triggers based on forecasted load with 2-hour advance warning
  • Capacity planning dashboard showing projected resource utilization and forecasted growth
  • Auto-scaling integration with forecasting data to prevent over-provisioning or under-provisioning

10. API Gateway & Edge Layer

API Gateway Selection

For a push notification service, I recommend AWS API Gateway as the primary API gateway with custom edge services for specific traffic handling needs. This decision balances managed service benefits with the flexibility required for push notification routing and transformation.

Justification:

  • Managed scalability: Handles millions of requests with automatic scaling
  • Built-in rate limiting and throttling for abuse protection
  • Authentication integration with Cognito/Custom JWT providers
  • Request/response transformation capabilities for cross-platform compatibility
  • Monitoring and logging with CloudWatch integration
  • Cost efficiency: Pay-per-request model with no minimum commitments

Request Routing Strategy

The API gateway will implement a multi-tier routing strategy:

  1. Platform-aware routing - Route to appropriate notification service based on device type (iOS, Android, Web)
  2. Service versioning - Support for v1/v2 notification endpoints
  3. Geographic routing - Route to nearest regional service for latency optimization
  4. Priority-based routing - Critical notifications (e.g., security alerts) to high-priority service paths

Authentication and Authorization

At Gateway Layer:

  • JWT-based authentication for mobile app clients using RS256 signing
  • Service-to-service authentication via AWS IAM roles and custom headers for internal services
  • API key validation for third-party integrations (with rate limiting)
  • OAuth2 integration for partner platforms using client credentials flow

Rate Limiting and Throttling Design

Global Limits:

  • 10,000 requests/minute per client (for mobile apps)
  • 500 requests/minute per API key (for partners)
  • 100,000 requests/minute for internal services

Per-Client Limits:

  • Device-level rate limiting (50 notifications/hour/device)
  • User-level rate limiting (1000 notifications/day/user)
  • Application-level rate limiting (50,000 notifications/day/app)

Request Transformation and Protocol Translation

Protocol Translation:

  • Convert mobile push format (APNs/FCM) to internal notification format
  • Handle different payload structures across platforms
  • Normalize timestamp formats and device identifiers

Request Transformation:

  • Input validation: Schema validation for notification payloads
  • Payload enrichment: Add device metadata, user preferences, and segmentation data
  • Content compression: Handle large payloads with gzip compression
  • Format normalization: Convert between JSON and binary formats

BFF (Backend for Frontend) Design

Mobile App BFF:

  • Platform-specific optimizations: iOS APNs and Android FCM specific handling
  • Batch processing: Combine multiple notifications into single push payloads
  • User preference integration: Apply user segmentation and opt-out preferences
  • Push history management: Track delivery status and retry logic

Web BFF:

  • WebSocket integration: Real-time notification streaming for web clients
  • Server-sent events (SSE): Alternative real-time delivery mechanism
  • Browser compatibility: Handle different web notification APIs (Web Push, Service Workers)

Edge Layer Considerations

Global Edge Services:

  • Content Delivery Network (CDN) integration for static notification templates
  • Regional edge locations to minimize latency for global users
  • Caching layer for frequently accessed notification templates and metadata

Traffic Management:

  • Load balancing across regional notification services
  • Traffic splitting for A/B testing of new notification formats
  • Graceful degradation during service outages (fallback to email/SMS)

Security Considerations

Edge Security:

  • DDoS protection through AWS Shield integration
  • WAF rules for common attack patterns (SQL injection, XSS)
  • Request size limiting to prevent resource exhaustion
  • IP whitelisting for trusted partner integrations

Performance Optimization

Caching Strategy:

  • Template caching: Store frequently used notification templates at edge
  • Device metadata caching: Cache device tokens and preferences for 15-minute TTL
  • Rate limit state caching: Track client request counts in distributed cache

Latency Optimization:

  • Pre-warming: Keep edge services warm with periodic health checks
  • Connection pooling: Reuse connections to downstream notification services
  • Async processing: Handle non-critical operations asynchronously at edge

Monitoring and Observability

Gateway Metrics:

  • Request/response latency per endpoint
  • Error rates and retry counts
  • Authentication/authorization success/failure rates
  • Rate limit violations and throttled requests

Alerting Configuration:

  • Critical alerts: >95% error rate in notification delivery
  • Warning alerts: >80% rate limit violations
  • Info alerts: 5xx errors, latency spikes >200ms

This API gateway and edge layer design provides the flexibility to handle diverse notification formats while maintaining security, scalability, and performance at global scale.

11. Authentication & Authorization Architecture

Identity Provider Integration

For a push notification service, we'll integrate with AWS Cognito as the identity provider due to its seamless integration with AWS services, built-in support for federated identities, and robust user pool management capabilities. Cognito provides the flexibility to support both social identity providers (Google, Facebook) and custom authentication flows while offering built-in MFA and user management features.

OAuth2 / OIDC Flow Design

sequenceDiagram
    participant Client
    participant APIGateway
    participant AuthService
    participant Cognito
    participant TokenService

    Client->>APIGateway: Request with Authorization Header
    APIGateway->>AuthService: Validate JWT token
    AuthService->>Cognito: Verify token signature and claims
    Cognito-->>AuthService: Token validation result
    AuthService->>TokenService: Fetch user permissions
    TokenService-->>AuthService: User role/permissions
    AuthService-->>APIGateway: Allow/Deny access
    APIGateway-->>Client: 200/403 response

The system implements the Authorization Code flow with PKCE for mobile and web clients, supporting both OAuth2 and OpenID Connect (OIDC) standards. This ensures secure token exchange and proper user identity validation.

Service-to-Service Authentication

For inter-service communication, we'll implement mTLS using SPIFFE/SPIRE for secure service identity and mutual TLS validation. Each service will have a unique SPIFFE ID that is validated at the transport layer, ensuring that only authorized services can communicate with each other. Service accounts will be used for backend-to-backend authentication, with short-lived tokens generated through a service token exchange mechanism.

RBAC / ABAC Model Design

We'll implement a Role-Based Access Control (RBAC) model with granular permissions for the push notification service. Users will be assigned roles such as:

  • admin: Full access to all push notification features and user management
  • publisher: Ability to create and send notifications, view analytics
  • viewer: Read-only access to notification history and analytics

The system will also support Attribute-Based Access Control (ABAC) for more complex permission rules, particularly for enterprise customers who need to define permissions based on user attributes like department, location, or device type.

Token Revocation Strategy

We'll implement a hybrid approach combining short-lived JWT tokens (15-30 minutes) with refresh token rotation. When a user logs out or their session expires, refresh tokens are invalidated through a centralized token revocation store. This approach provides better security than JWT blacklisting while maintaining good performance for frequent API calls.

For service-to-service communication, we'll use short-lived tokens with automatic refresh mechanisms to ensure security without impacting performance.

Token Management Architecture

The authentication system will include:

  • JWT token generation with custom claims (user ID, roles, permissions)
  • Token validation service that checks signature and expiration
  • Refresh token rotation with automatic session cleanup
  • Token introspection endpoint for service-to-service validation
  • Session management with timeout and idle session detection

Security Considerations

All authentication flows will use HTTPS/TLS termination at the API Gateway level, with strict certificate validation. The system will implement rate limiting on authentication endpoints to prevent brute force attacks and will use secure random generation for token creation. Session tokens will be stored in secure HTTP-only cookies with SameSite attributes to prevent CSRF attacks.

The authentication system will support multi-factor authentication for admin users and provide audit logging of all authentication events for security monitoring.

12. Search Architecture

Search Use Cases and Requirements

The push notification service does not require core search capabilities as its primary function is delivering targeted notifications based on user preferences, device tokens, and event triggers. However, if search capabilities are added for:

  • User segmentation and targeting
  • Content-based notification filtering
  • Analytics dashboard search over notification history
  • Admin console search over user devices and preferences

Search Indexing Pipeline Design

Source → Transform → Index

  1. Source:

    • User profile data (preferences, device tokens)
    • Notification templates and content
    • Event metadata (time, type, target segments)

  2. Transform:

    • Normalize user preferences into searchable fields
    • Extract keywords from notification content for relevance scoring
    • Enrich with metadata (user location, device type, time zones)

  3. Index:

    • Elasticsearch cluster for full-text search capabilities
    • Index per user segment with optimized field mappings

Index Schema and Query Patterns

Primary Search Fields:

  • user_id (keyword) - for exact user lookups
  • notification_type (keyword) - for category filtering
  • content_text (text) - for full-text search
  • tags (keyword array) - for tag-based filtering
  • created_at (date) - for time-range queries

Query Patterns:

  • User-specific notification history search
  • Content-based filtering by keyword or tag
  • Real-time user preference matching for targeting

Real-time vs. Near-real-time Indexing Trade-offs

Real-time indexing (1-2s latency):

  • Pros: Immediate content availability, better for user-facing search
  • Cons: Higher resource utilization, potential consistency issues

Near-real-time indexing (10s-30s latency):

  • Pros: Lower resource overhead, better for batched content updates
  • Cons: Delayed search results for user-facing features

Decision: Near-real-time indexing with 15s refresh window for content search, real-time for user preference matching.

Relevance Tuning Strategy

  • TF-IDF scoring for content relevance
  • Custom boosting for user preference matches
  • Time decay function for recent notifications
  • Query-time relevance adjustments via admin console

Search Scaling Design

Sharding Strategy:

  • 3 shards per index for horizontal scaling
  • Hash-based partitioning on user_id for even distribution
  • 2 replicas per shard for high availability

Performance Considerations:

  • Index size: ~50GB per month (estimated)
  • Query latency target: p95 < 200ms for user search, p95 < 500ms for content search
  • Search throughput: 10,000 concurrent queries at peak

Indexing Pipeline Implementation

[User Preferences] → [Elasticsearch Mapping] → [Search Index]
[Notification Content] → [Content Extraction] → [Search Index]
[Event Metadata] → [Tagging & Enrichment] → [Search Index]

Search Service Integration

API Endpoints:

  • POST /search/users - User preference and device search
  • POST /search/notifications - Content-based notification search

Search Service Boundaries:

  • Isolated from core notification service to prevent performance bottlenecks
  • Separate read replicas for search workloads
  • Dedicated Elasticsearch cluster (3 nodes) with 16GB RAM each

Search Result Caching Strategy

  • Cache search results for popular queries (user history, recent notifications)
  • 5-minute TTL for user-specific search results
  • 1-hour TTL for content-based search queries

Search Feature Rollout

Phase 1: Basic user preference search (no content search) Phase 2: Content-based notification search with keyword matching Phase 3: Advanced analytics dashboard search capabilities

Search Service Dependencies

  • Elasticsearch cluster (managed service or self-hosted)
  • Redis for result caching
  • Kafka for search index updates (event-driven)
  • Admin console for search configuration and tuning

Search Performance Monitoring

  • Query latency metrics (p50, p95, p99)
  • Index size and growth rate tracking
  • Cache hit ratio for search results
  • Resource utilization (CPU, memory, disk I/O)

13. Scalability Deep Dive

Bottleneck Identification

Database Layer:

  • Peak QPS: 100K (assuming 10M DAU with 10:1 peak-to-average ratio)
  • DB QPS after caching: ~20K (assuming 80% cache hit rate)
  • Per-user write operations: ~100 events/day → 500MB/year storage per user
  • Storage growth: 5TB/year for 10M users (assuming 500MB/user)
  • Read-heavy workload with 80% reads, 20% writes

Service Layer:

  • Push service processing: 100K QPS → 250ms avg latency
  • Service stateless design allows horizontal scaling
  • Memory usage per instance: 2GB (with 50K concurrent connections)
  • CPU utilization: 60% for processing, 40% for I/O

Network Layer:

  • Peak bandwidth: 5GB/s (read + write)
  • CDN edge cache hit rate: 70% for content delivery
  • Mobile push gateway (APNs/Firebase) throttling limits

Horizontal vs. Vertical Scaling Decisions

Push Service:

  • Horizontal scaling preferred due to stateless nature
  • Auto-scaling based on QPS and latency metrics
  • Target: 500 concurrent connections per instance
  • Vertical scaling for memory-intensive operations (batch processing)

Database Layer:

  • Vertical scaling initially (larger instances, read replicas)
  • Horizontal sharding for user data by region or user ID hash
  • Read replicas for analytical queries (user behavior analysis)

Caching Layer:

  • Horizontal scaling with Redis cluster
  • Memory capacity: 8GB per cache node (for hot user data)
  • Auto-scaling based on memory usage and hit rate

Database Scaling Path

Phase 1 (0-5M users):

  • Single PostgreSQL instance with read replicas
  • 2x read replicas for analytical queries
  • Memory-optimized instance (500GB storage)

Phase 2 (5M-50M users):

  • Horizontal sharding by user ID hash
  • 3x read replicas per shard
  • Connection pooling (500 connections per instance)
  • Multi-region read replicas for global distribution

Phase 3 (50M+ users):

  • Distributed database (CockroachDB or Spanner)
  • Multi-region active-active deployment
  • Cross-region read replicas with automatic failover

Stateless vs. Stateful Service Design

Push Service:

  • Fully stateless with minimal in-memory session data
  • Session affinity not required for core functionality
  • Service instances can be replaced without impact on user sessions

User State Management:

  • User preferences and device tokens stored in stateful services
  • Session data managed through Redis cache layer
  • State synchronization handled via event streaming

Session Affinity and Implications

No session affinity required for push processing

  • Push notifications are event-driven, not session-dependent
  • Each notification is processed independently
  • Load balancing across service instances is optimal

Device token management:

  • Device tokens stored in Redis with TTLs (30 days)
  • Token refresh handled via background service
  • Session invalidation on device unregistration

Performance Characteristics

Latency Budget:

  • p50: 100ms (push delivery)
  • p95: 250ms (with retries and fallbacks)
  • p99: 500ms (including error handling)

Throughput Targets:

  • Average QPS: 10K
  • Peak QPS: 100K (10x average)
  • Concurrent push operations: 5000

Resource Utilization:

  • CPU: 60% for processing, 40% for I/O
  • Memory: 2GB per instance (with 50K concurrent connections)
  • Network: 5GB/s peak bandwidth
  • Storage: 5TB/year growth rate

Auto-scaling Strategy

Push Service Scaling:

  • HPA based on QPS and latency metrics
  • Target: 100 QPS per instance (with 500 concurrent connections)
  • Scale up when QPS > 120 or latency > 250ms
  • Scale down when QPS < 80 or latency < 100ms

Database Scaling:

  • Vertical scaling for read replicas (up to 8x instance size)
  • Horizontal sharding when user count exceeds 5M
  • Auto-sharding based on user ID range distribution

Cache Layer Scaling:

  • HPA based on memory usage and cache hit rate
  • Target: 8GB per Redis instance (with 500K concurrent keys)
  • Scale up when memory usage > 80% or hit rate < 75%

Infrastructure Considerations

Compute Resources:

  • Instance type: m5.xlarge (4 vCPU, 16GB RAM)
  • Auto-scaling groups with minimum 5 instances
  • Spot instances for non-critical processing (20% of capacity)

Storage Scaling:

  • SSD storage with 10K IOPS per instance
  • Multi-zone redundancy for high availability
  • Tiered storage with hot/warm/cold data classification

Network Scaling:

  • 10Gbps network interface per instance
  • CDN integration for content delivery (70% cache hit rate)
  • Global load balancing with health checks

Capacity Planning

Peak Load Projections:

  • 10M DAU with 100K peak QPS
  • 500 concurrent push operations per instance
  • 5GB/s bandwidth requirement at peak

Resource Requirements:

  • 200 service instances for peak load (100 QPS per instance)
  • 50 Redis cache nodes for hot user data
  • 10 database instances with read replicas

Growth Projections:

  • 2x growth every 18 months
  • 50% increase in QPS with each major release
  • 30% increase in storage capacity annually

Scaling Patterns Applied

Microservices Scaling:

  • Independent scaling of push service, user service, and analytics service
  • Service-specific auto-scaling policies based on usage patterns
  • Cross-service coordination through event streaming

Database Sharding:

  • User ID hash-based sharding for even distribution
  • Range-based sharding for time-series data (notification history)
  • Cross-shard query handling through distributed processing

Cache Layer Scaling:

  • Multi-tier caching (L1 in-app, L2 Redis cluster)
  • Cache warming strategies for new user segments
  • Memory-efficient data structures (Redis hash, set, list)

This scalability design ensures linear growth in system capacity with minimal architectural changes, supporting rapid user base expansion while maintaining consistent performance and reliability.

14. Infrastructure & Deployment Architecture

Cloud Provider and Region Selection

AWS is selected as the primary cloud provider due to:

  • Mature ecosystem with extensive global infrastructure (50+ regions)
  • Strong support for microservices and containerized workloads
  • Comprehensive suite of managed services (SNS, SQS, Lambda, ECS)
  • Cost optimization features like Reserved Instances and Spot Instances
  • Established compliance certifications (SOC2, HIPAA, GDPR)

Region Strategy: Multi-region deployment with primary region (us-east-1) and secondary region (eu-west-1) for high availability. Cross-region replication will be implemented for disaster recovery.

Kubernetes Cluster Design

Cluster Architecture:

  • Multi-tenant clusters with dedicated namespaces per service (notification, user, device)
  • Resource quotas at namespace level to prevent resource contention
  • Node pools with different instance types (general purpose, compute optimized) based on service requirements

Namespace Strategy:

  • notification-system - core notification services
  • user-management - user and device management services
  • analytics - monitoring and reporting services
  • infrastructure - shared infrastructure components

Container Image Strategy

Base Images:

  • Alpine Linux for minimal footprint
  • Official language-specific base images (node:alpine, python:slim)
  • Security scanning integrated in CI pipeline

Multi-stage Builds:

  • Build stage with full toolchain
  • Runtime stage with minimal dependencies only
  • Size optimization reducing image size by 70-80%

Image Registry:

  • Private ECR repositories with image scanning
  • Tagging strategy: v1.2.3-{git-sha} for traceability

Deployment Strategy

Rolling Updates:

  • 10% traffic increments with health checks
  • Max unavailable pods set to 25% for graceful transitions

Blue-Green Deployment:

  • For major version updates (v2.x.x)
  • Zero-downtime deployment with traffic switch
  • Rollback capability via DNS record swap

Canary Analysis:

  • Traffic split: 5% canary, 95% baseline
  • Auto-rollback if error rate exceeds 1%
  • Success criteria: p99 latency < 200ms, error rate < 0.1%

Auto-scaling Design

Horizontal Pod Autoscaler (HPA):

  • Scale based on CPU utilization (80% target)
  • Memory-based scaling for stateful services
  • Minimum 3 replicas per service for availability

Vertical Pod Autoscaler (VPA):

  • Memory and CPU optimization based on historical usage
  • Prevents over-provisioning while maintaining performance

KEDA (Kubernetes Event-Driven Autoscaling):

  • Auto-scale based on SQS queue length for message processing
  • Scale to zero for idle services (e.g., reporting service)
  • Custom metrics support for business-specific triggers

Infrastructure as Code Approach

Terraform is selected as the primary IaC tool due to:

  • Strong ecosystem and module support for AWS services
  • State management with remote backend (S3 + DynamoDB)
  • Version control integration and change tracking
  • Multi-environment support (dev, staging, prod)

Module Organization:

infrastructure/
├── vpc.tf
├── clusters/
│   ├── main.tf
│   └── variables.tf
├── services/
│   ├── notification-service.tf
│   └── user-service.tf
└── monitoring/
    ├── alerts.tf
    └── dashboards.tf

GitOps Integration:

  • ArgoCD for declarative deployment management
  • Automated drift detection and reconciliation
  • Rollback capability through git history

Environment Management:

  • Separate Terraform workspaces for dev, staging, prod
  • Parameterized variables for different environments
  • Automated testing in dev environment before promotion

Security and Compliance:

  • Terraform state encryption at rest
  • IAM roles with least privilege principle
  • VPC security groups and network ACLs enforced through IaC
  • Automated compliance checks (TFLint, tfsec) in CI pipeline

Monitoring and Alerting Integration:

  • CloudWatch integration for infrastructure metrics
  • Prometheus exporters for containerized services
  • Automated alerting rules based on infrastructure thresholds

Cost Optimization:

  • Reserved instances for predictable workloads
  • Spot instances for batch processing and non-critical services
  • Resource tagging for cost allocation and tracking
  • Automated cleanup of unused resources (e.g., old EBS volumes, unused IAM roles)

15. Security Architecture

VPC Design and Network Segmentation

The system operates within a multi-AZ VPC architecture with strict network segmentation:

  • Public Subnets: Host API gateway and edge services with public-facing load balancers
  • Private Subnets: Contain all backend services, databases, and cache layers with no direct internet access
  • DMZ Subnets: Dedicated for third-party integrations and external API connections
  • Security Groups: Fine-grained rule sets with least-privilege access between services (e.g., API gateway can only communicate with notification service, not directly to database)
  • NACLs: Stateful network access control lists for additional layer of protection at subnet level

Zero-Trust Network Model

All internal communication is enforced through zero-trust principles:

  • mTLS: All inter-service communication uses mutual TLS with certificate-based authentication
  • Service Mesh Integration: Istio or Linkerd for enforcing zero-trust policies, including request authentication, authorization, and encryption
  • Network Policies: Kubernetes NetworkPolicies to enforce strict service-to-service communication rules
  • Identity-Based Access Control: All services authenticate using their service identities rather than shared credentials

TLS Termination Strategy

  • Edge Layer: TLS termination at the API gateway or load balancer level (AWS ALB or NGINX) with strong cipher suites (TLS 1.3+)
  • Internal Services: End-to-end encryption maintained through mTLS or service mesh, with internal services not handling TLS termination
  • Certificate Management: Integration with AWS Certificate Manager (ACM) or HashiCorp Vault for certificate lifecycle management

DDoS Protection Strategy

  • AWS Shield Advanced: Enabled at the VPC level for DDoS protection with automatic mitigation
  • Rate Limiting: API gateway rate limiting to prevent abuse and DDoS attacks at the edge
  • CloudFront Integration: For global DDoS protection and caching of common requests
  • Traffic Filtering: Integration with AWS WAF for filtering malicious traffic patterns

Secrets Management Architecture

  • Centralized Secret Store: HashiCorp Vault or AWS Secrets Manager for all sensitive data (API keys, database credentials, third-party tokens)
  • Automated Rotation: Secrets rotated every 90 days with automated secret rotation workflows
  • Service Account Integration: Services authenticate to secret stores using IAM roles or Vault service accounts with least privilege
  • Encryption at Rest: All secrets encrypted using AES-256 with key rotation policies

Vulnerability Scanning in CI/CD

  • Image Scanning: Trivy or Snyk integration in CI pipeline to scan container images for known vulnerabilities
  • Dependency Scanning: Automated scanning of application dependencies for security issues (OWASP ZAP, SonarQube)
  • Infrastructure Scanning: Terraform and Kubernetes manifest scanning for misconfigurations using Checkov or Conftest
  • Compliance Checks: Automated compliance checks against CIS benchmarks and NIST standards

WAF Rules for Critical Endpoints

  • API Gateway WAF: AWS WAF rules protecting critical endpoints from OWASP top 10 vulnerabilities
  • Rate-Based Rules: Protection against brute force attacks and excessive request rates
  • IP Allowlisting: Whitelisting of known trusted IP ranges for administrative endpoints
  • SQL Injection Protection: Rules to detect and block SQL injection attempts in query parameters
  • Cross-Site Scripting (XSS) Protection: Rules to prevent XSS attacks in user input fields

Security Controls Implementation

  • Audit Logging: All security-relevant events logged to CloudWatch or SIEM for compliance and forensic analysis
  • Compliance Monitoring: Automated checks for SOC2, HIPAA, GDPR compliance with real-time alerts
  • Threat Detection: Integration with AWS GuardDuty or similar threat detection services for anomaly detection
  • Access Control Review: Quarterly review of service and user permissions with automated alerts for privilege drift

16. Deployment & Release Strategy

Deployment Strategy and Traffic Management

The push notification service will utilize a blue-green deployment strategy for production releases to ensure zero-downtime deployments and smooth rollback capabilities. This approach involves maintaining two identical production environments where only one is active at any given time. When deploying a new version, the system routes traffic to the inactive environment, validates the deployment, and then switches traffic to the new version while decommissioning the old one.

Traffic management will be handled through Kubernetes Ingress controllers with weighted routing capabilities to support gradual traffic shifts during canary deployments for new features or experimental changes. The system will implement traffic splitting using Istio's traffic management capabilities to route a small percentage of traffic to new versions for validation before full promotion.

Artifact Promotion Model

The deployment pipeline follows a trunk-based development model with continuous integration and deployment. Artifacts are promoted through the following stages:

  1. Development: Code committed to main branch with automated unit and integration tests
  2. Staging: Automated testing in staging environment including load and performance tests
  3. Production Gates: Manual approval process for production deployments with automated validation checks:
    • Health check endpoints
    • Performance benchmarks (p99 latency < 500ms)
    • Error rate thresholds (< 0.1% error rate)
    • Database connection pool health

Zero-Downtime Database Migration Strategy

Database migrations will be executed using online schema change tools like gh-ost or pt-online-schema-change to avoid table locks during schema modifications. For major schema changes, the system will:

  1. Create new columns with default values in parallel
  2. Migrate data in batches to avoid locking tables
  3. Use read replicas for migration operations when possible
  4. Implement database versioning with migration status tracking in a dedicated metadata table

Feature Flag Integration and Progressive Rollout Design

Feature flags will be integrated through a centralized feature management service (e.g., LaunchDarkly or custom solution) to enable:

  • Gradual rollouts: Start with 5% of users, increase to 100% based on performance metrics
  • Risk-based rollouts: Roll out to specific user segments (e.g., beta testers, internal users) first
  • A/B testing: Support for concurrent feature variants with analytics tracking

The system will support feature flag configuration via API and integrate with the service's configuration management to dynamically enable/disable features without code changes.

Canary Analysis Automation and Auto-Rollback Triggers

Canary deployments will include automated analysis with the following triggers:

  • Error rate monitoring: If error rate exceeds 0.5% for more than 2 minutes, trigger rollback
  • Latency degradation: If p99 latency increases by more than 50% for 5 minutes, trigger rollback
  • Throughput reduction: If QPS drops below 80% of baseline for 10 minutes, trigger rollback

Auto-rollback will be implemented using Kubernetes deployment rollbacks and service versioning to quickly revert to stable versions.

Release Cadence and Branching Strategy

The system follows a trunk-based development approach with the following release cadence:

  • Minor releases: Every 2 weeks for feature enhancements and bug fixes
  • Major releases: Every 6-8 weeks for breaking changes or significant feature sets
  • Hotfixes: Immediate deployment for critical production issues

Branching strategy uses:

  • main branch for stable releases
  • Feature branches for development work (short-lived)
  • Release branches for patch releases and hotfixes

Release automation includes:

  • Semantic versioning (MAJOR.MINOR.PATCH)
  • Automated changelog generation
  • Release notes with impact assessment and rollback procedures

Deployment Pipeline Components

The deployment pipeline includes:

  1. CI/CD Pipeline: GitHub Actions or Jenkins-based pipeline with automated testing
  2. Artifact Registry: Docker image storage with versioning and security scanning
  3. Deployment Automation: Kubernetes operator or custom deployment controller
  4. Validation Suite: Pre-deployment health checks and performance benchmarks
  5. Rollback Mechanism: Automated rollback capability with manual override option

Release Risk Mitigation

Release risk is mitigated through:

  • Comprehensive testing: Unit, integration, and end-to-end tests with 95% coverage
  • Staging environment parity: Production-like staging for validation
  • Gradual rollout: Controlled release to minimize impact of failures
  • Automated monitoring: Real-time alerting for deployment issues
  • Manual approval gates: Critical production changes require human review

Rollback Capability Design

The system maintains rollback capability through:

  • Versioned deployments: Kubernetes deployment versions with clear rollback paths
  • Database migration tracking: Migration status stored in database for reverse operations
  • Configuration versioning: Service configuration with rollback capability
  • Traffic routing control: Ingress controller configuration that can quickly switch traffic
  • Deployment history: Complete deployment logs and rollback metadata stored in monitoring system

This approach ensures that the push notification service can be deployed with confidence, maintaining high availability and minimizing risk during release cycles.

17. Cost Architecture

Major Cost Drivers Analysis

Compute Costs: The primary driver is the push notification processing engine, which will run on EC2 instances or Kubernetes pods. With peak QPS of ~10k and assuming 200ms processing time per notification, we need approximately 5000 vCPUs at peak. This translates to roughly $15,000/month for on-demand instances in us-east-1.

Storage Costs:

  • Database storage (PostgreSQL/MySQL) for user preferences, device tokens, and notification metadata: ~200GB at $0.10/GB = $20/month
  • Message queue storage (Kafka/SQS): ~50GB at $0.10/GB = $5/month
  • Cache storage (Redis): ~20GB at $0.05/GB = $1/month
  • Object storage (S3) for notification templates and media: ~50GB at $0.02/GB = $1/month

Network & Data Transfer Costs:

  • Ingress traffic (from mobile apps): Minimal, ~$0.01/GB
  • Egress traffic (to mobile devices): High - assuming 100GB of data transfer per day at $0.09/GB = ~$300/month

Third-party Service Costs:

  • Push gateway providers (APNs, FCM): $0.001 per successful delivery (assumes 1M notifications/day)
  • SMS gateway integration: $0.005 per message (if included)

Cost Optimization Strategy

Reserved vs Spot Instances:

  • For the notification processing service, use 20% reserved instances for predictable workloads and 80% spot instances for burst capacity to reduce compute costs by ~30%.

Storage Tiering:

  • Implement hot/warm/cold storage tiers:
    • Hot: Recent user activity (last 30 days) stored in SSDs
    • Warm: Historical data (30-90 days) stored in cheaper HDDs
    • Cold: Archive data (>90 days) in Glacier or S3 Standard-IA

Database Cost Optimization:

  • Use read replicas for reporting queries to reduce primary DB load
  • Implement connection pooling to minimize database resource usage
  • Use multi-AZ deployments for redundancy but avoid over-provisioning

Data Egress Cost Minimization

CDN Integration:

  • Route push notifications through CloudFront or similar CDN to reduce direct egress costs from data centers
  • Cache frequently used notification templates and media assets

Batch Processing:

  • Batch multiple notifications to reduce per-message processing overhead
  • Reduce the frequency of individual message sends by aggregating notifications where appropriate

FinOps Dashboards and Budget Alerts

Cost Monitoring:

  • Implement Grafana dashboards with Prometheus to track real-time cost metrics by service and team
  • Set up AWS Cost Explorer alerts for monthly spend thresholds (e.g., 80% of budget)
  • Create cost allocation tags to attribute costs to specific teams or features

Budget Management:

  • Establish monthly budgets per team with automatic alerts when thresholds are exceeded
  • Implement cost allocation models to ensure teams understand their impact on overall system costs

Optimization Automation:

  • Set up auto-scaling policies that reduce compute resources during low-traffic periods
  • Configure cost optimization rules in Terraform to enforce best practices (e.g., default to spot instances for non-critical workloads)

18. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

  • Notification prioritization strategy: Should we implement a priority-based queuing system where high-priority notifications (e.g., security alerts) bypass lower-priority ones? This requires alignment on what constitutes "high priority" and how to weight different notification types.

  • Cross-platform support granularity: How granular should our platform-specific handling be? Should we support device-specific content (e.g., iOS rich notifications vs Android cards), or maintain a single universal payload?

  • Push delivery guarantee levels: Do we need to support "delivered" vs "opened" tracking? This affects the complexity of our event processing and the reliability requirements for downstream systems.

Key architectural risks

RiskImpactLikelihoodMitigation
Single point of failure in notification deliveryHigh (system-wide outage)MediumImplement multi-region deployment with failover and redundant delivery channels
Message loss during high-volume burstsHigh (user experience degradation)MediumUse durable message queues with dead letter handling and retry mechanisms
Cache invalidation stormsMedium (system instability)Low-MediumImplement cache warming strategies and limit concurrent invalidations
Rate limiting bypassesHigh (DDoS amplification)LowEnforce strict rate limits at API gateway and service level with circuit breaker patterns
Data consistency in multi-region setupsHigh (user confusion)MediumImplement eventual consistency with conflict resolution patterns and audit logs

Trade-offs made with justification

  • Consistency vs. Availability: Chose eventual consistency over strong consistency for notification metadata to enable horizontal scaling and reduce latency in high-throughput scenarios. This trade-off is acceptable because users don't expect real-time consistency of notification status.

  • Monolithic vs. Microservices: Selected a microservices approach for better scalability and team autonomy, but acknowledged the complexity of cross-service communication and distributed transaction management.

  • Push vs. Pull model: Chose push-based notifications over pull-based to reduce client-side resource consumption and provide more responsive user experience, despite the added complexity of managing connection states.

Alternatives considered and rejected

  • Traditional push notification services (e.g., Firebase Cloud Messaging): Rejected due to vendor lock-in and limited customization capabilities for enterprise requirements.

  • Simple HTTP polling: Rejected due to high client resource consumption and poor user experience for real-time notifications.

  • Full event sourcing: Rejected due to added complexity and overhead for a system that primarily needs to deliver notifications rather than maintain complete audit trails.

Technical debt acknowledged

  • State management complexity: The system's stateful nature (device tokens, user preferences) introduces potential for inconsistency and requires careful state synchronization mechanisms.

  • Cross-service dependency coupling: Notification service depends on user profile, device management, and analytics services, creating tight coupling that increases system fragility.

  • Platform-specific content handling: The need to maintain different content formats per platform introduces maintenance overhead and increases the risk of inconsistency in user experience.

  • Real-time delivery guarantees: The system currently provides best-effort delivery with no guaranteed delivery, which may require future investment in more robust delivery confirmation mechanisms.

19. Phased Implementation Roadmap

Phase 1: MVP (Minimum Viable Product)

Timeline: 8-12 weeks Core Focus: Basic push notification delivery with essential features

Deliverables:

  • Core notification service with REST API for sending notifications
  • Support for 2 major mobile platforms (iOS/Android) via APNS/Firebase
  • Basic user/device registration and token management
  • Simple message queue integration (SQS/Kafka)
  • Basic database schema for device tokens and notification history
  • Simple dashboard for sending test notifications and viewing delivery stats

Dependencies:

  • Mobile app integration (client-side SDK)
  • Basic infrastructure setup (VPC, IAM roles, SQS/Kafka)
  • Auth0/Okta integration for service authentication

Phase 2: Scale & Reliability

Timeline: 12-16 weeks
Core Focus: High availability, horizontal scaling, and reliability improvements

Deliverables:

  • Multi-region deployment capability (active-active)
  • Horizontal scaling of notification service instances
  • Enhanced retry mechanisms with exponential backoff
  • Dead letter queue handling and poison pill detection
  • Improved caching layer (Redis) for device token lookups
  • Service mesh integration (Istio) for cross-service communication
  • Auto-scaling policies based on QPS and latency metrics

Dependencies:

  • Kubernetes cluster with multiple zones
  • Service mesh infrastructure (Istio)
  • Enhanced monitoring and alerting system

Phase 3: Excellence & Optimization

Timeline: 16-24 weeks Core Focus: Advanced features, observability, cost optimization, and operational excellence

Deliverables:

  • Advanced segmentation and targeting capabilities (user attributes, tags)
  • A/B testing and campaign management dashboard
  • Comprehensive analytics and delivery performance metrics
  • Multi-tenant support with per-tenant configuration
  • Enhanced security features (mTLS, secrets management)
  • Cost optimization through intelligent storage tiering and egress reduction
  • Advanced alerting with error budget tracking and SLO monitoring
  • Feature flag integration for progressive rollouts

Dependencies:

  • Advanced analytics platform (Grafana, Prometheus)
  • Enhanced security infrastructure (Vault, WAF)
  • Multi-tenant Kubernetes cluster design
  • Comprehensive monitoring stack (Datadog, X-Ray)

Critical Path Dependencies:

  1. Mobile SDK integration and app-level push setup (Phase 1)
  2. Infrastructure provisioning (VPC, IAM, SQS/Kafka) (Phase 1)
  3. Authentication and authorization system (Phase 1)
  4. Multi-region deployment capability (Phase 2)
  5. Service mesh and cross-service communication (Phase 2)

Milestones:

  • Week 12: MVP functional end-to-end push delivery
  • Week 20: Scale-ready architecture with horizontal scaling
  • Week 32: Excellence features and operational maturity

Risk Mitigation:

  • Start with single-region MVP to validate core assumptions before multi-region
  • Implement feature flags early to enable safe rollouts and A/B testing
  • Design for gradual migration from legacy push systems (if applicable)
  • Establish clear rollback procedures and monitoring for each phase

This phased approach ensures rapid delivery of core functionality while building a robust, scalable foundation that can handle enterprise-scale push notification requirements.

Was this article helpful?

Topics in this article

More from Backend HLD

  • Backend LLD

    Notification System Low-Level Design

    Comprehensive backend LLD for scalable notification delivery system with real-time updates and multi-channel support.

    • CategoryBackend LLD
    • UpdatedDecember 21, 2025
    • Reading time40 min read
  • Backend LLD

    CQRS Framework Design and Implementation

    Comprehensive low-level design for a Command Query Responsibility Segregation (CQRS) framework with event sourcing, read/write model separation, and distributed consistency patterns.

    • CategoryBackend LLD
    • UpdatedOctober 4, 2026
    • Reading time45 min read
  • Backend HLD

    SMS Gateway System High-Level Design

    Comprehensive backend architecture for scalable SMS messaging platform supporting global carrier integration and real-time delivery.

    • CategoryBackend HLD
    • UpdatedAugust 24, 2026
    • Reading time57 min read

Stay in the Loop

Get curated DSA patterns, coding problems, frontend and backend LLD/HLD, and AI/ML System Design insights delivered straight to your inbox. No spam, just focused interview and engineering content.