Delayed Notification System Design

Comprehensive low-level design for a scalable delayed notification system with queue management, retry mechanisms, and distributed processing capabilities.

PublishedJune 8, 2025
Length7,492 words
Reading time38 min read

Backend LLD

A delayed notification system enables applications to schedule and deliver notifications at specific times while handling failures gracefully through robust queue management and retry strategies.

1. Problem Statement & Requirements

Clarifying questions

What is the expected scale of notifications? Will we be handling millions of notifications per day, or is this more of a smaller-scale system?
What are the consistency requirements for notification delivery? Should notifications be delivered exactly once, or is at-least-once acceptable?
What types of notifications are we supporting? Email, SMS, push notification, in-app, or a combination?
What is the maximum delay for notifications? Is it in minutes, hours, or potentially days?
What is the read/write ratio for the notification system? Are we more focused on writing notifications or querying their status?
What authentication model is expected? Will this system be accessed via API keys, OAuth2, or JWT tokens?
Are there any external integrations required? For example, does the system need to integrate with third-party providers like Twilio or SendGrid?

Functional requirements

Allow clients to schedule notifications with a specific delay
Support various notification channels (email, SMS, push)
Allow querying the status of scheduled notifications
Support cancellation of pending notifications
Support rescheduling of existing notifications
Provide a mechanism to retry failed notifications

Non-functional requirements

Latency targets:
- p50: < 100ms for scheduling a notification
- p95: < 500ms for scheduling a notification
- p99: < 1s for scheduling a notification
Throughput: Handle at least 10,000 notifications per second during peak hours
Availability: 99.9% uptime for the notification service
Delivery guarantee: At-least-once delivery with idempotency support

Constraints and assumptions

System will be built using microservices architecture with a focus on horizontal scalability
Assumption: Notifications are not time-critical and can be delayed by up to 1 hour
Assumption: The system will handle a large volume of notifications with minimal impact on database performance
Assumption: Notifications can be retried up to 3 times before being marked as failed

Out of scope

Real-time delivery (e.g., immediate push notifications)
Complex business logic within the notification system (e.g., content generation, templating)
Multi-tenant support beyond basic user isolation
Advanced analytics or reporting on notification delivery metrics
Integration with external services (e.g., Twilio, SendGrid) beyond basic API access
Support for recurring notifications or cron-like scheduling

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for the Delayed Notification system due to:

Strong typing and schema evolution support
Efficient binary serialization for high-throughput systems
Built-in support for streaming and bidirectional communication
Native support for versioning through proto file management
Better performance characteristics than REST for this use case

API Endpoints

Method	Path	Description	Auth
CreateNotification	`POST /v1/notifications`	Schedule a new notification with delay	JWT
GetNotification	`GET /v1/notifications/{id}`	Retrieve notification details	JWT
CancelNotification	`DELETE /v1/notifications/{id}`	Cancel scheduled notification	JWT
ListNotifications	`GET /v1/notifications`	List user's scheduled notifications	JWT
UpdateNotification	`PUT /v1/notifications/{id}`	Modify delay or content of pending notification	JWT

Request and Response Schemas

CreateNotificationRequest

{
  "userId": "string",
  "message": "string",
  "delaySeconds": 3600,
  "metadata": {
    "type": "string",
    "priority": "high|medium|low"
  }
}

CreateNotificationResponse

{
  "id": "uuid",
  "userId": "string",
  "message": "string",
  "scheduledTime": "timestamp",
  "delaySeconds": 3600,
  "status": "pending|sent|cancelled",
  "metadata": {
    "type": "string",
    "priority": "high|medium|low"
  }
}

GetNotificationResponse

{
  "id": "uuid",
  "userId": "string",
  "message": "string",
  "scheduledTime": "timestamp",
  "delaySeconds": 3600,
  "status": "pending|sent|cancelled",
  "metadata": {
    "type": "string",
    "priority": "high|medium|low"
  }
}

HTTP Status Codes and Error Response

Standard HTTP Statuses:

200 OK - Successful GET, PUT, PATCH
201 Created - Successful POST
400 Bad Request - Invalid input, validation errors
401 Unauthorized - Missing or invalid auth token
403 Forbidden - Insufficient permissions
404 Not Found - Resource not found
429 Too Many Requests - Rate limiting
500 Internal Server Error - Unexpected server errors

Standardized Error Response Format:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid input parameters",
    "details": [
      {
        "field": "delaySeconds",
        "message": "must be between 0 and 2592000 seconds"
      }
    ]
  }
}

API Versioning Strategy

Semantic versioning with path prefix: /v1/notifications to allow future evolution without breaking existing clients. Versioning is handled at the service layer with feature flags for backward compatibility.

Pagination Design

Cursor-based pagination is used because:

More stable than offset-based pagination when data changes
Better performance for large datasets
Easier to implement consistent ordering

Cursor format: Base64 encoded JSON with id and scheduledTime

Authentication and Authorization

Authentication: JWT tokens with audience claim for the notification service. Tokens validated via JWKs endpoint.

Authorization:

Users can only access their own notifications (userId field validation)
Admin role can access all notifications
RBAC enforced at service layer

Rate Limiting

Per-user rate limiting:

1000 requests per hour per user
50 concurrent notifications creation
Token bucket algorithm with burst capacity of 10

Enforcement:

In-memory or Redis-based token bucket implementation
Middleware layer intercepting all requests

Idempotency Design

Idempotency Key Strategy:

Clients send X-Idempotency-Key header with each request
Service stores last result for the key within 24 hours
For create operations, if same key exists and status is pending or sent, return existing resource
For cancel operations, if same key exists and status is pending, return success

// Example idempotency handling in service layer
public CreateNotificationResponse createNotification(CreateNotificationRequest request, String idempotencyKey) {
    // Check if this idempotency key was already processed
    NotificationIdempotencyRecord record = idempotencyRepository.findByKey(idempotencyKey);
    if (record != null && record.getStatus().isTerminal()) {
        return convertToResponse(record.getNotification());
    }
    
    // Proceed with normal processing...
    Notification notification = createNewNotification(request);
    // ... save and publish
    
    // Store idempotency record
    idempotencyRepository.save(new NotificationIdempotencyRecord(idempotencyKey, notification));
    
    return convertToResponse(notification);
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    NOTIFICATION {
        string id PK
        string userId FK
        string title
        string body
        string type
        timestamp scheduledAt
        timestamp deliveredAt
        integer delaySeconds
        string status
        timestamp createdAt
        timestamp updatedAt
    }

    USER {
        string id PK
        string email
        string name
        timestamp createdAt
        timestamp updatedAt
    }

    NOTIFICATION_EVENT {
        string id PK
        string notificationId FK
        string eventType
        timestamp occurredAt
        string metadata
    }

    NOTIFICATION_STATUS {
        string id PK
        string notificationId FK
        string status
        timestamp transitionedAt
        string reason
    }

    NOTIFICATION --> USER : "user"
    NOTIFICATION_EVENT --> NOTIFICATION : "notification"
    NOTIFICATION_STATUS --> NOTIFICATION : "notification"

Table Schema Definitions

`notifications` table

CREATE TABLE notifications (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    title VARCHAR(255) NOT NULL,
    body TEXT NOT NULL,
    type VARCHAR(50) NOT NULL,
    scheduled_at TIMESTAMP WITH TIME ZONE NOT NULL,
    delivered_at TIMESTAMP WITH TIME ZONE,
    delay_seconds INTEGER NOT NULL DEFAULT 0,
    status VARCHAR(20) NOT NULL DEFAULT 'PENDING',
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    
    INDEX idx_notifications_user_id (user_id),
    INDEX idx_notifications_scheduled_at (scheduled_at),
    INDEX idx_notifications_status (status),
    INDEX idx_notifications_user_scheduled (user_id, scheduled_at),
    INDEX idx_notifications_status_scheduled (status, scheduled_at)
);

`notification_events` table

CREATE TABLE notification_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    notification_id UUID NOT NULL REFERENCES notifications(id) ON DELETE CASCADE,
    event_type VARCHAR(50) NOT NULL,
    occurred_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    metadata JSONB,
    
    INDEX idx_notification_events_notification_id (notification_id),
    INDEX idx_notification_events_event_type (event_type)
);

`notification_status_history` table

CREATE TABLE notification_status_history (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    notification_id UUID NOT NULL REFERENCES notifications(id) ON DELETE CASCADE,
    status VARCHAR(20) NOT NULL,
    transitioned_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    reason TEXT,
    
    INDEX idx_notification_status_history_notification_id (notification_id),
    INDEX idx_notification_status_history_status (status)
);

Primary Key Strategy

UUID is selected as the primary key for all tables due to:

Global uniqueness across distributed systems
No need for coordination between services or databases
Better security properties (no sequential guessing)
Support for eventual consistency in distributed environments

Index Design Justification

Index	Fields	Type	Purpose
`idx_notifications_user_id`	user_id	Single	Fast user-based queries
`idx_notifications_scheduled_at`	scheduled_at	Single	Efficient query for pending notifications
`idx_notifications_status`	status	Single	Fast status filtering
`idx_notifications_user_scheduled`	user_id, scheduled_at	Composite	Fast retrieval of user's notifications ordered by time
`idx_notifications_status_scheduled`	status, scheduled_at	Composite	Efficient batch processing of pending notifications
`idx_notification_events_notification_id`	notification_id	Single	Fast event retrieval for a specific notification
`idx_notification_status_history_notification_id`	notification_id	Single	Fast status history lookup

Data Partitioning Strategy

Time-based partitioning on scheduled_at field:

Daily partitions for recent notifications (last 30 days)
Monthly partitions for older notifications (30+ days)
This enables:
- Efficient querying of pending notifications
- Easy cleanup of old completed notifications
- Parallel processing of notification batches

Soft Delete vs. Hard Delete

Soft delete is chosen because:

Audit trail requirement for all notification lifecycle events
Ability to recover accidentally deleted notifications
Compliance with data retention policies (e.g., GDPR)
Historical analytics on notification delivery patterns

Audit Trail Design

Each notification lifecycle event is recorded in notification_events table:

Creation of notification
Status transitions (PENDING → DELIVERED, PENDING → FAILED)
Manual cancellation or rescheduling
Retry attempts

Data Archival Strategy

30-day retention policy:

Notifications with delivered_at or status = FAILED older than 30 days are archived
Archive stored in separate table with reduced schema (no metadata, minimal status history)
Archival process runs nightly using partition pruning
Historical analytics data is moved to data warehouse for long-term analysis

Data Validation Rules

scheduled_at must be in future (>= current time)
delay_seconds must be >= 0
status must be one of: PENDING, DELIVERED, FAILED, CANCELLED
type must be one of: EMAIL, SMS, PUSH, IN_APP
title and body must not be empty when creating notification

Data Consistency Considerations

Use of atomic database transactions for status updates
Foreign key constraints to maintain referential integrity
Timestamps stored with timezone information for global consistency
Soft-delete pattern to prevent data loss during maintenance windows

4. Class Design & Service Layer

UML Class Diagram

classDiagram
    class NotificationService {
        +sendDelayedNotification(NotificationRequest)
        +cancelScheduledNotification(String)
        +getScheduledNotifications(String, String)
    }
    
    class NotificationRequest {
        -id: UUID
        -userId: String
        -message: String
        -scheduledTime: Instant
        -status: NotificationStatus
        -retryCount: int
        -createdAt: Instant
        -updatedAt: Instant
    }
    
    class NotificationStatus {
        <<enumeration>>
        PENDING
        SENT
        FAILED
        CANCELLED
    }
    
    class NotificationRepository {
        +save(NotificationRequest)
        +findById(String)
        +findByUserIdAndStatus(String, NotificationStatus)
        +update(NotificationRequest)
        +delete(String)
    }
    
    class EventPublisher {
        +publish(NotificationEvent)
    }
    
    class NotificationEvent {
        -id: UUID
        -notificationId: String
        -userId: String
        -message: String
        -timestamp: Instant
    }
    
    class NotificationProcessor {
        +processScheduledNotifications()
        +handleNotificationDelivery(NotificationRequest)
    }
    
    class RetryStrategy {
        <<interface>>
        +shouldRetry(NotificationRequest)
        +getDelay(NotificationRequest)
    }
    
    class ExponentialBackoffStrategy {
        +shouldRetry(NotificationRequest)
        +getDelay(NotificationRequest)
    }
    
    class NotificationScheduler {
        +scheduleNotification(NotificationRequest)
        +unscheduleNotification(String)
    }
    
    class NotificationValidator {
        +validate(NotificationRequest)
    }
    
    NotificationService --> NotificationRepository : uses
    NotificationService --> EventPublisher : publishes
    NotificationService --> NotificationProcessor : delegates
    NotificationService --> RetryStrategy : uses
    NotificationService --> NotificationValidator : validates
    
    NotificationProcessor --> RetryStrategy : uses
    NotificationProcessor --> EventPublisher : publishes
    
    NotificationRequest --> NotificationStatus : has
    NotificationEvent --> NotificationRequest : references
    
    RetryStrategy <|-- ExponentialBackoffStrategy : implements

State Machine Diagram

stateDiagram-v2
    [*] --> PENDING
    PENDING --> SENT : Successfully delivered
    PENDING --> FAILED : Delivery failed (max retries)
    PENDING --> CANCELLED : Cancelled by user
    FAILED --> PENDING : Retry attempted
    FAILED --> CANCELLED : Cancelled by user
    
    note right of PENDING
        Waiting for scheduled time to deliver
    end note
    
    note right of SENT
        Notification successfully delivered
    end note
    
    note right of FAILED
        Max retry attempts exhausted
    end note
    
    note right of CANCELLED
        Explicitly cancelled by user or system
    end note

Service Class Structure and Responsibilities

The NotificationService is the primary orchestrator that handles all external interactions and business logic for delayed notifications. It delegates to specialized components:

NotificationRepository: Data persistence layer with CRUD operations
EventPublisher: Event-driven communication with downstream systems
NotificationProcessor: Core processing logic for delivery and retry handling
RetryStrategy: Configurable retry policies with exponential backoff
NotificationValidator: Input validation and business rule enforcement

Interface / Abstract Class Design

public interface RetryStrategy {
    boolean shouldRetry(NotificationRequest request);
    Duration getDelay(NotificationRequest request);
}

public interface NotificationValidator {
    ValidationResult validate(NotificationRequest request);
}

public enum NotificationStatus {
    PENDING, SENT, FAILED, CANCELLED
}

Business Logic Encapsulation Patterns

Strategy Pattern Implementation - Retry Strategy

public class ExponentialBackoffStrategy implements RetryStrategy {
    private static final int MAX_RETRIES = 3;
    private static final Duration BASE_DELAY = Duration.ofSeconds(1);
    
    @Override
    public boolean shouldRetry(NotificationRequest request) {
        return request.getRetryCount() < MAX_RETRIES;
    }
    
    @Override
    public Duration getDelay(NotificationRequest request) {
        int retryCount = request.getRetryCount();
        long delaySeconds = BASE_DELAY.getSeconds() * Math.pow(2, retryCount);
        return Duration.ofSeconds(Math.min(delaySeconds, 300)); // Cap at 5 minutes
    }
}

Critical-Path Method Implementation

@Transactional
public void sendDelayedNotification(NotificationRequest request) {
    // Validate input
    ValidationResult validation = notificationValidator.validate(request);
    if (!validation.isValid()) {
        throw new ValidationException("Invalid notification request: " + validation.getErrors());
    }
    
    // Generate unique ID if not provided
    if (request.getId() == null) {
        request.setId(UUID.randomUUID().toString());
    }
    
    // Set initial status and timestamps
    request.setStatus(NotificationStatus.PENDING);
    request.setCreatedAt(Instant.now());
    request.setUpdatedAt(Instant.now());
    
    // Save to database
    notificationRepository.save(request);
    
    // Schedule for processing at the specified time
    notificationScheduler.scheduleNotification(request);
    
    // Publish event for audit and monitoring
    NotificationEvent event = new NotificationEvent(
        UUID.randomUUID().toString(),
        request.getId(),
        request.getUserId(),
        request.getMessage(),
        Instant.now()
    );
    
    eventPublisher.publish(event);
}

Transaction Boundary Design

A single transaction encompasses:

Input validation and business rule enforcement
Database persistence of the notification record
Scheduling of the delayed processing job
Event publishing for audit and monitoring

This ensures atomicity for the entire notification lifecycle from creation to scheduling.

Service Interface Contracts

public interface NotificationService {
    void sendDelayedNotification(NotificationRequest request);
    void cancelScheduledNotification(String notificationId);
    List<NotificationRequest> getScheduledNotifications(String userId, String status);
}

Saga / Orchestration Pattern

For distributed transactions involving external systems:

Use a saga pattern to coordinate notifications with external delivery services
Each step in the saga publishes events that trigger compensating actions if needed
The saga manager maintains state and orchestrates the sequence of operations

Domain Object / Value Object Design

public class NotificationRequest {
    private String id;
    private String userId;
    private String message;
    private Instant scheduledTime;
    private NotificationStatus status;
    private int retryCount;
    private Instant createdAt;
    private Instant updatedAt;
    
    // Getters and setters with validation
}

public class ValidationResult {
    private boolean valid;
    private List<String> errors;
    
    // Constructor, getters, setters
}

public class NotificationEvent {
    private String id;
    private String notificationId;
    private String userId;
    private String message;
    private Instant timestamp;
    
    // Constructor, getters, setters
}

5. Concurrency & Thread Safety

Concurrency Model

The delayed notification system will use a hybrid approach combining thread pools for processing and async-await patterns for I/O operations. The core processing will be handled by a fixed-size thread pool (e.g., 10-20 threads) to avoid resource exhaustion while maintaining throughput. For I/O-bound operations like database reads/writes and external API calls, we'll leverage async patterns to maximize throughput without blocking threads.

Race Condition Prevention Strategy

Race conditions will be prevented through:

Database-level atomic operations for state transitions and counters
Optimistic locking using version fields in entities to detect concurrent modifications
Thread-local state management for processing context that doesn't need cross-thread coordination
Synchronized access to shared mutable state using ReadWriteLocks where needed

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected for the delayed notification system because:

Notifications are primarily read-heavy with infrequent updates
Concurrent access is expected to be low (single notification update at a time)
The system prioritizes throughput over strict consistency
Database-level version checking provides sufficient conflict detection
Pessimistic locking would introduce unnecessary blocking and reduce system throughput

Deadlock Prevention Strategy

Deadlock prevention will be implemented through:

Fixed ordering of resource acquisition - all operations acquire locks in consistent order (e.g., by notification ID)
Timeout-based lock acquisition - maximum wait time of 500ms for any database locks
Hierarchical lock management - no cross-domain locking (e.g., never lock both notification and user entities together)
Deadlock detection in database layer - rely on DB engine's built-in deadlock detection and retry logic

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency will be handled through:

Idempotency keys in request headers - client-provided unique identifiers for operations
Database-level uniqueness constraints on idempotency keys with timestamps
In-memory cache of recent idempotency keys (LRU cache with 10k capacity) to avoid database round-trips for recent requests
Idempotency key expiration - keys expire after 24 hours to prevent memory leaks

public class IdempotencyService {
    private final Cache<String, String> idempotencyCache = 
        Caffeine.newBuilder()
            .maximumSize(10_000)
            .expireAfterWrite(Duration.ofHours(24))
            .build();
    
    public boolean isIdempotent(String idempotencyKey) {
        return idempotencyCache.getIfPresent(idempotencyKey) != null;
    }
    
    public void markIdempotent(String idempotencyKey) {
        idempotencyCache.put(idempotencyKey, "processed");
    }
}

Thread Safety in Core Components

NotificationProcessor - Uses thread-safe queue and atomic counters for processing statistics
NotificationRepository - All database operations are wrapped in proper transaction boundaries with thread-safe connection pooling
EventPublisher - Uses thread-safe concurrent queues for async event publishing with proper synchronization
StateTransitionManager - Uses synchronized methods and atomic operations for state updates

Synchronization Primitives Used

ReentrantReadWriteLock - For read-heavy notification state access with write exclusivity
CountDownLatch - For coordinating async processing completion in batch operations
Semaphore - To limit concurrent database connection usage (max 50 connections)
ThreadLocal - For storing request context and trace IDs without cross-thread contamination

Concurrency Control for Batch Operations

Batch operations will use:

Fixed thread pool with size configurable based on system load (max 20 threads)
Work-stealing queue for efficient task distribution
Batch size limits (max 100 notifications per batch) to prevent resource exhaustion
Per-batch atomic counters to track progress and handle partial failures

Memory Consistency Considerations

All shared mutable state will be:

Protected by proper synchronization mechanisms
Using volatile variables for flags and status indicators
Leveraging concurrent collections where appropriate (ConcurrentHashMap, CopyOnWriteArrayList)
Using atomic reference types for state transitions

Performance Considerations

The concurrency model is designed to:

Maximize throughput under moderate load (1000+ concurrent notifications)
Minimize latency for individual notification processing
Handle burst traffic gracefully without system degradation
Maintain consistent response times under high load conditions

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant U as User
    participant API as NotificationServiceAPI
    participant DB as Database
    participant Queue as MessageQueue
    participant Worker as NotificationWorker

    U->>API: POST /notifications/delayed
    API->>DB: Save notification with delay_time
    DB-->>API: Return notification_id
    API-->>U: 201 Created with notification_id
    API->>Queue: Publish to delayed_notification_queue
    Queue-->>Worker: Message with notification_id and delay_time
    Worker->>DB: Poll for notifications ready to send
    DB-->>Worker: Ready notifications
    Worker->>DB: Update notification status to sent
    Worker->>U: Send push/email/SMS to user

sequenceDiagram
    participant U as User
    participant API as NotificationServiceAPI
    participant DB as Database
    participant Queue as MessageQueue
    participant Worker as NotificationWorker

    U->>API: POST /notifications/delayed
    API->>DB: Save notification with delay_time
    DB-->>API: Return notification_id
    API-->>U: 201 Created with notification_id
    API->>Queue: Publish to delayed_notification_queue
    Queue-->>Worker: Message with notification_id and delay_time
    Worker->>DB: Poll for notifications ready to send
    DB-->>Worker: Ready notifications
    Worker->>DB: Update notification status to failed
    Worker->>Queue: Publish to dead_letter_queue
    Queue-->>Worker: Message with notification_id and error details
    Worker->>DB: Log error in notification_log

sequenceDiagram
    participant U as User
    participant API as NotificationServiceAPI
    participant DB as Database
    participant Queue as MessageQueue
    participant Worker as NotificationWorker

    U->>API: POST /notifications/delayed
    API->>DB: Save notification with delay_time
    DB-->>API: Return notification_id
    API-->>U: 201 Created with notification_id
    API->>Queue: Publish to delayed_notification_queue
    Queue-->>Worker: Message with notification_id and delay_time
    Worker->>DB: Poll for notifications ready to send
    DB-->>Worker: Ready notifications
    Worker->>DB: Update notification status to sent
    Worker->>U: Send push/email/SMS to user
    Worker->>DB: Update notification status to delivered
    Worker->>Queue: Publish to notification_delivered_queue
    Queue-->>Worker: Message with notification_id and delivery_time

7. Caching Design

Cache Strategy Selection

For the delayed notification system, I'll use cache-aside pattern with write-through behavior for critical data and write-behind for less time-sensitive operations. This approach provides the best balance between consistency, performance, and complexity.

The cache-aside pattern is chosen because:

It allows for explicit control over when data is loaded into cache
Enables proper error handling and fallback mechanisms
Provides clear separation between application logic and caching concerns
Supports gradual rollout of cache invalidation strategies

Cache Key Design

Cache keys will follow a structured namespace pattern:

public class CacheKey {
    private static final String NAMESPACE = "notification";
    private static final String DELAYED_NOTIFICATION = "delayed";
    private static final String USER_NOTIFICATIONS = "user";
    
    public static String delayedNotificationKey(String notificationId) {
        return String.format("%s:%s:%s", NAMESPACE, DELAYED_NOTIFICATION, notificationId);
    }
    
    public static String userNotificationsKey(String userId) {
        return String.format("%s:%s:%s", NAMESPACE, USER_NOTIFICATIONS, userId);
    }
    
    public static String notificationQueueKey(String queueName) {
        return String.format("%s:%s:%s", NAMESPACE, "queue", queueName);
    }
}

TTL Configuration

Data Type	TTL	Reason
Delayed Notification	30 minutes	Notifications should not be cached longer than their processing window
User Notification List	5 minutes	Frequent updates require relatively fresh data
Notification Queue	1 hour	Queued notifications need to remain accessible for processing
Processing State	10 minutes	Temporary state tracking during processing

Cache Eviction Policy

LRU (Least Recently Used) with TTL-based expiration will be implemented using Redis with custom eviction policies:

public class CacheConfig {
    private static final int DEFAULT_TTL_SECONDS = 30 * 60; // 30 minutes
    private static final int LRU_CACHE_SIZE = 10000;
    
    // Redis configuration with custom eviction policy
    public static RedisClient createRedisClient() {
        return RedisClient.create(
            RedisURI.create("redis://localhost:6379"),
            RedisClientOptions.builder()
                .commandTimeout(Duration.ofSeconds(5))
                .maxRedirects(3)
                .build()
        );
    }
}

Cache Stampede Prevention

To prevent cache stampedes, I'll implement probabilistic early expiry:

public class CacheStampedePrevention {
    private static final double STAMPED_EXPIRY_FACTOR = 0.8;
    
    public static boolean shouldRefreshCache(String cacheKey, long ttlSeconds) {
        // Check if cache is likely to expire soon (early refresh)
        long timeToExpiry = getTimeToExpiry(cacheKey);
        return timeToExpiry < (ttlSeconds * STAMPED_EXPIRY_FACTOR);
    }
    
    public static String getCacheKeyWithExpiry(String baseKey, long ttlSeconds) {
        // Add timestamp to key for probabilistic refresh
        return baseKey + ":" + System.currentTimeMillis();
    }
}

Distributed Cache Architecture

Redis Cluster with Sentinel support will be used for high availability:

@Configuration
public class RedisClusterConfig {
    
    @Bean
    public RedisTemplate<String, Object> redisTemplate() {
        RedisTemplate<String, Object> template = new RedisTemplate<>();
        template.setConnectionFactory(redisConnectionFactory());
        template.setKeySerializer(new StringRedisSerializer());
        template.setValueSerializer(new GenericJackson2JsonRedisSerializer());
        return template;
    }
    
    @Bean
    public LettuceConnectionFactory redisConnectionFactory() {
        RedisClusterConfiguration clusterConfig = new RedisClusterConfiguration(
            Arrays.asList("redis://node1:6379", "redis://node2:6379")
        );
        
        return new LettuceConnectionFactory(clusterConfig);
    }
}

Cache Invalidation Strategy

Event-driven invalidation will be implemented using a combination of:

Immediate invalidation on data changes (write-through)
TTL-based expiration for time-sensitive data
Explicit purge for bulk operations

@Component
public class CacheInvalidationService {
    
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;
    
    public void invalidateDelayedNotification(String notificationId) {
        String key = CacheKey.delayedNotificationKey(notificationId);
        redisTemplate.delete(key);
        
        // Also invalidate user's notification list if needed
        String userKey = CacheKey.userNotificationsKey(getUserIdForNotification(notificationId));
        redisTemplate.delete(userKey);
    }
    
    public void invalidateUserNotifications(String userId) {
        String key = CacheKey.userNotificationsKey(userId);
        redisTemplate.delete(key);
    }
    
    private String getUserIdForNotification(String notificationId) {
        // Implementation to fetch user ID from DB or cache
        return "user:" + notificationId.substring(0, 10);
    }
}

Cache Tiering Strategy

For optimal performance, a multi-tier caching approach will be implemented:

In-memory cache (Caffeine) for the most frequently accessed data
Redis cluster for distributed caching and persistence

@Configuration
public class MultiTierCacheConfig {
    
    @Bean
    public CacheManager cacheManager() {
        CaffeineCacheManager caffeineCacheManager = new CaffeineCacheManager();
        caffeineCacheManager.setCaffeine(Caffeine.newBuilder()
            .maximumSize(1000)
            .expireAfterWrite(Duration.ofMinutes(5))
            .recordStats());
        
        return caffeineCacheManager;
    }
}

Cache Read-Through Pattern

@Service
public class NotificationCacheService {
    
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;
    
    @Autowired
    private NotificationRepository notificationRepository;
    
    public DelayedNotification getDelayedNotification(String notificationId) {
        String cacheKey = CacheKey.delayedNotificationKey(notificationId);
        
        // Try cache first
        DelayedNotification cached = (DelayedNotification) redisTemplate.opsForValue().get(cacheKey);
        if (cached != null) {
            return cached;
        }
        
        // Cache miss - fetch from database
        DelayedNotification notification = notificationRepository.findById(notificationId);
        if (notification != null) {
            // Cache the result with TTL
            redisTemplate.opsForValue().set(cacheKey, notification, 30, TimeUnit.MINUTES);
        }
        
        return notification;
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern provides a clean abstraction over data access operations, separating domain logic from persistence concerns. For the delayed notification system, we'll implement repositories for core entities like Notification, ScheduledNotification, and NotificationTemplate.

public interface NotificationRepository {
    Notification save(Notification notification);
    Optional<Notification> findById(UUID id);
    List<Notification> findByStatusAndScheduledTimeBefore(
        NotificationStatus status, 
        LocalDateTime now);
    void updateStatus(UUID id, NotificationStatus status);
    void delete(UUID id);
}

public interface ScheduledNotificationRepository {
    ScheduledNotification save(ScheduledNotification notification);
    Optional<ScheduledNotification> findById(UUID id);
    List<ScheduledNotification> findReadyToProcess(
        LocalDateTime now, 
        int limit);
    void markProcessed(UUID id);
}

ORM vs Raw Query Decisions

For the delayed notification system, we'll use JPA/Hibernate for most operations due to its strong typing and ease of maintenance, but will use raw SQL for performance-critical batch operations like fetching ready notifications.

// JPA for standard CRUD operations
@Query("SELECT n FROM Notification n WHERE n.status = :status AND n.scheduledTime < :now")
List<Notification> findReadyNotifications(
    @Param("status") NotificationStatus status,
    @Param("now") LocalDateTime now);

// Raw SQL for high-performance batch operations
@Query(value = "SELECT * FROM scheduled_notifications WHERE status = 'SCHEDULED' AND scheduled_time <= ?1 LIMIT ?2 FOR UPDATE", 
       nativeQuery = true)
List<ScheduledNotification> findReadyForProcessing(
    LocalDateTime now, 
    int limit);

Read vs Write Separation

We'll implement CQRS for the notification system to separate read and write concerns:

// Write model - used for creation and updates
public class NotificationWriteModel {
    public void createNotification(Notification notification) {
        // Business logic validation
        notificationRepository.save(notification);
        eventPublisher.publish(new NotificationCreatedEvent(notification));
    }
}

// Read model - optimized for querying
public class NotificationReadModel {
    @QueryHandler
    public List<NotificationDto> getNotificationsByUser(UUID userId) {
        return notificationReadRepository.findByUserId(userId);
    }
}

Connection Pool Configuration

Using HikariCP for optimal connection handling:

spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      validation-timeout: 5000

N+1 Query Prevention Strategy

We'll use eager loading with JOIN FETCH for critical paths and DataLoader pattern for complex read operations:

// Eager loading for notification with template
@Query("SELECT n FROM Notification n JOIN FETCH n.template WHERE n.id = :id")
Optional<Notification> findByIdWithTemplate(@Param("id") UUID id);

// DataLoader for batch operations
public CompletableFuture<List<Notification>> getNotificationsByIds(List<UUID> ids) {
    return dataLoader.loadMany(ids);
}

Batch Operation Design

Batch operations for processing ready notifications:

public class BatchNotificationProcessor {
    
    @Transactional
    public void processReadyNotifications(int batchSize) {
        List<ScheduledNotification> readyNotifications = 
            scheduledNotificationRepository.findReadyForProcessing(
                LocalDateTime.now(), batchSize);
        
        // Process in batches to avoid memory issues
        readyNotifications.forEach(notification -> {
            try {
                processNotification(notification);
                scheduledNotificationRepository.markProcessed(notification.getId());
            } catch (Exception e) {
                // Handle individual failures without affecting batch
                log.error("Failed to process notification: {}", notification.getId(), e);
            }
        });
    }
}

Transaction Management Strategy

@Service
@Transactional(readOnly = true)
public class NotificationService {
    
    @Transactional
    public void scheduleNotification(NotificationRequest request) {
        // Transaction boundary for creation and event publishing
        Notification notification = notificationFactory.create(request);
        notificationRepository.save(notification);
        
        // Publish event for async processing
        eventPublisher.publish(new NotificationScheduledEvent(notification));
    }
}

Repository Implementation Example

@Repository
public class ScheduledNotificationRepositoryImpl implements ScheduledNotificationRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    @Override
    public ScheduledNotification save(ScheduledNotification notification) {
        entityManager.persist(notification);
        return notification;
    }
    
    @Override
    public Optional<ScheduledNotification> findById(UUID id) {
        return Optional.ofNullable(
            entityManager.find(ScheduledNotification.class, id));
    }
    
    @Override
    public List<ScheduledNotification> findReadyForProcessing(
            LocalDateTime now, int limit) {
        String jpql = """
            SELECT n FROM ScheduledNotification n 
            WHERE n.status = :status AND n.scheduledTime <= :now
            ORDER BY n.scheduledTime ASC
            """;
        
        return entityManager.createQuery(jpql, ScheduledNotification.class)
            .setParameter("status", NotificationStatus.SCHEDULED)
            .setParameter("now", now)
            .setMaxResults(limit)
            .setHint("javax.persistence.fetchgraph", 
                entityManager.getMetamodel().entity(ScheduledNotification.class))
            .getResultList();
    }
    
    @Override
    @Transactional
    public void markProcessed(UUID id) {
        String jpql = "UPDATE ScheduledNotification n SET n.status = :status WHERE n.id = :id";
        entityManager.createQuery(jpql)
            .setParameter("status", NotificationStatus.PROCESSED)
            .setParameter("id", id)
            .executeUpdate();
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The delayed notification system will use a hybrid synchronous/asynchronous approach. The core business logic (notification creation, scheduling, and delivery) will be handled synchronously to ensure immediate consistency for the user-facing operations. However, the actual sending of notifications will be decoupled via asynchronous message processing to provide better scalability and resilience.

Key triggers for async processing:

Notification creation → fires a NotificationScheduledEvent to queue the delivery
Notification delivery failure → fires a NotificationDeliveryRetryEvent
Manual retry trigger → fires a NotificationRetryEvent

The system will maintain synchronous behavior for the API responses to ensure users get immediate feedback on their actions, while offloading the actual delivery to a message queue for processing.

Message Schema Design

{
  "version": "1.0",
  "event_type": "notification.scheduled.v1",
  "event_id": "uuid-12345",
  "timestamp": "2023-10-15T14:30:00Z",
  "payload": {
    "notification_id": "uuid-67890",
    "recipient_user_id": "user-123",
    "notification_type": "email",
    "content": {
      "subject": "Your appointment reminder",
      "body": "You have an appointment tomorrow at 10:00 AM"
    },
    "scheduled_at": "2023-10-16T10:00:00Z",
    "retry_count": 0,
    "max_retries": 3
  }
}

Dead Letter Queue Strategy

All failed notifications will be moved to a dead letter queue (DLQ) after max retries are exhausted. The DLQ will be monitored by an admin dashboard and can trigger manual intervention or escalation to support teams.

Poison message handling:

Messages that fail 3+ times will be moved to DLQ
DLQ messages will have a separate monitoring dashboard with alerting
Admins can manually retry or delete from DLQ

Delivery Guarantees

At-least-once delivery is chosen for this system to ensure no notifications are lost. This is critical for user experience and compliance in many notification use cases.

Retry Policy

Retry strategy:

Max attempts: 3
Backoff formula: Exponential with jitter (1s, 2s, 4s)
Retryable errors: Network timeouts, service unavailability
Non-retryable errors: Invalid notification format, user unsubscribed

public class RetryPolicy {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    
    public boolean shouldRetry(int currentAttempt, Exception error) {
        if (currentAttempt >= MAX_RETRIES) return false;
        
        // Retry on network or service errors
        return error instanceof NetworkException || 
               error instanceof ServiceUnavailableException;
    }
    
    public long getDelay(int attempt) {
        long delay = (long) Math.pow(2, attempt) * BASE_DELAY_MS;
        // Add jitter (±25%)
        return delay + (long) (Math.random() * 0.5 * delay) - (delay / 2);
    }
}

Outbox Pattern

The outbox pattern is implemented to ensure reliable event publishing:

public class NotificationOutboxService {
    private final NotificationEventRepository eventRepo;
    private final MessagePublisher publisher;
    
    @Transactional
    public void createNotificationWithEvent(Notification notification) {
        // Create notification in DB
        Notification saved = notificationRepository.save(notification);
        
        // Create outbox event
        NotificationScheduledEvent event = new NotificationScheduledEvent(saved.getId());
        outboxRepository.save(new OutboxEvent(
            UUID.randomUUID(),
            "notification.scheduled.v1",
            event,
            EventStatus.PENDING
        ));
        
        // Publish to queue (in same transaction)
        publisher.publish(event);
    }
}

Sequence Diagram for Async Flows

sequenceDiagram
    participant User
    participant API
    participant DB
    participant Outbox
    participant MessageQueue
    participant NotificationService
    
    User->>API: Create delayed notification
    API->>DB: Save notification record
    API->>Outbox: Create outbox event
    Outbox->>MessageQueue: Publish notification.scheduled event
    MessageQueue-->>NotificationService: Consume event
    NotificationService->>DB: Check delivery time
    alt Not ready for delivery
        NotificationService->>DB: Update status to queued
        NotificationService->>MessageQueue: Schedule retry event
    else Ready for delivery
        NotificationService->>DB: Update status to sending
        NotificationService->>ExternalService: Send notification
        alt Success
            NotificationService->>DB: Update status to delivered
        else Failure
            NotificationService->>DB: Increment retry count
            NotificationService->>MessageQueue: Schedule retry event
        end
    end

10. Error Handling Design

Error Classification Hierarchy

The system follows a three-tier error classification:

Domain Errors - Business logic violations (e.g., invalid notification configuration, unsupported delay duration)
Infrastructure Errors - System resource constraints (e.g., database connection failure, Redis timeout)
Validation Errors - Input format issues (e.g., malformed JSON, missing required fields)

public enum ErrorType {
    VALIDATION_ERROR,
    DOMAIN_ERROR,
    INFRASTRUCTURE_ERROR
}

Error Propagation Strategy

Errors are handled at the service layer and propagated upward using custom exceptions that wrap underlying causes. This enables centralized logging and response formatting while maintaining stack trace visibility for debugging.

Retry Strategy

For transient infrastructure errors, exponential backoff with jitter is applied:

public class RetryConfig {
    private int maxAttempts = 3;
    private long initialDelayMs = 1000;
    private double multiplier = 2.0;
    private long maxDelayMs = 30000;
    
    public Duration calculateDelay(int attempt) {
        double delay = initialDelayMs * Math.pow(multiplier, attempt - 1);
        delay = Math.min(delay, maxDelayMs);
        // Add jitter (±25%)
        double jitter = ThreadLocalRandom.current().nextDouble(0.75, 1.25);
        return Duration.ofMillis((long) (delay * jitter));
    }
}

Circuit Breaker Design

For external dependencies like Redis or database connections, a circuit breaker pattern is implemented using the Resilience4j library:

public class CircuitBreakerManager {
    private final CircuitBreaker circuitBreaker = CircuitBreaker.ofDefaults("notification-service");
    
    public <T> T executeWithCircuitBreaker(Supplier<T> operation) {
        return circuitBreaker.executeSupplier(operation);
    }
    
    public void onCallSuccess() {
        circuitBreaker.recordSuccess();
    }
    
    public void onCallFailure() {
        circuitBreaker.recordFailure();
    }
}

Fallback Behavior Design

For critical operations, fallbacks are implemented:

public class NotificationService {
    private final CircuitBreaker circuitBreaker = CircuitBreaker.ofDefaults("notification-service");
    
    public void sendNotificationWithFallback(NotificationRequest request) {
        try {
            circuitBreaker.executeSupplier(() -> {
                // Primary operation
                return sendNotification(request);
            });
        } catch (Exception e) {
            // Fallback: log and queue for retry later
            logger.warn("Primary notification send failed, using fallback", e);
            queueForRetry(request);
        }
    }
}

Error Logging Format

Structured logging includes trace context and error metadata:

{
  "timestamp": "2023-10-15T14:30:45.123Z",
  "trace_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "span_id": "f0e9d8c7-b6a5-4321-0fed-cba987654321",
  "user_id": "user-12345",
  "error_code": "NOTIFICATION_TIMEOUT",
  "error_message": "Failed to send notification due to timeout",
  "error_type": "INFRASTRUCTURE_ERROR",
  "retry_count": 2,
  "stack_trace": "...",
  "service_name": "notification-service"
}

Alerting Thresholds

Critical alerts are configured for:

Notification timeout rate > 5% (warn at 1%, critical at 5%)
Database connection failures > 3 consecutive (critical)
Redis availability < 95% (warn at 98%, critical at 95%)

Error Budget Management

The system maintains a 5% error budget for the notification service, with alerts triggered when:

P99 latency exceeds 500ms for 10 minutes (warn)
Service availability drops below 99.5% for 1 hour (critical)

Error Handling in Retry Logic

Retry logic includes specific error categorization:

public class NotificationRetryHandler {
    private static final Set<Class<? extends Exception>> RETRYABLE_EXCEPTIONS = 
        Set.of(TimeoutException.class, SQLException.class, RedisConnectionException.class);
    
    public boolean shouldRetry(Exception exception) {
        return RETRYABLE_EXCEPTIONS.stream()
            .anyMatch(type -> type.isInstance(exception));
    }
    
    public void handleRetryableError(Exception error, int attempt) {
        if (shouldRetry(error)) {
            logger.info("Retryable error on attempt {}: {}", attempt, error.getMessage());
            // Apply exponential backoff
        } else {
            logger.error("Non-retryable error: {}", error.getMessage());
            // Log for alerting and fail fast
        }
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using a combination of:

Bean Validation (JSR-303) for structured validation rules
Custom sanitization filters to remove or escape potentially harmful characters
Regular expression patterns for format validation (e.g., email, phone number)
Length limits to prevent buffer overflow attacks

public class NotificationRequest {
    @NotBlank
    @Size(max = 1000)
    private String message;
    
    @Email
    @Size(max = 255)
    private String recipientEmail;
    
    @Min(0)
    @Max(3600)
    private Integer delaySeconds;
}

SQL Injection Prevention

Prepared statements are used exclusively for all database queries
Parameterized queries with no string concatenation in SQL generation
ORM framework (Hibernate) is used to abstract raw SQL construction
Input sanitization for any dynamic query components (e.g., table names in audit logs)

Authentication Middleware Design

JWT-based authentication with 2-hour expiration and refresh token support
OAuth2 integration for third-party service authentication (e.g., Slack, email providers)
Authentication filter in the request pipeline that validates JWT tokens and extracts user context
API key support for service-to-service communication with HMAC signing

Authorization Enforcement

Role-Based Access Control (RBAC) with predefined roles: USER, ADMIN, SERVICE
Permission checking at service layer using @PreAuthorize annotations
Resource ownership validation - users can only access notifications they created
Fine-grained permissions for notification management operations (create, read, update, delete)

@PreAuthorize("hasRole('ADMIN') or @notificationPermissionService.canAccessNotification(#id)")
public void deleteNotification(String id) {
    // implementation
}

Sensitive Data Encryption at Rest and in Transit

At-rest encryption: AES-256 with AWS KMS for data at rest (notifications content, user identifiers)
In-transit encryption: TLS 1.3 for all API communications
Encryption keys are rotated quarterly with key rotation logs
Database encryption is handled at the storage layer (AWS RDS encryption)

PII Handling and Masking Strategy

Email addresses are stored in hashed format (SHA-256 with salt) for internal reference
User identifiers are masked in logs and audit trails (e.g., user@example.com → u***@example.com)
Personal data is never stored in logs or monitoring systems beyond the minimum required for operation
PII removal from error messages and logs using regex-based sanitization

Secrets Management

AWS Secrets Manager for storing:
- Third-party API credentials (Slack, email service)
- Database connection strings
- Encryption keys and certificates
Secret rotation every 90 days with automatic refresh
Access control at the secret level using IAM policies and secret-specific permissions

Audit Logging Design

Audit events capture:
- notification_created, notification_sent, notification_failed
- User ID, timestamp, IP address, operation type, affected resource ID

Log format follows structured JSON schema:

{
  "timestamp": "2023-10-15T14:30:00Z",
  "userId": "user_12345",
  "operation": "notification_created",
  "resourceId": "notif_abcde",
  "ipAddress": "192.168.1.100",
  "status": "success"
}

Retention policy: 90 days for audit logs, archived to S3 with 1-year retention
Compliance: Logs are encrypted at rest and accessible only to authorized security personnel

12. Testing Strategy

Unit Testing Scope and Coverage Targets

Unit tests will cover 85%+ of business logic and service layer code. Each service class will have dedicated test classes with:

100% branch coverage for all decision points
95% statement coverage for core logic paths
Test cases for all boundary conditions and error scenarios
Mock-based testing for external dependencies (database, message brokers, third-party APIs)

Integration Testing Strategy

Integration tests will validate end-to-end flows using:

Testcontainers for database and cache integration (PostgreSQL 13+, Redis 6+)
Embedded database for fast unit test execution (H2 in-memory DB)
WireMock for external API stubbing (e.g., notification service endpoints)
Test data factories to generate realistic test data for various scenarios
Database transaction rollback to maintain test isolation

Contract Testing

Contract testing will be implemented using Pact for:

Consumer-driven contracts between notification service and client applications
Versioned contract definitions with semantic versioning (v1, v2)
Automated contract verification in CI pipeline
Mock provider setup to validate consumer expectations

Performance Testing Tool and Key Scenarios

Performance testing will use k6 with the following key scenarios:

High-concurrency notification delivery: 1000 concurrent delayed notifications with 50ms delays
Batch processing throughput: 10,000 notifications processed in batches of 100
Peak load simulation: 5000 notifications/sec with 95th percentile < 200ms
Memory leak detection: 100,000 notifications processed over 2-hour period
Database query performance: <50ms average for notification lookup queries

Test Data Management Strategy

Test data will be managed through:

Factory pattern for generating test entities (Notification, User, Template)
Fixture files in JSON/YAML format for complex test scenarios
Database seed scripts for initial test environment setup
Test data cleanup procedures to ensure isolation between test runs
Data anonymization for PII in test environments

// Example unit test structure for notification service
@Test
public void testCreateDelayedNotification_Success() {
    // Given
    NotificationRequest request = createValidRequest();
    User user = createUser("user123");
    
    // When
    NotificationResult result = notificationService.createDelayedNotification(request);
    
    // Then
    assertThat(result.getId()).isNotNull();
    verify(notificationRepository).save(any(Notification.class));
    verify(eventPublisher).publish(any(NotificationCreatedEvent.class));
}

@Test
public void testCreateDelayedNotification_InvalidInput() {
    // Given
    NotificationRequest request = createInvalidRequest();
    
    // When & Then
    assertThatThrownBy(() -> notificationService.createDelayedNotification(request))
        .isInstanceOf(ValidationException.class)
        .hasMessageContaining("Invalid notification request");
}

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective querying and analysis:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO|WARN|ERROR",
  "service": "delayed-notification-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user_12345",
  "entity_id": "notification_98765",
  "operation": "sendDelayedNotification",
  "message": "Notification sent successfully",
  "error_code": "SUCCESS",
  "duration_ms": 150,
  "context": {
    "delay_seconds": 300,
    "recipient": "user_12345",
    "notification_type": "email"
  }
}

PII masking rules:

User email addresses are masked as user***@domain.com
Phone numbers are masked as +1 (555) ***-****
Full names are truncated to first letter: J***

Distributed Tracing Instrumentation

OpenTelemetry is used for distributed tracing with custom spans:

Key operations to trace:

processDelayedNotification - Main processing flow
sendNotification - External service call
persistNotification - Database operation

Span attributes to capture:

notification.id: Unique identifier for the notification
notification.delay_seconds: Delay in seconds before delivery
notification.type: Type of notification (email, SMS, push)
notification.status: Current status (pending, sent, failed)
retry.count: Number of retry attempts

Metrics Design

RED Method Metrics:

Metric Name	Type	Labels	Description
`notifications_processed_total`	Counter	status, type, error_code	Total notifications processed
`notification_processing_duration_seconds`	Histogram	status, type	Time to process notification
`notifications_failed_total`	Counter	error_code, type	Failed notifications by reason
`notification_retry_count`	Histogram	attempt_number, type	Retry attempts per notification
`queue_size`	Gauge	queue_name	Current size of processing queues

Key Labels:

status: pending, sent, failed, expired
type: email, sms, push
error_code: INVALID_INPUT, SERVICE_UNAVAILABLE, TIMEOUT

Health Check Endpoint Design

GET /health
GET /ready
GET /live
GET /startup

Response format:

{
  "status": "healthy|unhealthy",
  "checks": {
    "database": {"status": "healthy", "message": "Connected to primary DB"},
    "message_queue": {"status": "healthy", "message": "Connected to Redis"},
    "cache": {"status": "healthy", "message": "Connected to Redis cache"},
    "dependencies": {
      "external_api": {"status": "healthy", "latency_ms": 45}
    }
  },
  "timestamp": "2023-10-27T14:30:45.123Z"
}