Skip to main content
Interview-Mapagam

Notification System Low-Level Design

Comprehensive backend LLD for scalable notification delivery system with real-time updates and multi-channel support.

  • PublishedDecember 21, 2025
  • Length7,959 words
  • Reading time40 min read
Backend LLD

This document presents a scalable backend architecture for handling real-time notifications across multiple channels, focusing on high throughput, reliability, and horizontal scalability for modern distributed systems.

1. Problem Statement & Requirements

Clarifying questions

  1. What is the scale expectation for notifications? Are we talking about millions of users with potentially billions of notifications per day, or a smaller scale system?
  2. What are the consistency requirements for notifications? Should notifications be delivered exactly once, or is at-least-once delivery acceptable?
  3. What types of notifications are we supporting? SMS, email, push, in-app, or a combination of these?
  4. What is the read/write ratio for notifications? Are we primarily storing and retrieving notifications or more focused on real-time delivery?
  5. What is the latency sensitivity? Are notifications time-critical (e.g., real-time chat) or can they be batched with a few seconds delay?
  6. What authentication model is expected? Will the system be accessed by internal services or external clients?
  7. Are there any external integrations required? Do we need to integrate with third-party providers like Twilio, Firebase Cloud Messaging, or SendGrid?

Functional requirements

  • User notification creation: Create notifications for users with content, type, and delivery preferences
  • Notification retrieval: Fetch user's notification history with filtering capabilities (by type, date range, read status)
  • Notification delivery: Deliver notifications to user's preferred channels (email, SMS, push)
  • Notification status tracking: Track delivery status (sent, failed, read, etc.)
  • User preference management: Allow users to configure notification preferences by channel and type
  • Bulk notification support: Support sending notifications to multiple users or user segments
  • Notification scheduling: Schedule notifications to be sent at a future time
  • Notification archiving: Archive old notifications based on retention policy

Non-functional requirements

  • Latency targets:
    • p50: <100ms for notification creation and retrieval
    • p95: <500ms for notification creation and retrieval
    • p99: <2s for notification creation and retrieval
  • Throughput targets:
    • 10,000 notifications/sec for creation
    • 50,000 notifications/sec for retrieval (read-heavy)
  • Availability: 99.9% uptime (43.2 minutes of downtime/month)
  • Reliability: 99.9% delivery success rate for notifications
  • Scalability: Support for 10M+ concurrent users with linear scaling

Constraints and assumptions

  • Assumption: Notifications are primarily used for user engagement, not mission-critical system events
  • Constraint: System must support multi-tenant architecture with isolation between tenants
  • Assumption: Users may have multiple devices and channels, each with independent preferences
  • Constraint: Notification content must be sanitized to prevent injection attacks
  • Assumption: System will handle burst traffic during peak usage times (e.g., marketing campaigns)
  • Constraint: Notifications must support internationalization and localization

Out of scope

  • Notification content personalization: Advanced AI-driven content generation or A/B testing
  • Real-time push delivery: Direct WebSocket or long-polling connections for real-time notifications
  • Advanced analytics dashboard: Deep analytics on notification engagement, open rates, click-through rates
  • External provider integrations: Twilio, Firebase, SendGrid integration beyond basic API abstraction
  • Webhook delivery: Support for webhooks to external systems for notification delivery confirmation
  • Notification templates: Pre-defined template system for content generation
  • User segmentation: Advanced user grouping and targeting capabilities beyond simple preference filtering
  • Multi-language content: Full multi-language content management system (only basic localization support)

2. API Design

API Style Selection

REST is selected for this notification system due to its simplicity, wide adoption, and ease of integration with existing frontend systems. REST provides clear resource-based URLs that make the API intuitive to understand and use. While GraphQL offers flexibility in data fetching, it introduces complexity in query validation and caching that isn't justified for a notification system with predictable data structures.

API Endpoints

MethodPathDescriptionAuth
GET/v1/notificationsFetch user's notification listJWT
POST/v1/notificationsCreate new notificationJWT
GET/v1/notifications/{id}Fetch specific notificationJWT
PUT/v1/notifications/{id}/readMark notification as readJWT
DELETE/v1/notifications/{id}Delete specific notificationJWT
POST/v1/notifications/batchCreate multiple notificationsJWT
GET/v1/notifications/statsGet user's notification statisticsJWT

Request and Response Schemas

Create Notification Request

{
  "userId": "uuid",
  "title": "string (max 255 chars)",
  "body": "string (max 10000 chars)",
  "type": "enum: [EMAIL, SMS, PUSH, IN_APP]",
  "priority": "enum: [LOW, NORMAL, HIGH, URGENT]",
  "target": {
    "channel": "enum: [EMAIL, SMS, PUSH, IN_APP]",
    "recipient": "string (email or phone number)"
  },
  "metadata": {
    "key1": "value1",
    "key2": "value2"
  },
  "scheduledAt": "ISO8601 datetime (optional)"
}

Create Notification Response

{
  "id": "uuid",
  "userId": "uuid",
  "title": "string",
  "body": "string",
  "type": "enum",
  "priority": "enum",
  "status": "enum: [PENDING, SENT, FAILED]",
  "createdAt": "ISO8601 datetime",
  "updatedAt": "ISO8601 datetime",
  "readAt": "ISO8601 datetime (nullable)",
  "metadata": {
    "key1": "value1",
    "key2": "value2"
  }
}

Notification List Response

{
  "notifications": [
    {
      "id": "uuid",
      "userId": "uuid",
      "title": "string",
      "body": "string",
      "type": "enum",
      "priority": "enum",
      "status": "enum",
      "createdAt": "ISO8601 datetime",
      "readAt": "ISO8601 datetime (nullable)"
    }
  ],
  "pagination": {
    "cursor": "string",
    "hasMore": "boolean"
  }
}

HTTP Status Codes and Error Response Format

Status CodeDescription
200Success - GET, PUT, DELETE
201Created - POST
400Bad Request - Invalid input
401Unauthorized - Missing or invalid auth
403Forbidden - Insufficient permissions
404Not Found - Resource doesn't exist
429Too Many Requests - Rate limit exceeded
500Internal Server Error - Unexpected server error

Standardized error response format:

{
  "error": {
    "code": "string",
    "message": "string",
    "details": "object (optional)",
    "timestamp": "ISO8601 datetime"
  }
}

API Versioning Strategy

Versioning is done through the path prefix /v1/ to ensure backward compatibility and allow for future version evolution. This approach is simple, explicit, and easy to maintain.

Pagination Design

Cursor-based pagination is used because it's more stable and handles data changes better than offset-based pagination. It provides consistent results even when items are inserted or deleted between paginated requests.

Authentication and Authorization

  • JWT tokens with scopes for authentication
  • User ID extracted from token claims for authorization
  • RBAC model where users can only access their own notifications
  • Admin scope required for system-level operations

Rate Limiting

Rate limiting per endpoint:

  • GET /notifications: 100 requests/minute per user
  • POST /notifications: 50 requests/minute per user
  • PUT /notifications/{id}/read: 1000 requests/minute per user
  • DELETE /notifications/{id}: 50 requests/minute per user

Rate limiting enforced using Redis with sliding window algorithm.

Idempotency Design

Mutation endpoints support idempotency using:

  • Idempotency-Key header with UUID value
  • Idempotency store in Redis with TTL of 24 hours
  • For POST /notifications, if same key exists with same payload, return existing notification
  • For PUT /notifications/{id}/read, if already read, return success without change

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    USER {
        string id PK
        string email
        string phone
        string status
        timestamp created_at
        timestamp updated_at
    }

    NOTIFICATION {
        string id PK
        string user_id FK
        string title
        string content
        string type
        string status
        boolean is_read
        timestamp created_at
        timestamp updated_at
        timestamp delivered_at
    }

    NOTIFICATION_TEMPLATE {
        string id PK
        string name
        string subject
        string content
        string type
        timestamp created_at
        timestamp updated_at
    }

    NOTIFICATION_CHANNEL {
        string id PK
        string name
        string provider
        boolean is_active
        timestamp created_at
        timestamp updated_at
    }

    USER_NOTIFICATION_PREFERENCES {
        string user_id FK
        string channel_id FK
        string notification_type
        boolean is_enabled
        timestamp created_at
        timestamp updated_at
    }

    USER_NOTIFICATION_PREFERENCES ||--o{ USER : "has"
    USER ||--o{ NOTIFICATION : "receives"
    NOTIFICATION ||--o{ NOTIFICATION_TEMPLATE : "uses"
    NOTIFICATION ||--o{ NOTIFICATION_CHANNEL : "sent_via"

Table Schema Definitions

Users Table

CREATE TABLE users (
    id VARCHAR(36) PRIMARY KEY,
    email VARCHAR(255) UNIQUE NOT NULL,
    phone VARCHAR(20),
    status ENUM('ACTIVE', 'INACTIVE', 'SUSPENDED') DEFAULT 'ACTIVE',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    
    INDEX idx_email (email),
    INDEX idx_status (status)
);

Notifications Table

CREATE TABLE notifications (
    id VARCHAR(36) PRIMARY KEY,
    user_id VARCHAR(36) NOT NULL,
    title VARCHAR(255) NOT NULL,
    content TEXT NOT NULL,
    type VARCHAR(50) NOT NULL,
    status ENUM('PENDING', 'DELIVERED', 'FAILED', 'CANCELLED') DEFAULT 'PENDING',
    is_read BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    delivered_at TIMESTAMP NULL,
    
    INDEX idx_user_id (user_id),
    INDEX idx_status (status),
    INDEX idx_created_at (created_at),
    INDEX idx_user_status (user_id, status),
    FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);

Notification Templates Table

CREATE TABLE notification_templates (
    id VARCHAR(36) PRIMARY KEY,
    name VARCHAR(100) NOT NULL UNIQUE,
    subject VARCHAR(255),
    content TEXT NOT NULL,
    type VARCHAR(50) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    
    INDEX idx_type (type),
    INDEX idx_name (name)
);

Notification Channels Table

CREATE TABLE notification_channels (
    id VARCHAR(36) PRIMARY KEY,
    name VARCHAR(50) NOT NULL UNIQUE,
    provider VARCHAR(50) NOT NULL,
    is_active BOOLEAN DEFAULT TRUE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    
    INDEX idx_provider (provider),
    INDEX idx_active (is_active)
);

User Notification Preferences Table

CREATE TABLE user_notification_preferences (
    user_id VARCHAR(36) NOT NULL,
    channel_id VARCHAR(36) NOT NULL,
    notification_type VARCHAR(50) NOT NULL,
    is_enabled BOOLEAN DEFAULT TRUE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    
    PRIMARY KEY (user_id, channel_id, notification_type),
    INDEX idx_notification_type (notification_type),
    INDEX idx_channel_enabled (channel_id, is_enabled),
    FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE,
    FOREIGN KEY (channel_id) REFERENCES notification_channels(id) ON DELETE CASCADE
);

Primary Key Strategy

UUID (Universally Unique Identifier) is selected for all primary keys due to:

  • Distributed system compatibility (no coordination needed for ID generation)
  • No sequential patterns that could expose system structure
  • Better sharding capabilities across multiple database instances
  • Support for eventual consistency in distributed environments

Index Design

Index NameFieldsTypePurpose
idx_user_iduser_idStandardFast lookup by user
idx_statusstatusStandardQuery by notification status
idx_created_atcreated_atStandardTime-based queries
idx_user_statususer_id, statusCompositeEfficient user notification list
idx_notification_typenotification_typeStandardFilter by type in preferences
idx_channel_enabledchannel_id, is_enabledCompositeFast channel filtering
idx_providerproviderStandardFilter by delivery provider

Data Partitioning / Sharding Strategy

Horizontal sharding by user_id hash:

  • Hash(user_id) % shard_count to determine database shard
  • Shard key: user_id (UUID)
  • Benefits:
    • Even distribution of users across shards
    • Support for multi-tenant notification systems
    • Scalable to millions of users

Soft Delete vs. Hard Delete Decision

Soft delete is used for notifications due to:

  • Audit and compliance requirements (cannot permanently delete user history)
  • Ability to restore accidentally deleted notifications
  • Analytics tracking of notification lifecycle
  • Support for data retention policies

Audit Trail Design

All data modifications capture:

  • created_by (user_id or system identifier)
  • updated_by (user_id or system identifier)
  • created_at and updated_at timestamps
  • operation_type (INSERT, UPDATE, DELETE)
  • changed_fields (JSON array of modified fields)

Data Archival Strategy and Retention Policy

Notification archival policy:

  • 30 days: Active notifications (read/unread)
  • 90 days: Read notifications (archived)
  • 365 days: Failed notifications (with reasons)
  • Permanent: Templates and preferences

Archival process:

  • Move notifications to archive tables based on status and age
  • Compress content for older notifications (if storage cost is a concern)
  • Maintain referential integrity between main and archive tables

4. Class Design & Service Layer

classDiagram
    class NotificationService {
        +sendNotification(NotificationRequest)
        +getNotificationsByUser(String userId, NotificationFilter filter)
        +markAsRead(String notificationId, String userId)
        +markAllAsRead(String userId)
        +deleteNotification(String notificationId, String userId)
    }
    
    class Notification {
        -id: UUID
        -userId: String
        -title: String
        -body: String
        -type: NotificationType
        -status: NotificationStatus
        -createdAt: Instant
        -readAt: Instant
        -metadata: Map~String, Object~
    }
    
    class NotificationRequest {
        userId: String
        title: String
        body: String
        type: NotificationType
        metadata: Map~String, Object~
    }
    
    class NotificationFilter {
        status: NotificationStatus
        type: NotificationType
        limit: Integer
        offset: Integer
    }
    
    class NotificationEvent {
        eventType: String
        payload: Map~String, Object~
        timestamp: Instant
    }
    
    class NotificationProducer {
        +publishEvent(NotificationEvent)
    }
    
    class NotificationConsumer {
        +handleNotificationEvent(NotificationEvent)
    }
    
    class NotificationRepository {
        +save(Notification)
        +findById(UUID)
        +findByUserId(String userId, NotificationFilter filter)
        +updateStatus(UUID, NotificationStatus)
        +deleteById(UUID)
    }
    
    class EventPublisher {
        +publish(String topic, Object event)
    }
    
    class NotificationProcessor {
        +process(NotificationRequest)
    }
    
    class NotificationValidator {
        +validate(NotificationRequest)
    }
    
    class NotificationTemplate {
        templateId: String
        content: String
        variables: Set~String~
    }
    
    class NotificationTemplateService {
        +getTemplate(String templateId)
        +renderTemplate(String templateId, Map~String, String~ variables)
    }
    
    NotificationService --> NotificationRepository
    NotificationService --> NotificationProducer
    NotificationService --> NotificationValidator
    NotificationService --> NotificationProcessor
    NotificationService --> NotificationTemplateService
    
    NotificationRequest --> Notification
    Notification --> NotificationEvent
    NotificationEvent --> NotificationConsumer
    
    NotificationRepository --> EventPublisher
    NotificationProcessor --> NotificationTemplateService
stateDiagram-v2
    [*] --> PENDING
    PENDING --> SENT : process()
    SENT --> DELIVERED : delivery()
    SENT --> FAILED : timeout()
    DELIVERED --> READ : userRead()
    FAILED --> RETRY : retryableError()
    FAILED --> ARCHIVED : permanentFailure()
    READ --> [*]
    RETRY --> PENDING : retryCount < maxRetries
    RETRY --> FAILED : retryCount >= maxRetries

Service Class Structure and Responsibilities

The notification system consists of the following key service classes:

  1. NotificationService: Main orchestrator for all notification operations
  2. NotificationProcessor: Handles business logic for creating and processing notifications
  3. NotificationTemplateService: Manages notification templates and content rendering
  4. NotificationValidator: Validates incoming notification requests
  5. EventPublisherService: Handles event publishing to message brokers

Interface / Abstract Class Design

public interface NotificationRepository {
    Notification save(Notification notification);
    Optional<Notification> findById(UUID id);
    List<Notification> findByUserId(String userId, NotificationFilter filter);
    void updateStatus(UUID id, NotificationStatus status);
    void deleteById(UUID id);
}

public interface EventPublisher {
    void publish(String topic, Object event);
}

public enum NotificationType {
    EMAIL, SMS, PUSH, IN_APP
}

public enum NotificationStatus {
    PENDING, SENT, DELIVERED, FAILED, READ, ARCHIVED
}

public interface NotificationValidator {
    void validate(NotificationRequest request);
}

Business Logic Encapsulation Patterns

The system uses the Strategy Pattern for handling different notification types:

public interface NotificationDeliveryStrategy {
    void deliver(Notification notification);
}

@Component
public class EmailDeliveryStrategy implements NotificationDeliveryStrategy {
    private final EmailService emailService;
    
    @Override
    public void deliver(Notification notification) {
        // Implementation for email delivery
        emailService.sendEmail(
            notification.getUserId(),
            notification.getTitle(),
            notification.getBody()
        );
    }
}

@Component
public class PushDeliveryStrategy implements NotificationDeliveryStrategy {
    private final PushNotificationService pushService;
    
    @Override
    public void deliver(Notification notification) {
        // Implementation for push delivery
        pushService.sendPush(
            notification.getUserId(),
            notification.getTitle(),
            notification.getBody()
        );
    }
}

@Component
public class NotificationDeliveryContext {
    private final Map<NotificationType, NotificationDeliveryStrategy> strategies;
    
    public void deliver(Notification notification) {
        NotificationDeliveryStrategy strategy = 
            strategies.getOrDefault(notification.getType(), 
                strategies.get(NotificationType.IN_APP));
        strategy.delive(notification);
    }
}

Critical-Path Method Implementation

@Transactional
public NotificationResult sendNotification(NotificationRequest request) {
    // Validate input
    validator.validate(request);
    
    // Create notification entity
    Notification notification = Notification.builder()
        .id(UUID.randomUUID())
        .userId(request.getUserId())
        .title(request.getTitle())
        .body(request.getBody())
        .type(request.getType())
        .status(NotificationStatus.PENDING)
        .metadata(request.getMetadata())
        .createdAt(Instant.now())
        .build();
    
    // Save to database
    Notification savedNotification = notificationRepository.save(notification);
    
    // Acquire lock for user to prevent race conditions
    String lockKey = "notification_lock:" + request.getUserId();
    try (RedisLock lock = redisLockManager.acquire(lockKey, Duration.ofSeconds(10))) {
        // Update notification status to SENT
        notificationRepository.updateStatus(savedNotification.getId(), NotificationStatus.SENT);
        
        // Publish event for async processing
        NotificationEvent event = NotificationEvent.builder()
            .eventType("NOTIFICATION_SENT")
            .payload(Map.of(
                "notificationId", savedNotification.getId().toString(),
                "userId", request.getUserId(),
                "type", request.getType().name()
            ))
            .timestamp(Instant.now())
            .build();
        
        eventPublisher.publish("notifications", event);
        
        // Process based on type
        notificationDeliveryContext.deliver(savedNotification);
        
        return NotificationResult.success(
            savedNotification.getId(),
            "Notification sent successfully"
        );
    } catch (Exception e) {
        // Handle lock acquisition failure
        notificationRepository.updateStatus(
            savedNotification.getId(), 
            NotificationStatus.FAILED
        );
        
        throw new NotificationProcessingException(
            "Failed to send notification", e);
    }
}

Transaction Boundary Design

A single transaction encompasses:

  1. Creating the notification record in database
  2. Updating status to SENT (within same transaction)
  3. Publishing event to message broker (in same transaction)

Service Interface Contracts

public interface NotificationService {
    NotificationResult sendNotification(NotificationRequest request);
    
    List<Notification> getNotificationsByUser(
        String userId, 
        NotificationFilter filter);
    
    boolean markAsRead(String notificationId, String userId);
    
    int markAllAsRead(String userId);
    
    boolean deleteNotification(String notificationId, String userId);
}

Saga / Orchestration Pattern

For complex workflows involving external systems:

public class NotificationSaga {
    private final List<SagaStep> steps;
    
    public void execute(NotificationRequest request) {
        for (SagaStep step : steps) {
            try {
                step.execute(request);
            } catch (Exception e) {
                // Compensate previous steps
                compensateSteps(steps, step);
                throw new SagaExecutionException("Saga failed", e);
            }
        }
    }
}

Domain Object / Value Object Design

@Value
public class NotificationRequest {
    String userId;
    String title;
    String body;
    NotificationType type;
    Map<String, Object> metadata;
    
    public static class Builder {
        private String userId;
        private String title;
        private String body;
        private NotificationType type;
        private Map<String, Object> metadata = new HashMap<>();
        
        public Builder userId(String userId) {
            this.userId = userId;
            return this;
        }
        
        public Builder title(String title) {
            this.title = title;
            return this;
        }
        
        public Builder body(String body) {
            this.body = body;
            return this;
        }
        
        public Builder type(NotificationType type) {
            this.type = type;
            return this;
        }
        
        public Builder metadata(Map<String, Object> metadata) {
            this.metadata = metadata;
            return this;
        }
        
        public NotificationRequest build() {
            return new NotificationRequest(userId, title, body, type, metadata);
        }
    }
}

@Value
public class NotificationFilter {
    NotificationStatus status;
    NotificationType type;
    Integer limit;
    Integer offset;
    
    public static class Builder {
        private NotificationStatus status;
        private NotificationType type;
        private Integer limit = 20;
        private Integer offset = 0;
        
        public Builder status(NotificationStatus status) {
            this.status = status;
            return this;
        }
        
        public Builder type(NotificationType type) {
            this.type = type;
            return this;
        }
        
        public Builder limit(Integer limit) {
            this.limit = limit;
            return this;
        }
        
        public Builder offset(Integer offset) {
            this.offset = offset;
            return this;
        }
        
        public NotificationFilter build() {
            return new NotificationFilter(status, type, limit, offset);
        }
    }
}

5. Concurrency & Thread Safety

Concurrency Model

The notification system will use a thread pool-based concurrency model with Java's ExecutorService for handling concurrent message processing. Each notification channel (email, SMS, push) will have its own dedicated thread pool to prevent one slow channel from blocking others.

@Configuration
public class NotificationExecutorConfig {
    @Bean
    public ExecutorService emailNotificationExecutor() {
        return Executors.newFixedThreadPool(
            10, 
            ThreadFactoryBuilder
                .newBuilder()
                .setNameFormat("email-notif-%d")
                .build()
        );
    }
    
    @Bean
    public ExecutorService smsNotificationExecutor() {
        return Executors.newFixedThreadPool(
            5,
            ThreadFactoryBuilder
                .newBuilder()
                .setNameFormat("sms-notif-%d")
                .build()
        );
    }
}

Race Condition Prevention Strategy

Race conditions in the notification system are primarily concerned with:

  1. Concurrent updates to notification status
  2. Duplicate notification creation for the same event
  3. Concurrent delivery attempts to the same recipient

To prevent these, we use:

  • Database-level atomic operations (e.g., UPDATE notifications SET status = ? WHERE id = ? AND status = ?)
  • Distributed locks using Redis for critical sections
  • Idempotency keys in request processing

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected for most operations because:

  • Notifications are primarily read-heavy with infrequent updates
  • Concurrent creation of notifications is rare and handled via idempotency keys
  • The system prioritizes throughput over strict consistency

Pessimistic locking is used for:

  • Delivery attempts where we must ensure only one concurrent delivery attempt per recipient
  • Critical status transitions (e.g., from PENDING to DELIVERED)
@Service
public class NotificationDeliveryService {
    
    @Autowired
    private RedisTemplate<String, String> redisTemplate;
    
    public boolean attemptDelivery(String recipientId, String notificationId) {
        String lockKey = "delivery_lock:" + recipientId;
        String lockValue = UUID.randomUUID().toString();
        
        try {
            // Attempt to acquire distributed lock
            Boolean acquired = redisTemplate.opsForValue()
                .setIfAbsent(lockKey, lockValue, Duration.ofSeconds(30));
            
            if (!acquired) {
                return false; // Another delivery is in progress
            }
            
            // Perform the actual delivery operation
            return processDelivery(recipientId, notificationId);
        } finally {
            // Release lock if we own it
            releaseLock(lockKey, lockValue);
        }
    }
    
    private void releaseLock(String lockKey, String lockValue) {
        String script = 
            "if redis.call('get', KEYS[1]) == ARGV[1] then " +
            "return redis.call('del', KEYS[1]) else return 0 end";
        
        redisTemplate.execute(
            new DefaultRedisScript<>(script, Long.class),
            Collections.singletonList(lockKey),
            lockValue
        );
    }
}

Deadlock Prevention Strategy

Deadlock prevention is addressed through:

  1. Lock ordering: All operations acquire locks in a consistent order (e.g., by recipient ID hash)
  2. Timeout-based acquisition: Distributed locks have 30-second timeouts to prevent indefinite blocking
  3. Avoid nested locks: Minimize the scope of lock acquisition

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is implemented at the API layer using UUID-based idempotency keys:

@RestController
@RequestMapping("/notifications")
public class NotificationController {
    
    @Autowired
    private NotificationService notificationService;
    
    @PostMapping
    public ResponseEntity<Notification> createNotification(
            @RequestHeader("Idempotency-Key") String idempotencyKey,
            @RequestBody CreateNotificationRequest request) {
        
        // Use idempotency key to prevent duplicate processing
        Notification existing = notificationService.findOrCreateByIdempotencyKey(
            idempotencyKey, request);
        
        if (existing != null) {
            return ResponseEntity.ok(existing);
        }
        
        // Proceed with new notification creation
        Notification created = notificationService.createNotification(request);
        return ResponseEntity.status(HttpStatus.CREATED).body(created);
    }
}
@Service
public class NotificationService {
    
    private final Map<String, Notification> idempotencyCache = new ConcurrentHashMap<>();
    
    public Notification findOrCreateByIdempotencyKey(
            String idempotencyKey, CreateNotificationRequest request) {
        
        return idempotencyCache.computeIfAbsent(idempotencyKey, key -> {
            // Check if notification already exists in database
            Notification existing = notificationRepository.findByRequestHash(
                calculateRequestHash(request));
            
            if (existing != null) {
                return existing;
            }
            
            // Create new notification
            return createNotification(request);
        });
    }
    
    private String calculateRequestHash(CreateNotificationRequest request) {
        // Hash based on recipient, content, and timestamp to ensure uniqueness
        return DigestUtils.md5Hex(
            request.getRecipientId() + 
            request.getContent() + 
            request.getTimestamp().toString()
        );
    }
}

6. Sequence Diagrams for Key Flows

Happy Path Sequence Diagram

sequenceDiagram
    participant U as User
    participant API as NotificationServiceAPI
    participant Service as NotificationService
    participant Repo as NotificationRepository
    participant Cache as RedisCache
    participant MQ as MessageQueue
    participant Notifier as NotificationSender

    U->>API: Send notification request
    API->>Service: processNotification()
    Service->>Repo: saveNotification()
    Repo-->>Service: notificationId
    Service->>Cache: cacheNotification()
    Cache-->>Service: success
    Service->>MQ: publishNotificationEvent()
    MQ->>Notifier: consume notification event
    Notifier->>Notifier: sendPushNotification()
    Notifier-->>U: Push notification delivered

Error / Edge Case Sequence Diagram

sequenceDiagram
    participant U as User
    participant API as NotificationServiceAPI
    participant Service as NotificationService
    participant Repo as NotificationRepository
    participant Cache as RedisCache
    participant MQ as MessageQueue
    participant Notifier as NotificationSender

    U->>API: Send notification request with invalid user ID
    API->>Service: processNotification()
    Service-->>U: Error response (400)
    
    U->>API: Send notification request with invalid template
    API->>Service: processNotification()
    Service->>Repo: saveNotification()
    Repo-->>Service: Database error
    Service-->>U: Error response (500)
    
    U->>API: Send notification request
    API->>Service: processNotification()
    Service->>Repo: saveNotification()
    Repo-->>Service: notificationId
    Service->>Cache: cacheNotification()
    Cache-->>Service: Cache timeout error
    Service->>MQ: publishNotificationEvent()
    MQ->>Notifier: consume notification event
    Notifier->>Notifier: sendPushNotification()
    Notifier-->>U: Push notification failed

Async Flow Sequence Diagram

sequenceDiagram
    participant U as User
    participant API as NotificationServiceAPI
    participant Service as NotificationService
    participant Repo as NotificationRepository
    participant MQ as MessageQueue
    participant Notifier as NotificationSender
    participant DB as Database

    U->>API: Send notification request
    API->>Service: processNotification()
    Service->>Repo: saveNotification()
    Repo-->>Service: notificationId
    Service->>MQ: publishNotificationEvent()
    MQ->>Notifier: consume notification event
    Notifier->>DB: fetchUserPreferences()
    DB-->>Notifier: user preferences
    Notifier->>Notifier: sendPushNotification()
    Notifier-->>U: Push notification delivered
    Notifier->>Repo: updateNotificationStatus()
    Repo-->>Notifier: success

7. Caching Design

Cache Strategy Selection

For the notification system, we'll use a cache-aside pattern with write-through behavior for critical notification data. This approach provides the best balance between consistency and performance for a system where real-time delivery is important but not mission-critical.

Cache Key Design

Cache keys will follow a structured namespace pattern:

notification:{userId}:{notificationId}
notification:unreadCount:{userId}
notification:preferences:{userId}

TTL Strategy

Data TypeTTLReason
User notification list1 hourRecent notifications should be readily available
Unread count5 minutesFrequent updates needed for real-time UI
User preferences24 hoursPreferences change infrequently
Notification templates1 weekTemplates are stable and rarely updated

Cache Eviction Policy

We'll use LRU (Least Recently Used) eviction policy with a maximum cache size of 10GB. This ensures that frequently accessed notifications and preferences remain in cache while evicting stale data efficiently.

Cache Stampede Prevention

To prevent cache stampedes:

  1. Probabilistic early expiry: Set cache TTL to 80% of maximum duration (e.g., 4 minutes instead of 5)
  2. Distributed locks: Use Redis distributed locks for cache population when cache misses occur
  3. Cache warming: Pre-warm popular user notification caches during low-traffic periods

Distributed Cache Architecture

We'll implement a Redis cluster with:

  • 3 master nodes for write operations
  • 3 replica nodes for read operations
  • Automatic failover with Redis Sentinel
  • Cross-zone replication for high availability

Cache Invalidation Strategy

We'll use event-driven invalidation:

  1. When a notification is created/updated, publish notification.created or notification.updated events
  2. When user preferences change, publish user.preferences.updated event
  3. Consumers listen to these events and invalidate relevant cache entries
  4. For immediate consistency, we'll also implement cache invalidation on write for critical operations

Cache Implementation Example

@Component
public class NotificationCacheService {
    private final RedisTemplate<String, Object> redisTemplate;
    private final ObjectMapper objectMapper;
    
    public void cacheNotification(String userId, Notification notification) {
        String key = "notification:" + userId + ":" + notification.getId();
        String cacheValue = objectMapper.writeValueAsString(notification);
        
        // Set with TTL and atomic operation to prevent stampede
        redisTemplate.opsForValue()
            .set(key, cacheValue, Duration.ofHours(1));
    }
    
    public Notification getCachedNotification(String userId, String notificationId) {
        String key = "notification:" + userId + ":" + notificationId;
        String cachedValue = (String) redisTemplate.opsForValue().get(key);
        
        if (cachedValue != null) {
            return objectMapper.readValue(cachedValue, Notification.class);
        }
        
        return null;
    }
    
    public void invalidateUserNotifications(String userId) {
        // Using Redis SCAN to avoid blocking operations
        String pattern = "notification:" + userId + ":*";
        Set<String> keys = redisTemplate.keys(pattern);
        
        if (!keys.isEmpty()) {
            redisTemplate.delete(keys);
        }
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and persistence concerns. Each entity has its own repository interface with standardized CRUD operations.

public interface NotificationRepository {
    Notification save(Notification notification);
    Optional<Notification> findById(UUID id);
    List<Notification> findByUserId(UUID userId, Pageable pageable);
    List<Notification> findByUserIdAndStatus(UUID userId, NotificationStatus status, Pageable pageable);
    void markAsRead(UUID id);
    void markAsReadByUserId(UUID userId);
    void deleteById(UUID id);
    long countUnreadByUserId(UUID userId);
}

public interface NotificationTemplateRepository {
    NotificationTemplate save(NotificationTemplate template);
    Optional<NotificationTemplate> findByCode(String code);
    List<NotificationTemplate> findAll();
}

ORM vs Raw Query Decisions

For the notification system, we use JPA/Hibernate as our ORM for most operations due to:

  • Type safety and compile-time checking
  • Reduced boilerplate code
  • Easy mapping between entities and database tables
  • Support for complex queries through JPQL

However, we use raw SQL for performance-critical operations like bulk read operations and analytics queries:

@Repository
public class NotificationRepositoryImpl {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    @Transactional(readOnly = true)
    public List<Notification> findUnreadByUserId(UUID userId, Pageable pageable) {
        String sql = """
            SELECT n FROM Notification n 
            WHERE n.userId = :userId AND n.status = :status
            ORDER BY n.createdAt DESC
            """;
            
        TypedQuery<Notification> query = entityManager.createQuery(sql, Notification.class);
        query.setParameter("userId", userId);
        query.setParameter("status", NotificationStatus.UNREAD);
        
        return query
            .setFirstResult(pageable.getOffset())
            .setMaxResults(pageable.getPageSize())
            .getResultList();
    }
    
    @Transactional
    public void markAsReadBatch(List<UUID> notificationIds) {
        String sql = """
            UPDATE Notification n 
            SET n.status = :status, n.readAt = :readAt
            WHERE n.id IN (:ids)
            """;
            
        entityManager.createQuery(sql)
            .setParameter("status", NotificationStatus.READ)
            .setParameter("readAt", Instant.now())
            .setParameter("ids", notificationIds)
            .executeUpdate();
    }
}

Read vs Write Separation

We implement CQRS (Command Query Responsibility Segregation) for the notification system to optimize read and write operations:

// Write model - used for creating/updating notifications
@Repository
public class NotificationWriteRepository {
    @PersistenceContext
    private EntityManager entityManager;
    
    public void save(Notification notification) {
        entityManager.persist(notification);
    }
}

// Read model - optimized for querying notifications
@Repository
public class NotificationReadRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    public List<NotificationView> findRecentByUserId(UUID userId, int limit) {
        String sql = """
            SELECT new com.example.notification.dto.NotificationView(
                n.id, n.title, n.body, n.status, n.createdAt
            )
            FROM Notification n 
            WHERE n.userId = :userId 
            ORDER BY n.createdAt DESC
            """;
            
        return entityManager.createQuery(sql, NotificationView.class)
            .setParameter("userId", userId)
            .setMaxResults(limit)
            .getResultList();
    }
}

Connection Pool Configuration

We configure HikariCP connection pool with optimized settings:

# application.yml
spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      validation-timeout: 5000
      leak-detection-threshold: 60000

N+1 Query Prevention Strategy

We prevent N+1 queries by implementing eager loading and batch fetching:

@Entity
@Table(name = "notifications")
@NamedEntityGraph(
    name = "notification.withUser",
    attributeNodes = @NamedAttributeNode(value = "user", subgraph = "user-subgraph"),
    subgraphs = @NamedSubgraph(
        name = "user-subgraph",
        attributeNodes = @NamedAttributeNode("user")
    )
)
public class Notification {
    @ManyToOne(fetch = FetchType.LAZY)
    @JoinColumn(name = "user_id")
    private User user;
    
    @OneToMany(mappedBy = "notification", fetch = FetchType.LAZY)
    @BatchSize(size = 20)
    private List<NotificationEvent> events = new ArrayList<>();
}

// In repository query
@Query("SELECT n FROM Notification n JOIN FETCH n.user WHERE n.userId = :userId")
List<Notification> findByUserIdWithUser(@Param("userId") UUID userId);

Batch Operation Design

For high-volume operations, we implement batch processing:

@Repository
public class BatchNotificationRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    public void batchCreate(List<Notification> notifications) {
        int batchSize = 1000;
        
        for (int i = 0; i < notifications.size(); i += batchSize) {
            int end = Math.min(i + batchSize, notifications.size());
            List<Notification> batch = notifications.subList(i, end);
            
            for (int j = 0; j < batch.size(); j++) {
                entityManager.persist(batch.get(j));
                
                if (j % 100 == 0) {
                    entityManager.flush();
                    entityManager.clear();
                }
            }
        }
    }
    
    public void batchUpdateStatus(List<UUID> notificationIds, NotificationStatus status) {
        String sql = """
            UPDATE Notification n 
            SET n.status = :status, n.updatedAt = :updatedAt
            WHERE n.id IN (:ids)
            """;
            
        entityManager.createQuery(sql)
            .setParameter("status", status)
            .setParameter("updatedAt", Instant.now())
            .setParameter("ids", notificationIds)
            .executeUpdate();
    }
}

Transaction Isolation Strategy

We use READ_COMMITTED isolation level for most operations and SERIALIZABLE for critical batch operations:

@Service
@Transactional(isolation = Isolation.READ_COMMITTED)
public class NotificationService {
    
    @Transactional(isolation = Isolation.SERIALIZABLE)
    public void processBulkReadOperations(List<UUID> userIds) {
        // Critical batch operation requiring strict consistency
    }
}

Query Optimization with Indexes

We create optimized indexes for common query patterns:

-- Composite index for user and status queries
CREATE INDEX idx_notifications_user_status ON notifications(user_id, status);

-- Composite index for user and creation time queries  
CREATE INDEX idx_notifications_user_created ON notifications(user_id, created_at);

-- Partial index for unread notifications
CREATE INDEX idx_notifications_unread ON notifications(user_id) 
WHERE status = 'UNREAD';

Soft Delete vs Hard Delete

We implement soft delete to maintain audit trail and allow for recovery:

@Entity
@Table(name = "notifications")
public class Notification {
    @Column(name = "deleted_at")
    private Instant deletedAt;
    
    @Column(name = "is_deleted")
    private boolean deleted = false;
    
    public void softDelete() {
        this.deleted = true;
        this.deletedAt = Instant.now();
    }
    
    public boolean isDeleted() {
        return deleted;
    }
}

Data Archival Strategy

We implement a daily archival job that moves old notifications to archive tables:

@Component
public class NotificationArchivalService {
    
    @Transactional
    public void archiveOldNotifications() {
        String sql = """
            INSERT INTO notification_archive (id, user_id, title, body, status, created_at, read_at)
            SELECT id, user_id, title, body, status, created_at, read_at 
            FROM notifications 
            WHERE created_at < :cutoffDate AND deleted = false
            """;
            
        entityManager.createNativeQuery(sql)
            .setParameter("cutoffDate", Instant.now().minus(30, ChronoUnit.DAYS))
            .executeUpdate();
            
        // Then mark original records as deleted
        entityManager.createNativeQuery(
            "UPDATE notifications SET deleted = true WHERE created_at < :cutoffDate")
            .setParameter("cutoffDate", Instant.now().minus(30, ChronoUnit.DAYS))
            .executeUpdate();
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The notification system will use a hybrid synchronous/asynchronous approach. Critical operations like sending an immediate email or SMS will be handled synchronously to ensure user experience is not degraded. Non-critical notifications such as digest emails, analytics tracking, or audit logging will be processed asynchronously to maintain system responsiveness and scalability.

Key triggers for async processing:

  • User registration completion → send welcome email
  • Order placement → trigger order confirmation and inventory update
  • Payment success → notify user and update status
  • Scheduled events → send reminders (e.g., 1 hour before)
  • System health alerts → broadcast to admin channels

Message schema design with sample payload

{
  "version": "1.0",
  "eventId": "uuid-1234567890",
  "timestamp": "2023-10-15T14:30:00Z",
  "eventType": "user_registered",
  "payload": {
    "userId": "user-123456",
    "email": "john.doe@example.com",
    "templateId": "welcome_email_v2",
    "context": {
      "userName": "John Doe",
      "verificationUrl": "https://app.example.com/verify?token=abc123"
    }
  },
  "source": "user-service",
  "correlationId": "trace-abc123"
}

Dead letter queue strategy and poison message handling

Poison messages will be routed to a dedicated dead-letter queue (DLQ) after 3 failed delivery attempts. The DLQ will be monitored by an alerting system that triggers manual investigation when messages accumulate beyond a threshold (e.g., 100 messages). Messages in DLQ will be stored with metadata including:

  • Original payload
  • Failure reason and stack trace
  • Retry count and timestamps
  • Source service and event type

At-least-once vs. exactly-once delivery

At-least-once delivery is selected for this system as it provides better performance and simpler implementation than exactly-once semantics. The system uses idempotent consumers to handle duplicate messages gracefully, and all notification operations are designed to be idempotent where possible.

Retry policy for consumers

Retry strategy:

  • Max attempts: 3
  • Backoff formula: exponential with jitter (1s, 2s, 4s)
  • Retryable errors: network timeouts, service unavailability, transient DB errors
  • Non-retryable errors: validation failures, user not found, malformed payloads
public class RetryPolicy {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    
    public boolean shouldRetry(int attempt, Exception error) {
        if (attempt >= MAX_RETRIES) return false;
        
        // Retry on transient errors
        return error instanceof NetworkException || 
               error instanceof ServiceUnavailableException ||
               error instanceof TimeoutException;
    }
    
    public long getDelay(int attempt) {
        long delay = (long) Math.pow(2, attempt) * BASE_DELAY_MS;
        // Add jitter: ±25% randomization
        return delay + (long) (Math.random() * 0.5 * delay - 0.25 * delay);
    }
}

Outbox pattern for reliable event publishing

Outbox pattern is implemented to ensure eventual consistency between database writes and message publishing. When a notification event needs to be sent:

  1. Create notification record in database
  2. Insert event into outbox table with status "pending"
  3. Publish to message broker from a background processor
  4. Update outbox record status to "published" once successful

This prevents data loss in case of system failures during event publishing.

@Transactional
public void handleUserRegistration(User user) {
    // 1. Create notification record
    Notification notification = notificationService.createNotification(
        user.getId(), 
        "welcome_email", 
        Map.of("userName", user.getName())
    );
    
    // 2. Insert into outbox
    OutboxEvent event = new OutboxEvent(
        UUID.randomUUID().toString(),
        "user_registered",
        notification.getId(),
        "pending"
    );
    outboxRepository.save(event);
    
    // 3. Publish to message broker in background
    eventPublisher.publishAsync(event);
}

Sequence diagram for async flows

sequenceDiagram
    participant UserService
    participant NotificationService
    participant OutboxRepository
    participant MessageBroker
    participant EmailService
    
    UserService->>NotificationService: User registered
    NotificationService->>OutboxRepository: Create outbox event
    OutboxRepository-->>NotificationService: Event saved
    NotificationService->>MessageBroker: Publish to queue
    MessageBroker-->>NotificationService: Acknowledged
    loop Retry logic
        MessageBroker->>EmailService: Deliver email event
        EmailService-->>MessageBroker: Success/Failure
    end
    MessageBroker->>OutboxRepository: Update event status

10. Error Handling Design

Error Classification Hierarchy

The notification system implements a three-tier error classification:

  1. Validation Errors - Input data issues (e.g., invalid email format, missing required fields)
  2. Domain Errors - Business logic violations (e.g., user not found, notification template invalid)
  3. Infrastructure Errors - System-level failures (e.g., database connection timeout, external service unavailability)
public class NotificationError extends RuntimeException {
    private final ErrorType errorType;
    private final String errorCode;
    
    public enum ErrorType {
        VALIDATION, 
        DOMAIN, 
        INFRASTRUCTURE
    }
    
    public NotificationError(ErrorType errorType, String errorCode, String message) {
        super(message);
        this.errorType = errorType;
        this.errorCode = errorCode;
    }
    
    // Getters
}

Error Propagation Strategy

Errors are propagated up the call stack in a layered fashion:

  • Service layer catches and wraps infrastructure errors
  • Controller layer translates domain errors to appropriate HTTP status codes
  • All errors include traceable context identifiers for debugging

Retry Strategy with Exponential Backoff and Jitter

For transient infrastructure errors, the system implements a retry mechanism:

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;
    
    public <T> T executeWithRetry(Supplier<T> operation) {
        int attempts = 0;
        while (true) {
            try {
                return operation.get();
            } catch (InfrastructureException e) {
                if (++attempts >= MAX_RETRIES) {
                    throw new RetryExhaustedException("Operation failed after " + MAX_RETRIES + " attempts", e);
                }
                
                long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempts - 1));
                delay += (long) (delay * JITTER_FACTOR * (Math.random() - 0.5) * 2);
                
                try {
                    Thread.sleep(delay);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new RetryExhaustedException("Retry interrupted", ie);
                }
            }
        }
    }
}

Circuit Breaker Design

External service calls (e.g., email provider, SMS gateway) use circuit breaker pattern:

public class CircuitBreaker {
    private final int failureThreshold = 5;
    private final long timeoutMs = 60000;
    private final int resetTimeoutMs = 30000;
    
    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private long lastFailureTime = 0;
    
    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }
    
    public <T> T execute(Supplier<T> operation) {
        switch (state) {
            case CLOSED:
                return executeWhenClosed(operation);
            case OPEN:
                if (System.currentTimeMillis() - lastFailureTime > resetTimeoutMs) {
                    state = CircuitState.HALF_OPEN;
                    return executeWhenHalfOpen(operation);
                }
                throw new CircuitBreakerOpenException("Circuit is open");
            case HALF_OPEN:
                return executeWhenHalfOpen(operation);
        }
        throw new IllegalStateException("Invalid circuit state");
    }
    
    private <T> T executeWhenClosed(Supplier<T> operation) {
        try {
            T result = operation.get();
            failureCount = 0;
            return result;
        } catch (Exception e) {
            failureCount++;
            lastFailureTime = System.currentTimeMillis();
            if (failureCount >= failureThreshold) {
                state = CircuitState.OPEN;
            }
            throw e;
        }
    }
    
    private <T> T executeWhenHalfOpen(Supplier<T> operation) {
        try {
            T result = operation.get();
            state = CircuitState.CLOSED;
            failureCount = 0;
            return result;
        } catch (Exception e) {
            state = CircuitState.OPEN;
            lastFailureTime = System.currentTimeMillis();
            throw e;
        }
    }
}

Fallback Behavior Design

For critical external dependencies, fallbacks are implemented:

public class NotificationService {
    private final EmailService emailService;
    private final SMSService smsService;
    
    public void sendNotification(NotificationRequest request) {
        if (request.getDeliveryMethod() == DeliveryMethod.EMAIL) {
            try {
                emailService.sendEmail(request);
            } catch (InfrastructureException e) {
                // Fallback to SMS if email fails
                fallbackToSMS(request);
            }
        } else {
            // SMS delivery with fallback to email
            try {
                smsService.sendSMS(request);
            } catch (InfrastructureException e) {
                fallbackToEmail(request);
            }
        }
    }
    
    private void fallbackToSMS(NotificationRequest request) {
        if (request.getFallbackMethod() == DeliveryMethod.SMS) {
            try {
                smsService.sendSMS(request);
            } catch (InfrastructureException e) {
                // Log and escalate - no more fallbacks
                log.error("Failed to send notification via SMS", e);
                throw new NotificationDeliveryFailureException(
                    "Failed to deliver notification via fallback methods", e);
            }
        }
    }
    
    private void fallbackToEmail(NotificationRequest request) {
        if (request.getFallbackMethod() == DeliveryMethod.EMAIL) {
            try {
                emailService.sendEmail(request);
            } catch (InfrastructureException e) {
                log.error("Failed to send notification via email", e);
                throw new NotificationDeliveryFailureException(
                    "Failed to deliver notification via fallback methods", e);
            }
        }
    }
}

Error Logging Format and Alerting Thresholds

Structured logging includes trace context and error classification:

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);
    
    public void logError(String operation, String errorCode, String message, 
                        Map<String, Object> context) {
        Map<String, Object> logContext = new HashMap<>(context);
        logContext.put("operation", operation);
        logContext.put("errorCode", errorCode);
        logContext.put("timestamp", Instant.now());
        
        if (isAlertable(errorCode)) {
            logger.error("Notification system error - {}", message, logContext);
            sendAlert(errorCode, message, logContext);
        } else {
            logger.warn("Notification system warning - {}", message, logContext);
        }
    }
    
    private boolean isAlertable(String errorCode) {
        return errorCode.startsWith("INFRA_") || 
               errorCode.startsWith("DOMAIN_") ||
               errorCode.equals("NOTIFICATION_DELIVERY_FAILURE");
    }
    
    private void sendAlert(String errorCode, String message, Map<String, Object> context) {
        // Integration with monitoring system (e.g., PagerDuty, Slack)
        AlertService.sendAlert(errorCode, message, context);
    }
}

Alerting Thresholds

MetricWarning ThresholdCritical ThresholdEscalation
Notification delivery failures10 per minute50 per minuteOn-call team
External service timeouts5 per minute20 per minuteOn-call team
Database connection errors3 per minute10 per minuteOn-call team

Error Code Definitions

Error CodeTypeDescription
VALIDATION_INVALID_EMAILValidationEmail address format invalid
VALIDATION_MISSING_TEMPLATEValidationNotification template not provided
DOMAIN_USER_NOT_FOUNDDomainUser ID does not exist
INFRA_EMAIL_SERVICE_TIMEOUTInfrastructureEmail service unreachable
NOTIFICATION_DELIVERY_FAILUREInfrastructureFailed to deliver notification via all methods

Error Response Format

All API errors return standardized JSON:

{
  "error": {
    "code": "VALIDATION_INVALID_EMAIL",
    "message": "Email address format is invalid",
    "details": {
      "field": "email",
      "value": "invalid-email"
    },
    "timestamp": "2023-12-01T10:30:00Z",
    "traceId": "abc123def456"
  }
}

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using a combination of:

  • Bean Validation annotations (javax.validation) for structural validation
  • Custom validators for business logic constraints
  • HTML sanitization using OWASP Java HTML Sanitizer for content fields
public class NotificationRequest {
    @NotBlank
    @Size(max = 100)
    private String title;
    
    @NotBlank
    @Size(max = 1000)
    private String content;
    
    @ValidNotificationType
    private NotificationType type;
    
    // getters/setters
}

SQL Injection Prevention

All database queries use parameterized statements exclusively:

  • JPA/Hibernate with named parameters for all dynamic queries
  • No string concatenation in SQL generation
  • Whitelisted query parameters with strict type validation
@Query("SELECT n FROM Notification n WHERE n.userId = :userId AND n.status = :status")
List<Notification> findByUserIdAndStatus(@Param("userId") UUID userId, @Param("status") NotificationStatus status);

Authentication Middleware Design

Authentication is handled via JWT Bearer tokens with:

  • Spring Security filter chain intercepting all requests
  • Token validation including signature verification and expiration check
  • Role-based access control at the endpoint level
@Configuration
@EnableWebSecurity
public class SecurityConfig {
    @Bean
    public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
        http
            .authorizeHttpRequests(authz -> authz
                .requestMatchers("/api/notifications/**").authenticated()
                .anyRequest().permitAll()
            )
            .oauth2ResourceServer(oauth2 -> oauth2
                .jwt(jwt -> jwt.decoder(jwtDecoder))
            );
        return http.build();
    }
}

Authorization Enforcement

RBAC implementation with:

  • User roles (USER, ADMIN, SYSTEM)
  • Permission checking before operation execution
  • Role hierarchy (ADMIN includes USER permissions)
@PreAuthorize("hasRole('ADMIN') or @notificationPermissionEvaluator.hasPermission(authentication, #userId)")
public void deleteNotification(UUID userId, UUID notificationId) {
    // implementation
}

Sensitive Data Encryption

At rest:

  • AES-256 encryption using Jasypt for sensitive fields
  • Key rotation strategy with 90-day lifecycle

In transit:

  • TLS 1.3 enforced for all external communications
  • HSTS header configuration to prevent protocol downgrade attacks

PII Handling and Masking Strategy

PII fields are:

  • Masked in logs using pattern matching (e.g., email: "user***@domain.com")
  • Redacted in API responses for non-privileged users
  • Completely excluded from audit logs for sensitive fields
public class NotificationResponse {
    @JsonSerialize(using = MaskedEmailSerializer.class)
    private String recipientEmail;
    
    // other fields
}

Secrets Management

AWS Secrets Manager integration:

  • Database credentials rotated every 90 days
  • Encryption keys managed via KMS
  • Service tokens stored in secure vault with automatic rotation

Audit Logging Design

Audit logs capture:

  • Required fields: timestamp, user_id, operation_type, entity_id, ip_address
  • Security events only: login attempts, permission changes, system-level actions
  • PII masking in log entries using regex patterns
@EventListener
public void handleSecurityEvent(SecurityAuditEvent event) {
    String maskedIp = maskIpAddress(event.getIpAddress());
    String logMessage = String.format(
        "SECURITY_EVENT: %s - User: %s - IP: %s", 
        event.getOperation(), 
        event.getUserId(), 
        maskedIp
    );
    logger.info(logMessage);
}

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

  • Service Layer: 85% code coverage with focus on business logic, error handling, and boundary conditions
  • Repository Layer: 90% coverage including edge cases for read/write operations and query validation
  • Domain Objects: 100% coverage for value objects, enums, and immutable data structures
  • Event Handlers: 80% coverage for async processing logic and error recovery paths
  • API Controllers: 75% coverage with emphasis on request validation, authentication, and response formatting

Integration Testing Strategy

  • Database Integration: Test with embedded H2/PostgreSQL in-memory databases for transactional consistency and query performance
  • External Service Mocks: Use WireMock or Mockito to simulate third-party notification providers (Twilio, SendGrid, FCM)
  • Message Broker: Test with embedded Kafka or RabbitMQ containers to validate event-driven workflows
  • End-to-End Flows: Validate complete notification lifecycle from creation to delivery with test containers

Contract Testing

  • API Contracts: Use Pact for consumer-driven contract testing between notification service and client applications
  • Event Schemas: Validate event payloads using JSON Schema or Avro schema registry with version compatibility checks
  • External Provider Contracts: Maintain contract tests for Twilio, Firebase, and email service providers to ensure API compatibility

Performance Testing Tool

  • Load Testing: Use k6 or Gatling to simulate concurrent notification requests (1000+ concurrent users)
  • Key Scenarios:
    • High-volume notification burst (10k notifications/sec)
    • Concurrent user subscription management
    • Batch notification processing under load
  • Metrics Tracked:
    • P95 latency for notification delivery (target <200ms)
    • Throughput capacity (10k notifications/sec target)
    • Memory usage under sustained load

Test Data Management Strategy

  • Fixtures: Maintain minimal, deterministic test data sets for common scenarios (user profiles, notification templates)
  • Factories: Implement factory patterns for generating test data with realistic distribution (user preferences, device tokens)
  • Seeding: Use database migration scripts to seed test environments with consistent baseline data
  • Isolation: Each test runs in isolated database transaction or clean test database instance to prevent cross-contamination

13. Observability Design

Structured Logging Schema

All logs follow a structured JSON format with consistent fields to enable efficient querying and analysis:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO",
  "service": "notification-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user_12345",
  "notification_id": "notif_98765",
  "event_type": "notification_sent",
  "message": "Notification sent successfully to user",
  "duration_ms": 150,
  "error_code": null,
  "error_message": null,
  "stack_trace": null
}

PII masking rules:

  • User IDs are masked in logs when not required for debugging
  • Email addresses are truncated (e.g., "user@example.com" → "u***@example.com")
  • Phone numbers are masked (e.g., "+1234567890" → "+1234****90")

Distributed Tracing Instrumentation

Using OpenTelemetry for end-to-end traceability:

Custom Spans:

  • notification.send - Main span for notification processing
  • notification.delivery - Sub-span for delivery method selection and execution
  • notification.template.render - Template rendering operation
  • notification.queue.publish - Message queue publishing

Attributes Captured:

Span currentSpan = Span.current();
currentSpan.setAttribute("notification.type", "email");
currentSpan.setAttribute("notification.template_id", "template_123");
currentSpan.setAttribute("delivery.method", "smtp");
currentSpan.setAttribute("user.id", "user_12345");
currentSpan.setAttribute("notification.status", "delivered");

Metrics Design

RED Metrics (Rate, Errors, Duration):

Metric NameTypeLabelsDescription
notification.sent.countCountertype, status, methodTotal notifications sent
notification.send.durationHistogramtype, status, methodTime to send notification
notification.errors.countCountererror_type, notification_typeFailed notification attempts
notification.queue.sizeGaugequeue_nameCurrent size of notification queues
notification.template.render.durationHistogramtemplate_typeTemplate rendering time

Example metric collection:

@Micrometer
public class NotificationMetrics {
    private final Counter sentCounter = Counter.builder("notification.sent.count")
        .description("Total notifications sent")
        .register(registry);
        
    private final Histogram sendDuration = Histogram.builder("notification.send.duration")
        .description("Time to send notification")
        .register(registry);
        
    public void recordSend(String type, String status, long durationMs) {
        sentCounter.increment(Tag.of("type", type), Tag.of("status", status));
        sendDuration.record(durationMs, Tag.of("type", type), Tag.of("status", status));
    }
}

Health Check Endpoint Design

Endpoints:

  • /health - Overall service health (critical dependencies)
  • /ready - Service readiness (database, cache, queue connections)
  • /live - Process liveness (memory, CPU usage)
  • /startup - Service startup completion status

Response Format:

{
  "status": "healthy",
  "timestamp": "2023-10-27T14:30:45.123Z",
  "checks": {
    "database": {"status": "healthy", "details": "Connected to primary DB"},
    "redis": {"status": "healthy", "details": "Cache connected"},
    "message_queue": {"status": "healthy", "details": "Kafka connected"}
  }
}

Alerting Thresholds

Critical Alerts (on-call escalation):

  • notification.errors.count > 100 per minute (warn) or 500 per minute (critical)
  • notification.send.duration p99 > 5000ms (warn) or 10000ms (critical)
  • notification.queue.size > 10000 messages (warn) or 50000 messages (critical)

Warning Alerts:

  • notification.send.duration p95 > 2000ms
  • Database connection pool utilization > 80%
  • Redis cache hit ratio < 70%

Escalation Path:

  1. Slack channel alert (5 min threshold)
  2. On-call engineer notification (30 sec threshold)
  3. Incident management system escalation (1 min threshold for critical errors)

Service Level Indicators:

  • 99.9% availability (max 43.2 minutes downtime/month)
  • 95% of notifications delivered within 2 seconds (p95 latency)
  • 99.9% success rate for notification delivery attempts

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

  • Notification delivery priority levels: Should we implement a priority queue system with different delivery guarantees (e.g., real-time vs. batch) or treat all notifications as equal? This affects system complexity and resource allocation.
  • Cross-platform support strategy: Should we build a unified notification system that handles web, mobile push, email, SMS, and in-app notifications, or build separate systems for each channel with shared core logic?
  • User preference management: How granular should user notification preferences be? Should users control channels per notification type, or just disable all notifications?

Known trade-offs and justifications

  • Consistency vs. availability: We're choosing eventual consistency for notification delivery to ensure high availability during system failures, accepting that some notifications might be delivered out of order or with slight delays.
  • In-memory vs. persistent storage: Using in-memory queues for high-throughput scenarios trades off reliability (data loss on system restart) for performance, with a fallback to persistent storage for critical notifications.
  • Monolithic vs. microservice design: A monolithic notification service provides simpler deployment and better performance for low-latency use cases, but a microservice approach offers better scalability and independent deployment capabilities.

Risks and mitigation strategies

  • Notification spamming: Risk of abuse or misconfiguration causing excessive notifications. Mitigation includes rate limiting per user, content filtering, and admin dashboard for monitoring.
  • Delivery failures: Risk of notifications not reaching users due to channel unavailability or user opt-outs. Mitigation includes fallback channels, retry mechanisms with exponential backoff, and delivery status tracking.
  • Data loss: Risk of losing notifications during system outages. Mitigation includes persistent storage, message queues with durability guarantees, and regular backups.
  • Performance bottlenecks: Risk of notification processing becoming a bottleneck under high load. Mitigation includes horizontal scaling, async processing, and caching of frequently accessed user data.

Technical debt acknowledged

  • Limited channel extensibility: Current design assumes a fixed set of notification channels (email, SMS, push), making it harder to add new channels without refactoring core logic.
  • No built-in content personalization: The system currently sends generic notifications without dynamic content generation, requiring separate personalization services.
  • Basic error handling: Error handling is minimal and focused on core functionality, with more sophisticated error categorization and recovery strategies needed for production use.
  • No multi-tenant support: The system assumes single-tenant usage, requiring architectural changes to support multiple clients or organizations with isolated notification systems.

Was this article helpful?

Topics in this article

More from Backend LLD

  • Backend LLD

    CQRS Framework Design and Implementation

    Comprehensive low-level design for a Command Query Responsibility Segregation (CQRS) framework with event sourcing, read/write model separation, and distributed consistency patterns.

    • CategoryBackend LLD
    • UpdatedOctober 4, 2026
    • Reading time45 min read

Stay in the Loop

Get curated DSA patterns, coding problems, frontend and backend LLD/HLD, and AI/ML System Design insights delivered straight to your inbox. No spam, just focused interview and engineering content.