This document presents a production-ready low-level design for a reminder service that handles millions of scheduled notifications with high availability and fault tolerance across distributed systems.
1. Problem Statement & Requirements
Clarifying questions
-
What is the scale expectation for this reminder service?
- How many active reminders can we expect to handle concurrently?
- What is the peak rate of reminder creation and delivery?
-
What consistency requirements exist for reminders?
- Should all reminders be delivered exactly once, or is at-least-once acceptable?
- Are there any strict SLA requirements for delivery timing?
-
What are the key entities and their relationships in this system?
- Are reminders tied to users, or can they be shared across users?
- Do reminders have dependencies on other entities like events, tasks, or calendar items?
-
What is the read/write ratio for reminders?
- Is this primarily a write-heavy system (e.g., frequent creation) or read-heavy (e.g., frequent querying)?
- What is the expected ratio of reminder creation vs. retrieval?
-
How sensitive is the system to latency?
- Are reminders time-sensitive (e.g., 5 minutes before an event), or can they be delivered within a few hours?
- Is there a maximum acceptable delivery delay?
-
What authentication and authorization model is in place?
- Will reminders be tied to user sessions, OAuth2 tokens, or API keys?
- Can users access reminders created by others?
-
Are there any external integrations or dependencies?
- Will reminders trigger notifications through email, SMS, push, or in-app?
- Is there integration with calendar systems (e.g., Google Calendar, Outlook)?
Functional requirements
- Create Reminder: Users can create a reminder with a title, description, trigger time, and optional recurrence pattern.
- List Reminders: Users can retrieve all their reminders or filter by date/time range.
- Update Reminder: Users can modify existing reminders (e.g., title, time, recurrence).
- Delete Reminder: Users can delete a reminder or cancel all future occurrences.
- Trigger Reminder: System automatically delivers a reminder at its trigger time (e.g., via push notification, email).
- Recurring Reminders: Support for daily, weekly, monthly, or custom recurring patterns.
- Reminder Scheduling: Reminders can be scheduled for future times (e.g., 5 minutes from now).
Non-functional requirements
- Latency Targets:
- p50: < 100ms for read operations (list, get)
- p95: < 500ms for read operations
- p99: < 1s for read operations
- p50/p95/p99 for write operations: < 200ms / 500ms / 1s
- Throughput:
- 10,000 reminder creations per second (peak)
- 50,000 reminder reads per second (peak)
- Availability:
- 99.9% uptime (28.8 minutes of downtime/month max)
- Delivery Reliability:
- At-least-once delivery guarantee for reminder triggers
Constraints and assumptions
-
Assumptions:
- Users are authenticated via JWT tokens or OAuth2.
- Reminder delivery is handled by a separate notification service (e.g., FCM, Twilio).
- Reminders are stored in a single region for simplicity (multi-region support can be added later).
- The system will not support reminders more than 10 years in the future.
-
Constraints:
- Reminder creation and retrieval must be fast enough to support real-time user interaction.
- System must handle up to 10 million active reminders across all users.
- Reminder triggers must be accurate within ±5 minutes (for non-critical reminders) or ±1 minute (for time-sensitive ones).
Out of scope
- User management: Creation, deletion, or modification of user accounts.
- Calendar integration: Sync with external calendar systems (e.g., Google Calendar, Outlook).
- Multi-language support: Reminder content is in a single language (e.g., English).
- Webhooks or third-party integrations: Reminder triggers only support internal notification methods.
- Advanced scheduling logic: Complex scheduling like "first Monday of the month" is not supported.
- User preferences and settings: Notification method, timezone, or reminder tone customization is not part of this design.
- Analytics dashboard: Reporting or analytics on reminder usage is out of scope.
2. API Design
API Style Selection
gRPC with Protocol Buffers is selected for the Reminder service due to:
- Strong typing and automatic code generation
- Efficient binary serialization with lower payload sizes than JSON
- Built-in support for streaming and bidirectional communication
- Native support for versioning through proto file management
- Better performance characteristics for high-throughput scenarios
Endpoint Definitions
| Method | Path | Description | Auth | Rate Limit |
|---|---|---|---|---|
| CreateReminder | POST /v1/reminders | Creates a new reminder | JWT | 100/hour |
| GetReminder | GET /v1/reminders/{id} | Retrieves a reminder by ID | JWT | 1000/hour |
| UpdateReminder | PUT /v1/reminders/{id} | Updates an existing reminder | JWT | 50/hour |
| DeleteReminder | DELETE /v1/reminders/{id} | Deletes a reminder by ID | JWT | 50/hour |
| ListReminders | GET /v1/reminders | Lists user's reminders with pagination | JWT | 100/hour |
| TriggerReminder | POST /v1/reminders/{id}/trigger | Forces immediate trigger of a reminder | JWT | 20/hour |
| GetReminderHistory | GET /v1/reminders/{id}/history | Gets reminder trigger history | JWT | 50/hour |
Request and Response Schemas
CreateReminderRequest
message CreateReminderRequest {
string title = 1;
string description = 2;
int64 trigger_time = 3; // Unix timestamp
string timezone = 4;
repeated string recipients = 5;
ReminderType type = 6;
map<string, string> metadata = 7;
}
enum ReminderType {
EMAIL = 0;
SMS = 1;
PUSH_NOTIFICATION = 2;
}CreateReminderResponse
message CreateReminderResponse {
string id = 1;
string title = 2;
int64 created_at = 3;
int64 trigger_time = 4;
}GetReminderResponse
message GetReminderResponse {
string id = 1;
string title = 2;
string description = 3;
int64 trigger_time = 4;
string timezone = 5;
repeated string recipients = 6;
ReminderType type = 7;
int64 created_at = 8;
int64 updated_at = 9;
bool is_active = 10;
}HTTP Status Codes and Error Response Format
| Status Code | Description |
|---|---|
| 200 | Success |
| 201 | Created |
| 400 | Bad Request |
| 401 | Unauthorized |
| 403 | Forbidden |
| 404 | Not Found |
| 429 | Rate Limited |
| 500 | Internal Server Error |
Standard error response:
{
"error": {
"code": "VALIDATION_ERROR",
"message": "Trigger time must be in future",
"details": [
{
"field": "trigger_time",
"message": "Must be in future"
}
]
}
}API Versioning Strategy
Versioned using path prefix: /v1/reminders with support for future versions like /v2/reminders.
Versioning strategy allows for breaking changes to be introduced without affecting existing clients.
Pagination Design
Cursor-based pagination is used because:
- More consistent and predictable than offset-based pagination
- Handles data changes during pagination better
- Provides stable results for users
message ListRemindersRequest {
string cursor = 1;
int32 page_size = 2; // Default: 20, Max: 100
}Authentication and Authorization
All endpoints require JWT Bearer token with scopes:
reminder:read- for GET operationsreminder:write- for CREATE/UPDATE/DELETE operations
Rate Limiting Strategy
Rate limiting implemented per user with:
- Token bucket algorithm for burst handling
- Per-user limits to prevent abuse
- Sliding window for better fairness
Idempotency Design
For mutation endpoints, idempotency key is supported via header:
X-Idempotency-Key: <uuid>If same key is used within 24 hours, operation is idempotent and returns same result.
3. Data Model Design
Entity-Relationship Diagram
erDiagram
USER ||--o{ REMINDER : "creates"
REMINDER ||--o{ REMINDER_EVENT : "has"
REMINDER_EVENT ||--o{ REMINDER_NOTIFICATION : "triggers"
REMINDER ||--o{ REMINDER_LOG : "logs"
USER {
string id PK
string email
string name
timestamp created_at
timestamp updated_at
}
REMINDER {
string id PK
string user_id FK
string title
string description
timestamp trigger_time
string status
string recurrence_pattern
timestamp created_at
timestamp updated_at
timestamp deleted_at
}
REMINDER_EVENT {
string id PK
string reminder_id FK
timestamp event_time
string event_type
string status
timestamp created_at
}
REMINDER_NOTIFICATION {
string id PK
string reminder_event_id FK
string channel
string status
timestamp sent_at
timestamp created_at
}
REMINDER_LOG {
string id PK
string reminder_id FK
string log_level
string message
timestamp created_at
}Table Schema Definitions
Users Table
CREATE TABLE users (
id VARCHAR(36) PRIMARY KEY,
email VARCHAR(255) UNIQUE NOT NULL,
name VARCHAR(255) NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
INDEX idx_email (email),
INDEX idx_created_at (created_at)
);Reminders Table
CREATE TABLE reminders (
id VARCHAR(36) PRIMARY KEY,
user_id VARCHAR(36) NOT NULL,
title VARCHAR(255) NOT NULL,
description TEXT,
trigger_time TIMESTAMP NOT NULL,
status ENUM('ACTIVE', 'COMPLETED', 'CANCELLED') DEFAULT 'ACTIVE',
recurrence_pattern VARCHAR(255),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE,
INDEX idx_user_trigger_time (user_id, trigger_time),
INDEX idx_trigger_time (trigger_time),
INDEX idx_status (status),
INDEX idx_deleted_at (deleted_at),
INDEX idx_created_at (created_at)
);Reminder Events Table
CREATE TABLE reminder_events (
id VARCHAR(36) PRIMARY KEY,
reminder_id VARCHAR(36) NOT NULL,
event_time TIMESTAMP NOT NULL,
event_type ENUM('TRIGGER', 'SCHEDULED') NOT NULL,
status ENUM('PENDING', 'PROCESSING', 'COMPLETED', 'FAILED') DEFAULT 'PENDING',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (reminder_id) REFERENCES reminders(id) ON DELETE CASCADE,
INDEX idx_reminder_event_time (reminder_id, event_time),
INDEX idx_event_time (event_time),
INDEX idx_status (status)
);Reminder Notifications Table
CREATE TABLE reminder_notifications (
id VARCHAR(36) PRIMARY KEY,
reminder_event_id VARCHAR(36) NOT NULL,
channel ENUM('EMAIL', 'SMS', 'PUSH') NOT NULL,
status ENUM('PENDING', 'SENT', 'FAILED') DEFAULT 'PENDING',
sent_at TIMESTAMP NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (reminder_event_id) REFERENCES reminder_events(id) ON DELETE CASCADE,
INDEX idx_reminder_event_channel (reminder_event_id, channel),
INDEX idx_status (status),
INDEX idx_created_at (created_at)
);Reminder Logs Table
CREATE TABLE reminder_logs (
id VARCHAR(36) PRIMARY KEY,
reminder_id VARCHAR(36) NOT NULL,
log_level ENUM('INFO', 'WARN', 'ERROR') NOT NULL,
message TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (reminder_id) REFERENCES reminders(id) ON DELETE CASCADE,
INDEX idx_reminder_created_at (reminder_id, created_at),
INDEX idx_created_at (created_at)
);Primary Key Strategy
UUID v4 is used for all primary keys to:
- Provide global uniqueness across distributed systems
- Prevent enumeration attacks on IDs
- Support horizontal sharding without ID conflicts
- Avoid sequential ID prediction vulnerabilities
Index Design
Reminders Table Indices:
idx_user_trigger_time(user_id, trigger_time) - For efficient user-specific reminder queries and time-range filteringidx_trigger_time(trigger_time) - For efficient query of upcoming reminders across usersidx_status(status) - For filtering active vs completed remindersidx_deleted_at(deleted_at) - For soft-deleted record filtering in queriesidx_created_at(created_at) - For audit and time-based queries
Reminder Events Table Indices:
idx_reminder_event_time(reminder_id, event_time) - For efficient event retrieval by reminder with time orderingidx_event_time(event_time) - For efficient query of events by time windowidx_status(status) - For filtering pending vs processed events
Reminder Notifications Table Indices:
idx_reminder_event_channel(reminder_event_id, channel) - For efficient lookup of notifications by event and channelidx_status(status) - For filtering pending vs sent notifications
Data Partitioning / Sharding Strategy
Shard by user_id hash modulo 16 for horizontal scaling:
- Each shard contains users with similar user_id hash values
- Reminder queries can be routed to specific shards based on user_id
- Cross-shard queries handled via coordinator service or sharding key lookup tables
Soft Delete vs. Hard Delete
Soft delete is implemented using deleted_at timestamp field:
- Prevents data loss during accidental deletions
- Enables audit trail and recovery capability
- Allows for logical restoration of reminders (with business logic validation)
- Maintains referential integrity in related tables
Audit Trail Design
All data modification operations log to reminder_logs table:
- Fields: reminder_id, log_level (INFO/WARN/ERROR), message, created_at
- Messages capture: creation, update, deletion, status changes, error conditions
- Provides complete audit trail for compliance and debugging
Data Archival Strategy
- Reminders with
status = 'COMPLETED'andupdated_at < 30 days agoare archived to cold storage - Archive retention: 1 year for compliance requirements
- Archival process runs nightly with batch processing to minimize impact on live systems
- Archived data is accessible via separate read-only service for compliance queries
## 4. Class Design & Service Layer
```mermaid
classDiagram
class ReminderService {
+createReminder(ReminderRequest) Reminder
+updateReminder(String, ReminderUpdateRequest) Reminder
+deleteReminder(String) void
+getReminder(String) Reminder
+listReminders(UserId, Filter) List~Reminder~
+triggerReminder(String) void
}
class ReminderRepository {
+save(Reminder) Reminder
+findById(String) Optional~Reminder~
+findByUserIdAndStatus(UserId, Status) List~Reminder~
+updateStatus(String, Status) void
+delete(String) void
}
class Reminder {
-id: String
-userId: String
-title: String
-description: String
-scheduledTime: Instant
-status: Status
-createdAt: Instant
-updatedAt: Instant
+isOverdue() boolean
+isActive() boolean
}
class ReminderEventPublisher {
+publish(ReminderEvent) void
}
class ReminderProcessor {
+processPendingReminders() void
}
class ReminderValidator {
+validateCreateRequest(ReminderRequest) ValidationResult
+validateUpdateRequest(ReminderUpdateRequest) ValidationResult
}
class ReminderScheduler {
+scheduleReminder(String, Instant) void
+cancelScheduledReminder(String) void
}
class ReminderEvent {
-id: String
-reminderId: String
-eventType: EventType
-timestamp: Instant
}
class ReminderStatus {
ACTIVE
PENDING
COMPLETED
CANCELLED
OVERDUE
}
ReminderService --> ReminderRepository : uses
ReminderService --> ReminderValidator : validates
ReminderService --> ReminderEventPublisher : publishes
ReminderService --> ReminderProcessor : schedules
ReminderService --> ReminderScheduler : schedules
Reminder --> ReminderEventPublisher : emits events
Reminder --> ReminderProcessor : triggers processing
class ReminderRequest {
title: String
description: String
scheduledTime: Instant
userId: String
}
class ReminderUpdateRequest {
title: String
description: String
scheduledTime: Instant
}
class ReminderFilter {
status: Status
userId: String
from: Instant
to: Instant
}
class ValidationResult {
valid: boolean
errors: List~String~
}stateDiagram-v2
[*] --> PENDING
PENDING --> ACTIVE : ScheduledTime reached
ACTIVE --> COMPLETED : Triggered
ACTIVE --> OVERDUE : Time passed without trigger
ACTIVE --> CANCELLED : User cancels
PENDING --> CANCELLED : User cancels
OVERDUE --> CANCELLED : User cancelsService Layer Structure
The Reminder service is designed with a clean separation of concerns:
- ReminderService - Main entry point for all reminder operations
- ReminderRepository - Data access layer abstraction
- ReminderValidator - Input validation and business rule enforcement
- ReminderEventPublisher - Event publishing for async operations
- ReminderProcessor - Background processing of reminders
- ReminderScheduler - Scheduling and timing management
Business Logic Patterns
Strategy Pattern for Reminder Processing
public interface ReminderProcessorStrategy {
void process(Reminder reminder);
}
@Component
public class EmailReminderProcessor implements ReminderProcessorStrategy {
private final EmailService emailService;
@Override
public void process(Reminder reminder) {
emailService.sendReminder(reminder.getUserId(),
reminder.getTitle(), reminder.getDescription());
}
}
@Component
public class PushNotificationReminderProcessor implements ReminderProcessorStrategy {
private final NotificationService notificationService;
@Override
public void process(Reminder reminder) {
notificationService.sendPushNotification(reminder.getUserId(),
reminder.getTitle(), reminder.getDescription());
}
}
@Component
public class ReminderProcessor {
private final Map<String, ReminderProcessorStrategy> strategies;
public ReminderProcessor(Map<String, ReminderProcessorStrategy> strategies) {
this.strategies = strategies;
}
public void process(Reminder reminder) {
String type = reminder.getType(); // Could be "email", "push"
ReminderProcessorStrategy strategy = strategies.get(type);
if (strategy != null) {
strategy.process(reminder);
}
}
}Critical-Path Method Implementation
@Transactional
public Reminder createReminder(ReminderRequest request) {
// Validate input
ValidationResult validation = validator.validateCreateRequest(request);
if (!validation.isValid()) {
throw new ValidationException("Invalid reminder request", validation.getErrors());
}
// Create domain object
Reminder reminder = Reminder.builder()
.id(UUID.randomUUID().toString())
.userId(request.getUserId())
.title(request.getTitle())
.description(request.getDescription())
.scheduledTime(request.getScheduledTime())
.status(Status.PENDING)
.createdAt(Instant.now())
.updatedAt(Instant.now())
.build();
// Save to database
Reminder savedReminder = repository.save(reminder);
// Schedule for processing if in future
if (reminder.getScheduledTime().isAfter(Instant.now())) {
scheduler.scheduleReminder(reminder.getId(), reminder.getScheduledTime());
} else {
// Immediate processing if already due
processReminder(reminder.getId());
}
// Publish event for async processing
ReminderEvent event = ReminderEvent.builder()
.id(UUID.randomUUID().toString())
.reminderId(reminder.getId())
.eventType(EventType.CREATED)
.timestamp(Instant.now())
.build();
eventPublisher.publish(event);
return savedReminder;
}
private void processReminder(String reminderId) {
Reminder reminder = repository.findById(reminderId)
.orElseThrow(() -> new ResourceNotFoundException("Reminder not found"));
// Update status to ACTIVE
repository.updateStatus(reminderId, Status.ACTIVE);
// Process based on type (email, push, etc.)
processor.process(reminder);
// Update status to COMPLETED
repository.updateStatus(reminderId, Status.COMPLETED);
}Transaction Boundary Design
- Create/Update/Delete operations: Single database transaction
- Cross-service operations: Saga pattern for distributed transactions (e.g., when reminder creation triggers external notifications)
- Batch operations: Transaction per batch unit, not entire operation set
Service Interface Contracts
public interface ReminderService {
Reminder createReminder(ReminderRequest request);
Reminder updateReminder(String id, ReminderUpdateRequest request);
void deleteReminder(String id);
Reminder getReminder(String id);
List<Reminder> listReminders(UserId userId, ReminderFilter filter);
void triggerReminder(String id);
}Saga Pattern for Distributed Transactions
For reminder creation that involves external systems (e.g., notification service):
public class ReminderCreationSaga {
private final String sagaId = UUID.randomUUID().toString();
public void execute(ReminderRequest request) {
try {
// Step 1: Create reminder in DB
Reminder reminder = createReminderInDB(request);
// Step 2: Send notification (if external system)
sendNotification(reminder);
// Step 3: Update status in DB
updateReminderStatus(reminder.getId(), Status.ACTIVE);
// Step 4: Schedule processing
scheduleProcessing(reminder.getId());
} catch (Exception e) {
// Compensation logic - rollback steps
handleCompensation(e);
}
}
}Domain Object Design
@Builder
@Value
public class Reminder {
String id;
String userId;
String title;
String description;
Instant scheduledTime;
Status status;
Instant createdAt;
Instant updatedAt;
public boolean isOverdue() {
return status == Status.ACTIVE &&
scheduledTime.isBefore(Instant.now());
}
public boolean isActive() {
return status == Status.ACTIVE;
}
public enum Status {
PENDING, ACTIVE, COMPLETED, CANCELLED, OVERDUE
}
public enum EventType {
CREATED, UPDATED, DELETED, TRIGGERED, CANCELLED
}
}Thread Safety Considerations
- State transitions: Use atomic updates and optimistic locking
- Concurrent creation: UUID primary keys prevent conflicts
- Batch operations: Isolated transaction boundaries per batch item
- Event publishing: Thread-safe event bus with proper ordering guarantees
Idempotency Implementation
public Reminder createReminder(ReminderRequest request, String idempotencyKey) {
// Check if this operation was already processed
String cacheKey = "reminder_idempotency:" + idempotencyKey;
String existingId = cache.get(cacheKey);
if (existingId != null) {
return repository.findById(existingId).orElseThrow();
}
// Proceed with creation...
Reminder reminder = repository.save(reminder);
// Cache the result
cache.put(cacheKey, reminder.getId(), Duration.ofHours(1));
return reminder;
}5. Concurrency & Thread Safety
Concurrency Model
The Reminder service will use a thread pool-based concurrency model with Java's ExecutorService and CompletableFuture for asynchronous operations. Each reminder processing task will be executed in a separate thread from a bounded thread pool to prevent resource exhaustion. For high-concurrency scenarios, the service will leverage reactive programming patterns using Project Reactor or RxJava to handle thousands of concurrent reminder triggers efficiently.
Race Condition Prevention Strategy
Race conditions in the reminder service will be mitigated through:
- Database-level atomic operations for state transitions and updates
- Optimistic locking with version fields in reminder entities to prevent concurrent modifications
- Database transactions for multi-step operations that must remain consistent
- Idempotency checks to prevent duplicate processing of the same reminder event
Optimistic vs. Pessimistic Locking Selection
Optimistic locking is selected for the reminder service because:
- Reminder creation and updates are infrequent relative to read operations
- The system expects low contention for reminder entities
- It provides better scalability and avoids blocking threads
- Reminder state changes are typically independent of each other
Implementation example:
@Entity
@Table(name = "reminders")
public class Reminder {
@Id
private UUID id;
@Version
private Long version; // Optimistic locking field
private String status;
private LocalDateTime triggerTime;
// getters and setters
}Deadlock Prevention Strategy
Deadlock prevention will be implemented through:
- Consistent lock ordering - all operations acquire locks in the same order (e.g., by reminder ID)
- Timeout-based lock acquisition - using
tryLock()with timeout to prevent indefinite blocking - Avoid nested locking - minimizing the scope of locks and avoiding complex lock hierarchies
- Thread pool size limiting - controlling concurrent operations to reduce contention
Idempotency Implementation for Concurrent Duplicate Requests
Idempotency will be handled through:
- Idempotency keys in request headers for external API calls
- Database uniqueness constraints on reminder creation parameters (user_id + trigger_time + type)
- In-memory deduplication cache with TTL for recent reminder triggers
- Event-based idempotency tracking using a deduplication table
@Service
public class ReminderService {
private final Map<String, Instant> deduplicationCache = new ConcurrentHashMap<>();
public void processReminder(String idempotencyKey, ReminderTrigger trigger) {
String cacheKey = "reminder:" + idempotencyKey;
Instant now = Instant.now();
// Check if this exact trigger was processed recently
if (deduplicationCache.putIfAbsent(cacheKey, now) != null) {
// Already processed - return early
return;
}
// Process the reminder (with proper transaction handling)
processReminderInternal(trigger);
// Remove from cache after TTL to allow reprocessing
ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
scheduler.schedule(() -> deduplicationCache.remove(cacheKey), 5, TimeUnit.MINUTES);
}
}6. Sequence Diagrams for Key Flows
sequenceDiagram
participant U as User
participant API as ReminderServiceAPI
participant Auth as AuthMiddleware
participant Service as ReminderService
participant DB as Database
participant Scheduler as ReminderScheduler
participant Notifier as NotificationService
U->>API: CreateReminderRequest
API->>Auth: Validate JWT
Auth-->>API: Authenticated
API->>Service: createReminder(request)
Service->>DB: save(reminder)
DB-->>Service: reminderId
Service->>Scheduler: schedule(reminder)
Scheduler-->>Service: scheduled
Service-->>API: ReminderResponse
API-->>U: Created 201sequenceDiagram
participant U as User
participant API as ReminderServiceAPI
participant Auth as AuthMiddleware
participant Service as ReminderService
participant DB as Database
participant Notifier as NotificationService
U->>API: GetRemindersRequest
API->>Auth: Validate JWT
Auth-->>API: Authenticated
API->>Service: getReminders(userId, filter)
Service->>DB: findReminders(userId, filter)
DB-->>Service: List<Reminder>
Service-->>API: ReminderListResponse
API-->>U: OK 200sequenceDiagram
participant Scheduler as ReminderScheduler
participant Service as ReminderService
participant DB as Database
participant Notifier as NotificationService
participant EventPublisher as EventPublisher
Scheduler->>Service: triggerReminder(reminderId)
Service->>DB: findById(reminderId)
DB-->>Service: Reminder
Service->>Notifier: sendNotification(reminder)
Notifier-->>Service: NotificationSent
Service->>DB: update(status=notified)
DB-->>Service: Updated
Service->>EventPublisher: publish(reminderNotified)
EventPublisher-->>Service: EventPublished7. Caching Design
Cache Strategy Selection
For the Reminder service, we'll use a cache-aside pattern with LRU eviction policy. This approach provides the best balance between performance and consistency for a reminder system where:
- Read-heavy workloads (users frequently checking reminders)
- Data freshness is important but not mission-critical
- Cache misses are acceptable for performance reasons
Cache Key Design
Cache keys will follow a structured namespace pattern:
reminder:{userId}:{reminderId}
reminder:byTime:{timeBucket}:{userId}
reminder:byStatus:{status}:{userId}The namespace strategy provides:
- Clear separation of concerns
- Efficient querying by time ranges and status
- Easy cache invalidation by user or status
- Support for cache warming strategies
TTL Strategy
| Data Type | TTL | Reason |
|---|---|---|
| User reminders | 30 minutes | Frequent access, reasonable freshness |
| Reminder by time bucket | 1 hour | Time-based queries benefit from longer TTL |
| Reminder status cache | 15 minutes | Status changes should be visible quickly |
| User preferences | 24 hours | Stable user settings don't change frequently |
Cache Eviction Policy
We'll implement LRU (Least Recently Used) eviction policy with:
- Soft TTL: 90% of configured TTL for automatic expiration
- Hard TTL: Full configured TTL for forced cleanup
- Cache warming: Pre-populate frequently accessed time buckets during startup
Cache Stampede Prevention
To prevent cache stampedes:
- Probabilistic early expiry: Set cache TTL to 75% of configured value for primary keys
- Distributed locks: Use Redis distributed locks when cache misses occur
- Cache warming: Pre-populate hot keys during system startup and maintenance windows
Distributed Cache Architecture
We'll use Redis Cluster with:
- 3 master nodes, 3 replica nodes for high availability
- 200MB memory limit per node with LRU eviction policy
- Redis Sentinel for automatic failover monitoring
- Cross-zone replication for geographic distribution
Cache Invalidation Strategy
We'll implement event-driven invalidation:
- Write-through: Cache is updated on create/update/delete operations
- Event-driven invalidation: Cache entries invalidated via Redis Pub/Sub when data changes
- Time-based invalidation: TTL-based cleanup for stale entries
@Component
public class ReminderCacheService {
private final RedisTemplate<String, Reminder> redisTemplate;
private final ObjectMapper objectMapper;
public void invalidateReminderCache(String userId, String reminderId) {
String key = "reminder:" + userId + ":" + reminderId;
redisTemplate.delete(key);
// Invalidate related caches
String timeBucketKey = "reminder:byTime:" + getTimeBucket() + ":" + userId;
redisTemplate.delete(timeBucketKey);
String statusKey = "reminder:byStatus:" + getStatus() + ":" + userId;
redisTemplate.delete(statusKey);
}
public Reminder getCachedReminder(String userId, String reminderId) {
String key = "reminder:" + userId + ":" + reminderId;
return redisTemplate.opsForValue().get(key);
}
public void cacheReminder(String userId, String reminderId, Reminder reminder) {
String key = "reminder:" + userId + ":" + reminderId;
redisTemplate.opsForValue().set(key, reminder, 30, TimeUnit.MINUTES);
}
}8. Repository / Data Access Layer
Repository Pattern Implementation
The repository pattern is implemented to provide a clean abstraction over data access operations, separating domain logic from persistence concerns. Each entity has its own repository interface and implementation.
public interface ReminderRepository {
Reminder save(Reminder reminder);
Optional<Reminder> findById(UUID id);
List<Reminder> findByUserIdAndStatus(UUID userId, ReminderStatus status);
Page<Reminder> findByUserId(UUID userId, Pageable pageable);
void deleteById(UUID id);
void updateStatus(UUID id, ReminderStatus status);
}
@Repository
public class JpaReminderRepository implements ReminderRepository {
private final EntityManager entityManager;
public JpaReminderRepository(EntityManager entityManager) {
this.entityManager = entityManager;
}
@Override
public Reminder save(Reminder reminder) {
if (reminder.getId() == null) {
entityManager.persist(reminder);
} else {
entityManager.merge(reminder);
}
return reminder;
}
@Override
public Optional<Reminder> findById(UUID id) {
return Optional.ofNullable(entityManager.find(Reminder.class, id));
}
@Override
public List<Reminder> findByUserIdAndStatus(UUID userId, ReminderStatus status) {
return entityManager.createQuery(
"SELECT r FROM Reminder r WHERE r.userId = :userId AND r.status = :status",
Reminder.class)
.setParameter("userId", userId)
.setParameter("status", status)
.getResultList();
}
@Override
public Page<Reminder> findByUserId(UUID userId, Pageable pageable) {
TypedQuery<Reminder> query = entityManager.createQuery(
"SELECT r FROM Reminder r WHERE r.userId = :userId ORDER BY r.createdAt DESC",
Reminder.class)
.setParameter("userId", userId);
long total = entityManager.createQuery(
"SELECT COUNT(r) FROM Reminder r WHERE r.userId = :userId",
Long.class)
.setParameter("userId", userId)
.getSingleResult();
query.setFirstResult(pageable.getOffset());
query.setMaxResults(pageable.getPageSize());
return new PageImpl<>(query.getResultList(), pageable, total);
}
@Override
public void deleteById(UUID id) {
entityManager.createQuery("DELETE FROM Reminder r WHERE r.id = :id")
.setParameter("id", id)
.executeUpdate();
}
@Override
public void updateStatus(UUID id, ReminderStatus status) {
entityManager.createQuery(
"UPDATE Reminder r SET r.status = :status WHERE r.id = :id")
.setParameter("id", id)
.setParameter("status", status)
.executeUpdate();
}
}ORM vs Raw Query Decisions
For this service, we use JPA/Hibernate as the ORM layer because:
- It provides type safety and reduces boilerplate code
- Supports complex queries with JPQL and Criteria API
- Handles object-relational mapping efficiently
- Offers lazy loading and caching capabilities
Raw queries are used only for performance-critical operations like bulk updates or analytics queries where we need fine-grained control.
Read vs Write Separation
We implement read-write separation using CQRS pattern for better scalability:
public interface ReminderReadRepository {
List<ReminderProjection> findActiveRemindersForUser(UUID userId);
ReminderProjection findById(UUID id);
}
@Repository
public class JpaReminderReadRepository implements ReminderReadRepository {
private final EntityManager entityManager;
public JpaReminderReadRepository(EntityManager entityManager) {
this.entityManager = entityManager;
}
@Override
public List<ReminderProjection> findActiveRemindersForUser(UUID userId) {
return entityManager.createQuery(
"SELECT new com.example.reminder.projection.ReminderProjection(" +
"r.id, r.title, r.scheduledTime, r.status) " +
"FROM Reminder r WHERE r.userId = :userId AND r.status = 'ACTIVE'",
ReminderProjection.class)
.setParameter("userId", userId)
.getResultList();
}
@Override
public ReminderProjection findById(UUID id) {
return entityManager.createQuery(
"SELECT new com.example.reminder.projection.ReminderProjection(" +
"r.id, r.title, r.scheduledTime, r.status) " +
"FROM Reminder r WHERE r.id = :id",
ReminderProjection.class)
.setParameter("id", id)
.getSingleResult();
}
}Connection Pool Configuration
Connection pool is configured with HikariCP for optimal performance:
spring:
datasource:
hikari:
maximum-pool-size: 20
minimum-idle: 5
connection-timeout: 30000
idle-timeout: 600000
max-lifetime: 1800000
leak-detection-threshold: 60000N+1 Query Prevention Strategy
To prevent N+1 queries, we use eager loading with JOIN FETCH and projection-based queries:
// Instead of separate queries for each reminder's user, we fetch with JOIN
@Query("SELECT r FROM Reminder r JOIN FETCH r.user WHERE r.userId = :userId")
List<Reminder> findWithUser(@Param("userId") UUID userId);
// For projections to avoid loading full entities
@Query("SELECT new com.example.reminder.dto.ReminderSummary(r.id, r.title, r.scheduledTime) " +
"FROM Reminder r WHERE r.userId = :userId AND r.status = 'ACTIVE'")
List<ReminderSummary> findActiveRemindersWithSummary(@Param("userId") UUID userId);Batch Operation Design
Batch operations are implemented for bulk processing:
@Repository
public class BatchReminderRepository {
@PersistenceContext
private EntityManager entityManager;
public void batchUpdateStatus(List<UUID> reminderIds, ReminderStatus status) {
if (reminderIds.isEmpty()) return;
String jpql = "UPDATE Reminder r SET r.status = :status WHERE r.id IN :ids";
Query query = entityManager.createQuery(jpql);
query.setParameter("status", status);
query.setParameter("ids", reminderIds);
// Process in batches of 1000 to avoid memory issues
int batchSize = 1000;
for (int i = 0; i < reminderIds.size(); i += batchSize) {
int end = Math.min(i + batchSize, reminderIds.size());
List<UUID> batch = reminderIds.subList(i, end);
query.setParameter("ids", batch);
query.executeUpdate();
}
}
public void batchDelete(List<UUID> reminderIds) {
if (reminderIds.isEmpty()) return;
String jpql = "DELETE FROM Reminder r WHERE r.id IN :ids";
Query query = entityManager.createQuery(jpql);
query.setParameter("ids", reminderIds);
// Process in batches to avoid transaction size limits
int batchSize = 1000;
for (int i = 0; i < reminderIds.size(); i += batchSize) {
int end = Math.min(i + batchSize, reminderIds.size());
List<UUID> batch = reminderIds.subList(i, end);
query.setParameter("ids", batch);
query.executeUpdate();
}
}
}9. Asynchronous Processing & Messaging
Event-driven boundaries in this feature
The Reminder service will use a hybrid synchronous/asynchronous approach to handle various operations:
-
Synchronous flows:
- Reminder creation and immediate notification (e.g., SMS/email)
- Reminder status updates (e.g., completed, cancelled)
- Immediate validation and conflict detection
-
Asynchronous flows:
- Scheduled reminder delivery (e.g., push notifications, emails)
- Reminder retry logic for failed deliveries
- Audit trail and analytics updates
- Cross-service coordination (e.g., integration with calendar services)
The decision to use asynchronous processing for delivery and retries stems from the need to handle potentially long-running operations without blocking the main request path, ensuring system responsiveness and resilience.
Message Schema Design
{
"version": "1.0",
"event_type": "reminder.scheduled",
"timestamp": "2023-10-27T14:30:00Z",
"payload": {
"reminder_id": "uuid-12345",
"user_id": "user-98765",
"notification_type": "email",
"scheduled_time": "2023-10-27T15:00:00Z",
"message_content": "Your meeting is scheduled for 3 PM today.",
"recipient_email": "user@example.com"
},
"source": "reminder-service",
"correlation_id": "corr-abc123"
}Dead Letter Queue Strategy and Poison Message Handling
The system will implement a DLQ strategy with:
- Max retry attempts: 3 retries with exponential backoff
- Dead letter queue: Messages that fail after max retries are moved to a DLQ topic
- Poison message handling: Messages in DLQ will trigger alerts and manual inspection
public class ReminderDeliveryConsumer {
private static final int MAX_RETRY_ATTEMPTS = 3;
public void processReminderDelivery(ReminderEvent event) {
int attempt = 0;
while (attempt < MAX_RETRY_ATTEMPTS) {
try {
// Attempt delivery
deliverNotification(event);
return;
} catch (Exception e) {
attempt++;
if (attempt >= MAX_RETRY_ATTEMPTS) {
// Move to DLQ
moveMessageToDLQ(event);
throw new RuntimeException("Failed after max retries", e);
}
// Exponential backoff
Thread.sleep(1000 * Math.pow(2, attempt));
}
}
}
}At-least-once vs. Exactly-once Delivery
Decision: At-least-once delivery model with idempotency handling.
Justification:
- Reminder service needs to ensure notifications are delivered at least once
- Duplicate deliveries can be safely handled with idempotency keys
- The cost of exactly-once (complex coordination) outweighs the benefit for this use case
Retry Policy for Consumers
public class ReminderRetryPolicy {
private static final int MAX_RETRIES = 3;
private static final long INITIAL_DELAY_MS = 1000;
private static final double BACKOFF_MULTIPLIER = 2.0;
public boolean shouldRetry(int attempt, Exception exception) {
// Retry on transient failures
return attempt < MAX_RETRIES &&
(exception instanceof NetworkException ||
exception instanceof ServiceUnavailableException);
}
public long getDelay(int attempt) {
return (long) (INITIAL_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempt));
}
}Outbox Pattern for Reliable Event Publishing
The outbox pattern will be implemented to ensure reliable event publishing:
public class ReminderEventPublisher {
private final EventRepository eventRepository;
private final MessageProducer messageProducer;
public void publishReminderEvent(Reminder reminder, String eventType) {
// 1. Create event in outbox table
ReminderEvent event = new ReminderEvent(
UUID.randomUUID(),
reminder.getId(),
eventType,
"reminder-service",
new Date()
);
// 2. Save to outbox (within same transaction)
eventRepository.save(event);
// 3. Publish to message broker (in separate transaction)
messageProducer.publish(event);
}
}Sequence Diagram for Async Flows
sequenceDiagram
participant Client
participant ReminderService
participant EventPublisher
participant MessageBroker
participant NotificationService
Client->>ReminderService: Create Reminder (POST /reminders)
ReminderService->>EventPublisher: Store reminder event in outbox
EventPublisher->>MessageBroker: Publish reminder.scheduled event
MessageBroker->>NotificationService: Deliver to consumer queue
NotificationService->>NotificationService: Process delivery attempt
NotificationService->>NotificationService: Retry on failure (max 3 attempts)
NotificationService->>NotificationService: Move to DLQ if exhausted10. Error Handling Design
Error Classification Hierarchy
The error handling strategy follows a hierarchical classification system to enable proper routing and response:
Error
├── ValidationException (400)
│ ├── InvalidInputException
│ └── ConstraintViolationException
├── DomainException (400/409)
│ ├── ReminderAlreadyExistsException
│ ├── ReminderNotFoundException
│ └── InvalidReminderStateException
├── InfrastructureException (500)
│ ├── DatabaseException
│ ├── ExternalServiceException
│ └── CacheException
└── AuthorizationException (401/403)
├── UnauthorizedAccessException
└── PermissionDeniedExceptionError Propagation Strategy
- Validation errors are caught at the API boundary and converted to structured 400 responses
- Domain errors are propagated up to the service layer and converted to appropriate HTTP status codes (400/409)
- Infrastructure errors are either retried or escalated to higher layers for circuit breaker handling
- Authorization errors are immediately converted to 401/403 responses
Retry Strategy with Exponential Backoff and Jitter
For infrastructure-level failures (e.g., database timeouts, external service unavailability):
public class RetryableOperation {
private static final int MAX_RETRIES = 3;
private static final long BASE_DELAY_MS = 1000;
private static final double JITTER_RANGE = 0.5;
public <T> T executeWithRetry(Supplier<T> operation) {
Exception lastException = null;
for (int attempt = 0; attempt <= MAX_RETRIES; attempt++) {
try {
return operation.get();
} catch (InfrastructureException e) {
lastException = e;
if (attempt >= MAX_RETRIES) {
throw new RetryFailedException("Operation failed after " + MAX_RETRIES + " attempts", e);
}
long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempt) +
(Math.random() - 0.5) * BASE_DELAY_MS * JITTER_RANGE);
try {
Thread.sleep(delay);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
throw new RetryFailedException("Retry interrupted", ie);
}
}
}
throw new RetryFailedException("Operation failed after " + MAX_RETRIES + " attempts", lastException);
}
}Circuit Breaker Design
For external service dependencies (e.g., notification service):
public class CircuitBreaker {
private final int failureThreshold = 5;
private final long timeoutMs = 60000; // 1 minute
private final int successThreshold = 3;
private volatile CircuitState state = CircuitState.CLOSED;
private int failureCount = 0;
private long lastFailureTime = 0;
public enum CircuitState {
CLOSED, OPEN, HALF_OPEN
}
public <T> T execute(Supplier<T> operation) {
switch (state) {
case CLOSED:
return executeClosed(operation);
case OPEN:
return executeOpen(operation);
case HALF_OPEN:
return executeHalfOpen(operation);
default:
throw new IllegalStateException("Unknown circuit state");
}
}
private <T> T executeClosed(Supplier<T> operation) {
try {
T result = operation.get();
onSuccess();
return result;
} catch (Exception e) {
onFailure(e);
throw e;
}
}
private <T> T executeOpen(Supplier<T> operation) {
if (System.currentTimeMillis() - lastFailureTime > timeoutMs) {
state = CircuitState.HALF_OPEN;
return executeHalfOpen(operation);
}
throw new CircuitBreakerOpenException("Circuit is open");
}
private <T> T executeHalfOpen(Supplier<T> operation) {
try {
T result = operation.get();
state = CircuitState.CLOSED;
failureCount = 0;
return result;
} catch (Exception e) {
state = CircuitState.OPEN;
lastFailureTime = System.currentTimeMillis();
throw new CircuitBreakerOpenException("Circuit re-opened due to failure", e);
}
}
private void onSuccess() {
if (state == CircuitState.HALF_OPEN) {
failureCount = 0;
}
}
private void onFailure(Exception e) {
if (state == CircuitState.CLOSED) {
failureCount++;
lastFailureTime = System.currentTimeMillis();
if (failureCount >= failureThreshold) {
state = CircuitState.OPEN;
}
} else if (state == CircuitState.HALF_OPEN) {
state = CircuitState.OPEN;
}
}
}Fallback Behavior Design
For critical dependencies like notification service:
public class NotificationFallback {
private final NotificationService primaryService;
private final NotificationService backupService;
public void sendNotificationWithFallback(NotificationRequest request) {
try {
primaryService.send(request);
} catch (InfrastructureException e) {
// Log the failure but don't let it break the main flow
logger.warn("Primary notification service failed, using fallback", e);
try {
backupService.send(request);
} catch (Exception fallbackException) {
logger.error("Both primary and backup notification services failed",
fallbackException);
// Consider alerting or storing for manual follow-up
}
}
}
}Error Logging Format
Structured logging with trace context:
{
"timestamp": "2023-10-27T14:30:45.123Z",
"trace_id": "12345678-1234-1234-1234-123456789012",
"span_id": "abcdef12-3456-7890-abcd-ef1234567890",
"user_id": "user_12345",
"entity_id": "reminder_abcde",
"error_code": "REMINDER_NOT_FOUND",
"error_message": "Reminder with ID abcde not found",
"error_type": "DomainException",
"stack_trace": "...",
"http_status_code": 404,
"request_path": "/api/reminders/abcde",
"request_method": "GET"
}Alerting Thresholds
Critical alerts are triggered when:
- Database connection failures exceed 5 in 5 minutes
- External service timeouts exceed 10% of requests for 1 minute
- Circuit breaker opens more than 3 times in 5 minutes
- Retry failures exceed 10% of operations for 1 minute
Alerts are sent to:
- On-call engineer via Slack
- Operations dashboard with severity levels (critical/warning)
- Email to engineering team for critical issues
11. Security Design
Input Validation and Sanitization Strategy
All API inputs are validated using a combination of:
- Bean Validation (JSR-303) for structural validation
- Custom validators for business logic constraints (e.g., valid time formats, recurrence rules)
- HTML sanitization using OWASP Java HTML Sanitizer for any user-generated content fields
public class ReminderRequest {
@NotBlank
@Size(max = 255)
private String title;
@NotNull
@Future
private LocalDateTime triggerTime;
@ValidRecurrenceRule
private String recurrenceRule;
}SQL Injection Prevention
- Prepared statements for all database queries using JDBC Template or JPA
- Parameterized queries - never concatenating user input into SQL strings
- ORM usage with proper entity mapping to prevent raw SQL injection
Authentication Middleware Design
- JWT-based authentication with HS256 signing algorithm
- OAuth2 integration for third-party identity providers (Google, Microsoft)
- Authentication filter that validates JWT tokens and extracts user context
- Token refresh mechanism with short-lived access tokens (15 min) and long-lived refresh tokens (7 days)
Authorization Enforcement
- Role-Based Access Control (RBAC) with user roles: USER, ADMIN
- Resource-level authorization using Spring Security annotations:
@PreAuthorize("@reminderAuthorizationService.canAccessReminder(principal, #reminderId)")
public void updateReminder(String reminderId, ReminderUpdateRequest request) { }- Permission checking at service layer for reminder ownership and modification rights
Sensitive Data Encryption
- At rest: AES-256 encryption for sensitive fields (reminder content, user preferences)
- In transit: TLS 1.3 with HSTS enforcement for all API endpoints
- Key management: AWS KMS integration for key rotation and secure storage
PII Handling and Masking Strategy
- PII fields: User email, reminder content (masked in logs)
- Masking rules: Email addresses shown as
user***@domain.com, content truncated to first 50 chars in logs - Data retention: PII deleted after 3 years of inactivity
Secrets Management
- AWS Secrets Manager integration for storing:
- Database credentials (rotated every 90 days)
- Third-party API keys (Google Calendar, Slack)
- JWT signing keys
- Dynamic secret rotation with automated refresh mechanisms
Audit Logging Design
- Logged events: Reminder creation, modification, deletion, trigger execution
- Log format:
{
"timestamp": "2023-12-01T10:30:00Z",
"userId": "user-uuid-123",
"action": "CREATE_REMINDER",
"resourceId": "reminder-uuid-456",
"details": {
"title": "Meeting",
"triggerTime": "2023-12-01T14:00:00Z"
}
}- Retention policy: 90 days for audit logs, archived to S3 for compliance
Security Headers and CORS Configuration
- Security headers:
- X-Content-Type-Options: nosniff
- X-Frame-Options: DENY
- X-XSS-Protection: 1; mode=block
- Content-Security-Policy: default-src 'self'
- CORS policy: Strict whitelist of allowed origins with credentials support
Rate Limiting and DDoS Protection
- API rate limiting using Redis-based token bucket algorithm (1000 req/hour per user)
- Request size limits (max 10KB per request)
- IP-based blocking for suspicious traffic patterns
- Bot detection using behavioral analysis and CAPTCHA integration
Vulnerability Scanning and Dependency Management
- Static code analysis using SonarQube for security issues
- Dependency scanning with OWASP DependencyCheck for vulnerable libraries
- Automated security patching for runtime dependencies
12. Testing Strategy
Unit Testing Scope and Coverage Targets (%)
- Service Layer: 85% code coverage with focus on business logic, error handling, and boundary conditions
- Repository Layer: 90% coverage with emphasis on query correctness and edge case handling
- Domain Objects: 100% coverage for value objects, enums, and immutable entities
- Exception Handling: 100% coverage for all custom exception paths
- Integration Points: 95% coverage for external dependency mocks and stubs
Integration Testing Strategy
- Database Integration: Use Testcontainers with PostgreSQL 13+ for end-to-end database tests
- External API Integration: WireMock or embedded mock servers for third-party service dependencies (e.g., email/SMS providers)
- Event System: Embedded Kafka or RabbitMQ containers for async flow validation
- Caching Layer: In-memory Redis mock for cache interaction testing
Contract Testing
- API Contracts: Use Pact for consumer-driven contract testing between reminder service and client applications
- Event Contracts: Define Avro schemas with schema registry integration for event contract validation
- External Service Contracts: Mock external service contracts using WireMock or similar tools
Performance Testing Tool and Key Scenarios
- Tool: k6 (JavaScript-based) for load testing, JMeter for stress testing
- Key Scenarios:
- Concurrent reminder creation (1000+ simultaneous requests)
- High-volume reminder retrieval (batch queries with pagination)
- Scheduled reminder processing under load (1000+ reminders per minute)
- Cache warm-up and hit ratio validation
- Metrics Tracked:
- p95/p99 response times (target: <200ms for CRUD operations)
- Throughput (target: 5000 RPS for read-heavy operations)
- Error rates (target: <0.1% failure rate)
Test Data Management Strategy
- Fixtures: JSON/YAML fixtures for common test data (users, reminders, categories)
- Factories: Factory pattern with Faker.js for generating realistic test data
- Seeding: Database seed scripts for initial test environment setup
- Isolation: Per-test database transactions with rollback strategy to ensure test isolation
- Data Generation: Use of data generation libraries for large-scale synthetic data (e.g., 100K+ reminder records for performance testing)
13. Observability Design
Structured Logging Schema
All logs must follow a consistent schema to enable effective querying and analysis:
{
"timestamp": "2023-10-15T14:30:45.123Z",
"level": "INFO|WARN|ERROR",
"service": "reminder-service",
"trace_id": "1234567890abcdef1234567890abcdef",
"span_id": "abcdef1234567890abcdef1234567890",
"user_id": "user-123",
"entity_id": "reminder-456",
"operation": "create_reminder",
"message": "Reminder created successfully",
"error_code": "REMINDER_001",
"duration_ms": 45,
"error_details": {
"stack_trace": "...",
"cause": "Invalid reminder time"
}
}PII masking rules:
- User IDs are logged as hashed values (e.g.,
user-123→user-abc123) - Email addresses are masked (e.g.,
john@example.com→j***@example.com) - Full names are truncated to first initial (e.g.,
John Doe→J*** ***)
Distributed Tracing Instrumentation
Using OpenTelemetry for end-to-end tracing:
Custom Spans to Add:
reminder.create- from API entry point to DB writereminder.process- background task executionreminder.send_notification- external notification service callreminder.cleanup- periodic cleanup job
Attributes to Capture:
reminder.id: reminder identifierreminder.user_id: user who owns the reminderreminder.type: email, sms, pushreminder.scheduled_time: when reminder should firereminder.status: pending, sent, failed
Metrics Design
RED Method Metrics:
| Metric Name | Type | Description | Labels |
|---|---|---|---|
reminder_creation_count | Counter | Total reminders created | user_id, reminder_type |
reminder_creation_duration | Histogram | Time to create reminder | user_id, reminder_type |
reminder_send_errors | Counter | Failed notification attempts | user_id, notification_type |
reminder_send_duration | Histogram | Time to send notification | user_id, notification_type |
reminder_processing_errors | Counter | Failed reminder processing | error_code |
reminder_queue_size | Gauge | Current queue size | queue_name |
Additional Service-Level Metrics:
active_reminders(Gauge): Count of active remindersnotification_success_rate(Gauge): % successful notificationsqueue_processing_rate(Counter): Rate of queue processing
Health Check Endpoint Design
GET /health
GET /ready
GET /live
GET /startupResponse Format:
{
"status": "healthy|degraded|unhealthy",
"checks": {
"database": {"status": "healthy", "details": "Connected to primary DB"},
"cache": {"status": "healthy", "details": "Redis connected"},
"message_queue": {"status": "degraded", "details": "5% of messages delayed"},
"dependencies": {"status": "healthy", "details": "All external services reachable"}
},
"timestamp": "2023-10-15T14:30:45.123Z"
}Health Check Logic:
/live: Basic service liveness (port listening)/ready: Service readiness (DB, cache, queue connectivity)/health: Full health including external dependencies/startup: Service startup validation (config loaded, DB migration completed)
Alerting Thresholds
Critical Alerts:
reminder_creation_errors> 100 errors/minute → Alertreminder_send_errors> 50 errors/minute → Alertqueue_processing_rate< 10 items/sec for 5 min → Alertdatabase_connection_errors> 5 errors/minute → Alert
Warning Alerts:
reminder_creation_durationp95 > 1000ms → Alertreminder_send_durationp95 > 2000ms → Alertqueue_size> 10000 items → Alert
On-call Escalation Path:
- Primary engineer (5 min)
- Secondary engineer (30 min)
- Team lead (1 hour)
- SRE team (2 hours)
Monitoring Dashboard
Key dashboard panels:
- Service Health Overview: Status of all health checks
- Reminder Creation Rate: Throughput over time
- Notification Delivery Rate: Success/failure ratio
- Queue Metrics: Size, processing rate, latency
- Error Rate by Type: Distribution of error types
- Latency Distribution: p50/p95/p99 response times
14. Open Questions, Risks & Trade-offs
Decisions requiring product / engineering alignment
- Notification channel prioritization: Should SMS, email, and push notifications be treated as equal priority channels or should we implement a channel selection strategy based on user preference, device availability, or delivery success rates?
- Reminder recurrence patterns: How granular should recurrence support be? Should we support simple daily/weekly/monthly patterns or complex cron-like expressions that require more sophisticated parsing and validation?
- User preference management: How should user preferences for reminder timing (e.g., "remind me 15 minutes before") be stored and managed? Should these be global defaults or per-reminder overrides?
- Time zone handling: Should all reminders be stored in UTC and converted to user's local time zone at delivery, or should we store the original user timezone and handle conversions dynamically?
Known trade-offs and justifications
- Eventual consistency vs. strong consistency: We're leaning towards eventual consistency for reminder delivery to improve system scalability and availability, but this introduces a risk of reminders being delivered slightly late or not at all if the system is under heavy load. This trade-off is acceptable for most reminder use cases where slight delays are tolerable.
- Caching vs. real-time data: We're using cache-aside pattern with TTLs to reduce database load, but this introduces potential staleness. The trade-off is acceptable since reminder systems are generally not mission-critical and can tolerate a few seconds of delay in data freshness.
- Monolithic design vs. microservices: We're keeping the reminder service as a monolithic component rather than splitting into separate services for notification, scheduling, and user preferences. This simplifies deployment and reduces inter-service communication overhead but increases system complexity in the long term.
- Idempotency vs. performance: We're implementing idempotency keys for all mutation endpoints to handle duplicate requests, but this adds a small performance overhead due to additional database lookups and state tracking.
Risks and mitigation strategies
- Delivery failure risk: Reminders might not be delivered due to system outages, network issues, or user device unavailability. Mitigation includes implementing retry logic with exponential backoff and dead-letter queue handling for failed deliveries.
- Clock drift risk: System clock drift between services or time zone changes could cause reminders to fire at wrong times. Mitigation includes using NTP for clock synchronization and storing timestamps in UTC with proper timezone handling.
- Resource exhaustion risk: High-volume reminder systems could exhaust database connections, memory, or CPU resources. Mitigation includes implementing proper connection pooling, query optimization, and circuit breaker patterns.
- Data loss risk: Reminder data could be lost due to database failures or improper cleanup. Mitigation includes implementing proper backup strategies, soft deletes with audit trails, and regular data validation checks.
Technical debt acknowledged
- Limited extensibility: The current design assumes a fixed set of reminder types and notification channels, which makes future feature additions more complex without refactoring. This is acceptable for MVP but should be addressed in future iterations.
- Hardcoded business logic: Reminder scheduling and notification logic is currently tightly coupled within the service layer, which makes it harder to extend or modify without affecting core functionality.
- Limited multi-tenancy support: The current design assumes single-tenant operation and doesn't handle cross-tenant reminder conflicts or permissions properly. This is acceptable for initial implementation but needs enhancement for enterprise use cases.
- Basic error handling: The current error handling is minimal and doesn't include comprehensive error categorization or detailed logging for debugging complex edge cases. This should be enhanced to support better observability and troubleshooting in production environments.