RESTful CRUD API Design Patterns and Implementation Guidelines

This document presents a comprehensive low-level design approach for building scalable RESTful CRUD APIs with proper resource modeling, HTTP semantics adherence, and production-ready implementation patterns.

1. Problem Statement & Requirements

Clarifying questions

1. What is the specific resource being designed for CRUD operations? Is this a generic "resource" or a specific domain entity like user, order, product, etc.?
2. What scale expectations are there for this API? Should we design for millions of concurrent users or is it a smaller-scale internal service?
3. What are the consistency requirements for this resource? Should reads be strongly consistent or eventual consistency acceptable?
4. What is the expected read/write ratio for this resource? Is it primarily read-heavy or write-heavy?
5. What are the latency sensitivity requirements? Are there strict SLAs (e.g., <100ms p99) or is this a batch-oriented system?
6. What authentication/authorization model is in place? Will this API integrate with existing OAuth2, JWT, or custom auth systems?
7. Are there any external integrations or dependencies? Will this resource integrate with other services or databases?

Functional requirements

• Create new resources via POST endpoint
• Read individual resources by ID via GET endpoint
• Read collections of resources via GET endpoint with filtering and sorting capabilities
• Update existing resources via PUT/PATCH endpoints
• Delete resources via DELETE endpoint
• Support for soft-deletes (optionally)
• Support for versioning of resources (if needed)
• Audit trail capability for all operations
• Bulk operations support (batch create/update/delete)

Non-functional requirements

• Latency targets:
  • p50: <50ms
  • p95: <100ms
  • p99: <200ms
• Throughput: 10,000 requests/second (read + write combined) for peak load
• Availability: 99.9% uptime (28.8 minutes of downtime/month max)
• Scalability: Horizontal scaling capability up to 1000 concurrent instances
• Data durability: Data persistence with no loss in case of system failure

Constraints and assumptions

• Assumptions:
  • The system will run on a cloud infrastructure (AWS/GCP/Azure)
  • Data will be stored in a relational database with ACID properties
  • Authentication and authorization are handled at the API gateway level
  • No external dependencies for core CRUD operations (except database)
  • All data is stored in a single region initially
• Constraints:
  • Maximum resource size is 1MB (including metadata)
  • Maximum concurrent users is 50,000
  • Database connection pool size is limited to 50 connections per instance

Out of scope

• Authentication and authorization (handled at API gateway)
• Data encryption at rest or in transit
• Multi-region deployment and failover strategies
• Complex business logic beyond basic CRUD operations
• Real-time notifications or webhooks (can be added later)
• Data migration tools or ETL processes
• UI components or frontend integration
• Third-party service integrations (e.g., payment gateways, external APIs)
• Data archiving or long-term retention policies
• Bulk import/export functionality (CSV, JSON)

2. API Design

API Style Selection

REST was selected for this design due to its widespread adoption, clear semantics, and strong ecosystem support. REST's resource-based approach aligns well with CRUD operations and provides predictable URL structures that are easy to understand and consume.

Endpoint Definitions

Method	Path	Description	Auth
GET	/api/v1/resources/{id}	Retrieve a specific resource	JWT
POST	/api/v1/resources	Create a new resource	JWT
PUT	/api/v1/resources/{id}	Update an entire resource	JWT
PATCH	/api/v1/resources/{id}	Partially update a resource	JWT
DELETE	/api/v1/resources/{id}	Delete a resource	JWT
GET	/api/v1/resources	List resources with pagination	JWT

Request and Response Schemas

Resource Object

{
  "id": "uuid",
  "name": "string (max 255 chars)",
  "description": "string (max 10000 chars)",
  "status": "enum (ACTIVE, INACTIVE, DELETED)",
  "createdAt": "ISO8601 timestamp",
  "updatedAt": "ISO8601 timestamp"
}

Create Resource Request

{
  "name": "Sample Resource",
  "description": "A sample resource for testing"
}

Create Resource Response

{
  "id": "123e4567-e89b-12d3-a456-426614174000",
  "name": "Sample Resource",
  "description": "A sample resource for testing",
  "status": "ACTIVE",
  "createdAt": "2023-10-01T12:00:00Z",
  "updatedAt": "2023-10-01T12:00:00Z"
}

List Resources Response (with pagination)

{
  "data": [
    {
      "id": "123e4567-e89b-12d3-a456-426614174000",
      "name": "Sample Resource",
      "description": "A sample resource for testing",
      "status": "ACTIVE",
      "createdAt": "2023-10-01T12:00:00Z",
      "updatedAt": "2023-10-01T12:00:00Z"
    }
  ],
  "pagination": {
    "page": 1,
    "size": 20,
    "total": 1,
    "hasNext": false
  }
}

HTTP Status Codes and Error Response Format

Status Code	Description
200	Successful GET, PUT, PATCH
201	Successful POST
204	Successful DELETE
400	Bad Request - validation errors
401	Unauthorized
403	Forbidden
404	Not Found
409	Conflict - resource exists or constraint violation
422	Unprocessable Entity - validation errors
429	Rate Limited
500	Internal Server Error

Standardized error response format:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Name must be between 1 and 255 characters",
    "details": [
      {
        "field": "name",
        "message": "Name must be between 1 and 255 characters"
      }
    ],
    "timestamp": "2023-10-01T12:00:00Z"
  }
}

API Versioning Strategy

Versioning is handled through the path prefix /api/v1/ to allow for future version migrations without breaking existing clients. This approach is simple, explicit, and follows REST conventions.

Pagination Design

Cursor-based pagination was chosen over offset-based pagination because:

1. It's more efficient for large datasets
2. Provides consistent results even when data changes during pagination
3. Handles concurrent modifications better than offset-based approaches

Cursor format: cursor=base64encoded(pageToken)

Authentication and Authorization

All endpoints require JWT-based authentication with the following scopes:

• resources:read - for GET operations
• resources:write - for POST, PUT, PATCH, DELETE operations

JWT tokens are validated using HS256 signing algorithm with a shared secret.

Rate Limiting

Rate limiting is implemented at the endpoint level:

• GET /api/v1/resources: 1000 requests per hour
• POST /api/v1/resources: 500 requests per hour
• PUT/PATCH/DELETE /api/v1/resources/{id}: 250 requests per hour

Rate limiting is enforced using a sliding window algorithm with Redis for distributed state.

Idempotency Design

Idempotency keys are supported for mutation operations:

• X-Idempotency-Key header with UUID value
• For POST, PUT, PATCH operations, same key ensures idempotent behavior
• Keys are stored in Redis with TTL of 24 hours

Example Idempotency Flow

POST /api/v1/resources
X-Idempotency-Key: 123e4567-e89b-12d3-a456-426614174000
Content-Type: application/json

{
  "name": "Sample Resource",
  "description": "A sample resource for testing"
}

If the same request is made with the same idempotency key, the system returns the previously created resource instead of creating a duplicate.

3. Data Model Design

Entity-Relationship diagram (Mermaid)

erDiagram
    USER {
        string id PK
        string email
        string name
        timestamp created_at
        timestamp updated_at
        boolean is_active
    }

    RESOURCE {
        string id PK
        string title
        string description
        string owner_id FK
        timestamp created_at
        timestamp updated_at
        boolean is_deleted
    }

    USER ||--o{ RESOURCE : "owns"

Table / Collection schema definitions with field types, constraints, indices

-- Users table
CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email VARCHAR(255) UNIQUE NOT NULL,
    name VARCHAR(255) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    is_active BOOLEAN DEFAULT TRUE,
    
    CONSTRAINT chk_email_format CHECK (email ~* '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$')
);

-- Resources table
CREATE TABLE resources (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    title VARCHAR(255) NOT NULL,
    description TEXT,
    owner_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    is_deleted BOOLEAN DEFAULT FALSE,
    
    INDEX idx_resources_owner_id (owner_id),
    INDEX idx_resources_created_at (created_at),
    INDEX idx_resources_title (title),
    INDEX idx_resources_owner_deleted (owner_id, is_deleted)
);

-- Audit trail table for tracking changes
CREATE TABLE audit_trail (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    entity_type VARCHAR(50) NOT NULL,
    entity_id UUID NOT NULL,
    operation VARCHAR(20) NOT NULL, -- CREATE, UPDATE, DELETE
    old_values JSONB,
    new_values JSONB,
    user_id UUID REFERENCES users(id),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Primary key strategy (UUID vs. auto-increment vs. ULID — with justification)

Using UUID as primary key for both tables:

• UUID advantages:

  • Globally unique across distributed systems
  • No need for coordination between services or databases
  • Support for horizontal sharding without key conflicts
  • Better security (no sequential guessing of IDs)
  • No database round-trip for ID generation in distributed systems

• UUID disadvantages:

  • Larger storage footprint (16 bytes vs 8 bytes for BIGINT)
  • Less efficient index performance due to random distribution
  • Not human-readable

Index design (composite, partial, covering indices — with justification per index)

Index Name	Columns	Type	Purpose
idx_resources_owner_id	owner_id	Standard	Fast lookup of resources by owner
idx_resources_created_at	created_at	Standard	Efficient time-based queries
idx_resources_title	title	Standard	Fast search by title
idx_resources_owner_deleted	owner_id, is_deleted	Composite	Efficient filtering of active resources per user
idx_audit_trail_entity_type_created_at	entity_type, created_at	Composite	Fast audit log queries by type and time

Data partitioning / sharding strategy

Partition resources table by owner_id hash:

-- Example of partitioning by owner_id hash (PostgreSQL)
CREATE TABLE resources_partitioned (
    id UUID,
    title VARCHAR(255),
    description TEXT,
    owner_id UUID,
    created_at TIMESTAMP,
    updated_at TIMESTAMP,
    is_deleted BOOLEAN
) PARTITION BY HASH (owner_id);

CREATE TABLE resources_0 PARTITION OF resources_partitioned FOR VALUES WITH (MODULUS 8, REMAINDER 0);
CREATE TABLE resources_1 PARTITION OF resources_partitioned FOR VALUES WITH (MODULUS 8, REMAINDER 1);
-- ... additional partitions

Soft delete vs. hard delete decision with justification

Soft delete is used for resources:

• Soft delete advantages:

  • Data recovery capability
  • Audit trail preservation
  • Maintains referential integrity for foreign key relationships
  • Support for "undelete" functionality if needed

• Hard delete disadvantages:

  • Irreversible data loss
  • Audit trail gaps
  • Potential for orphaned references in other systems

Audit trail design (who, what, when)

The audit_trail table captures:

• Who: User ID (from authentication context)
• What: Operation type (CREATE, UPDATE, DELETE) and JSON payload of changes
• When: Timestamp of operation

Data archival strategy and retention policy

Archival strategy:

1. Daily archiving job: Move resources older than 3 years to archive storage (S3 or similar)
2. Retention policy:
   • Active resources: 1 year (soft deleted but not archived)
   • Archived resources: 3 years (full historical retention)
3. Archive format: Compressed JSON or Parquet with metadata
4. Indexing: Maintain indexes on archived data for search capability

4. Class Design & Service Layer

UML Class Diagram

classDiagram
    class Resource {
        +String id
        +String name
        +String description
        +Instant createdAt
        +Instant updatedAt
        +Boolean isActive
        +validate()
        +updateTimestamps()
    }
    
    class ResourceService {
        +createResource(CreateResourceRequest)
        +getResourceById(String)
        +updateResource(UpdateResourceRequest)
        +deleteResource(String)
        +listResources(ResourceFilter)
        +getResourceStats()
    }
    
    class ResourceRepository {
        +save(Resource)
        +findById(String)
        +update(Resource)
        +deleteById(String)
        +findAll(ResourceFilter)
        +countActive()
    }
    
    class ResourceValidator {
        +validateCreate(CreateResourceRequest)
        +validateUpdate(UpdateResourceRequest)
    }
    
    class ResourceEventPublisher {
        +publishCreated(Resource)
        +publishUpdated(Resource)
        +publishDeleted(String)
    }
    
    class ResourceEvent {
        +String id
        +String resourceId
        +String eventType
        +Instant timestamp
    }
    
    class CreateResourceRequest {
        +String name
        +String description
    }
    
    class UpdateResourceRequest {
        +String id
        +String name
        +String description
    }
    
    class ResourceFilter {
        +String name
        +Boolean isActive
        +Integer limit
        +Integer offset
    }
    
    ResourceService --> Resource : manages
    ResourceService --> ResourceRepository : delegates to
    ResourceService --> ResourceValidator : validates
    ResourceService --> ResourceEventPublisher : publishes events
    ResourceRepository ..> Resource : persists
    ResourceValidator ..> CreateResourceRequest : validates
    ResourceValidator ..> UpdateResourceRequest : validates
    ResourceEventPublisher --> ResourceEvent : publishes
    
    class ResourceController {
        +createResource(CreateResourceRequest)
        +getResourceById(String)
        +updateResource(UpdateResourceRequest)
        +deleteResource(String)
        +listResources(ResourceFilter)
    }
    
    ResourceController --> ResourceService : delegates to

State Machine Diagram

stateDiagram-v2
    [*] --> Active
    Active --> Inactive : deactivate()
    Inactive --> Active : activate()
    Active --> Deleted : delete()
    Inactive --> Deleted : delete()
    Deleted --> [*]

Service Class Structure and Responsibilities

The ResourceService class encapsulates the business logic for resource operations. It handles:

• Request validation and transformation
• Business rule enforcement (e.g., uniqueness constraints, soft-delete behavior)
• Transaction boundary management
• Event publishing for audit and async processing
• Integration with external systems (e.g., analytics, notifications)

Interface / Abstract Class Design

public interface ResourceRepository {
    Resource save(Resource resource);
    Optional<Resource> findById(String id);
    Resource update(Resource resource);
    void deleteById(String id);
    List<Resource> findAll(ResourceFilter filter);
    long countActive();
}

public interface ResourceValidator {
    void validateCreate(CreateResourceRequest request);
    void validateUpdate(UpdateResourceRequest request);
}

public interface ResourceEventPublisher {
    void publishCreated(Resource resource);
    void publishUpdated(Resource resource);
    void publishDeleted(String resourceId);
}

Business Logic Encapsulation Patterns

The service layer implements the Strategy pattern for validation logic to support different validation rules based on resource type or business context.

public interface ValidationStrategy {
    void validate(CreateResourceRequest request);
}

public class DefaultValidationStrategy implements ValidationStrategy {
    @Override
    public void validate(CreateResourceRequest request) {
        if (request.getName() == null || request.getName().trim().isEmpty()) {
            throw new ValidationException("Name is required");
        }
        if (request.getName().length() > 255) {
            throw new ValidationException("Name must be less than 256 characters");
        }
    }
}

public class ResourceService {
    private final ValidationStrategy validationStrategy;
    
    public ResourceService(ValidationStrategy validationStrategy) {
        this.validationStrategy = validationStrategy;
    }
    
    public Resource createResource(CreateResourceRequest request) {
        validationStrategy.validate(request);
        // ... rest of implementation
    }
}

Critical-Path Method Implementation

@Transactional
public Resource createResource(CreateResourceRequest request) {
    // Validate input
    validator.validateCreate(request);
    
    // Check for duplicates (optimistic concurrency)
    if (repository.existsByName(request.getName())) {
        throw new DuplicateResourceException("Resource with name '%s' already exists".formatted(request.getName()));
    }
    
    // Create new resource
    Resource resource = Resource.builder()
        .id(UUID.randomUUID().toString())
        .name(request.getName())
        .description(request.getDescription())
        .isActive(true)
        .createdAt(Instant.now())
        .updatedAt(Instant.now())
        .build();
    
    // Save to database
    Resource savedResource = repository.save(resource);
    
    // Update audit timestamp for consistency
    resource.updateTimestamps();
    
    // Publish event for async processing and audit trail
    eventPublisher.publishCreated(savedResource);
    
    return savedResource;
}

Transaction Boundary Design

Each operation that modifies state is wrapped in a single database transaction:

• Create: Save resource and publish event (within same transaction)
• Update: Validate, update, and publish (all in one transaction)
• Delete: Soft-delete and publish event (atomic operation)

Service Interface Contracts

public interface ResourceService {
    Resource createResource(CreateResourceRequest request);
    Optional<Resource> getResourceById(String id);
    Resource updateResource(UpdateResourceRequest request);
    void deleteResource(String id);
    List<Resource> listResources(ResourceFilter filter);
    ResourceStats getResourceStats();
}

Saga / Orchestration Pattern

For complex operations involving external systems, a saga pattern would be implemented using the outbox pattern to ensure eventual consistency.

Domain Object / Value Object Design

public class Resource {
    private final String id;
    private final String name;
    private final String description;
    private final Instant createdAt;
    private final Instant updatedAt;
    private final boolean isActive;
    
    public Resource(String id, String name, String description, 
                   Instant createdAt, Instant updatedAt, boolean isActive) {
        this.id = id;
        this.name = name;
        this.description = description;
        this.createdAt = createdAt;
        this.updatedAt = updatedAt;
        this.isActive = isActive;
    }
    
    public void updateTimestamps() {
        this.updatedAt = Instant.now();
    }
    
    // Getters and immutable operations
}

Key Design Decisions

1. Immutable domain objects: Resource instances are immutable to prevent accidental modification after creation
2. Separation of concerns: Validation logic is separated from business logic in distinct interfaces
3. Event-driven architecture: All state changes are published as events for audit and async processing
4. Transaction boundary: Single database transaction per operation ensures consistency
5. Idempotency support: UUID primary keys provide natural idempotency for external systems

5. Concurrency & Thread Safety

Concurrency Model

The system uses a thread-per-request model with a fixed-size thread pool for handling API requests. For I/O-bound operations, we leverage non-blocking I/O using Reactor pattern with Project Reactor or CompletableFuture for async processing. The application runs on a multi-core JVM where each HTTP request is handled by a dedicated thread from the Tomcat/Nginx worker pool.

Race Condition Prevention Strategy

Race conditions are mitigated through:

1. Database-level atomic operations - Using UPDATE statements with WHERE clauses that check current state
2. Optimistic locking via version fields - All entities include a version field for concurrent updates
3. Transaction isolation levels - Read Committed or Repeatable Read depending on consistency needs
4. Application-level synchronization - Critical sections use synchronized blocks or ReentrantLock for cross-service coordination

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected for the following reasons:

• High read-to-write ratio (typical 80:20)
• Low contention scenarios in most use cases
• Better performance under concurrent loads
• Eliminates lock wait times and deadlocks
• Simpler implementation with no explicit lock management

For write operations, we implement optimistic locking using a version field in the entity:

@Entity
@Table(name = "resources")
public class Resource {
    @Id
    private UUID id;
    
    @Version
    private Long version;
    
    private String name;
    private String description;
}

Deadlock Prevention Strategy

Deadlock prevention is achieved through:

1. Consistent ordering of resource acquisition - All operations acquire locks in a predefined order (e.g., by ID)
2. Timeout-based lock acquisition - 5-second timeout for any database locks
3. Hierarchical locking strategy - Primary keys are always acquired in ascending order
4. Application-level lock manager - Custom lock registry that tracks and prevents circular dependencies

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is implemented using the idempotency-key header pattern:

@RestController
public class ResourceController {
    
    @PostMapping("/resources")
    public ResponseEntity<Resource> createResource(
            @RequestHeader("Idempotency-Key") String idempotencyKey,
            @RequestBody CreateResourceRequest request) {
        
        // Check if this idempotency key already exists
        Resource existing = resourceService.findByExternalId(idempotencyKey);
        if (existing != null) {
            return ResponseEntity.ok(existing);
        }
        
        // Process the request with idempotency protection
        Resource resource = resourceService.createResourceWithIdempotency(
            idempotencyKey, request);
        
        return ResponseEntity.status(HttpStatus.CREATED).body(resource);
    }
}

The service layer implements a distributed idempotency store using Redis:

@Service
public class ResourceService {
    
    private final RedisTemplate<String, String> redisTemplate;
    private static final String IDEMPOTENCY_PREFIX = "idempotency:";
    private static final Duration TTL = Duration.ofHours(24);
    
    public Resource createResourceWithIdempotency(String idempotencyKey, 
                                                  CreateResourceRequest request) {
        String key = IDEMPOTENCY_PREFIX + idempotencyKey;
        
        // Try to acquire the idempotency lock with timeout
        String lockValue = UUID.randomUUID().toString();
        Boolean acquired = redisTemplate.opsForValue()
            .setIfAbsent(key, lockValue, TTL);
        
        if (!acquired) {
            // Wait for existing operation to complete or timeout
            return waitForCompletion(idempotencyKey, request);
        }
        
        try {
            // Proceed with resource creation
            return createResource(request);
        } finally {
            // Release the lock after operation completes
            releaseIdempotencyLock(key, lockValue);
        }
    }
    
    private void releaseIdempotencyLock(String key, String lockValue) {
        String script = "if redis.call('GET', KEYS[1]) == ARGV[1] then return redis.call('DEL', KEYS[1]) else return 0 end";
        redisTemplate.execute(
            new DefaultRedisScript<>(script, Long.class),
            Collections.singletonList(key),
            lockValue
        );
    }
}

This approach ensures that duplicate requests with the same idempotency key are handled safely without causing data inconsistencies or duplicate operations. The system maintains a 24-hour TTL for idempotency entries to prevent indefinite accumulation of stale keys.

6. Sequence Diagrams for Key Flows

Happy Path Sequence Diagram

sequenceDiagram
    participant C as Client
    participant API as REST API Gateway
    participant S as Service Layer
    participant R as Repository
    participant DB as Database

    C->>API: POST /api/v1/users
    API->>S: createUser(request)
    S->>R: save(user)
    R->>DB: INSERT INTO users (id, name, email)
    DB-->>R: success
    R-->>S: user
    S-->>API: 201 Created
    API-->>C: {id: "uuid", name: "John Doe", email: "john@example.com"}

Error / Edge Case Sequence Diagram

sequenceDiagram
    participant C as Client
    participant API as REST API Gateway
    participant S as Service Layer
    participant R as Repository
    participant DB as Database

    C->>API: POST /api/v1/users
    API->>S: createUser(request)
    S->>R: save(user)
    R->>DB: INSERT INTO users (id, name, email)
    DB-->>R: DuplicateKeyException
    R-->>S: throw DuplicateUserException
    S-->>API: 409 Conflict
    API-->>C: {error: "User already exists", code: "USER_DUPLICATE"}

Async Flow Sequence Diagram

sequenceDiagram
    participant C as Client
    participant API as REST API Gateway
    participant S as Service Layer
    participant E as Event Publisher
    participant MQ as Message Queue
    participant WC as Worker Consumer

    C->>API: POST /api/v1/users
    API->>S: createUser(request)
    S->>E: publishUserCreatedEvent(user)
    E->>MQ: send user.created event
    MQ-->>WC: consume user.created
    WC->>S: handleUserCreated(user)
    S->>DB: update user status to active
    WC-->>E: event processed successfully

7. Caching Design

Cache Strategy Selection

For this RESTful CRUD API, we'll implement a cache-aside pattern with LRU eviction policy. This approach provides the best balance between performance gains and cache consistency for a typical CRUD workload where reads significantly outnumber writes.

Cache Key Design

Cache keys will follow a namespace strategy to prevent key collisions and enable efficient cache invalidation:

{namespace}:{resource_type}:{id}

Example key format:

user:12345
order:67890
product:abcde

Cache TTL Strategy

Data Type	TTL	Reason
User Profile	5 minutes	Frequent updates, acceptable staleness
Order Details	10 minutes	Moderate update frequency, read-heavy
Product Catalog	30 minutes	Low update frequency, high read volume
Session Tokens	1 hour	Session-based, time-sensitive
Configuration Data	15 minutes	Dynamic config changes, not critical

Cache Eviction Policy

We'll use LRU (Least Recently Used) with a probabilistic early expiry to prevent cache stampedes. When a cache entry is accessed, we'll set a random TTL between 80-100% of the base TTL to spread out cache refreshes.

Cache Stampede Prevention

1. Probabilistic Early Expiry: When a cache miss occurs, we'll set a random TTL (e.g., 80-100% of base TTL) to prevent simultaneous cache refreshes
2. Distributed Locking: For high-traffic resources, we'll implement a distributed lock using Redis SET with NX and EX options to ensure only one cache refresh occurs per resource at a time

Distributed Cache Architecture

We'll use Redis Cluster with:

• 3 master nodes for data distribution
• 3 replica nodes per shard for high availability
• Redis Sentinel for automatic failover
• Cross-zone replication for multi-AZ deployment

Cache Invalidation Strategy

We'll implement event-driven invalidation using a publish-subscribe model:

1. Write Operations: When a resource is created/updated/deleted, publish an event to invalidate the corresponding cache entry
2. Batch Invalidation: For bulk operations, use a batch invalidation key pattern like batch:{resource_type}:{timestamp} to invalidate entire batches efficiently
3. Cache Warm-up: On service startup, warm up frequently accessed resources from database

Cache Implementation Example (Java)

@Component
public class ResourceCache {
    private final RedisTemplate<String, Object> redisTemplate;
    private final ObjectMapper objectMapper;

    public ResourceCache(RedisTemplate<String, Object> redisTemplate, ObjectMapper objectMapper) {
        this.redisTemplate = redisTemplate;
        this.objectMapper = objectMapper;
    }

    public <T> T get(String resourceType, String id, Class<T> type) {
        String key = buildKey(resourceType, id);
        Object cached = redisTemplate.opsForValue().get(key);
        
        if (cached != null) {
            try {
                return objectMapper.convertValue(cached, type);
            } catch (Exception e) {
                // Log and continue with database fetch
                log.warn("Failed to deserialize cached object for key: {}", key);
            }
        }
        
        return null;
    }

    public <T> void put(String resourceType, String id, T value, Duration ttl) {
        String key = buildKey(resourceType, id);
        try {
            Object serializedValue = objectMapper.convertValue(value, Map.class);
            redisTemplate.opsForValue().set(key, serializedValue, ttl);
        } catch (Exception e) {
            log.error("Failed to cache object for key: {}", key, e);
        }
    }

    public void invalidate(String resourceType, String id) {
        String key = buildKey(resourceType, id);
        redisTemplate.delete(key);
    }

    private String buildKey(String resourceType, String id) {
        return String.format("%s:%s:%s", "resource", resourceType, id);
    }

    public void invalidateBatch(String resourceType, String timestamp) {
        String batchKey = String.format("batch:%s:%s", resourceType, timestamp);
        Set<String> keys = redisTemplate.keys(batchKey + "*");
        if (keys != null && !keys.isEmpty()) {
            redisTemplate.delete(keys);
        }
    }
}

Cache Read-Through Pattern

@Service
public class UserService {
    private final UserRepository userRepository;
    private final ResourceCache cache;

    public User getUserById(String userId) {
        // Try cache first
        User cached = cache.get("user", userId, User.class);
        if (cached != null) {
            return cached;
        }

        // Cache miss - fetch from database
        User user = userRepository.findById(userId);
        if (user != null) {
            // Cache for future reads
            cache.put("user", userId, user, Duration.ofMinutes(5));
        }
        
        return user;
    }
}

Cache Write-Through Pattern

@Service
public class OrderService {
    private final OrderRepository orderRepository;
    private final ResourceCache cache;

    @Transactional
    public Order createOrder(Order order) {
        // Create in database
        Order created = orderRepository.save(order);
        
        // Update cache with new value
        cache.put("order", created.getId(), created, Duration.ofMinutes(10));
        
        return created;
    }

    @Transactional
    public Order updateOrder(String orderId, OrderUpdateRequest request) {
        // Update in database
        Order updated = orderRepository.update(orderId, request);
        
        // Invalidate cache to force refresh on next read
        cache.invalidate("order", orderId);
        
        return updated;
    }
}

Cache Hit/Miss Metrics

We'll track cache performance with Prometheus metrics:

• cache_hits_total{type="user", operation="get"}
• cache_misses_total{type="user", operation="get"}
• cache_evictions_total{type="user"}
• cache_latency_seconds{type="user", operation="get"}

This caching strategy provides high availability for frequently accessed resources while maintaining reasonable consistency through event-driven invalidation and proper TTL management.

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and database operations. Each entity has its own repository interface with standard CRUD operations.

public interface UserRepository extends Repository<User, UUID> {
    Optional<User> findByEmail(String email);
    List<User> findByStatus(UserStatus status);
    Page<User> findByStatusWithPagination(UserStatus status, Pageable pageable);
}

public interface OrderRepository extends Repository<Order, UUID> {
    List<Order> findByUserId(UUID userId);
    Optional<Order> findByOrderIdAndUserId(String orderId, UUID userId);
    Page<Order> findByUserIdWithPagination(UUID userId, Pageable pageable);
}

ORM vs Raw Query Decisions

For this design, we use JPA/Hibernate as the ORM for its maturity, ecosystem support, and ability to handle complex object-relational mapping. Raw queries are used for performance-critical operations or when ORM doesn't support specific database features.

// Using JPA for standard operations
@Query("SELECT u FROM User u WHERE u.status = :status")
List<User> findByStatus(@Param("status") UserStatus status);

// Using native query for complex analytics
@Query(value = "SELECT u.id, u.email, COUNT(o.id) as order_count FROM users u LEFT JOIN orders o ON u.id = o.user_id WHERE u.status = :status GROUP BY u.id, u.email", nativeQuery = true)
List<UserOrderCountProjection> findUserOrderCounts(@Param("status") UserStatus status);

Read vs Write Separation

We implement CQRS (Command Query Responsibility Segregation) for read-heavy workloads to optimize performance:

// Write model - used for mutations
@Repository
public class UserWriteRepository {
    @PersistenceContext
    private EntityManager entityManager;
    
    public void save(User user) {
        entityManager.merge(user);
    }
}

// Read model - optimized for queries
@Repository
public class UserReadRepository {
    @PersistenceContext
    private EntityManager entityManager;
    
    @Query("SELECT u FROM UserReadModel u WHERE u.email = :email")
    Optional<UserReadModel> findByEmail(@Param("email") String email);
}

Connection Pool Configuration

Connection pooling is configured with HikariCP for optimal performance:

spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      validation-timeout: 5000

N+1 Query Prevention Strategy

We implement eager loading and batch fetching to prevent N+1 queries:

@Entity
public class User {
    @OneToMany(mappedBy = "user", fetch = FetchType.LAZY)
    @BatchSize(size = 20)
    private List<Order> orders;
    
    @ManyToOne(fetch = FetchType.EAGER)
    @JoinColumn(name = "department_id")
    private Department department;
}

// In repository query with eager loading
@Query("SELECT u FROM User u JOIN FETCH u.orders o WHERE u.id = :userId")
Optional<User> findWithOrders(@Param("userId") UUID userId);

Batch Operation Design

Batch operations are implemented for bulk data processing:

@Repository
public class UserBatchRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    public void batchInsert(List<User> users) {
        int batchSize = 1000;
        for (int i = 0; i < users.size(); i += batchSize) {
            int end = Math.min(i + batchSize, users.size());
            List<User> batch = users.subList(i, end);
            
            for (User user : batch) {
                entityManager.persist(user);
            }
            
            entityManager.flush();
            entityManager.clear();
        }
    }
    
    public void batchUpdate(List<User> users) {
        int batchSize = 1000;
        for (int i = 0; i < users.size(); i += batchSize) {
            int end = Math.min(i + batchSize, users.size());
            List<User> batch = users.subList(i, end);
            
            for (User user : batch) {
                entityManager.merge(user);
            }
            
            entityManager.flush();
            entityManager.clear();
        }
    }
}

Transaction Management

All repository operations are wrapped in proper transaction boundaries:

@Repository
@Transactional
public class UserRepositoryImpl implements UserRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    @Override
    public User save(User user) {
        if (user.getId() == null) {
            entityManager.persist(user);
        } else {
            entityManager.merge(user);
        }
        return user;
    }
    
    @Override
    public Optional<User> findById(UUID id) {
        return Optional.ofNullable(entityManager.find(User.class, id));
    }
    
    @Override
    public void delete(UUID id) {
        User user = entityManager.find(User.class, id);
        if (user != null) {
            entityManager.remove(user);
        }
    }
}

Read-Through Cache Integration

Repository implementations integrate with the caching layer:

@Repository
public class CachedUserRepository implements UserRepository {
    
    private final UserCache userCache;
    private final UserRepository delegate;
    
    @Override
    public Optional<User> findById(UUID id) {
        User cached = userCache.get(id);
        if (cached != null) {
            return Optional.of(cached);
        }
        
        Optional<User> user = delegate.findById(id);
        user.ifPresent(u -> userCache.put(u.getId(), u));
        return user;
    }
    
    @Override
    public User save(User user) {
        User saved = delegate.save(user);
        userCache.put(saved.getId(), saved);
        return saved;
    }
}

Error Handling in Repository Layer

Repository layer handles database-specific exceptions:

@Repository
public class UserRepositoryImpl implements UserRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    public User save(User user) {
        try {
            return entityManager.merge(user);
        } catch (PersistenceException e) {
            throw new DataAccessException("Failed to save user", e);
        }
    }
    
    public Optional<User> findById(UUID id) {
        try {
            return Optional.ofNullable(entityManager.find(User.class, id));
        } catch (Exception e) {
            throw new DataAccessException("Failed to find user by ID", e);
        }
    }
}

Pagination and Sorting Support

Repository methods support pagination and sorting:

@Repository
public class OrderRepositoryImpl implements OrderRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    public Page<Order> findByUserIdWithPagination(UUID userId, Pageable pageable) {
        CriteriaBuilder cb = entityManager.getCriteriaBuilder();
        CriteriaQuery<Order> query = cb.createQuery(Order.class);
        Root<Order> root = query.from(Order.class);
        
        Predicate predicate = cb.equal(root.get("userId"), userId);
        query.where(predicate);
        
        if (pageable.getSort().isSorted()) {
            List<OrderExpression> orderExpressions = new ArrayList<>();
            pageable.getSort().forEach(order -> {
                if (order.isAscending()) {
                    orderExpressions.add(root.get(order.getProperty()).asc());
                } else {
                    orderExpressions.add(root.get(order.getProperty()).desc());
                }
            });
            query.orderBy(orderExpressions);
        }
        
        TypedQuery<Order> typedQuery = entityManager.createQuery(query);
        typedQuery.setFirstResult((int) pageable.getOffset());
        typedQuery.setMaxResults(pageable.getPageSize());
        
        List<Order> orders = typedQuery.getResultList();
        long total = countByUserId(userId);
        
        return new PageImpl<>(orders, pageable, total);
    }
    
    private long countByUserId(UUID userId) {
        CriteriaBuilder cb = entityManager.getCriteriaBuilder();
        CriteriaQuery<Long> countQuery = cb.createQuery(Long.class);
        Root<Order> countRoot = countQuery.from(Order.class);
        countQuery.select(cb.count(countRoot));
        countQuery.where(cb.equal(countRoot.get("userId"), userId));
        
        return entityManager.createQuery(countQuery).getSingleResult();
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The CRUD API for a resource will maintain synchronous operations for all create, read, update, and delete operations to ensure immediate consistency and response times. Asynchronous processing will be introduced only for secondary effects that don't block the primary operation, such as:

• Audit logging
• Notification sending (email/SMS)
• Data synchronization to external systems
• Analytics event publishing

This approach ensures that the core CRUD operations remain fast and predictable, while allowing for scalable background processing of non-critical side effects.

Message schema design with sample payload

{
  "version": "1.0",
  "event_type": "resource.created",
  "timestamp": "2023-10-15T14:30:00Z",
  "resource_id": "uuid-12345",
  "resource_type": "user",
  "data": {
    "id": "uuid-12345",
    "name": "John Doe",
    "email": "john.doe@example.com"
  },
  "metadata": {
    "source_service": "user-service",
    "correlation_id": "corr-abcde"
  }
}

Dead letter queue strategy and poison message handling

Poison messages will be handled through a dead-letter queue (DLQ) strategy:

• Messages that fail more than 3 times will be moved to a DLQ topic
• DLQ messages will have a separate retention policy (e.g., 30 days)
• Failed messages will be logged with full context for manual inspection
• A monitoring alert will be triggered when DLQ queue size exceeds 100 messages

At-least-once vs. exactly-once delivery

At-least-once delivery is selected for this feature because:

• The resource CRUD operations are idempotent in nature
• The system can handle duplicate events gracefully through idempotency keys
• Simpler implementation and better performance characteristics
• The business logic can safely process the same event multiple times

Retry policy for consumers

Retry configuration:

• Maximum retries: 3 attempts
• Backoff strategy: Exponential with jitter (1s, 2s, 4s)
• Retryable errors: Network timeouts, service unavailability
• Non-retryable errors: Validation failures, business rule violations

Outbox pattern for reliable event publishing

The outbox pattern will be implemented to ensure exactly-once delivery of events:

// OutboxEvent entity
@Entity
@Table(name = "outbox_events")
public class OutboxEvent {
    @Id
    private UUID id;
    
    @Column(name = "event_type")
    private String eventType;
    
    @Column(name = "payload")
    private String payload;
    
    @Column(name = "status")
    private OutboxStatus status = OutboxStatus.PENDING;
    
    @Column(name = "created_at")
    private LocalDateTime createdAt;
    
    @Column(name = "processed_at")
    private LocalDateTime processedAt;
}

Sequence diagram for async flows

sequenceDiagram
    participant Client
    participant Service
    participant Repository
    participant OutboxRepo
    participant MessageBroker
    participant Consumer

    Client->>Service: POST /resources
    Service->>Repository: save(resource)
    Repository->>OutboxRepo: insert(outboxEvent)
    OutboxRepo-->>Service: event persisted
    Service->>Client: 201 Created
    loop Process Outbox Events
        Service->>MessageBroker: publish(event)
        MessageBroker->>Consumer: consume(event)
        Consumer->>Consumer: processEvent()
        Consumer->>Consumer: updateProcessedAt()
    end

10. Error Handling Design

Error Classification Hierarchy

Errors are categorized into three main levels to guide handling and propagation:

1. Domain Errors - Business logic violations (e.g., ResourceNotFoundException, ValidationException)
2. Infrastructure Errors - System-level issues (e.g., DatabaseConnectionException, ExternalServiceTimeout)
3. Validation Errors - Input validation failures (e.g., BadRequestException, UnauthorizedException)

Error Propagation Strategy

• Domain and Validation errors are handled at the service layer and converted to appropriate HTTP responses
• Infrastructure errors bubble up to the controller layer for consistent error response formatting
• Global exception handlers catch unhandled exceptions and log them before returning generic 500 errors to prevent leaking internal details

Retry Strategy with Exponential Backoff and Jitter

For transient infrastructure errors (e.g., database timeouts, external service failures):

public class RetryConfig {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;
    
    public static long calculateDelay(int attempt) {
        long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempt));
        double jitter = ThreadLocalRandom.current().nextDouble(-JITTER_FACTOR, JITTER_FACTOR);
        return (long) (delay * (1 + jitter));
    }
}

Circuit Breaker Design

For external service calls using the Hystrix pattern:

public class CircuitBreaker {
    private final String name;
    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private final int failureThreshold = 5;
    private final long timeoutMs = 30000;
    private final long resetTimeoutMs = 60000;
    
    public <T> T execute(Supplier<T> operation) {
        if (state == CircuitState.OPEN) {
            throw new CircuitOpenException("Circuit is open for " + name);
        }
        
        try {
            T result = operation.get();
            failureCount = 0;
            state = CircuitState.CLOSED;
            return result;
        } catch (Exception e) {
            failureCount++;
            if (failureCount >= failureThreshold) {
                state = CircuitState.OPEN;
                // Schedule reset
                ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
                scheduler.schedule(() -> state = CircuitState.HALF_OPEN, resetTimeoutMs, TimeUnit.MILLISECONDS);
            }
            throw e;
        }
    }
}

Fallback Behavior Design

Fallbacks are implemented for external dependencies:

public class ExternalServiceFallback {
    public List<Resource> getResourceListFallback(List<String> ids, Throwable cause) {
        // Log fallback trigger
        logger.warn("Falling back to cached data for resources: {}", ids, cause);
        
        // Return cached or default values
        return Collections.emptyList();
    }
}

Error Logging Format

Structured logging with required fields:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "trace_id": "a1b2c3d4e5f6",
  "span_id": "f6e5d4c3b2a1",
  "user_id": "user-123",
  "entity_id": "resource-456",
  "error_code": "RESOURCE_NOT_FOUND",
  "error_message": "Resource with ID 'resource-456' not found",
  "error_type": "DomainError",
  "stack_trace": "...",
  "http_method": "GET",
  "http_path": "/api/resources/resource-456",
  "request_id": "req-789"
}

Alerting Thresholds

Critical alerts are triggered for:

• 5xx errors: > 10 occurrences in 5 minutes
• Database timeouts: > 5 occurrences in 1 minute
• External service failures: > 3 occurrences in 2 minutes
• Circuit breaker opens: Immediate alert on first open

Error Response Format

Standardized JSON response for all errors:

{
  "error": {
    "code": "RESOURCE_NOT_FOUND",
    "message": "Resource with ID 'resource-456' not found",
    "details": {
      "resourceId": "resource-456"
    },
    "timestamp": "2023-10-27T14:30:45.123Z",
    "requestId": "req-789"
  }
}

Custom Exception Classes

public class BaseException extends RuntimeException {
    private final String errorCode;
    private final Map<String, Object> details;

    public BaseException(String errorCode, String message) {
        super(message);
        this.errorCode = errorCode;
        this.details = Collections.emptyMap();
    }

    public BaseException(String errorCode, String message, Map<String, Object> details) {
        super(message);
        this.errorCode = errorCode;
        this.details = details != null ? details : Collections.emptyMap();
    }

    // Getters
}

public class ResourceNotFoundException extends BaseException {
    public ResourceNotFoundException(String resourceId) {
        super("RESOURCE_NOT_FOUND", 
            "Resource with ID '" + resourceId + "' not found",
            Collections.singletonMap("resourceId", resourceId));
    }
}

Exception Handler Implementation

@RestControllerAdvice
public class GlobalExceptionHandler {
    private static final Logger logger = LoggerFactory.getLogger(GlobalExceptionHandler.class);

    @ExceptionHandler(BaseException.class)
    public ResponseEntity<ErrorResponse> handleDomainError(BaseException ex) {
        logger.warn("Domain error occurred: {}", ex.getMessage(), ex);
        
        ErrorResponse errorResponse = new ErrorResponse(
            ex.getErrorCode(),
            ex.getMessage(),
            ex.getDetails(),
            UUID.randomUUID().toString()
        );
        
        return ResponseEntity.status(HttpStatus.NOT_FOUND).body(errorResponse);
    }

    @ExceptionHandler(Exception.class)
    public ResponseEntity<ErrorResponse> handleGenericError(Exception ex) {
        logger.error("Unexpected error occurred", ex);
        
        ErrorResponse errorResponse = new ErrorResponse(
            "INTERNAL_ERROR",
            "An internal error occurred",
            Collections.emptyMap(),
            UUID.randomUUID().toString()
        );
        
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(errorResponse);
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using a combination of:

• Bean Validation (JSR 380) for structured validation rules
• Custom sanitization filters to remove or escape potentially harmful content
• Regular expressions for format validation (e.g., email, phone number)
• Length limits to prevent buffer overflow attacks

public class ResourceRequest {
    @NotBlank(message = "Name is required")
    @Size(max = 255, message = "Name must be less than 256 characters")
    private String name;
    
    @Email(message = "Invalid email format")
    private String email;
    
    // Sanitize input using Apache Commons Text
    public void sanitize() {
        this.name = StringEscapeUtils.escapeHtml4(this.name);
        this.email = StringEscapeUtils.escapeHtml4(this.email);
    }
}

SQL Injection Prevention

• Prepared statements for all database queries using JDBC or JPA
• Parameterized queries with no string concatenation in SQL
• Whitelisting of column names and table names where dynamic queries are needed

Authentication Middleware Design

• JWT-based authentication with 2-hour expiration for access tokens
• OAuth2 integration for third-party authentication (Google, GitHub)
• Authentication filter that intercepts all requests and validates JWT tokens
• Refresh token rotation to enhance security

Authorization Enforcement (RBAC)

• Role-based access control model with roles: ADMIN, USER, GUEST
• Permission checking at service layer using Spring Security annotations:

@PreAuthorize("hasRole('ADMIN') or @resourcePermissionChecker.canAccess(resourceId)")
public Resource getResource(Long resourceId) {
    // implementation
}

• Resource-level permissions where applicable (e.g., users can only access their own resources)

Sensitive Data Encryption

• At-rest encryption: AES-256 with AWS KMS for database and file storage
• In-transit encryption: TLS 1.3 for all API communications (enforced via HTTPS)
• Secrets management: AWS Secrets Manager for storing database credentials and API keys

PII Handling and Masking Strategy

• PII fields: email, phone, address are encrypted at rest
• Masking in logs: PII fields are masked (e.g., "user@**.com") in logs and monitoring
• Data retention: PII data is automatically deleted after 36 months of inactivity

Secrets Management

• AWS Secrets Manager for storing:
  • Database connection strings
  • API keys for external services
  • JWT signing keys
• Rotation policy: Secrets rotated every 90 days or upon compromise

Audit Logging Design

• Events logged: CREATE, UPDATE, DELETE, ACCESS, AUTH_FAILURE
• Log format:

{
  "timestamp": "2023-10-15T14:30:00Z",
  "trace_id": "abc123def456",
  "user_id": "user-123",
  "action": "CREATE_RESOURCE",
  "resource_id": "res-456",
  "ip_address": "192.168.1.100",
  "user_agent": "Mozilla/5.0...",
  "status": "SUCCESS"
}

• Retention policy: Audit logs retained for 2 years with daily backups to S3
• Compliance: Logs are structured to support GDPR and SOC2 compliance requirements

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

• Service Layer: 85% code coverage with focus on business logic, validation, and error handling paths
• Repository Layer: 90% coverage for all data access methods with mock database interactions
• Controller Layer: 80% coverage for API endpoints, including edge case handling and error responses
• Domain Objects: 95% coverage for value objects, entities, and their validation logic
• Integration Points: 70% coverage for external service interactions (e.g., cache, message broker)
• Critical Path Operations: 100% coverage for core operations like create, update, and delete

Integration Testing Strategy

• Database Integration Tests: Use testcontainers to spin up real PostgreSQL instances with pre-seeded test data
• External Service Mocking: Employ WireMock for external API integrations (e.g., payment gateway, notification service)
• Caching Layer Tests: Validate cache behavior with embedded Redis instances to ensure proper caching and eviction
• Message Broker Integration: Use embedded Kafka or RabbitMQ containers for testing async flows and event handling
• End-to-End Test Scenarios: Execute full request-response cycles with real HTTP clients against test deployments

Contract Testing

• Pact Broker Integration: Implement contract testing using Pact for all external API dependencies (e.g., payment provider, auth service)
• Consumer-Provider Contract Verification: Define contracts for each external dependency to ensure compatibility between services
• Versioned Contract Testing: Maintain versioned contract files in source control to support CI/CD pipeline validation

Performance Testing Tool and Key Scenarios

• Tool: k6 (JavaScript-based load testing framework) for high-fidelity performance simulation
• Key Scenarios:
  • Concurrent CRUD operations (1000 concurrent users)
  • Peak load stress test (5000 concurrent requests)
  • Response time SLA validation (p95 < 200ms for CRUD operations)
  • Memory and CPU usage under load (max 80% utilization)
  • Database connection pool exhaustion testing

Test Data Management Strategy

• Fixtures: Maintain JSON/YAML fixtures for common test data sets (e.g., user profiles, resource configurations)
• Factories: Implement factory classes using libraries like FactoryBot or custom Java builders for generating test data
• Seeding Strategy: Use database migration scripts with seed data to populate test environments consistently
• Test Data Isolation: Ensure each test runs in isolated database transactions with automatic rollback after completion
• Data Generation Patterns: Implement randomized data generation for edge case testing (e.g., boundary values, invalid inputs)

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective search, filtering, and analysis:

{
  "timestamp": "2023-10-15T14:30:45.123Z",
  "level": "INFO|WARN|ERROR",
  "service": "user-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user-12345",
  "entity_id": "resource-98765",
  "request_id": "req-abcde12345",
  "method": "GET",
  "path": "/api/v1/users/12345",
  "status_code": 200,
  "duration_ms": 150,
  "message": "User retrieved successfully",
  "error_code": "USER_NOT_FOUND",
  "error_message": "User with ID 12345 not found",
  "stack_trace": "...",
  "http_headers": {
    "authorization": "Bearer ***",
    "content-type": "application/json"
  }
}

PII masking rules:

• User IDs, email addresses, phone numbers must be masked in logs
• Authorization headers must have token values redacted
• All sensitive fields must be marked with *** or replaced with hash values

Distributed Tracing Instrumentation

Using OpenTelemetry for distributed tracing:

Custom Spans to Add:

• http.request - For all incoming HTTP requests
• db.query - For database operations
• external.api.call - For external service calls
• cache.hit/miss - For cache operations

Attributes to Capture:

// Example span attributes for a user retrieval operation
span.setAttribute("user.id", userId);
span.setAttribute("db.query", "SELECT * FROM users WHERE id = ?");
span.setAttribute("db.rows.returned", 1);
span.setAttribute("cache.hit", false);

Metrics Design

RED Method Metrics:

Metric Name	Type	Labels	Description
http_requests_total	Counter	{method, path, status_code}	Total HTTP requests
http_request_duration_seconds	Histogram	{method, path}	Request processing time
db_query_duration_seconds	Histogram	{query_type, db_name}	Database query latency
cache_hits_total	Counter	{cache_type, operation}	Cache hit count
external_api_calls_total	Counter	{service, endpoint, status_code}	External API call counts

Service-Level Metrics:

• service.up (Gauge): Service health status
• active_requests (Gauge): Concurrent active requests
• error_rate (Gauge): Error rate per second

Health Check Endpoint Design

// Health check endpoints with different levels of checks
@RestController
@RequestMapping("/health")
public class HealthController {
    
    @GetMapping("/live")
    public ResponseEntity<Map<String, Object>> live() {
        // Basic service liveness - always returns 200
        return ResponseEntity.ok(Map.of("status", "alive"));
    }
    
    @GetMapping("/ready")
    public ResponseEntity<Map<String, Object>> ready() {
        // Service readiness - checks database connectivity and cache
        boolean dbReady = databaseService.isReady();
        boolean cacheReady = cacheService.isReady();
        
        if (dbReady && cacheReady) {
            return ResponseEntity.ok(Map.of("status", "ready"));
        }
        return ResponseEntity.status(503).body(Map.of("status", "not ready"));
    }
    
    @GetMapping("/startup")
    public ResponseEntity<Map<String, Object>> startup() {
        // Startup health - checks all dependencies
        return ResponseEntity.ok(Map.of(
            "status", "healthy",
            "dependencies", Map.of(
                "database", "connected",
                "cache", "connected",
                "auth_service", "reachable"
            )
        ));
    }
}

Alerting Thresholds

Critical Alerts (Escalation Path: On-call engineer → Team Lead → Engineering Manager):

Metric	Threshold	Alert Level	Escalation
http_request_duration_seconds (p99)	> 500ms	Critical	On-call engineer
error_rate (per minute)	> 5%	Critical	On-call engineer
db_query_duration_seconds (p99)	> 200ms	Critical	On-call engineer
cache_miss_rate	> 30%	Warning	Team Lead

Warning Alerts (Escalation Path: On-call engineer → Team Lead):

Metric	Threshold	Alert Level
http_request_duration_seconds (p99)	> 200ms	Warning
error_rate (per minute)	> 1%	Warning
cache_hit_rate	< 70%	Warning

Monitoring Dashboard:

• Real-time request rate and latency dashboard
• Error rate and error distribution charts
• Cache performance metrics
• Database query performance trends

All alerts must include contextual information including:

• Service name and instance ID
• Affected resource or user ID when applicable
• Error stack traces for error alerts
• Correlation IDs to trace requests across services

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• API versioning strategy: Should we use header-based, URI-based, or query parameter-based versioning? Each has different implications for client migration and URL stability.
• Soft delete vs. hard delete: The decision between soft deletes (flagging records as deleted) and hard deletes (physically removing data) affects auditability, storage costs, and compliance requirements.
• Data retention policy: How long should we retain historical data for analytics or compliance? This impacts storage costs and system complexity.
• Rate limiting granularity: Should rate limits be applied per user, per IP, or per API key? This affects both security and user experience.
• Caching strategy: Should we use a centralized cache or local in-memory caching? This decision affects consistency, scalability, and infrastructure costs.

Known trade-offs and justifications

• Consistency vs. availability: Using eventual consistency for read-heavy operations trades immediate consistency for better availability and performance, particularly under high load.
• Database normalization vs. denormalization: Normalized schemas reduce data redundancy and maintain consistency but increase join complexity; denormalized schemas improve read performance at the cost of write complexity.
• Monolithic vs. microservices: A monolithic approach simplifies development and deployment but can become unwieldy as the system grows, while microservices offer better scalability but introduce complexity in distributed systems.
• Event-driven vs. synchronous processing: Event-driven architecture improves decoupling and scalability but introduces complexity in debugging and ensuring delivery guarantees.
• Read-heavy vs. write-heavy workload: Optimizing for read-heavy workloads (e.g., caching, materialized views) can hurt write performance and vice versa.

Risks and mitigation strategies

• Data consistency violations: Risk of data inconsistency in distributed systems. Mitigation includes implementing proper transaction boundaries, using eventual consistency patterns, and maintaining audit trails.
• Cache stampede: Risk of cache misses causing a thundering herd of requests to the database. Mitigation includes probabilistic early expiry and distributed locking mechanisms.
• API rate limiting bypass: Risk of malicious or misconfigured clients exceeding rate limits. Mitigation includes robust rate-limiting middleware and client-side retry strategies.
• Data loss: Risk of data loss due to hardware failures or accidental deletion. Mitigation includes regular backups, replication strategies, and soft-delete mechanisms.
• Security vulnerabilities: Risk of unauthorized access or data breaches. Mitigation includes proper authentication, authorization, input validation, and encryption at rest and in transit.

Technical debt acknowledged

• Monolithic architecture: The current design assumes a monolithic structure, which may require future refactoring into microservices as the system scales.
• Hardcoded configuration values: Some configuration parameters are currently hardcoded, which reduces flexibility and requires code changes for environment-specific settings.
• Limited audit logging: Audit logging is implemented at a basic level and may not capture all necessary details for compliance or forensic analysis.
• No circuit breaker pattern: The system does not currently implement circuit breakers for external dependencies, which could lead to cascading failures during outages.
• Basic error handling: Error handling is centralized but lacks granular error categorization and specific error response formats for different failure scenarios.