CDN Cache Invalidation System Design

Comprehensive low-level design for a scalable CDN cache invalidation system with real-time invalidation and cache consistency strategies.

PublishedAugust 24, 2025
Length8,366 words
Reading time42 min read

Backend LLD

A CDN cache invalidation system ensures content consistency across distributed edge nodes by efficiently managing cache expiration and invalidation requests while maintaining low latency and high availability for global content delivery.

1. Problem Statement & Requirements

Clarifying questions

What scale and traffic volume are we targeting?
- Expected concurrent requests, peak throughput, and data volume (e.g., 100GB/s, 1M requests/sec).
What consistency model is required?
- Should cache invalidation be near real-time (sub-second) or can it tolerate delays (e.g., 5-10 minutes)?
What are the key entities and their relationships?
- Are we dealing with content (files, assets) or specific resources (URLs, paths)? Do they have versioning or tags?
What is the read/write ratio for content?
- Is it predominantly reads (e.g., 95% cache hits) or more dynamic content updates?
What are the latency sensitivity requirements?
- Is sub-second response time critical or can it be up to a few minutes?
What authentication and authorization models are in place?
- Do we need to support different access levels (public, private, authenticated) for content?
Are there external integrations or dependencies?
- Do we integrate with content providers, CDNs, or other systems like S3, Cloudflare, or Akamai?

Functional requirements

Cache Invalidation Triggering: The system must support triggering cache invalidation via API, webhooks, or events.
Content Tagging: Support content tagging to invalidate groups of assets (e.g., invalidate all images tagged with user-123).
URL-based Invalidation: Invalidate specific content by URL or path (e.g., /images/avatar.jpg).
Batch Invalidation: Support batch invalidation of multiple URLs or tags in a single request.
Invalidate by TTL: Automatically invalidate content based on time-to-live (TTL) configuration.
Invalidate by Origin Change: Invalidate content when the source of truth (e.g., S3 object) is updated.
Invalidate by Content Version: Invalidate content when a new version of the same resource is published.

Non-functional requirements

Latency Targets:
- p50: < 100ms for invalidation requests
- p95: < 500ms
- p99: < 2s
Throughput:
- Handle up to 10,000 invalidation requests per second
Availability:
- 99.9% uptime for invalidation service
Scalability:
- Support horizontal scaling to handle traffic spikes

Constraints and assumptions

Assumption: The system will be part of a larger CDN ecosystem where content is stored in distributed storage (e.g., S3, GCS).
Assumption: The system will support multiple CDN providers (Cloudflare, Akamai, AWS CloudFront).
Constraint: Cache invalidation must be idempotent to prevent duplicate invalidations.
Constraint: The system will not support real-time cache purging but can handle near-real-time invalidation (within 5 seconds).
Constraint: All content paths are assumed to be normalized and follow a consistent format (e.g., /images/avatar.jpg).

Out of scope

Content delivery: The system does not handle actual content delivery or caching logic.
Content upload or storage: Content storage and upload are handled by external systems (e.g., S3, GCS).
User or role management: Authentication and authorization for invalidation are handled by external systems.
Content compression or transformation: The system does not handle content transformations like image resizing or compression.
Rate limiting at CDN edge: Rate limiting for client requests is handled by the CDN providers, not this system.
Multi-region support: Multi-region cache invalidation is out of scope for the initial design.

2. API Design

API Style Selection

REST is selected for its simplicity, wide adoption, and clear HTTP semantics. It provides well-defined endpoints with standard CRUD operations that align with the cache invalidation use case. While GraphQL offers flexibility, it's overkill for this focused system. gRPC would be suitable if we were building a microservice-to-microservice communication layer, but the primary consumers are likely to be admin UIs and external systems that prefer REST.

Endpoint Definitions

Method	Path	Description	Auth
GET	`/v1/cache/invalidations`	List all cache invalidation requests	JWT
POST	`/v1/cache/invalidations`	Create a new cache invalidation request	JWT
GET	`/v1/cache/invalidations/{id}`	Get specific invalidation request by ID	JWT
DELETE	`/v1/cache/invalidations/{id}`	Cancel pending invalidation request	JWT
POST	`/v1/cache/invalidations/{id}/retry`	Retry failed invalidation request	JWT
GET	`/v1/cache/invalidations/{id}/status`	Get status of invalidation request	JWT

Request and Response Schemas

Create Invalidations Request

{
  "urls": ["https://cdn.example.com/image1.jpg", "https://cdn.example.com/image2.png"],
  "tags": ["images", "user-content"],
  "invalidateAll": false,
  "ttlSeconds": 3600
}

Create Invalidations Response

{
  "id": "inv-1234567890",
  "status": "pending",
  "urls": ["https://cdn.example.com/image1.jpg", "https://cdn.example.com/image2.png"],
  "tags": ["images", "user-content"],
  "createdAt": "2023-10-27T10:00:00Z",
  "expiresAt": "2023-10-27T11:00:00Z",
  "requester": "admin@example.com"
}

Error Response

{
  "error": {
    "code": "INVALID_URL",
    "message": "One or more URLs are invalid",
    "details": [
      {
        "url": "https://cdn.example.com/invalid.jpg",
        "reason": "URL does not match expected pattern"
      }
    ]
  }
}

HTTP Status Codes

200 OK - Successful GET, PUT, PATCH operations
201 Created - Successful POST operations
204 No Content - Successful DELETE operations
400 Bad Request - Invalid request body, missing required fields, invalid URLs
401 Unauthorized - Missing or invalid authentication token
403 Forbidden - Insufficient permissions to perform operation
404 Not Found - Invalid invalidation request ID or resource not found
422 Unprocessable Entity - Validation errors, invalid URL format
429 Too Many Requests - Rate limit exceeded
500 Internal Server Error - Unexpected server error

API Versioning Strategy

Versioned using path prefix: /v1/. This allows for backward compatibility and migration paths. Future versions can be introduced at /v2/ without breaking existing clients.

Pagination Design

Cursor-based pagination is used for listing invalidations to handle large datasets efficiently and provide consistent results even when data changes during pagination.

Authentication and Authorization

All endpoints require JWT-based authentication. RBAC is implemented where:

admin role can create, read, update, delete all invalidations
user role can only manage their own invalidation requests
read-only role can only read invalidations

Rate Limiting

Rate limiting per endpoint:

POST /v1/cache/invalidations: 100 requests per minute
GET /v1/cache/invalidations: 500 requests per minute
DELETE /v1/cache/invalidations/{id}: 50 requests per minute
POST /v1/cache/invalidations/{id}/retry: 20 requests per minute

Rate limiting is enforced using a token bucket algorithm with distributed Redis storage to handle multi-instance deployments.

Idempotency Design

For mutation endpoints, idempotency keys are supported via X-Idempotency-Key header:

// Example usage in controller
String idempotencyKey = request.getHeader("X-Idempotency-Key");
if (idempotencyKey != null) {
    // Check if operation was already processed
    if (idempotencyService.isProcessed(idempotencyKey)) {
        return existingResponse;
    }
}

The system stores the result of each idempotent operation and returns the same response for subsequent identical requests within a time window (e.g., 5 minutes).

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    CONTENT_ITEM ||--o{ INVALIDATION_REQUEST : "has"
    CONTENT_ITEM ||--o{ CACHE_INVALIDATION_LOG : "logged"
    INVALIDATION_REQUEST ||--o{ INVALIDATION_REQUEST_ENTRY : "contains"
    INVALIDATION_REQUEST ||--o{ CACHE_INVALIDATION_EVENT : "triggers"
    
    CONTENT_ITEM {
        string id PK
        string url
        string content_type
        int size_bytes
        timestamp created_at
        timestamp updated_at
        bool is_deleted
    }
    
    INVALIDATION_REQUEST {
        string id PK
        string requester_id
        string request_type
        timestamp created_at
        timestamp processed_at
        string status
    }
    
    INVALIDATION_REQUEST_ENTRY {
        string id PK
        string invalidation_request_id FK
        string content_item_id FK
        timestamp created_at
    }
    
    CACHE_INVALIDATION_EVENT {
        string id PK
        string content_item_id FK
        string event_type
        timestamp created_at
        string status
    }
    
    CACHE_INVALIDATION_LOG {
        string id PK
        string content_item_id FK
        string event_id FK
        timestamp created_at
        string status
        string reason
    }

Table Schema Definitions

`content_items` table

CREATE TABLE content_items (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    url TEXT UNIQUE NOT NULL,
    content_type VARCHAR(255),
    size_bytes BIGINT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    is_deleted BOOLEAN DEFAULT FALSE,
    
    INDEX idx_content_items_url (url),
    INDEX idx_content_items_created_at (created_at),
    INDEX idx_content_items_updated_at (updated_at)
);

`invalidation_requests` table

CREATE TABLE invalidation_requests (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    requester_id VARCHAR(255) NOT NULL,
    request_type VARCHAR(50) NOT NULL, -- 'PURGE', 'REFRESH', 'BATCH'
    status VARCHAR(20) DEFAULT 'PENDING',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    processed_at TIMESTAMP,
    
    INDEX idx_invalidation_requests_requester (requester_id),
    INDEX idx_invalidation_requests_status (status),
    INDEX idx_invalidation_requests_created_at (created_at)
);

`invalidation_request_entries` table

CREATE TABLE invalidation_request_entries (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    invalidation_request_id UUID NOT NULL REFERENCES invalidation_requests(id) ON DELETE CASCADE,
    content_item_id UUID NOT NULL REFERENCES content_items(id) ON DELETE CASCADE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_invalidation_request_entries_request (invalidation_request_id),
    INDEX idx_invalidation_request_entries_content (content_item_id)
);

`cache_invalidation_events` table

CREATE TABLE cache_invalidation_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    content_item_id UUID NOT NULL REFERENCES content_items(id) ON DELETE CASCADE,
    event_type VARCHAR(50) NOT NULL, -- 'PURGE', 'REFRESH'
    status VARCHAR(20) DEFAULT 'PENDING',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    processed_at TIMESTAMP,
    
    INDEX idx_cache_invalidation_events_content (content_item_id),
    INDEX idx_cache_invalidation_events_status (status),
    INDEX idx_cache_invalidation_events_created_at (created_at)
);

`cache_invalidation_logs` table

CREATE TABLE cache_invalidation_logs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    content_item_id UUID NOT NULL REFERENCES content_items(id) ON DELETE CASCADE,
    event_id UUID NOT NULL REFERENCES cache_invalidation_events(id) ON DELETE CASCADE,
    status VARCHAR(20) NOT NULL, -- 'SUCCESS', 'FAILED', 'PENDING'
    reason TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_cache_invalidation_logs_content (content_item_id),
    INDEX idx_cache_invalidation_logs_event (event_id),
    INDEX idx_cache_invalidation_logs_created_at (created_at)
);

Primary Key Strategy

UUIDs are used as primary keys for all tables. This choice provides:

Global uniqueness across distributed systems
No coordination required for ID generation
Scalability without contention on auto-increment sequences
Security - no predictable patterns that could be exploited

Index Design

Table	Index Name	Fields	Type	Purpose
`content_items`	idx_content_items_url	url	Unique	Fast URL lookups
`content_items`	idx_content_items_created_at	created_at	Standard	Time-based queries
`content_items`	idx_content_items_updated_at	updated_at	Standard	Cache freshness
`invalidation_requests`	idx_invalidation_requests_requester	requester_id	Standard	User-based filtering
`invalidation_requests`	idx_invalidation_requests_status	status	Standard	Query pending requests
`invalidation_request_entries`	idx_invalidation_request_entries_request	invalidation_request_id	Standard	Fast lookup by request
`invalidation_request_entries`	idx_invalidation_request_entries_content	content_item_id	Standard	Fast content lookup
`cache_invalidation_events`	idx_cache_invalidation_events_content	content_item_id	Standard	Fast content event lookup
`cache_invalidation_events`	idx_cache_invalidation_events_status	status	Standard	Query pending events
`cache_invalidation_logs`	idx_cache_invalidation_logs_content	content_item_id	Standard	Fast content logs
`cache_invalidation_logs`	idx_cache_invalidation_logs_event	event_id	Standard	Fast event logs

Data Partitioning / Sharding Strategy

Shard by content item URL hash prefix (first 2 characters of URL path):

Each shard contains content items with similar URL patterns
Allows for efficient cache invalidation by URL prefix
Enables horizontal scaling across multiple database instances

Soft Delete vs. Hard Delete Decision

Soft delete is used for content_items table:

Allows for audit trail and recovery capability
Maintains referential integrity in related tables
Supports historical analysis of content lifecycle

Audit Trail Design

All changes to content_items and cache_invalidation_events are audited:

created_at, updated_at timestamps
requester_id in invalidation requests
status and reason in logs for failure analysis

Data Archival Strategy and Retention Policy

Active content items: Retained indefinitely with soft delete capability
Invalidation logs: Retained for 90 days (compliance + debugging)
Processed invalidation requests: Retained for 30 days (operational visibility)
Cache events: Retained for 7 days (performance monitoring)

Archival process:

-- Move old logs to archive table
INSERT INTO cache_invalidation_logs_archive 
SELECT * FROM cache_invalidation_logs 
WHERE created_at < NOW() - INTERVAL '90 days';
DELETE FROM cache_invalidation_logs 
WHERE created_at < NOW() - INTERVAL '90 days';

4. Class Design & Service Layer

UML Class Diagram

classDiagram
    class CacheInvalidationService {
        -cacheInvalidationRepository
        -eventPublisher
        -cacheManager
        +invalidateCache(String cacheKey, String reason)
        +invalidateByPattern(String pattern, String reason)
        +invalidateByTags(Set~String~ tags, String reason)
        +getInvalidationStatus(String requestId)
    }
    
    class CacheInvalidationRepository {
        +saveInvalidationRequest(InvalidationRequest request)
        +findInvalidationRequest(String requestId)
        +findPendingRequests()
        +updateInvalidationStatus(String requestId, InvalidationStatus status)
    }
    
    class CacheManager {
        +invalidateKey(String key)
        +invalidatePattern(String pattern)
        +invalidateByTags(Set~String~ tags)
        +getCacheStats()
    }
    
    class EventPublisher {
        +publishInvalidationEvent(InvalidationEvent event)
    }
    
    class InvalidationRequest {
        -requestId
        -cacheKey
        -pattern
        -tags
        -reason
        -status
        -createdAt
        -updatedAt
    }
    
    class InvalidationEvent {
        -eventId
        -requestId
        -cacheKey
        -pattern
        -tags
        -reason
        -timestamp
    }
    
    class InvalidationStatus {
        PENDING
        PROCESSING
        COMPLETED
        FAILED
    }
    
    class CacheInvalidationProcessor {
        +processInvalidationRequest(InvalidationRequest request)
        +handleInvalidationFailure(InvalidationRequest request, Exception error)
    }
    
    CacheInvalidationService --> CacheInvalidationRepository : uses
    CacheInvalidationService --> CacheManager : delegates
    CacheInvalidationService --> EventPublisher : publishes
    CacheInvalidationProcessor --> CacheInvalidationService : orchestrates
    CacheInvalidationRepository --> InvalidationRequest : manages
    CacheManager --> InvalidationEvent : triggers
    EventPublisher --> InvalidationEvent : publishes
    
    class CacheInvalidationStrategy {
        <<interface>>
        +invalidate(String key, String reason)
    }
    
    class PatternBasedInvalidationStrategy {
        +invalidate(String pattern, String reason)
    }
    
    class TagBasedInvalidationStrategy {
        +invalidate(Set~String~ tags, String reason)
    }
    
    CacheInvalidationStrategy <|-- PatternBasedInvalidationStrategy
    CacheInvalidationStrategy <|-- TagBasedInvalidationStrategy

State Machine Diagram for InvalidationRequest

stateDiagram-v2
    [*] --> PENDING
    PENDING --> PROCESSING : Processed
    PROCESSING --> COMPLETED : Successful
    PROCESSING --> FAILED : Failed
    FAILED --> PENDING : Retryable
    COMPLETED --> [*]
    FAILED --> [*]

Service Class Structure and Responsibilities

The system consists of several key service classes:

CacheInvalidationService: Main orchestrator that handles external API calls and business logic
CacheInvalidationProcessor: Handles the actual processing of invalidation requests in background
CacheManager: Direct interface to cache systems (Redis, CDN)
EventPublisher: Handles event publishing for audit and monitoring
CacheInvalidationRepository: Manages persistence of invalidation requests

Interface and Abstract Class Design

public interface CacheInvalidationStrategy {
    void invalidate(String key, String reason);
}

public abstract class AbstractCacheInvalidationService {
    protected final CacheInvalidationRepository repository;
    protected final CacheManager cacheManager;
    protected final EventPublisher eventPublisher;
    
    public AbstractCacheInvalidationService(
            CacheInvalidationRepository repository,
            CacheManager cacheManager,
            EventPublisher eventPublisher) {
        this.repository = repository;
        this.cacheManager = cacheManager;
        this.eventPublisher = eventPublisher;
    }
    
    protected void publishInvalidationEvent(InvalidationRequest request) {
        InvalidationEvent event = InvalidationEvent.builder()
            .eventId(UUID.randomUUID().toString())
            .requestId(request.getRequestId())
            .cacheKey(request.getCacheKey())
            .pattern(request.getPattern())
            .tags(request.getTags())
            .reason(request.getReason())
            .timestamp(Instant.now())
            .build();
        eventPublisher.publishInvalidationEvent(event);
    }
}

Business Logic Encapsulation Patterns

The system uses the Strategy pattern for different invalidation methods:

public class TagBasedInvalidationStrategy implements CacheInvalidationStrategy {
    private final CacheManager cacheManager;
    
    public TagBasedInvalidationStrategy(CacheManager cacheManager) {
        this.cacheManager = cacheManager;
    }
    
    @Override
    public void invalidate(String key, String reason) {
        // Implementation for tag-based invalidation
        cacheManager.invalidateByTags(Set.of(key), reason);
    }
}

public class PatternBasedInvalidationStrategy implements CacheInvalidationStrategy {
    private final CacheManager cacheManager;
    
    public PatternBasedInvalidationStrategy(CacheManager cacheManager) {
        this.cacheManager = cacheManager;
    }
    
    @Override
    public void invalidate(String pattern, String reason) {
        cacheManager.invalidatePattern(pattern, reason);
    }
}

Critical-Path Method Implementation

public class CacheInvalidationService {
    private final CacheInvalidationRepository repository;
    private final CacheManager cacheManager;
    private final EventPublisher eventPublisher;
    private final ObjectMapper objectMapper;
    
    public CompletableFuture<String> invalidateCache(String cacheKey, String reason) {
        return CompletableFuture.supplyAsync(() -> {
            try {
                // 1. Create and persist invalidation request
                String requestId = UUID.randomUUID().toString();
                InvalidationRequest request = InvalidationRequest.builder()
                    .requestId(requestId)
                    .cacheKey(cacheKey)
                    .reason(reason)
                    .status(InvalidationStatus.PENDING)
                    .createdAt(Instant.now())
                    .updatedAt(Instant.now())
                    .build();
                
                repository.saveInvalidationRequest(request);
                
                // 2. Start processing in background
                CompletableFuture<Void> processingFuture = CompletableFuture.runAsync(() -> {
                    try {
                        // 3. Transaction boundary - process the invalidation
                        // Using optimistic locking to prevent concurrent updates
                        request.setStatus(InvalidationStatus.PROCESSING);
                        repository.updateInvalidationStatus(requestId, InvalidationStatus.PROCESSING);
                        
                        // 4. Perform actual cache invalidation
                        cacheManager.invalidateKey(cacheKey, reason);
                        
                        // 5. Update status to completed
                        request.setStatus(InvalidationStatus.COMPLETED);
                        repository.updateInvalidationStatus(requestId, InvalidationStatus.COMPLETED);
                        
                        // 6. Publish event for audit and monitoring
                        publishInvalidationEvent(request);
                        
                    } catch (Exception e) {
                        // 7. Handle failure and update status
                        request.setStatus(InvalidationStatus.FAILED);
                        repository.updateInvalidationStatus(requestId, InvalidationStatus.FAILED);
                        throw new RuntimeException("Failed to invalidate cache key: " + cacheKey, e);
                    }
                });
                
                return requestId;
            } catch (Exception e) {
                throw new RuntimeException("Failed to initiate cache invalidation for key: " + cacheKey, e);
            }
        });
    }
}

Transaction Boundary Design

The transaction boundary is designed as follows:

Atomic Persistence: The invalidation request creation and status update happen in a single atomic operation
Idempotent Processing: The processing itself is idempotent through status tracking and optimistic locking
Eventual Consistency: Cache invalidation is eventually consistent with the database state

Service Interface Contracts

public interface CacheInvalidationServiceInterface {
    CompletableFuture<String> invalidateCache(String cacheKey, String reason);
    CompletableFuture<String> invalidateByPattern(String pattern, String reason);
    CompletableFuture<String> invalidateByTags(Set<String> tags, String reason);
    InvalidationRequest getInvalidationStatus(String requestId);
}

Saga/Orchestration Pattern

The system uses a saga pattern for distributed invalidation operations:

public class CacheInvalidationSaga {
    private final List<CacheInvalidationStep> steps;
    
    public void execute(InvalidationRequest request) {
        for (CacheInvalidationStep step : steps) {
            try {
                step.execute(request);
            } catch (Exception e) {
                // Compensate for failed steps
                compensateSteps(steps, step);
                throw new CacheInvalidationException("Saga failed", e);
            }
        }
    }
    
    private void compensateSteps(List<CacheInvalidationStep> steps, CacheInvalidationStep failedStep) {
        // Reverse the order and execute compensation logic
        for (int i = steps.indexOf(failedStep); i >= 0; i--) {
            steps.get(i).compensate();
        }
    }
}

Domain Object Design

@Builder
@Data
@AllArgsConstructor
@NoArgsConstructor
public class InvalidationRequest {
    private String requestId;
    private String cacheKey;
    private String pattern;
    private Set<String> tags;
    private String reason;
    private InvalidationStatus status;
    private Instant createdAt;
    private Instant updatedAt;
    
    public boolean isPatternBased() {
        return pattern != null && !pattern.isEmpty();
    }
    
    public boolean isTagBased() {
        return tags != null && !tags.isEmpty();
    }
    
    public boolean isKeyBased() {
        return cacheKey != null && !cacheKey.isEmpty();
    }
}

public enum InvalidationStatus {
    PENDING,
    PROCESSING,
    COMPLETED,
    FAILED
}

5. Concurrency & Thread Safety

Concurrency Model

The CDN cache invalidation system operates with a hybrid concurrency model combining:

Thread pools for handling concurrent invalidation requests (Java's ExecutorService with fixed thread pool)
Async/Await pattern for non-blocking I/O operations (via CompletableFuture and Reactor)
Reactive streams for handling high-volume invalidation events

Race Condition Prevention Strategy

Race conditions in cache invalidation can occur when:

Multiple invalidation requests target the same cache key simultaneously
Concurrent read/write operations on cache metadata during invalidation
Cache state synchronization between distributed nodes

Prevention mechanisms:

Lock-based coordination using ReentrantReadWriteLock for metadata operations
Atomic operations via java.util.concurrent.atomic package for counters and flags
Versioned cache entries with compare-and-swap semantics to prevent stale updates

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected for cache invalidation operations due to:

Low contention in typical CDN usage patterns
Better scalability under concurrent invalidations
Reduced lock acquisition overhead for fast-path operations

Implementation approach:

// Pseudo-code for optimistic locking in cache invalidation
public class CacheInvalidationService {
    private final Map<String, CacheEntry> cache = new ConcurrentHashMap<>();
    
    public boolean invalidateWithOptimisticLock(String key, String expectedVersion) {
        CacheEntry entry = cache.get(key);
        if (entry == null) return true; // No-op for non-existent entries
        
        // Attempt to update with version check
        if (entry.getVersion().equals(expectedVersion)) {
            entry.setInvalidated(true);
            entry.setLastInvalidatedAt(Instant.now());
            return true;
        }
        
        return false; // Version mismatch - retry or abort
    }
}

Deadlock Prevention Strategy

Deadlock prevention is achieved through:

Lock ordering - Always acquire locks in consistent order (e.g., by cache key hash)
Timeout-based acquisition - Use tryLock() with timeout to prevent indefinite blocking
Hierarchical lock management - Single global invalidation lock for cross-node operations

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is ensured through:

Idempotency keys in invalidation requests (UUID-based)
Request deduplication cache with TTL to prevent processing same invalidation twice
State tracking to detect and ignore already-processed invalidation operations

public class IdempotentInvalidationService {
    private final Map<String, Instant> processedRequests = new ConcurrentHashMap<>();
    private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
    
    public void processInvalidationRequest(String idempotencyKey, String cacheKey) {
        // Check if request was already processed
        if (processedRequests.containsKey(idempotencyKey)) {
            return; // Idempotent - no-op
        }
        
        // Process invalidation
        processInvalidation(cacheKey);
        
        // Record request processing
        processedRequests.put(idempotencyKey, Instant.now());
        
        // Schedule cleanup of processed request tracking
        scheduler.schedule(() -> processedRequests.remove(idempotencyKey), 1, TimeUnit.HOURS);
    }
}

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant U as User/Client
    participant CDN as CDN Edge Node
    participant Cache as Cache Service
    participant Storage as Origin Storage
    participant EventQueue as Message Queue

    U->>CDN: Request resource (e.g., /images/photo.jpg)
    CDN->>Cache: Check if resource exists in cache
    Cache-->>CDN: Cache miss
    CDN->>Storage: Fetch resource from origin
    Storage-->>CDN: Return resource (with ETag)
    CDN->>Cache: Store resource with TTL
    Cache-->>CDN: Cache stored successfully
    CDN-->>U: Return cached resource

    Note over CDN,Cache: Cache hit on subsequent requests

sequenceDiagram
    participant Admin as Admin User
    participant API as Cache Invalidation API
    participant CacheService as Cache Service
    participant EventQueue as Message Queue
    participant CacheNode as CDN Edge Node

    Admin->>API: POST /invalidate?url=/images/photo.jpg
    API->>CacheService: Validate and queue invalidation request
    CacheService->>EventQueue: Publish invalidation event (topic: cache.invalidation)
    EventQueue-->>CacheNode: Broadcast invalidation message
    CacheNode->>CacheNode: Remove cached resource from local cache
    CacheNode-->>API: Acknowledge invalidation completion
    API-->>Admin: Return success response

    Note over CacheService,EventQueue: Event-driven invalidation pattern

sequenceDiagram
    participant U as User/Client
    participant CDN as CDN Edge Node
    participant Cache as Cache Service
    participant Storage as Origin Storage
    participant EventQueue as Message Queue

    U->>CDN: Request resource (e.g., /images/photo.jpg)
    CDN->>Cache: Check if resource exists in cache
    Cache-->>CDN: Cache hit - return cached content
    CDN-->>U: Return cached resource

    Note over CDN,Cache: Cache hit path - no origin fetch required

sequenceDiagram
    participant Admin as Admin User
    participant API as Cache Invalidation API
    participant CacheService as Cache Service
    participant EventQueue as Message Queue
    participant Storage as Origin Storage
    participant CacheNode as CDN Edge Node

    Admin->>API: POST /invalidate?url=/images/*&pattern=true
    API->>CacheService: Validate and queue batch invalidation request
    CacheService->>EventQueue: Publish batch invalidation event (topic: cache.invalidation.batch)
    EventQueue-->>CacheNode: Broadcast batch invalidation message
    CacheNode->>CacheNode: Remove matching cached resources from local cache
    CacheNode-->>API: Acknowledge invalidation completion
    API-->>Admin: Return success response with count of invalidated items

    Note over CacheService,EventQueue: Batch invalidation for wildcard patterns

sequenceDiagram
    participant Admin as Admin User
    participant API as Cache Invalidation API
    participant CacheService as Cache Service
    participant EventQueue as Message Queue
    participant CacheNode as CDN Edge Node

    Admin->>API: POST /invalidate?url=/images/photo.jpg&force=true
    API->>CacheService: Validate and queue force invalidation request
    CacheService->>EventQueue: Publish force invalidation event (topic: cache.invalidation.force)
    EventQueue-->>CacheNode: Broadcast force invalidation message
    CacheNode->>CacheNode: Remove cached resource from local cache (bypass TTL)
    CacheNode-->>API: Acknowledge invalidation completion
    API-->>Admin: Return success response

    Note over CacheService,EventQueue: Force invalidation bypasses cache TTL

sequenceDiagram
    participant Admin as Admin User
    participant API as Cache Invalidation API
    participant CacheService as Cache Service
    participant EventQueue as Message Queue
    participant CacheNode as CDN Edge Node

    Admin->>API: POST /invalidate?url=/images/photo.jpg&async=true
    API->>CacheService: Validate and queue async invalidation request
    CacheService->>EventQueue: Publish async invalidation event (topic: cache.invalidation.async)
    EventQueue-->>CacheNode: Broadcast async invalidation message
    CacheNode->>CacheNode: Schedule cache removal (non-blocking)
    CacheNode-->>API: Acknowledge async invalidation request
    API-->>Admin: Return success response immediately

    Note over CacheService,EventQueue: Async invalidation for non-blocking operations

sequenceDiagram
    participant U as User/Client
    participant CDN as CDN Edge Node
    participant Cache as Cache Service
    participant Storage as Origin Storage
    participant EventQueue as Message Queue

    U->>CDN: Request resource (e.g., /images/photo.jpg)
    CDN->>Cache: Check if resource exists in cache
    Cache-->>CDN: Cache miss - resource not found
    CDN->>Storage: Fetch resource from origin (with cache headers)
    Storage-->>CDN: Return resource with cache control headers
    CDN->>Cache: Store resource in cache with TTL from headers
    Cache-->>CDN: Cache stored successfully
    CDN-->>U: Return cached resource with cache headers

    Note over CDN,Cache: Cache warm-up with proper cache headers

sequenceDiagram
    participant Admin as Admin User
    participant API as Cache Invalidation API
    participant CacheService as Cache Service
    participant EventQueue as Message Queue
    participant CacheNode as CDN Edge Node

    Admin->>API: POST /invalidate?url=/images/photo.jpg&async=true&force=true
    API->>CacheService: Validate and queue force async invalidation request
    CacheService->>EventQueue: Publish force async invalidation event (topic: cache.invalidation.force.async)
    EventQueue-->>CacheNode: Broadcast force async invalidation message
    CacheNode->>CacheNode: Schedule immediate cache removal (bypass TTL)
    CacheNode-->>API: Acknowledge async invalidation request
    API-->>Admin: Return success response immediately

    Note over CacheService,EventQueue: Force async invalidation for immediate cache removal

7. Caching Design

Cache-Aside vs Write-Through vs Write-Behind

For CDN cache invalidation, we use cache-aside pattern with a write-through strategy for cache invalidation. This approach provides the best balance between consistency and performance for a CDN system where cache freshness is critical.

The cache-aside pattern allows us to:

Maintain control over when cache entries are invalidated
Handle cache misses gracefully with fallback logic
Support multiple cache tiers (edge, regional, origin)
Enable fine-grained control over invalidation timing

Cache Key Design with Namespace Strategy

Cache keys will follow a structured namespace pattern:

public class CacheKey {
    private final String namespace;
    private final String resourceType;
    private final String resourceId;
    private final String version;
    
    public CacheKey(String namespace, String resourceType, String resourceId, String version) {
        this.namespace = namespace;
        this.resourceType = resourceType;
        this.resourceId = resourceId;
        this.version = version;
    }
    
    public String getCacheKey() {
        return String.format("%s:%s:%s:%s", namespace, resourceType, resourceId, version);
    }
}

TTL Per Data Type

Data Type	TTL	Reason
Cache Entry	24h	CDN content typically changes infrequently
Invalidated Entry	5m	Prevents stale invalidation from blocking new content
Invalidation Queue Item	1h	Ensures retry capability for failed invalidations
Metadata Cache	10m	Frequent metadata updates require shorter TTL

Cache Eviction Policy

We use LRU (Least Recently Used) eviction policy with a time-based soft TTL to handle cache warming and hot resource retention:

public class CacheEvictionPolicy {
    private final int maxEntries;
    private final long ttlMillis;
    
    public CacheEvictionPolicy(int maxEntries, long ttlMillis) {
        this.maxEntries = maxEntries;
        this.ttlMillis = ttlMillis;
    }
    
    public boolean shouldEvict(CacheEntry entry) {
        return System.currentTimeMillis() - entry.getLastAccessed() > ttlMillis ||
               getEntryCount() > maxEntries;
    }
}

Cache Stampede Prevention

To prevent cache stampedes during high-traffic invalidation events:

Probabilistic Early Expiry: Cache entries expire 5 minutes before their TTL to allow for warm-up
Distributed Locking: Use Redis distributed locks for invalidation operations to prevent concurrent invalidations
Cache Entry Versioning: Each cache entry has a version that increments on invalidation

public class CacheStampedePrevention {
    private final RedisTemplate<String, String> redisTemplate;
    
    public boolean acquireInvalidationLock(String resourceKey, long timeoutMs) {
        String lockKey = "invalidation_lock:" + resourceKey;
        String lockValue = UUID.randomUUID().toString();
        
        Boolean acquired = redisTemplate.opsForValue()
            .setIfAbsent(lockKey, lockValue, timeoutMs, TimeUnit.MILLISECONDS);
        
        return acquired != null && acquired;
    }
    
    public void releaseInvalidationLock(String resourceKey, String lockValue) {
        String lockKey = "invalidation_lock:" + resourceKey;
        String script = 
            "if redis.call('get', KEYS[1]) == ARGV[1] then " +
            "return redis.call('del', KEYS[1]) else return 0 end";
        
        redisTemplate.execute(
            new DefaultRedisScript<>(script, Long.class),
            Collections.singletonList(lockKey),
            lockValue
        );
    }
}

Distributed Cache Architecture

We use Redis Cluster with:

3 master nodes for high availability
3 replica nodes per master for read scaling
Automatic failover and sharding
Redis Sentinel for monitoring and failover

Cache Invalidation Strategy

We implement event-driven cache invalidation with:

Immediate invalidation: When content changes, immediately invalidate related cache entries
Batch invalidation: For bulk operations, batch invalidations to reduce system load
Graceful degradation: If cache invalidation fails, continue serving stale content with warning headers

public class CacheInvalidationService {
    private final RedisTemplate<String, String> redisTemplate;
    private final ObjectMapper objectMapper;
    
    public void invalidateCache(String resourceKey, String version) {
        // 1. Acquire distributed lock to prevent race conditions
        String lockKey = "invalidate_lock:" + resourceKey;
        String lockValue = UUID.randomUUID().toString();
        
        try {
            if (redisTemplate.opsForValue()
                .setIfAbsent(lockKey, lockValue, 30, TimeUnit.SECONDS)) {
                
                // 2. Invalidate cache entries
                Set<String> keysToInvalidate = findRelatedCacheKeys(resourceKey);
                if (!keysToInvalidate.isEmpty()) {
                    redisTemplate.delete(keysToInvalidate);
                    
                    // 3. Publish invalidation event for async processing
                    publishInvalidationEvent(resourceKey, version);
                }
            } else {
                // Handle concurrent invalidation - retry or queue for later
                queueInvalidationForRetry(resourceKey, version);
            }
        } finally {
            releaseLock(lockKey, lockValue);
        }
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain objects and database operations. Each entity has its own repository interface and implementation.

public interface CacheInvalidationRepository {
    void save(CacheInvalidationRequest request);
    Optional<CacheInvalidationRequest> findById(String id);
    List<CacheInvalidationRequest> findByStatus(CacheInvalidationStatus status);
    List<CacheInvalidationRequest> findByTargetUrls(List<String> urls);
    void updateStatus(String id, CacheInvalidationStatus status);
    void delete(String id);
}

public class CacheInvalidationRepositoryImpl implements CacheInvalidationRepository {
    private final JdbcTemplate jdbcTemplate;
    private final ObjectMapper objectMapper;
    
    public CacheInvalidationRepositoryImpl(JdbcTemplate jdbcTemplate, ObjectMapper objectMapper) {
        this.jdbcTemplate = jdbcTemplate;
        this.objectMapper = objectMapper;
    }
    
    @Override
    public void save(CacheInvalidationRequest request) {
        String sql = """
            INSERT INTO cache_invalidation_requests (
                id, target_urls, status, created_at, updated_at, metadata
            ) VALUES (?, ?, ?, ?, ?, ?)
            """;
            
        jdbcTemplate.update(sql, 
            request.getId(),
            String.join(",", request.getTargetUrls()),
            request.getStatus().name(),
            request.getCreatedAt(),
            request.getUpdatedAt(),
            objectMapper.writeValueAsString(request.getMetadata())
        );
    }
    
    @Override
    public Optional<CacheInvalidationRequest> findById(String id) {
        String sql = "SELECT * FROM cache_invalidation_requests WHERE id = ?";
        try {
            return Optional.of(jdbcTemplate.queryForObject(sql, new CacheInvalidationRowMapper(), id));
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }
    
    @Override
    public List<CacheInvalidationRequest> findByStatus(CacheInvalidationStatus status) {
        String sql = "SELECT * FROM cache_invalidation_requests WHERE status = ? ORDER BY created_at ASC";
        return jdbcTemplate.query(sql, new CacheInvalidationRowMapper(), status.name());
    }
    
    @Override
    public List<CacheInvalidationRequest> findByTargetUrls(List<String> urls) {
        String sql = "SELECT * FROM cache_invalidation_requests WHERE target_urls LIKE ? ORDER BY created_at DESC";
        return jdbcTemplate.query(sql, new CacheInvalidationRowMapper(), "%," + String.join(",", urls) + ",%");
    }
    
    @Override
    public void updateStatus(String id, CacheInvalidationStatus status) {
        String sql = "UPDATE cache cache_invalidation_requests SET status = ?, updated_at = ? WHERE id = ?";
        jdbcTemplate.update(sql, status.name(), Instant.now(), id);
    }
    
    @Override
    public void delete(String id) {
        String sql = "DELETE FROM cache_invalidation_requests WHERE id = ?";
        jdbcTemplate.update(sql, id);
    }
}

ORM vs. Raw Query Decisions

For this system, raw SQL queries are preferred over ORM for the following reasons:

Performance: Direct SQL provides better performance for bulk operations and complex queries
Control: Fine-grained control over query execution and optimization
Complexity: Cache invalidation requires complex filtering and batch operations that are better expressed in raw SQL
Maintainability: Raw queries are more explicit and easier to debug

Read vs. Write Separation

The system uses a CQRS (Command Query Responsibility Segregation) pattern for read and write operations:

public interface CacheInvalidationCommandRepository {
    void save(CacheInvalidationRequest request);
    void updateStatus(String id, CacheInvalidationStatus status);
}

public interface CacheInvalidationQueryRepository {
    Optional<CacheInvalidationRequest> findById(String id);
    List<CacheInvalidationRequest> findByStatus(CacheInvalidationStatus status);
    List<CacheInvalidationRequest> findByTargetUrls(List<String> urls);
}

public class CacheInvalidationQueryRepositoryImpl implements CacheInvalidationQueryRepository {
    private final JdbcTemplate jdbcTemplate;
    
    public CacheInvalidationQueryRepositoryImpl(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }
    
    @Override
    public Optional<CacheInvalidationRequest> findById(String id) {
        String sql = "SELECT * FROM cache_invalidation_requests WHERE id = ?";
        try {
            return Optional.of(jdbcTemplate.queryForObject(sql, new CacheInvalidationRowMapper(), id));
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }
    
    @Override
    public List<CacheInvalidationRequest> findByStatus(CacheInvalidationStatus status) {
        String sql = "SELECT * FROM cache_invalidation_requests WHERE status = ? ORDER BY created_at ASC";
        return jdbcTemplate.query(sql, new CacheInvalidationRowMapper(), status.name());
    }
    
    @Override
    public List<CacheInvalidationRequest> findByTargetUrls(List<String> urls) {
        String sql = "SELECT * FROM cache_invalidation_requests WHERE target_urls LIKE ? ORDER BY created_at DESC";
        return jdbcTemplate.query(sql, new CacheInvalidationRowMapper(), "%," + String.join(",", urls) + ",%");
    }
}

Connection Pool Configuration

Connection pool is configured with:

Max pool size: 20
Min idle connections: 5
Connection timeout: 30 seconds
Validation query: SELECT 1
Idle timeout: 600 seconds

@Configuration
public class DataSourceConfig {
    @Bean
    public DataSource dataSource() {
        HikariConfig config = new HikariConfig();
        config.setJdbcUrl("jdbc:postgresql://localhost:5432/cdn_cache");
        config.setUsername("user");
        config.setPassword("password");
        config.setMaximumPoolSize(20);
        config.setMinimumIdle(5);
        config.setConnectionTimeout(30000);
        config.setIdleTimeout(600000);
        config.setValidationTimeout(5000);
        config.setLeakDetectionThreshold(60000);
        config.setConnectionTestQuery("SELECT 1");
        return new HikariDataSource(config);
    }
}

N+1 Query Prevention Strategy

To prevent N+1 query problems, eager loading is implemented using batch operations:

public class CacheInvalidationBatchService {
    private final CacheInvalidationRepository repository;
    
    public List<CacheInvalidationRequest> findRequestsByIds(List<String> ids) {
        if (ids.isEmpty()) return Collections.emptyList();
        
        String sql = "SELECT * FROM cache_invalidation_requests WHERE id IN (" + 
            String.join(",", Collections.nCopies(ids.size(), "?")) + ")";
        
        return repository.batchQuery(sql, ids);
    }
    
    public List<CacheInvalidationRequest> findRequestsByStatusBatch(
            CacheInvalidationStatus status, int batchSize) {
        List<CacheInvalidationRequest> allRequests = new ArrayList<>();
        int offset = 0;
        
        while (true) {
            String sql = "SELECT * FROM cache_invalidation_requests WHERE status = ? " +
                "ORDER BY created_at ASC LIMIT ? OFFSET ?";
            
            List<CacheInvalidationRequest> batch = repository.queryBatch(
                sql, status.name(), batchSize, offset);
            
            if (batch.isEmpty()) break;
            
            allRequests.addAll(batch);
            offset += batchSize;
        }
        
        return allRequests;
    }
}

Batch Operation Design

Batch operations are implemented for high-volume scenarios:

public class CacheInvalidationBatchRepository {
    private final JdbcTemplate jdbcTemplate;
    
    public void batchSave(List<CacheInvalidationRequest> requests) {
        String sql = """
            INSERT INTO cache_invalidation_requests (
                id, target_urls, status, created_at, updated_at, metadata
            ) VALUES (?, ?, ?, ?, ?, ?)
            """;
            
        jdbcTemplate.batchUpdate(sql, new BatchPreparedStatementSetter() {
            @Override
            public void setValues(PreparedStatement ps, int i) throws SQLException {
                CacheInvalidationRequest request = requests.get(i);
                ps.setString(1, request.getId());
                ps.setString(2, String.join(",", request.getTargetUrls()));
                ps.setString(3, request.getStatus().name());
                ps.setTimestamp(4, Timestamp.from(request.getCreatedAt()));
                ps.setTimestamp(5, Timestamp.from(request.getUpdatedAt()));
                ps.setString(6, objectMapper.writeValueAsString(request.getMetadata()));
            }
            
            @Override
            public int getBatchSize() {
                return requests.size();
            }
        });
    }
    
    public void batchUpdateStatus(List<String> ids, CacheInvalidationStatus status) {
        String sql = "UPDATE cache_invalidation_requests SET status = ?, updated_at = ? WHERE id = ?";
        
        jdbcTemplate.batchUpdate(sql, new BatchPreparedStatementSetter() {
            @Override
            public void setValues(PreparedStatement ps, int i) throws SQLException {
                String id = ids.get(i);
                ps.setString(1, status.name());
                ps.setTimestamp(2, Timestamp.from(Instant.now()));
                ps.setString(3, id);
            }
            
            @Override
            public int getBatchSize() {
                return ids.size();
            }
        });
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The CDN cache invalidation system operates with a hybrid synchronous/asynchronous architecture. The core cache invalidation logic is primarily synchronous to ensure immediate consistency for critical content updates, but the system leverages asynchronous processing for:

Batch invalidation operations - When multiple files need to be invalidated across different edge locations
Cleanup of stale cache entries - Periodic background tasks to remove expired content from edge nodes
Cross-region synchronization - Ensuring cache consistency across geographically distributed CDN nodes
Analytics and reporting - Aggregating invalidation metrics for monitoring and optimization

The synchronous flow handles immediate user-initiated invalidations (e.g., file uploads, content updates), while asynchronous flows manage system-level maintenance and distribution.

Message schema design

{
  "version": "1.0",
  "event_type": "cdn.cache.invalidation.requested",
  "timestamp": "2023-11-15T14:30:00Z",
  "payload": {
    "request_id": "inv-1234567890",
    "content_key": "/static/images/logo.png",
    "invalidate_all": false,
    "invalidate_tags": ["image", "logo"],
    "target_regions": ["us-east-1", "eu-west-1"],
    "priority": "high",
    "created_by": "user-98765"
  },
  "source": "content-management-service",
  "correlation_id": "req-abc123def456"
}

Dead letter queue strategy and poison message handling

The system implements a robust DLQ strategy with the following characteristics:

Poison message detection: Messages that fail more than 3 times in succession are automatically moved to a DLQ
Dead letter queue configuration: Messages in DLQ are retained for 7 days before permanent deletion
Alerting: When DLQ contains messages, an alert is sent to the on-call team with detailed error information
Manual reprocessing: Failed messages can be manually retried through a dedicated admin endpoint

At-least-once vs. exactly-once delivery

The system uses at-least-once delivery with idempotency mechanisms for the following reasons:

Cache invalidation operations are generally idempotent (repeating an invalidation request has the same effect as the first)
The system uses request IDs to detect and prevent duplicate processing
The complexity of exactly-once semantics outweighs the benefits for this use case

Retry policy for consumers

public class CacheInvalidationRetryPolicy {
    private static final int MAX_RETRIES = 5;
    private static final long BASE_DELAY_MS = 1000;
    private static final double BACKOFF_MULTIPLIER = 2.0;
    private static final long MAX_DELAY_MS = 30000; // 30 seconds
    
    public RetryResult executeWithRetry(Supplier<CompletableFuture<Void>> operation) {
        int attempt = 0;
        while (true) {
            try {
                return RetryResult.success(operation.get());
            } catch (Exception e) {
                attempt++;
                if (attempt >= MAX_RETRIES || !shouldRetry(e)) {
                    return RetryResult.failure(e, attempt);
                }
                
                long delay = Math.min(
                    (long) (BASE_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempt - 1)),
                    MAX_DELAY_MS
                );
                
                // Add jitter to prevent thundering herd
                long jitter = (long) (Math.random() * delay * 0.1);
                delay += jitter;
                
                try {
                    Thread.sleep(delay);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new RuntimeException("Retry interrupted", ie);
                }
            }
        }
    }
    
    private boolean shouldRetry(Exception e) {
        return !(e instanceof ValidationException || 
                e instanceof IllegalArgumentException);
    }
}

Outbox pattern for reliable event publishing

The outbox pattern is implemented to ensure reliable event publishing:

@Entity
@Table(name = "outbox_events")
public class OutboxEvent {
    @Id
    private String id;
    
    @Column(name = "event_type")
    private String eventType;
    
    @Lob
    @Column(name = "payload")
    private String payload;
    
    @Column(name = "status")
    private OutboxStatus status = OutboxStatus.PENDING;
    
    @Column(name = "created_at")
    private Instant createdAt;
    
    @Column(name = "processed_at")
    private Instant processedAt;
    
    // Getters and setters
}

public class OutboxService {
    private final EntityManager entityManager;
    private final EventPublisher eventPublisher;
    
    public void publishEvent(String eventType, String payload) {
        // 1. Create outbox event
        OutboxEvent event = new OutboxEvent();
        event.setId(UUID.randomUUID().toString());
        event.setEventType(eventType);
        event.setPayload(payload);
        event.setCreatedAt(Instant.now());
        
        // 2. Persist to database within same transaction
        entityManager.persist(event);
        entityManager.flush(); // Ensure persistence before publishing
        
        // 3. Publish to message queue (in same transaction)
        try {
            eventPublisher.publish(event);
            event.setStatus(OutboxStatus.PROCESSED);
            event.setProcessedAt(Instant.now());
        } catch (Exception e) {
            event.setStatus(OutboxStatus.FAILED);
            throw new RuntimeException("Failed to publish event", e);
        }
    }
}

Sequence diagram for async flows

sequenceDiagram
    participant User
    participant ContentService
    participant CacheInvalidationService
    participant MessageQueue
    participant EdgeNodeManager
    participant RedisCache
    
    User->>ContentService: Request cache invalidation
    ContentService->>CacheInvalidationService: Process invalidation request
    CacheInvalidationService->>MessageQueue: Publish invalidation event
    MessageQueue-->>EdgeNodeManager: Async invalidation request
    EdgeNodeManager->>RedisCache: Invalidate cache entries
    RedisCache-->>EdgeNodeManager: Cache invalidated
    EdgeNodeManager->>MessageQueue: Publish completion event
    MessageQueue-->>ContentService: Invalidations completed
    ContentService-->>User: Return success response

10. Error Handling Design

Error Classification Hierarchy

The system implements a three-tier error classification to support proper handling and alerting:

Domain Errors - Business logic violations (e.g., invalid cache key, unsupported content type)
Infrastructure Errors - System resource constraints (e.g., Redis connection timeout, DB connection pool exhausted)
Validation Errors - Input parameter issues (e.g., malformed URL, missing required fields)

public abstract class BaseException extends RuntimeException {
    private final ErrorCode errorCode;
    private final ErrorCategory category;

    public BaseException(ErrorCode code, ErrorCategory category, String message) {
        super(message);
        this.errorCode = code;
        this.category = category;
    }

    // Getters
}

public enum ErrorCategory {
    DOMAIN, INFRASTRUCTURE, VALIDATION
}

public enum ErrorCode {
    INVALID_CACHE_KEY(1001), 
    CACHE_NOT_FOUND(1002),
    INVALID_URL_FORMAT(2001),
    REDIS_CONNECTION_TIMEOUT(3001),
    DB_CONNECTION_FAILURE(3002),
    INVALID_TTL_VALUE(4001)
}

Error Propagation Strategy

Errors are propagated up the call stack with appropriate wrapping to preserve context:

public class CacheInvalidationService {
    public void invalidateCache(String cacheKey) throws CacheInvalidationException {
        try {
            // Business logic
            cacheRepository.delete(cacheKey);
        } catch (DataAccessException e) {
            throw new CacheInvalidationException(
                ErrorCode.DB_CONNECTION_FAILURE, 
                ErrorCategory.INFRASTRUCTURE,
                "Database operation failed during cache invalidation",
                e
            );
        } catch (IllegalArgumentException e) {
            throw new CacheInvalidationException(
                ErrorCode.INVALID_CACHE_KEY,
                ErrorCategory.VALIDATION,
                "Cache key validation failed",
                e
            );
        }
    }
}

Retry Strategy with Exponential Backoff

For transient infrastructure issues, the system implements retry logic with jitter:

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;

    public <T> T executeWithRetry(Supplier<T> operation) throws Exception {
        Exception lastException = null;
        
        for (int attempt = 0; attempt <= MAX_RETRIES; attempt++) {
            try {
                return operation.get();
            } catch (Exception e) {
                lastException = e;
                
                if (attempt == MAX_RETRIES || !isRetryable(e)) {
                    throw lastException;
                }
                
                long delay = calculateBackoffDelay(attempt);
                Thread.sleep(delay + randomJitter(delay));
            }
        }
        
        throw lastException;
    }

    private long calculateBackoffDelay(int attempt) {
        return (long) (BASE_DELAY_MS * Math.pow(2, attempt));
    }

    private long randomJitter(long baseDelay) {
        return (long) (baseDelay * JITTER_FACTOR * (Math.random() - 0.5) * 2);
    }

    private boolean isRetryable(Exception e) {
        return e instanceof RedisConnectionException || 
               e instanceof DataAccessException ||
               e instanceof TimeoutException;
    }
}

Circuit Breaker Design

For external dependencies like Redis and database, circuit breaker pattern prevents cascading failures:

public class CircuitBreaker {
    private final String name;
    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private long lastFailureTime = 0;
    private final int failureThreshold;
    private final long timeoutMs;

    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }

    public boolean allowRequest() {
        switch (state) {
            case CLOSED:
                return true;
            case HALF_OPEN:
                // Allow one request to test if the system is recovered
                return true;
            case OPEN:
                // If timeout has passed, move to HALF_OPEN state
                if (System.currentTimeMillis() - lastFailureTime > timeoutMs) {
                    state = CircuitState.HALF_OPEN;
                    return true;
                }
                return false;
        }
        return true;
    }

    public void recordSuccess() {
        if (state == CircuitState.HALF_OPEN) {
            state = CircuitState.CLOSED;
        }
        failureCount = 0;
    }

    public void recordFailure() {
        failureCount++;
        lastFailureTime = System.currentTimeMillis();
        
        if (failureCount >= failureThreshold) {
            state = CircuitState.OPEN;
        }
    }
}

Fallback Behavior Design

Fallbacks are implemented for critical operations:

public class CacheInvalidationService {
    private final CircuitBreaker redisCircuitBreaker;

    public void invalidateCacheWithFallback(String cacheKey) {
        try {
            if (redisCircuitBreaker.allowRequest()) {
                redisService.invalidate(cacheKey);
                redisCircuitBreaker.recordSuccess();
            } else {
                // Fallback to direct DB cleanup
                fallbackToDatabaseInvalidate(cacheKey);
            }
        } catch (Exception e) {
            redisCircuitBreaker.recordFailure();
            fallbackToDatabaseInvalidate(cacheKey);
        }
    }

    private void fallbackToDatabaseInvalidate(String cacheKey) {
        // Log the fallback for monitoring
        logger.warn("Falling back to database invalidation for key: {}", cacheKey);
        // Perform direct DB cleanup
        dbCacheRepository.delete(cacheKey);
    }
}

Error Logging Format and Alerting Thresholds

Structured logging includes trace context and error metadata:

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);

    public void logError(String operation, Exception error, Map<String, Object> context) {
        Map<String, Object> logContext = new HashMap<>();
        logContext.put("operation", operation);
        logContext.put("error_code", ((BaseException) error).getErrorCode());
        logContext.put("error_category", ((BaseException) error).getCategory().name());
        logContext.put("timestamp", Instant.now());
        logContext.putAll(context);

        logger.error("Cache invalidation operation failed", 
            new ObjectMapper().valueToTree(logContext));
    }
}

Alerting thresholds:

Critical: > 50% cache invalidation failures in 5 minutes → On-call alert
Warning: > 10% cache invalidation failures in 1 hour → Slack notification
Info: > 50 cache invalidation errors in 1 day → Weekly report

Error Response Format

Standardized error responses for API consumers:

{
  "error": {
    "code": "CACHE_INVALIDATION_FAILED",
    "message": "Failed to invalidate cache entry",
    "details": {
      "cacheKey": "cdn://example.com/image.jpg",
      "errorCode": 3001,
      "timestamp": "2023-06-15T14:30:00Z"
    }
  }
}

11. Security Design

Input Validation and Sanitization Strategy

All inputs are validated using a combination of:

Regex patterns for URL path validation (e.g., ^/cdn/[a-zA-Z0-9\-_]+(?:\.[a-zA-Z0-9]+)?$)
Length limits (max 255 characters for paths)
Character whitelisting (only alphanumeric, hyphens, underscores, dots, forward slashes)
Library validation: Using Apache Commons Validator for URL format and Spring Validation annotations

SQL Injection Prevention

Prepared statements exclusively used for all database queries
Parameterized queries with no string concatenation in SQL
ORM framework (Hibernate) used for all data access to prevent raw SQL injection

Authentication Middleware Design

JWT-based authentication for API endpoints with 24-hour expiration
OAuth2 integration with third-party providers (Google, GitHub) for admin access
API key validation for content publishers with rate limiting per key (1000 req/hour)
Middleware pipeline that validates tokens and sets user context in request scope

Authorization Enforcement

Role-Based Access Control (RBAC) model:
- ADMIN: Full access to all cache invalidation operations
- PUBLISHER: Can invalidate content they own (based on publisher_id in cache keys)
- VIEWER: Read-only access to cache status and logs
Permission checks performed at service layer using @PreAuthorize annotations in Spring Security

Sensitive Data Encryption

At rest: AES-256 encryption for cache metadata and logs stored in database
In transit: TLS 1.3 with HSTS enabled for all API communications
Secrets: All sensitive data (API keys, JWT secrets) encrypted using AWS KMS

PII Handling and Masking Strategy

No PII stored in cache keys or metadata (paths are normalized)
User identifiers in logs are hashed using SHA-256 with salt
Log redaction: Any field containing user IDs or paths are masked in logs

Secrets Management

AWS Secrets Manager integrated for:
- JWT signing keys
- Third-party API credentials (Cloudflare, Akamai)
- Database connection strings
Rotation policy: Secrets rotated every 90 days with automatic failover

Audit Logging Design

Events logged:
- Cache invalidation request (publisher_id, path, timestamp)
- API access attempts (success/failure, IP address)
- Admin actions (user_id, action_type, timestamp)
Log format:

{
  "timestamp": "2023-10-15T14:30:45Z",
  "trace_id": "abc123",
  "user_id": "hashed_user_id",
  "action": "invalidate_cache",
  "path": "/cdn/assets/12345.png",
  "status": "success",
  "ip_address": "192.168.1.100"
}

Retention: 90 days for audit logs with daily backups to S3
Compliance: Logs stored in encrypted format and accessible only via admin roles

12. Testing Strategy

Unit Testing Scope and Coverage Targets

Unit tests will cover 85%+ of business logic with 90%+ line coverage for core service classes. Each domain object and service method will have dedicated unit tests with:

Happy path scenarios
Boundary condition testing (edge cases)
Error state handling
Mocked external dependencies

Integration Testing Strategy

Integration tests will validate end-to-end flows using:

Testcontainers for database and cache infrastructure
Embedded Redis for caching layer testing
WireMock for external service stubbing (e.g., CDN provider APIs)
In-memory database (H2) for data persistence testing

Contract Testing

Contract tests will be implemented using Pact to ensure:

API contract compliance between CDN cache invalidation service and external systems (CDN providers, content publishers)
Versioned API contracts maintained with semantic versioning
Consumer-driven contract testing for all external integrations

Performance Testing Tool and Key Scenarios

Performance testing will use k6 with the following key scenarios:

High concurrency invalidation: 1000 concurrent cache invalidation requests with 50% read-heavy workload
Batch invalidation throughput: Simulate batch invalidations of 1000+ URLs with 50ms average latency target
Cache warm-up under load: Test system behavior during cache warming with 1000 concurrent requests
Failure recovery under stress: Simulate invalidation failures and retry behavior under load

Test Data Management Strategy

Test data will be managed using:

Factory pattern for generating test domain objects (URLs, cache keys, invalidation requests)
Fixture files with predefined content for integration tests
Seeding strategy using database migrations for test data setup
Data cleanup procedures to ensure test isolation and prevent cross-contamination between test runs

// Example unit test structure for cache invalidation service
@Test
public void testInvalidationWithValidUrls() {
    // Given
    List<String> urls = Arrays.asList("http://cdn.example.com/image1.jpg", "http://cdn.example.com/image2.jpg");
    CacheInvalidationRequest request = new CacheInvalidationRequest(urls);
    
    // When
    List<CacheInvalidationResult> results = cacheInvalidationService.invalidate(request);
    
    // Then
    assertThat(results).hasSize(2);
    assertThat(results.get(0).getStatus()).isEqualTo(CacheInvalidationStatus.SUCCESS);
}

13. Observability Design

Structured Logging Schema

All logs follow a structured JSON format with consistent fields to enable aggregation and analysis:

{
  "timestamp": "2023-10-27T14:30:00.000Z",
  "level": "INFO",
  "service": "cdn-cache-invalidation",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user-123",
  "request_id": "req-456",
  "operation": "invalidateCache",
  "entity_id": "cdn-object-789",
  "status": "success",
  "duration_ms": 150,
  "error_code": null,
  "error_message": null,
  "payload": {
    "url": "/static/images/logo.png",
    "cache_key": "cdn-static-images-logo.png"
  }
}

PII masking rules:

User IDs are masked in logs when not required for debugging
IP addresses are truncated to network prefix (e.g., 192.168.1.***)
Sensitive fields like tokens or secrets are completely omitted from logs

Distributed Tracing Instrumentation

Using OpenTelemetry for end-to-end tracing:

Custom Spans to Add:

cdn.invalidate.request - Entry point for invalidation requests
cdn.cache.lookup - Cache lookup operations
cdn.cache.invalidate - Actual invalidation execution
cdn.event.publish - Event publishing to message broker

Attributes Captured:

cdn.object.url - The URL being invalidated
cdn.cache.key - Cache key used for lookup
cdn.invalidate.strategy - Strategy used (e.g., "by-url", "by-tag")
cdn.invalidate.status - Success/failure status
cdn.event.type - Type of event published (e.g., "cache_invalidated")

Metrics Design

RED Method Metrics:

Metric Name	Type	Labels	Description
`cdn_cache_invalidation_requests_total`	Counter	`strategy`, `status`	Total cache invalidation requests
`cdn_cache_invalidation_duration_seconds`	Histogram	`strategy`, `status`	Duration of invalidation operations
`cdn_cache_invalidation_errors_total`	Counter	`strategy`, `error_type`	Errors during invalidation
`cdn_cache_invalidation_events_published_total`	Counter	`event_type`, `status`	Events published to message broker
`cdn_cache_invalidation_queue_size`	Gauge	`queue_name`	Current size of invalidation queues

Key Labels:

strategy: by_url, by_tag, by_pattern, by_user
status: success, failure, timeout, retry
error_type: validation_error, network_error, internal_error

Health Check Endpoint Design

GET /health
Accept: application/json

Response Format:

{
  "status": "healthy",
  "timestamp": "2023-10-27T14:30:00.000Z",
  "checks": {
    "database": {
      "status": "healthy",
      "latency_ms": 5
    },
    "message_broker": {
      "status": "healthy",
      "latency_ms": 10
    },
    "cache": {
      "status": "healthy",
      "latency_ms": 2
    }
  },
  "version": "1.2.3"
}

Endpoints:

/health - Overall system health (liveness probe)
/ready - Readiness check (dependencies ready)
/live - Basic liveness (service running)
/startup - Startup validation (config loaded, dependencies initialized)

Alerting Thresholds

Metric	Warning Level	Critical Level	Escalation Path
`cdn_cache_invalidation_errors_total`	> 10 errors/minute	> 50 errors/minute	On-call engineer
`cdn_cache_invalidation_duration_seconds` p95	> 200ms	> 500ms	SRE team
`cdn_cache_invalidation_queue_size`	> 1000 items	> 5000 items	SRE team
`message_broker_unavailable`	> 5% of requests	> 20% of requests	Platform team
`database_connection_errors`	> 5 errors/minute	> 20 errors/minute	DBA team

Alerting Logic:

Use sliding window (5-minute intervals) for rate-based alerts
Apply exponential backoff to prevent alert spamming
Implement auto-resolution when thresholds return to normal levels

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

Cache invalidation granularity - Should we support cache invalidation at the individual object level, directory level, or wildcard patterns? The choice affects both API complexity and cache efficiency.
Cache invalidation propagation strategy - How should invalidations be propagated across CDN edge nodes? Should we use a push model (immediate propagation) or pull model (edge nodes query central registry)?
Cache invalidation priority - How should we handle conflicting invalidation requests? Should high-priority invalidations (e.g., security patches) preempt lower-priority ones?
Cache invalidation batching policy - Should we batch invalidations for performance reasons, and if so, what's the optimal batch size and timing?

Known trade-offs and justifications

Consistency vs. Performance Trade-off - We're choosing eventual consistency over strong consistency for cache invalidations to achieve better performance and scalability. This means there may be a brief window where stale content is served, which is acceptable for most CDN use cases but not for real-time systems.
Immediate vs. Delayed Invalidation Trade-off - We're implementing a delayed invalidation model (with configurable TTL) instead of immediate invalidation to reduce system load and avoid thundering herd effects. This trades some latency for better system stability.
Centralized vs. Distributed Cache Invalidation Trade-off - We're using a centralized invalidation queue to maintain consistency, but this creates a single point of failure and potential bottleneck. Distributed approaches would increase complexity but improve availability.
Memory vs. Disk Storage Trade-off - We're using in-memory storage for cache invalidation metadata to achieve low latency, but this creates a risk of data loss during system restarts. Disk persistence would provide better durability but at the cost of performance.

Risks and mitigation strategies

Cache Invalidation Storm Risk - A large number of cache invalidations could overwhelm the system and cause cascading failures.
- Mitigation: Implement rate limiting on invalidation requests, use exponential backoff for retry logic, and add circuit breaker patterns to prevent system overload.
Data Loss Risk - If the cache invalidation service crashes or loses data, content might remain in cache longer than intended.
- Mitigation: Implement proper persistence (disk-based or database-backed) for critical invalidation metadata, use distributed consensus protocols where appropriate, and maintain audit logs.
Race Condition Risk - Concurrent invalidation requests for the same content could cause inconsistent cache states.
- Mitigation: Use distributed locks or atomic operations to ensure only one invalidation process can modify a cache entry at a time.
Cache Invalidation Loop Risk - If content is constantly being invalidated and re-cached, it could create an infinite loop.
- Mitigation: Implement circuit breaker logic to detect and break such loops, set maximum retry limits on invalidations, and add monitoring for invalidation frequency anomalies.

Technical debt acknowledged

Monolithic Design Debt - The current design assumes a single service for all cache invalidation operations, which could become a scalability bottleneck as the system grows. Future refactoring should consider sharding by content type or geographic region.
Hardcoded TTL Values - The system currently uses hardcoded TTL values for cache invalidation metadata, which makes it inflexible to changing business requirements without code changes.
Limited Monitoring Coverage - While basic metrics are implemented, deeper observability into invalidation performance and success rates is not yet fully developed, which could make debugging difficult in production.
No Backward Compatibility - The current design assumes all clients will use the new invalidation API, but there's no plan for supporting legacy invalidation methods or gradual migration strategies.
Synchronous Processing Debt - The system currently processes invalidations synchronously, which could cause latency issues under high load. A more robust async processing pipeline should be implemented for better scalability.
No Content-Type Specific Handling - The system treats all content types uniformly, but different content types might require different invalidation strategies (e.g., images vs. HTML vs. JavaScript files).