Idempotency Key System Design for Backend Services

Comprehensive low-level design for implementing idempotency keys to ensure idempotent API operations in distributed systems.

PublishedApril 13, 2025
Length8,248 words
Reading time42 min read

Backend LLD

Idempotency key systems ensure that repeated API requests with the same key produce identical results, preventing duplicate operations in distributed backend services. This design addresses critical concerns around idempotent operations, retry handling, and state consistency across microservices.

1. Problem Statement & Requirements

Clarifying questions

What is the expected scale of idempotency key usage?
- Are we expecting millions of concurrent requests with idempotency keys?
- Will the system handle bursty traffic patterns?
What consistency model is required?
- Should idempotency checks be strongly consistent or eventually consistent?
- Are there specific SLA requirements for idempotency validation?
What types of operations are expected to support idempotency?
- Is it only for write operations (e.g., create order, update user), or also read operations?
- Are there specific operation types (e.g., payments, bookings) that are more critical?
What is the expected latency sensitivity?
- Should idempotency checks happen in sub-100ms to avoid impacting user experience?
- Are there specific performance targets (p50/p95/p99) for idempotency lookups?
What is the expected lifecycle of an idempotency key?
- Should keys be automatically cleaned up after a fixed TTL or based on operation success/failure?
- Is there a need to support long-lived keys for retries over extended periods?
What authentication and authorization model is in place?
- Will idempotency keys be tied to specific users or sessions?
- Should the system support key sharing between different clients?
What external systems or integrations are involved?
- Will the system interact with third-party APIs that also require idempotency?
- Are there any downstream systems that need to validate the same idempotency key?

Functional requirements

Idempotency Key Validation: The system must validate that a given idempotency key has not been used for the same operation within a specified time window.
Idempotency Key Storage: The system must store and manage idempotency keys with associated metadata (e.g., operation type, user ID, timestamp).
Idempotency Key Expiry: The system must automatically clean up expired keys to prevent storage bloat.
Idempotency Key Scope: The system must support scoping keys by user, session, or operation type to prevent cross-user key collisions.
Idempotency Key Generation: The system must provide mechanisms for generating unique idempotency keys.
Idempotency Key Conflict Detection: The system must detect and reject duplicate requests with the same key for the same operation type.
Idempotency Key Audit: The system must maintain audit logs of all idempotency key usage.

Non-functional requirements

Latency Targets:
- p50: < 20ms for idempotency key validation
- p95: < 50ms for idempotency key validation
- p99: < 100ms for idempotency key validation
Throughput Targets:
- Support at least 10,000 concurrent idempotency key validations per second
- Handle burst traffic up to 50,000 requests/sec
Availability Targets:
- 99.9% availability for idempotency key service
- 99.95% availability for critical operations (e.g., payments)
- Support for multi-region deployment with failover capability

Constraints and assumptions

Assumptions:
- Idempotency keys are generated by client libraries or SDKs
- The system will not be responsible for key generation logic, only validation and storage
- Idempotency keys are unique per operation type within a user or session scope
- The system will not support cross-service idempotency key validation (e.g., a key generated in one service cannot be used in another)
- Idempotency keys will not be shared across different users or sessions
- The system assumes that the client will retry with the same key for the same operation
Constraints:
- The system must not block or delay the main request processing for idempotency validation
- Idempotency keys must be stored in a highly available and scalable storage system (e.g., Redis or database)
- The system must support horizontal scaling to handle high throughput
- Idempotency key validation must not introduce significant latency overhead for the main business logic

Out of scope

Idempotency Key Generation: The system will not implement the generation logic for idempotency keys; it will only validate and store them.
Cross-service Idempotency: The system does not support idempotency keys that are valid across different services or APIs.
Idempotency Key Revocation: The system does not support explicit revocation of idempotency keys by users or administrators.
Idempotency Key Sharing: The system does not support sharing idempotency keys between different users or sessions.
Idempotency Key Versioning: The system does not support versioned idempotency keys or key migration strategies.
Idempotency Key Auditing: The system does not provide detailed audit trails or compliance reporting for idempotency key usage.
Idempotency Key Encryption: The system does not encrypt or obfuscate idempotency keys in storage or transit.
Idempotency Key Compression: The system does not compress or encode idempotency keys for storage efficiency.

2. API Design

API Style Selection

gRPC over HTTP/2 with Protocol Buffers is selected for the Idempotency Key system due to:

Strong typing and schema validation at compile time
Efficient binary serialization reducing payload size
Built-in support for streaming and bidirectional communication
First-class support for idempotency key handling in gRPC
Better performance characteristics for high-throughput systems

Endpoint Definitions

Method	Path	Description	Auth
CreateIdempotencyKey	`POST /v1/idempotency`	Creates a new idempotency key with TTL	JWT
GetIdempotencyKey	`GET /v1/idempotency/{key}`	Retrieves metadata for an existing key	JWT
DeleteIdempotencyKey	`DELETE /v1/idempotency/{key}`	Deletes an idempotency key	JWT
ValidateIdempotencyKey	`POST /v1/idempotency/validate`	Validates if a key has been used for the same operation	JWT

Request and Response Schemas

CreateIdempotencyKey Request

message CreateIdempotencyKeyRequest {
  string key = 1;
  int64 ttl_seconds = 2;
  string operation_id = 3;
  string user_id = 4;
}

CreateIdempotencyKey Response

message CreateIdempotencyKeyResponse {
  string key = 1;
  int64 ttl_seconds = 2;
  string operation_id = 3;
  int64 created_at = 4;
  int64 expires_at = 5;
}

GetIdempotencyKey Response

message GetIdempotencyKeyResponse {
  string key = 1;
  int64 ttl_seconds = 2;
  string operation_id = 3;
  int64 created_at = 4;
  int64 expires_at = 5;
  bool is_valid = 6;
}

ValidateIdempotencyKey Request

message ValidateIdempotencyKeyRequest {
  string key = 1;
  string operation_id = 2;
}

ValidateIdempotencyKey Response

message ValidateIdempotencyKeyResponse {
  bool is_valid = 1;
  string existing_operation_id = 2;
}

HTTP Status Codes and Error Response Format

Status Code	Description
200	Success
400	Bad Request (malformed request, invalid key format)
401	Unauthorized
403	Forbidden (insufficient permissions)
409	Conflict (key already exists)
429	Too Many Requests
500	Internal Server Error

Standardized error response format:

{
  "error": {
    "code": "INVALID_ARGUMENT",
    "message": "Idempotency key format is invalid",
    "details": [
      {
        "field": "key",
        "message": "Key must be between 1 and 255 characters"
      }
    ]
  }
}

API Versioning Strategy

Versioned using path prefix: /v1/idempotency/...

Backward compatible changes within same major version
Breaking changes require new major version (v2)
Deprecation policy: 6-month grace period for deprecated endpoints

Pagination Design

Cursor-based pagination is used because:

More efficient than offset-based for large datasets
Consistent results even when data changes during pagination
Better performance characteristics

Authentication and Authorization

All endpoints require JWT-based authentication with scopes:

idempotency:read for GET operations
idempotency:write for CREATE/DELETE operations

Rate Limiting

Rate limits per endpoint:

Create/Delete: 100 requests/minute
Get/Validate: 500 requests/minute

Enforced using Redis-based token bucket algorithm with sliding window.

Idempotency Key Strategy for Mutations

For mutation endpoints that support idempotency:

Client includes Idempotency-Key header in request
System stores operation details for the key
If same key is used with same operation, return stored result
If same key but different operation, return conflict error

// Example of idempotency key handling in a gRPC service method
public CreateIdempotencyKeyResponse createIdempotencyKey(
    CreateIdempotencyKeyRequest request,
    ServerCallStreamObserver<CreateIdempotencyKeyResponse> responseObserver) {
    
    String idempotencyKey = extractIdempotencyKeyFromHeader();
    if (idempotencyKey == null) {
        responseObserver.onError(
            Status.INVALID_ARGUMENT.withDescription("Idempotency-Key header is required").asException());
        return;
    }
    
    try {
        // Validate key format and operation ID
        validateRequest(request);
        
        IdempotencyKey key = idempotencyService.createKey(
            idempotencyKey, 
            request.getTtlSeconds(), 
            request.getOperationId(), 
            request.getUserId()
        );
        
        responseObserver.onNext(mapToResponse(key));
        responseObserver.onCompleted();
    } catch (IdempotencyKeyAlreadyExistsException e) {
        responseObserver.onError(
            Status.ALREADY_EXISTS.withDescription("Idempotency key already exists").asException());
    } catch (Exception e) {
        responseObserver.onError(
            Status.INTERNAL.withDescription("Failed to create idempotency key").asException());
    }
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    IDEMPOTENCY_KEY {
        string id PK
        string resource_id
        string user_id
        string key_value
        string status
        timestamp created_at
        timestamp expires_at
        timestamp last_accessed_at
        integer retry_count
    }

    RESOURCE {
        string id PK
        string resource_type
        string resource_data
        timestamp created_at
        timestamp updated_at
    }

    IDEMPOTENCY_KEY ||--o{ RESOURCE : "references"

Table Schema Definitions

CREATE TABLE idempotency_keys (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    resource_id VARCHAR(255) NOT NULL,
    user_id VARCHAR(255) NOT NULL,
    key_value VARCHAR(255) NOT NULL UNIQUE,
    status VARCHAR(50) NOT NULL DEFAULT 'ACTIVE',
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
    last_accessed_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    retry_count INTEGER NOT NULL DEFAULT 0,
    
    INDEX idx_key_value (key_value),
    INDEX idx_resource_id (resource_id),
    INDEX idx_user_id (user_id),
    INDEX idx_expires_at (expires_at),
    INDEX idx_status (status),
    INDEX idx_created_at (created_at)
);

CREATE TABLE idempotency_resources (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    resource_type VARCHAR(100) NOT NULL,
    resource_data JSONB NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

Primary Key Strategy

UUID is selected as the primary key for idempotency_keys table because:

Provides globally unique identifiers without coordination
Avoids potential contention in high-concurrency scenarios
Supports distributed system deployment with no key generation conflicts
Enables better sharding strategies across multiple database instances

Index Design

Index Name	Fields	Type	Purpose
idx_key_value	key_value	Unique	Fast lookup by idempotency key
idx_resource_id	resource_id	Standard	Efficient resource-based queries
idx_user_id	user_id	Standard	User-scoped idempotency key lookups
idx_expires_at	expires_at	Standard	Efficient cleanup of expired keys
idx_status	status	Standard	Fast filtering by key state
idx_created_at	created_at	Standard	Audit trail and time-based queries

Data Partitioning / Sharding Strategy

Partition by user_id hash to distribute load across shards:

Hash(user_id) % shard_count
Each shard maintains its own set of idempotency keys
Cross-shard queries are handled through a coordinator service

Soft Delete vs. Hard Delete Decision

Soft delete is implemented with a status field instead of physical deletion because:

Maintains audit trail for compliance and debugging
Allows recovery of accidentally deleted keys
Supports historical analysis of idempotency usage patterns
Enables reprocessing of failed operations with same keys

Audit Trail Design

Each idempotency key record tracks:

created_at: Timestamp of creation
last_accessed_at: When key was last used
retry_count: Number of times operation was retried with same key
status: Current state (ACTIVE, EXPIRED, CONSUMED)

Data Archival Strategy and Retention Policy

Active keys: Retained for 7 days (configurable)
Consumed keys: Retained for 30 days (audit requirement)
Expired keys: Retained for 7 days before soft-deletion
Archival process runs daily to move old keys to cold storage (S3 or similar)
Keys older than 90 days are permanently deleted from database

4. Class Design & Service Layer

classDiagram
    class IdempotencyKeyService {
        +createIdempotencyKey(String, String, String, Duration)
        +executeWithIdempotency(String, String, String, Supplier~T~, Duration)
        +validateRequest(String, String, String)
    }
    
    class IdempotencyKeyRepository {
        +save(IdempotencyKey)
        +findById(String)
        +findByRequestHash(String)
        +deleteExpiredKeys()
    }
    
    class IdempotencyKey {
        -id: String
        -requestHash: String
        -userId: String
        -key: String
        -createdAt: Instant
        -expiresAt: Instant
        -status: IdempotencyStatus
        -result: String
        +isExpired(): boolean
    }
    
    class IdempotencyKeyValidator {
        +validate(String, String, String)
    }
    
    class IdempotencyKeyException {
        -errorCode: String
        -message: String
    }
    
    class IdempotencyKeyManager {
        -cache: Cache~String, IdempotencyKey~
        -repository: IdempotencyKeyRepository
        -lockProvider: LockProvider
    }
    
    class IdempotencyStatus {
        PENDING
        SUCCESS
        FAILURE
    }
    
    IdempotencyKeyService --> IdempotencyKeyRepository : uses
    IdempotencyKeyService --> IdempotencyKeyManager : delegates to
    IdempotencyKeyManager --> IdempotencyKeyRepository : persists
    IdempotencyKeyManager --> IdempotencyKeyValidator : validates
    IdempotencyKeyManager --> Cache : caches
    IdempotencyKeyManager --> LockProvider : manages concurrent access
    
    class IdempotencyKey {
        <<value object>>
    }
    
    class IdempotencyKeyValidator {
        <<component>>
    }

stateDiagram-v2
    [*] --> PENDING
    PENDING --> SUCCESS : successful execution
    PENDING --> FAILURE : failed execution
    SUCCESS --> [*]
    FAILURE --> [*]

Service Class Structure and Responsibilities

The IdempotencyKeyService is the primary entry point for handling idempotent operations. It encapsulates:

Key generation and validation logic
Execution context management for idempotent operations
Integration with caching and persistence layers

The IdempotencyKeyManager handles the core business logic for idempotency:

Concurrent access control using distributed locks
Cache management for fast lookups
Status tracking and result caching

Interface / Abstract Class Design

public interface IdempotencyKeyRepository {
    IdempotencyKey save(IdempotencyKey key);
    Optional<IdempotencyKey> findById(String id);
    Optional<IdempotencyKey> findByRequestHash(String requestHash);
    void deleteExpiredKeys();
}

public interface LockProvider {
    boolean acquireLock(String key, Duration timeout);
    void releaseLock(String key);
}

public interface IdempotencyKeyValidator {
    void validate(String userId, String key, String requestHash) throws IdempotencyKeyException;
}

Business Logic Encapsulation Patterns

We implement the Strategy Pattern for handling different idempotency scenarios:

public interface IdempotencyStrategy {
    <T> T execute(String userId, String key, Supplier<T> operation) throws IdempotencyKeyException;
}

public class IdempotentOperationStrategy implements IdempotencyStrategy {
    private final IdempotencyKeyManager manager;
    
    @Override
    public <T> T execute(String userId, String key, Supplier<T> operation) 
            throws IdempotencyKeyException {
        return manager.executeWithIdempotency(userId, key, operation);
    }
}

Critical-Path Method Implementation

public <T> T executeWithIdempotency(String userId, String key, Supplier<T> operation, Duration ttl) {
    String requestHash = generateRequestHash(userId, key);
    
    // Step 1: Validate the request and acquire lock
    String lockKey = "idempotency:" + requestHash;
    boolean acquired = lockProvider.acquireLock(lockKey, Duration.ofSeconds(5));
    
    if (!acquired) {
        throw new IdempotencyKeyException("Request processing in progress, try again later");
    }
    
    try {
        // Step 2: Check if key already exists in cache
        IdempotencyKey cachedKey = cache.getIfPresent(requestHash);
        if (cachedKey != null) {
            if (cachedKey.isExpired()) {
                cache.invalidate(requestHash);
            } else {
                return (T) cachedKey.getResult();
            }
        }
        
        // Step 3: Check database for existing key
        Optional<IdempotencyKey> existingKey = repository.findByRequestHash(requestHash);
        
        if (existingKey.isPresent()) {
            IdempotencyKey keyEntity = existingKey.get();
            
            if (keyEntity.isExpired()) {
                repository.delete(keyEntity);
                // Proceed to create new key
            } else {
                switch (keyEntity.getStatus()) {
                    case SUCCESS:
                        return (T) keyEntity.getResult();
                    case FAILURE:
                        throw new IdempotencyKeyException("Previous operation failed");
                    case PENDING:
                        // Wait for completion or timeout
                        throw new IdempotencyKeyException("Operation in progress");
                }
            }
        }
        
        // Step 4: Create new key and execute operation
        IdempotencyKey newKey = new IdempotencyKey(
            UUID.randomUUID().toString(),
            requestHash,
            userId,
            key,
            Instant.now(),
            Instant.now().plus(ttl),
            IdempotencyStatus.PENDING
        );
        
        repository.save(newKey);
        cache.put(requestHash, newKey);
        
        T result;
        try {
            result = operation.get();
            
            // Update key status to success
            newKey.setStatus(IdempotencyStatus.SUCCESS);
            newKey.setResult(result.toString());
            
            // Publish success event
            eventPublisher.publish(new IdempotencySuccessEvent(userId, key, result));
            
        } catch (Exception e) {
            // Update key status to failure
            newKey.setStatus(IdempotencyStatus.FAILURE);
            newKey.setResult(e.getMessage());
            
            // Publish failure event
            eventPublisher.publish(new IdempotencyFailureEvent(userId, key, e));
            
            throw new IdempotencyKeyException("Operation failed: " + e.getMessage(), e);
        }
        
        repository.save(newKey);
        cache.put(requestHash, newKey);
        return result;
        
    } finally {
        lockProvider.releaseLock(lockKey);
    }
}

Transaction Boundary Design

Each idempotent operation is wrapped in a single database transaction that:

Ensures atomicity of key creation and status updates
Provides consistent read-after-write semantics for the same key
Is bounded by a reasonable timeout to prevent deadlocks

Service Interface Contracts

public interface IdempotencyKeyService {
    <T> T executeWithIdempotency(String userId, String key, Supplier<T> operation, Duration ttl);
    IdempotencyKey createIdempotencyKey(String userId, String key, String operationType, Duration ttl);
    boolean validateRequest(String userId, String key, String operationType) throws IdempotencyKeyException;
}

Saga / Orchestration Pattern

For complex multi-step operations, we would implement a saga pattern where:

Each step maintains its own idempotency key
The saga coordinator tracks all steps and their statuses
Compensation logic is idempotent and can be replayed safely

Domain Object / Value Object Design

public class IdempotencyKey {
    private final String id;
    private final String requestHash;
    private final String userId;
    private final String key;
    private final Instant createdAt;
    private final Instant expiresAt;
    private IdempotencyStatus status;
    private String result;
    
    public IdempotencyKey(String id, String requestHash, String userId, String key,
                         Instant createdAt, Instant expiresAt, IdempotencyStatus status) {
        this.id = id;
        this.requestHash = requestHash;
        this.userId = userId;
        this.key = key;
        this.createdAt = createdAt;
        this.expiresAt = expiresAt;
        this.status = status;
    }
    
    public boolean isExpired() {
        return Instant.now().isAfter(expiresAt);
    }
    
    // Getters and setters
}

The design ensures that all idempotent operations are safe, consistent, and performant while maintaining clear separation of concerns between validation, execution, caching, and persistence layers.

5. Concurrency & Thread Safety

Concurrency Model

The idempotency system operates in a multi-threaded environment where multiple requests can arrive simultaneously for the same idempotency key. The system uses a thread-safe approach with:

Thread-safe hash maps for in-memory cache lookups
Database-level atomic operations for key validation and creation
Read-Write locks where appropriate to minimize contention
Async/Await pattern for non-blocking operations in high-concurrency scenarios

Race Condition Prevention Strategy

Race conditions are prevented through:

Database-level atomic operations - Using INSERT ... ON DUPLICATE KEY UPDATE or equivalent for MySQL/PostgreSQL to ensure atomic creation and validation
Optimistic locking with version fields - To detect concurrent modifications to the same key
Distributed locks for cross-service idempotency validation (when distributed across microservices)
Idempotency key namespace isolation - Different namespaces can be processed in parallel without conflict

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected for the following reasons:

High throughput scenarios: Idempotency checks happen frequently, pessimistic locks would create significant contention
Low conflict rate: Most idempotency keys are unique or have minimal concurrent access
Better scalability: Optimistic approach scales better with horizontal service instances
Reduced latency: No blocking waits for locks

The system uses version numbers or timestamps to detect conflicts and retries operations when conflicts occur.

Deadlock Prevention Strategy

Deadlock prevention is handled through:

Consistent lock ordering: All operations acquire locks in the same order (e.g., by key namespace, then by key)
Timeout-based lock acquisition: Each operation has a maximum wait time (e.g., 50ms) before giving up
Lock hierarchy: Idempotency keys are grouped by namespace with consistent processing order
Retry logic with exponential backoff: When deadlocks occur, operations retry with increasing delays

Idempotency Implementation for Concurrent Duplicate Requests

Duplicate requests are handled through:

public class IdempotencyService {
    private final IdempotencyRepository repository;
    private final Cache<String, IdempotencyRecord> cache;
    
    public IdempotencyResult executeWithIdempotency(
        String idempotencyKey, 
        String namespace,
        Supplier<IdempotencyResult> operation) {
        
        // Step 1: Try to get from cache first (fast path)
        IdempotencyRecord cached = cache.getIfPresent(idempotencyKey);
        if (cached != null && !isExpired(cached)) {
            return new IdempotencyResult(cached.getResult(), true);
        }
        
        // Step 2: Use database-level atomic operation to handle concurrent access
        try {
            // Attempt to create or retrieve existing record with optimistic locking
            IdempotencyRecord record = repository.createOrGet(
                namespace, 
                idempotencyKey, 
                operation::getOperationIdempotencyResult);
            
            // Step 3: If record already existed and is not expired, return cached result
            if (record.isProcessed() && !isExpired(record)) {
                cache.put(idempotencyKey, record);
                return new IdempotencyResult(record.getResult(), true);
            }
            
            // Step 4: Process the operation if not already done
            IdempotencyResult result = operation.get();
            
            // Step 5: Update record with final result (atomic update)
            repository.updateResult(idempotencyKey, result);
            
            // Step 6: Cache the result for fast retrieval
            cache.put(idempotencyKey, new IdempotencyRecord(
                idempotencyKey, 
                result, 
                System.currentTimeMillis()));
            
            return result;
        } catch (IdempotencyConflictException e) {
            // Handle case where concurrent operation was already processed
            return new IdempotencyResult(e.getExistingResult(), true);
        }
    }
}

The system handles concurrent duplicate requests by:

First checking cache for fast response
Using atomic database operations to ensure only one operation proceeds
Returning the cached result if already processed
Implementing retry logic with exponential backoff for transient failures
Using proper exception handling to distinguish between real conflicts and system errors

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant C as Client
    participant A as API Gateway
    participant S as Service Layer
    participant R as Repository
    participant M as Message Queue

    C->>A: POST /orders<br/>Idempotency-Key: abc123<br/>Request-Body: {amount: 100}
    A->>S: ProcessIdempotentRequest<br/>idempotencyKey: abc123
    S->>R: CheckIfIdempotentRecordExists(idempotencyKey)
    R-->>S: Found existing record
    S->>R: GetResultFromPreviousRequest(idempotencyKey)
    R-->>S: Return previous order ID: 12345
    S-->>A: Return 200 OK<br/>Order ID: 12345<br/>Previous request handled
    A-->>C: Return 200 OK<br/>Order ID: 12345<br/>Previous request handled

sequenceDiagram
    participant C as Client
    participant A as API Gateway
    participant S as Service Layer
    participant R as Repository
    participant DB as Database

    C->>A: POST /orders<br/>Idempotency-Key: def456<br/>Request-Body: {amount: 200}
    A->>S: ProcessIdempotentRequest<br/>idempotencyKey: def456
    S->>R: CheckIfIdempotentRecordExists(idempotencyKey)
    R-->>S: No existing record
    S->>DB: Begin Transaction
    S->>R: CreateIdempotentRecord(idempotencyKey, requestPayload)
    S->>DB: CreateOrder(requestPayload)
    DB-->>S: Order created with ID: 67890
    S->>R: UpdateIdempotentRecord(idempotencyKey, orderId: 67890)
    S->>DB: Commit Transaction
    S-->>A: Return 201 Created<br/>Order ID: 67890
    A-->>C: Return 201 Created<br/>Order ID: 67890

sequenceDiagram
    participant C as Client
    participant A as API Gateway
    participant S as Service Layer
    participant R as Repository
    participant M as Event Publisher

    C->>A: POST /orders<br/>Idempotency-Key: ghi789<br/>Request-Body: {amount: 300}
    A->>S: ProcessIdempotentRequest<br/>idempotencyKey: ghi789
    S->>R: CheckIfIdempotentRecordExists(idempotencyKey)
    R-->>S: No existing record
    S->>R: CreateIdempotentRecord(idempotencyKey, requestPayload)
    S->>R: ProcessOrder(requestPayload)
    S->>M: Publish OrderCreatedEvent(orderId: 12345)
    M-->>S: Event published successfully
    S->>R: UpdateIdempotentRecord(idempotencyKey, orderId: 12345)
    S-->>A: Return 201 Created<br/>Order ID: 12345
    A-->>C: Return 201 Created<br/>Order ID: 12345

7. Caching Design

Cache Strategy Selection

For idempotency key system, we'll use cache-aside pattern with write-through behavior for the core idempotency store. This ensures that every idempotency key operation is first checked in cache, and if miss occurs, the database is queried and result cached.

Cache Key Design

Cache keys will follow a structured namespace pattern:

idempotency:{key_type}:{key_value}

Example: idempotency:request:abc123def456

TTL Strategy

Data Type	TTL	Reason
Idempotency Keys	24 hours	Allows for retry window while keeping cache clean
Recent Operations	1 hour	Recent operations are more likely to be retried
Failed Requests	2 hours	Extended window for debugging failed retries

Cache Eviction Policy

We'll use LRU (Least Recently Used) eviction policy with a maximum capacity of 100,000 entries to prevent memory bloat. This ensures that the most frequently accessed idempotency keys remain in cache.

Cache Stampede Prevention

To prevent cache stampede:

Probabilistic early expiry: Set cache TTL to 20% shorter than configured TTL (e.g., 19.2 hours instead of 24 hours)
Distributed locks: Use Redis distributed locks (SET key value NX EX ttl) for cache population operations to ensure only one request populates the cache for a given key

Distributed Cache Architecture

We'll use Redis Cluster with:

3 master nodes for write operations
3 replica nodes for read operations
Automatic failover capability
Cross-zone replication for high availability

Cache Invalidation Strategy

We'll implement event-driven invalidation:

When an idempotency key is consumed or expires, a cache invalidation event is published
Cache consumers listen to these events and invalidate the corresponding keys
For immediate invalidation, we'll use a cache-invalidation queue that processes events asynchronously

@Component
public class IdempotencyCacheService {
    
    private final RedisTemplate<String, String> redisTemplate;
    private final ObjectMapper objectMapper;
    private static final String CACHE_PREFIX = "idempotency";
    
    public IdempotencyCacheService(RedisTemplate<String, String> redisTemplate, 
                                   ObjectMapper objectMapper) {
        this.redisTemplate = redisTemplate;
        this.objectMapper = objectMapper;
    }
    
    public boolean isIdempotent(String keyType, String keyValue) {
        String cacheKey = buildCacheKey(keyType, keyValue);
        
        // Try to get from cache first
        String cachedValue = redisTemplate.opsForValue().get(cacheKey);
        
        if (cachedValue != null) {
            try {
                IdempotencyRecord record = objectMapper.readValue(cachedValue, IdempotencyRecord.class);
                return !record.isConsumed() && record.getExpiryTime().isAfter(Instant.now());
            } catch (Exception e) {
                // Log error and fall through to database check
                log.error("Failed to deserialize cached idempotency record", e);
            }
        }
        
        return false;
    }
    
    public void storeIdempotency(String keyType, String keyValue, IdempotencyRecord record) {
        String cacheKey = buildCacheKey(keyType, keyValue);
        
        try {
            // Use distributed lock to prevent cache stampede
            String lockKey = "lock:" + cacheKey;
            String lockValue = UUID.randomUUID().toString();
            
            if (redisTemplate.opsForValue().setIfAbsent(lockKey, lockValue, 
                Duration.ofSeconds(5), TimeUnit.SECONDS)) {
                
                try {
                    String recordJson = objectMapper.writeValueAsString(record);
                    redisTemplate.opsForValue().set(cacheKey, recordJson, 
                        Duration.ofHours(24));
                    
                    // Publish cache invalidation event for eventual consistency
                    publishCacheInvalidationEvent(keyType, keyValue);
                } finally {
                    // Release the lock
                    releaseLock(lockKey, lockValue);
                }
            } else {
                // Another thread is handling this, wait briefly and retry
                Thread.sleep(100);
                // Retry logic can be implemented here if needed
            }
        } catch (Exception e) {
            log.error("Failed to store idempotency in cache", e);
        }
    }
    
    private String buildCacheKey(String keyType, String keyValue) {
        return String.format("%s:%s:%s", CACHE_PREFIX, keyType, keyValue);
    }
    
    private void releaseLock(String lockKey, String lockValue) {
        String script = 
            "if redis.call('GET', KEYS[1]) == ARGV[1] then " +
            "return redis.call('DEL', KEYS[1]) else return 0 end";
        
        redisTemplate.execute(
            new DefaultRedisScript<>(script, Long.class),
            Collections.singletonList(lockKey),
            lockValue
        );
    }
    
    private void publishCacheInvalidationEvent(String keyType, String keyValue) {
        // Implementation would publish to message queue for async invalidation
        // This ensures cache consistency across distributed systems
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and database operations. Each entity will have its own repository interface and implementation.

public interface IdempotencyKeyRepository {
    Optional<IdempotencyKey> findById(String id);
    IdempotencyKey save(IdempotencyKey key);
    void deleteById(String id);
    boolean existsById(String id);
}

@Repository
public class IdempotencyKeyRepositoryImpl implements IdempotencyKeyRepository {
    private final JdbcTemplate jdbcTemplate;
    private final ObjectMapper objectMapper;
    
    public IdempotencyKeyRepositoryImpl(JdbcTemplate jdbcTemplate, ObjectMapper objectMapper) {
        this.jdbcTemplate = jdbcTemplate;
        this.objectMapper = objectMapper;
    }
    
    @Override
    public Optional<IdempotencyKey> findById(String id) {
        try {
            String sql = "SELECT * FROM idempotency_keys WHERE id = ?";
            return Optional.ofNullable(jdbcTemplate.queryForObject(sql, new Object[]{id}, 
                (rs, rowNum) -> mapRowToIdempotencyKey(rs)));
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }
    
    @Override
    public IdempotencyKey save(IdempotencyKey key) {
        String sql = "INSERT INTO idempotency_keys (id, entity_id, operation_type, payload_hash, created_at, expires_at) " +
                     "VALUES (?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE " +
                     "entity_id = VALUES(entity_id), operation_type = VALUES(operation_type), " +
                     "payload_hash = VALUES(payload_hash), created_at = VALUES(created_at), " +
                     "expires_at = VALUES(expires_at)";
        
        jdbcTemplate.update(sql, key.getId(), key.getEntityId(), key.getOperationType(),
                           key.getPayloadHash(), key.getCreatedAt(), key.getExpiresAt());
        
        return key;
    }
    
    @Override
    public void deleteById(String id) {
        String sql = "DELETE FROM idempotency_keys WHERE id = ?";
        jdbcTemplate.update(sql, id);
    }
    
    @Override
    public boolean existsById(String id) {
        String sql = "SELECT COUNT(*) FROM idempotency_keys WHERE id = ?";
        return jdbcTemplate.queryForObject(sql, Integer.class, id) > 0;
    }
    
    private IdempotencyKey mapRowToIdempotencyKey(ResultSet rs) throws SQLException {
        return new IdempotencyKey(
            rs.getString("id"),
            rs.getString("entity_id"),
            rs.getString("operation_type"),
            rs.getString("payload_hash"),
            rs.getTimestamp("created_at").toInstant().atZone(ZoneId.systemDefault()).toLocalDateTime(),
            rs.getTimestamp("expires_at").toInstant().atZone(ZoneId.systemDefault()).toLocalDateTime()
        );
    }
}

ORM vs. Raw Query Decisions

For the Idempotency Key system, raw SQL queries are preferred over ORM for several reasons:

Performance: Raw queries provide better performance for simple CRUD operations with predictable query plans
Control: Direct control over query execution and result mapping for complex operations
Flexibility: Ability to use database-specific features like ON DUPLICATE KEY UPDATE for atomic upserts
Debugging: Easier to debug and optimize specific queries

Read vs. Write Separation

The system uses a read-write separation strategy with dedicated database connections:

public class IdempotencyKeyReadRepository {
    private final JdbcTemplate readTemplate;
    
    public IdempotencyKeyReadRepository(JdbcTemplate readTemplate) {
        this.readTemplate = readTemplate;
    }
    
    public Optional<IdempotencyKey> findById(String id) {
        // Read-only query on replica or read-only connection
        String sql = "SELECT * FROM idempotency_keys WHERE id = ? AND expires_at > NOW()";
        return Optional.ofNullable(readTemplate.queryForObject(sql, new Object[]{id}, 
            (rs, rowNum) -> mapRowToIdempotencyKey(rs)));
    }
}

public class IdempotencyKeyWriteRepository {
    private final JdbcTemplate writeTemplate;
    
    public IdempotencyKeyWriteRepository(JdbcTemplate writeTemplate) {
        this.writeTemplate = writeTemplate;
    }
    
    public IdempotencyKey save(IdempotencyKey key) {
        // Write operation on primary database
        String sql = "INSERT INTO idempotency_keys (id, entity_id, operation_type, payload_hash, created_at, expires_at) " +
                     "VALUES (?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE " +
                     "entity_id = VALUES(entity_id), operation_type = VALUES(operation_type), " +
                     "payload_hash = VALUES(payload_hash), created_at = VALUES(created_at), " +
                     "expires_at = VALUES(expires_at)";
        
        writeTemplate.update(sql, key.getId(), key.getEntityId(), key.getOperationType(),
                           key.getPayloadHash(), key.getCreatedAt(), key.getExpiresAt());
        
        return key;
    }
}

Connection Pool Configuration

Connection pool configuration is optimized for the idempotency key system:

# application.yml
spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      leak-detection-threshold: 60000
      pool-name: IdempotencyKeyPool

N+1 Query Prevention Strategy

The system avoids N+1 query problems through:

Batch Operations: Bulk operations for key validation
Eager Loading: Pre-fetching related data when needed
Query Optimization: Using JOINs instead of multiple queries

public class IdempotencyKeyBatchValidator {
    private final IdempotencyKeyReadRepository readRepo;
    
    public List<IdempotencyKey> validateBatch(List<String> keys) {
        if (keys.isEmpty()) return Collections.emptyList();
        
        // Single query to validate all keys
        String sql = "SELECT id, entity_id, operation_type, payload_hash, created_at, expires_at " +
                     "FROM idempotency_keys WHERE id IN (" + 
                     keys.stream().map(k -> "?").collect(Collectors.joining(",")) + ")";
        
        Object[] params = keys.toArray();
        return readRepo.findMultipleByIds(sql, params);
    }
}

Batch Operation Design

Batch operations are designed for high-throughput scenarios:

public class IdempotencyKeyBatchRepository {
    private final JdbcTemplate jdbcTemplate;
    
    public void saveBatch(List<IdempotencyKey> keys) {
        String sql = "INSERT INTO idempotency_keys (id, entity_id, operation_type, payload_hash, created_at, expires_at) " +
                     "VALUES (?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE " +
                     "entity_id = VALUES(entity_id), operation_type = VALUES(operation_type), " +
                     "payload_hash = VALUES(payload_hash), created_at = VALUES(created_at), " +
                     "expires_at = VALUES(expires_at)";
        
        // Batch processing for better performance
        jdbcTemplate.batchUpdate(sql, new BatchPreparedStatementSetter() {
            @Override
            public void setValues(PreparedStatement ps, int i) throws SQLException {
                IdempotencyKey key = keys.get(i);
                ps.setString(1, key.getId());
                ps.setString(2, key.getEntityId());
                ps.setString(3, key.getOperationType());
                ps.setString(4, key.getPayloadHash());
                ps.setTimestamp(5, Timestamp.valueOf(key.getCreatedAt()));
                ps.setTimestamp(6, Timestamp.valueOf(key.getExpiresAt()));
            }
            
            @Override
            public int getBatchSize() {
                return keys.size();
            }
        });
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature — what triggers async vs. what stays synchronous, and why

The idempotency key system is primarily a synchronous request handling mechanism, but it can benefit from asynchronous processing in specific scenarios:

Idempotency key cleanup: Expired keys should be cleaned up asynchronously to avoid blocking API requests.
Idempotency key monitoring: Metrics and analytics about usage patterns should be processed asynchronously.
Cross-service idempotency coordination: When multiple services use the same idempotency key, async coordination is needed to maintain consistency.

For core idempotency validation, synchronous processing is preferred for immediate response to clients. The system should validate the key synchronously within the request lifecycle, but background cleanup and analytics can be async.

Message schema design with a sample payload (Avro / Protobuf / JSON Schema — with versioning strategy)

{
  "type": "object",
  "title": "IdempotencyKeyCleanupEvent",
  "version": "1.0",
  "properties": {
    "idempotencyKey": {
      "type": "string",
      "description": "The idempotency key to clean up"
    },
    "serviceId": {
      "type": "string",
      "description": "Identifier of the service that owns this key"
    },
    "createdAt": {
      "type": "string",
      "format": "date-time",
      "description": "Timestamp when the key was created"
    },
    "expiresAt": {
      "type": "string",
      "format": "date-time",
      "description": "Timestamp when the key expires"
    }
  },
  "required": ["idempotencyKey", "serviceId", "createdAt", "expiresAt"]
}

Versioning strategy uses semantic versioning with backward compatibility in mind. When schema changes, a new version is created and consumers are updated gradually.

Dead letter queue strategy and poison message handling

For idempotency key cleanup events:

Messages that fail processing after max retries (e.g., 3) are sent to a dead letter queue (DLQ)
DLQ messages are logged with full context for manual inspection
A separate monitoring system alerts on DLQ queue size and content

Poison message handling includes:

Logging the full event context with trace IDs
Alerting on DLQ queue size exceeding threshold (e.g., 100 messages)
Manual intervention process for investigating failed cleanup operations

At-least-once vs. exactly-once delivery — decision with justification for this feature

At-least-once delivery is chosen for idempotency key system because:

Idempotency is the core requirement: If a message is processed twice, it should have the same effect as once
Idempotency keys are immutable: Once created, they don't change state, so duplicate processing doesn't cause data inconsistency
Simpler implementation: At-least-once is easier to implement and debug than exactly-once
Performance: Avoids the complexity and overhead of maintaining transaction state across services

The system handles duplicates by checking if a key already exists before processing, making the at-least-once delivery safe.

Retry policy for consumers (max attempts, backoff formula, non-retryable error classification)

Retry policy configuration:

Max attempts: 3
Backoff formula: exponential with jitter (base 1s, multiplier 2, jitter ±0.5s)
Retryable errors: Network timeouts, service unavailability, database connection issues
Non-retryable errors: Invalid key format, malformed payload, authorization failures

public class IdempotencyKeyCleanupConsumer {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_RANGE = 0.5;
    
    public void processCleanupEvent(CleanupEvent event) {
        int attempts = 0;
        while (attempts < MAX_RETRIES) {
            try {
                cleanupKey(event);
                return;
            } catch (RetryableException e) {
                attempts++;
                if (attempts >= MAX_RETRIES) {
                    moveMessageToDLQ(event);
                    throw new RuntimeException("Failed to process cleanup event after " + MAX_RETRIES + " attempts", e);
                }
                long delay = calculateBackoffDelay(attempts);
                Thread.sleep(delay);
            } catch (NonRetryableException e) {
                moveMessageToDLQ(event);
                throw new RuntimeException("Non-retryable error in cleanup event processing", e);
            }
        }
    }
    
    private long calculateBackoffDelay(int attempt) {
        double exponentialDelay = Math.pow(2, attempt - 1) * BASE_DELAY_MS;
        double jitter = (Math.random() - 0.5) * JITTER_RANGE * exponentialDelay;
        return Math.max(100, (long)(exponentialDelay + jitter));
    }
}

Outbox pattern for reliable event publishing — include or explicitly justify omission

Outbox pattern is omitted for the idempotency key system because:

Synchronous validation: Idempotency key validation happens synchronously within the request lifecycle, so immediate consistency is required
Simple event flow: The system primarily generates cleanup events for expired keys, which don't require distributed transaction semantics
Performance considerations: Adding outbox pattern would introduce additional latency and complexity for a system that's primarily about request validation
Idempotency is the primary concern: The system's core value is ensuring idempotent operations, not reliable event delivery

Instead, the system uses direct message publishing with proper error handling and retry logic.

Sequence diagram for all async flows (Mermaid)

sequenceDiagram
    participant A as IdempotencyKeyService
    participant M as MessageBroker
    participant C as CleanupConsumer
    
    A->>M: Publish CleanupEvent (key expired)
    M->>C: Deliver CleanupEvent
    C->>C: Validate cleanup event
    C->>C: Check key existence in DB
    C->>M: Acknowledge successful processing
    C->>C: Log cleanup completion
    
    Note over A,C: Async cleanup of expired keys

10. Error Handling Design

Error Classification Hierarchy

The system implements a hierarchical error handling strategy with clear categorization:

public abstract class BaseException extends RuntimeException {
    private final ErrorCode errorCode;
    
    public BaseException(ErrorCode errorCode, String message) {
        super(message);
        this.errorCode = errorCode;
    }
    
    public ErrorCode getErrorCode() {
        return errorCode;
    }
}

public class ValidationException extends BaseException {
    public ValidationException(ErrorCode code, String message) {
        super(code, message);
    }
}

public class BusinessException extends BaseException {
    public BusinessException(ErrorCode code, String message) {
        super(code, message);
    }
}

public class InfrastructureException extends BaseException {
    public InfrastructureException(ErrorCode code, String message) {
        super(code, message);
    }
}

Error Propagation Strategy

Errors follow a layered propagation model:

Domain layer: Throws business exceptions that are converted to appropriate HTTP responses
Service layer: Handles infrastructure exceptions and re-throws domain exceptions where appropriate
Controller layer: Catches all exceptions and maps to standardized error responses

Retry Strategy with Exponential Backoff

For transient infrastructure errors, the system implements a robust retry mechanism:

@Retryable(
    maxAttempts = 3,
    backoff = @Backoff(
        delay = 1000, 
        multiplier = 2,
        maxDelay = 10000
    ),
    include = {InfrastructureException.class},
    exclude = {ValidationException.class, BusinessException.class}
)
public void processIdempotentRequest(String idempotencyKey) {
    // Business logic here
}

Retry configuration parameters:

Max attempts: 3 retries (total 7 attempts including original)
Initial delay: 1 second
Multiplier: 2x (exponential backoff)
Max delay: 10 seconds
Jitter: ±25% randomization to prevent thundering herd

Circuit Breaker Design

The system employs a circuit breaker for external dependencies:

@CircuitBreaker(
    failureThreshold = 5,
    waitDuration = Duration.ofSeconds(30),
    recordExceptions = {InfrastructureException.class}
)
public CompletableFuture<IdempotencyResult> processRequestAsync(String key) {
    // Async processing logic
}

Circuit breaker behavior:

Failure threshold: 5 consecutive failures within 60 seconds
Timeout: 30 seconds before half-open state
Half-open state: Allows one request to test if the dependency is recovered
Fallback: Returns cached result or throws timeout exception

Fallback Behavior Design

@CircuitBreaker(
    failureThreshold = 5,
    waitDuration = Duration.ofSeconds(30)
)
@Fallback(
    fallbackMethod = "handleIdempotencyFailure"
)
public CompletableFuture<IdempotencyResult> processRequestAsync(String key) {
    return externalService.process(key);
}

public CompletableFuture<IdempotencyResult> handleIdempotencyFailure(String key, Exception ex) {
    // Log and return default behavior or cached response
    logger.warn("Fallback triggered for key: {}", key, ex);
    return CompletableFuture.completedFuture(
        new IdempotencyResult(key, "fallback", System.currentTimeMillis())
    );
}

Error Logging Format

Structured logging with consistent schema:

public class ErrorLog {
    private final String traceId;
    private final String spanId;
    private final String userId;
    private final String entityId;
    private final String errorCode;
    private final String errorMessage;
    private final String stackTrace;
    private final long timestamp;
    private final String service;
    private final String component;
    
    // Constructor and getters
}

Log fields include:

trace_id: Distributed tracing identifier (OpenTelemetry)
span_id: Current operation span
user_id: Authenticated user identifier (masked in logs)
entity_id: Affected business entity ID
error_code: Standardized error code (e.g., IDP001, IDP002)
message: Human-readable error description
stack_trace: Full stack trace for debugging (with PII filtering)
timestamp: UTC timestamp of error occurrence
service: Service name (idempotency-service)
component: Module or class where error occurred

Alerting Thresholds

Critical alerts are configured for:

Error rate: >5% of total requests in 5-minute window
Latency: >95th percentile response time >2s for 10 consecutive samples
Infrastructure errors: >50 concurrent failures in 1-minute window

Error Budget Management

The system enforces error budget consumption:

Target: 99.9% availability (0.1% error rate)
Budget: 100ms daily error budget
Alerts:
- Warning: 50ms consumed (50% of daily budget)
- Critical: 100ms consumed (100% of daily budget)

Retryable vs Non-Retryable Errors

Error Type	Retryable	Reason
InfrastructureException	✅	Network timeouts, DB connection issues
ValidationException	❌	Input validation errors - no point retrying
BusinessException	❌	Business logic errors - idempotency key already exists
RateLimitException	❌	Rate limiting - client needs to wait

Error Response Format

Standardized JSON error response:

{
  "error": {
    "code": "IDP001",
    "message": "Idempotency key already exists",
    "details": {
      "key": "user-123-order-456",
      "timestamp": "2023-12-01T10:30:00Z"
    },
    "retryAfter": 5000
  }
}

Error Code Registry

public enum ErrorCode {
    IDP001("Idempotency key already exists"),
    IDP002("Invalid idempotency key format"),
    IDP003("Request timeout - operation still in progress"),
    IDP004("Idempotency key expired"),
    IDP005("Service unavailable - retry later");
    
    private final String description;
    
    ErrorCode(String description) {
        this.description = description;
    }
    
    public String getDescription() {
        return description;
    }
}

Circuit Breaker State Transitions

The circuit breaker follows these state transitions:

Closed: Requests allowed, failure count tracked
Open: All requests fail fast, timeout period active
Half-Open: One request allowed to test recovery, if successful transitions back to closed

Retry Strategy Implementation

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double BACKOFF_MULTIPLIER = 2.0;
    private static final long MAX_DELAY_MS = 10000;
    
    public <T> T executeWithRetry(Supplier<T> operation, Predicate<Exception> retryable) {
        Exception lastException = null;
        
        for (int attempt = 0; attempt <= MAX_RETRIES; attempt++) {
            try {
                return operation.get();
            } catch (Exception e) {
                lastException = e;
                
                if (attempt == MAX_RETRIES || !retryable.test(e)) {
                    throw new InfrastructureException(
                        ErrorCode.IDP003, 
                        "Operation failed after " + MAX_RETRIES + " attempts", 
                        e
                    );
                }
                
                long delay = Math.min(
                    (long) (BASE_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempt)),
                    MAX_DELAY_MS
                );
                
                // Add jitter (±25%)
                delay = (long) (delay * (0.75 + Math.random() * 0.5));
                
                try {
                    Thread.sleep(delay);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new InfrastructureException(
                        ErrorCode.IDP003, 
                        "Retry interrupted", 
                        ie
                    );
                }
            }
        }
        
        throw new InfrastructureException(
            ErrorCode.IDP003, 
            "Operation failed after " + MAX_RETRIES + " attempts", 
            lastException
        );
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All API inputs undergo strict validation using a combination of:

Regex patterns for format validation (e.g., UUID format for idempotency keys)
Length limits to prevent buffer overflow attacks
Type checking to ensure correct data types
Custom validators for business logic constraints

public class IdempotencyKeyValidator {
    private static final Pattern IDEMPOTENCY_KEY_PATTERN = 
        Pattern.compile("^[a-zA-Z0-9_-]{1,255}$");
    
    public static boolean isValid(String key) {
        return key != null && 
               key.length() <= 255 && 
               IDEMPOTENCY_KEY_PATTERN.matcher(key).matches();
    }
}

SQL Injection Prevention

All database queries use parameterized prepared statements exclusively. No dynamic query construction or string concatenation is allowed.

Authentication Middleware Design

Authentication middleware intercepts all requests and validates:

JWT tokens for API key authentication (if applicable)
OAuth2 Bearer tokens for service-to-service communication
API key headers with rate limiting enforcement

@Component
public class IdempotencyAuthMiddleware {
    private final JwtDecoder jwtDecoder;
    
    public void validateRequest(HttpServletRequest request) {
        String authHeader = request.getHeader("Authorization");
        if (authHeader == null || !authHeader.startsWith("Bearer ")) {
            throw new SecurityException("Missing or invalid auth token");
        }
        
        String token = authHeader.substring(7);
        try {
            jwtDecoder.decode(token);
        } catch (JwtException e) {
            throw new SecurityException("Invalid JWT token", e);
        }
    }
}

Authorization Enforcement

RBAC-based authorization ensures only authorized users can access idempotency key operations:

Admin role: Full CRUD access to all keys
User role: Read-only access to own keys
Service role: Write access for internal system operations

@PreAuthorize("hasRole('ADMIN') or " +
             "hasRole('USER') and @idempotencyKeyService.isOwner(#keyId)")
public void deleteKey(String keyId) {
    // Implementation
}

Sensitive Data Encryption

All idempotency keys are stored encrypted at rest using AES-256 with key rotation every 90 days:

@Service
public class IdempotencyKeyEncryptionService {
    private final AesGcmEncryption aesGcm;
    
    public String encryptKey(String key) {
        return aesGcm.encrypt(key, getEncryptionKey());
    }
    
    public String decryptKey(String encryptedKey) {
        return aesGcm.decrypt(encryptedKey, getEncryptionKey());
    }
}

PII Handling and Masking Strategy

Idempotency keys are treated as sensitive but not PII. However, any user identifiers embedded in the key content must be masked:

public class IdempotencyKeyMasking {
    public static String maskUserId(String key) {
        return key.replaceAll("(?<=user_)(\\d{4})(\\d+)", "$1****");
    }
}

Secrets Management

All cryptographic keys and sensitive configuration values are managed through AWS Secrets Manager with automatic rotation:

@Value("${aws.secrets.idempotency.key}")
private String encryptionKeyArn;

public String getEncryptionKey() {
    return secretsManager.getSecretValue(encryptionKeyArn);
}

Audit Logging Design

All security-relevant operations are logged with structured JSON format:

{
  "timestamp": "2023-10-27T14:30:00Z",
  "trace_id": "abc123def456",
  "user_id": "user-123",
  "operation": "idempotency_key_create",
  "key_id": "key-abc123",
  "ip_address": "192.168.1.100",
  "status": "success",
  "duration_ms": 45
}

Retention policy: Audit logs retained for 3 years with daily backups to secure storage.

12. Testing Strategy

Unit Testing Scope and Coverage Targets

Target Coverage: 85% code coverage for business logic, 95% for critical paths and error handling.

Scope:

Service layer methods with complex business logic (e.g., idempotency key validation, transaction boundary handling)
Repository layer methods with database interactions
Value object and domain object validations
Error handling paths (duplicate key, timeout, conflict scenarios)
Integration points with external systems (cache, event bus)

Testing Framework: JUnit 5 + Mockito for mocking dependencies, AssertJ for fluent assertions.

Integration Testing Strategy

Test Containers Approach:

Use Testcontainers for PostgreSQL and Redis instances to test real database and caching interactions
Docker-based integration tests ensure environment parity with production

Test Data Management:

Use factories for generating test data (e.g., IdempotencyKeyFactory with configurable TTLs)
Seed test data using Flyway migrations in test environment
Clean up test data after each test using @AfterEach hooks

Sample Integration Test:

@Test
void shouldHandleIdempotentRequestWithSameKey() {
    String idempotencyKey = "key-123";
    String payload = "{ \"amount\": 100 }";

    // First request
    ResponseEntity<String> response1 = restTemplate.postForEntity(
        "/api/v1/payments", 
        new HttpEntity<>(payload, createHeaders(idempotencyKey)), 
        String.class
    );

    // Second request with same key
    ResponseEntity<String> response2 = restTemplate.postForEntity(
        "/api/v1/payments", 
        new HttpEntity<>(payload, createHeaders(idempotencyKey)), 
        String.class
    );

    assertThat(response1.getStatusCode()).isEqualTo(HttpStatus.OK);
    assertThat(response2.getStatusCode()).isEqualTo(HttpStatus.OK);
    assertThat(response2.getBody()).isEqualTo(response1.getBody());
}

Contract Testing

Framework: Pact JVM for consumer-driven contract testing.

Scope:

Define contracts between the Idempotency service and downstream services (e.g., Payment Service)
Test that external systems can correctly consume the idempotency key header and behavior
Ensure that HTTP responses and error codes match documented contracts

Performance Testing Strategy

Tools: Gatling for load testing, JMeter for stress testing.

Key Scenarios:

Concurrent requests with same idempotency key (1000 concurrent threads)
High throughput with varying TTLs (5s, 30s, 60s)
Cache hit vs. miss ratio under load (target: >95% cache hits)

Metrics Tracked:

Response time p50/p95/p99 (target: <20ms for 95% of requests)
Throughput (target: >1000 RPS)
Error rate (<0.1% for 95% of requests)

Test Data Management Strategy

Fixtures:

Use JSON fixtures for common test payloads (e.g., payment request, order creation)
Define data sets with specific TTLs and expiration scenarios

Factories:

public class IdempotencyKeyFactory {
    public static IdempotencyKey createActiveKey(String key, int ttlSeconds) {
        return new IdempotencyKey(
            UUID.randomUUID(),
            key,
            "ACTIVE",
            System.currentTimeMillis() + (ttlSeconds * 1000),
            null,
            null
        );
    }
}

Seeding:

Use Flyway migrations to set up test database schema and seed data
Ensure test environments are isolated with unique prefixes for tables and keys

Test Isolation:

Use separate Redis databases per test suite (e.g., idempotency-test-1, idempotency-test-2)
Clear test data after each test suite execution using Redis FLUSHDB command

13. Observability Design

Structured Logging Schema

All logs must follow a structured format with consistent fields to enable effective querying and analysis:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO|WARN|ERROR",
  "service": "idempotency-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user-12345",
  "request_id": "req-98765",
  "operation": "processIdempotentRequest",
  "entity_id": "idempotency-key-abc123",
  "error_code": "IDEMPOTENCY_KEY_EXISTS",
  "message": "Idempotency key already exists for operation",
  "duration_ms": 15,
  "tags": {
    "method": "POST",
    "path": "/api/v1/transactions",
    "client_id": "client-xyz"
  },
  "pii_masked_fields": ["user_id", "request_id"]
}

PII fields like user_id and request_id will be masked in logs when present, using a configurable mask pattern (e.g., user-***).

Distributed Tracing Instrumentation

OpenTelemetry will be used for distributed tracing with custom spans:

Key Spans to Instrument:

idempotency.validate - Validates idempotency key existence
idempotency.create - Creates new idempotency record
idempotency.lookup - Looks up existing operation result
idempotency.process - Processes the actual operation

Attributes to Capture:

operation.name: "processIdempotentRequest"
idempotency.key: "key-abc123"
request.method: "POST"
request.path: "/api/v1/transactions"
operation.status: "success|failure"
operation.duration_ms: 25
is_duplicate_request: true/false

Metrics Design

RED Metrics (Rate, Errors, Duration) for key operations:

Metric Name	Type	Description	Labels
`idempotency_requests_total`	Counter	Total idempotency operations	`method`, `status`, `operation`
`idempotency_errors_total`	Counter	Failed idempotency operations	`error_type`, `operation`
`idempotency_processing_duration_ms`	Histogram	Operation processing time	`operation`, `status`
`idempotency_cache_hits_total`	Counter	Cache hits for key lookups	`operation`
`idempotency_cache_misses_total`	Counter	Cache misses for key lookups	`operation`
`idempotency_active_keys_count`	Gauge	Active idempotency keys in cache	`operation`

Health Check Endpoint Design

Endpoints with their checks:

GET /health
Response: 200 OK
{
  "status": "healthy",
  "timestamp": "2023-10-27T14:30:45.123Z",
  "components": {
    "database": "healthy",
    "cache": "healthy",
    "idempotency_service": "healthy"
  }
}

GET /ready
Response: 200 OK if all dependencies ready, 503 if not
{
  "status": "ready",
  "timestamp": "2023-10-27T14:30:45.123Z",
  "dependencies": {
    "database": "connected",
    "cache": "connected"
  }
}

GET /live
Response: 200 OK if service is running, 503 if not
{
  "status": "alive",
  "timestamp": "2023-10-27T14:30:45.123Z"
}

Alerting Thresholds

Critical Alerts (on-call escalation):

idempotency_errors_total > 100 errors per minute → Critical (escalate to on-call)
idempotency_processing_duration_ms p99 > 500ms → Warning (monitor)
idempotency_cache_misses_total > 50% of cache lookups → Warning (tune cache strategy)
idempotency_active_keys_count > 1M keys → Warning (potential memory leak)

Warning Alerts:

idempotency_requests_total rate drops below 50% of baseline → Warning (possible outage)
idempotency_processing_duration_ms p95 > 200ms → Warning (performance degradation)

Monitoring Dashboard:

Real-time dashboard showing error rates, cache hit ratio, and processing latency
Historical trend analysis for capacity planning
Anomaly detection on key metrics using ML-based alerting

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

Idempotency key scope: Should idempotency keys be scoped per-user, per-application, or global? This affects the scope of deduplication and potential cross-user conflicts.
Key expiration strategy: What is the acceptable window for key reuse? Should keys be immediately invalidated after use or retained for a configurable period?
Error handling policy: When an idempotent request fails after key reuse, should the system return a success response or propagate the error? This affects client-side idempotency expectations.
Cross-service idempotency: Should the system support cross-service idempotency (e.g., same key across multiple microservices)? This impacts the design of global key storage and consistency requirements.

Known trade-offs and justifications

Memory vs. Storage Trade-off: Using in-memory storage (e.g., Redis) for idempotency keys provides faster access but increases memory pressure and requires replication strategies. Using database storage reduces memory pressure but adds latency to key lookups.
Consistency vs. Availability Trade-off: Strong consistency for idempotency key validation (e.g., using distributed locks) ensures correctness but can reduce availability during network partitions. Eventual consistency (e.g., using eventual consistency with TTLs) increases availability but introduces risk of duplicate processing.
Single point of failure vs. distributed complexity: Centralized idempotency service introduces a single point of failure but simplifies key management. Distributed approach reduces SPOF but increases complexity in handling cross-service idempotency.
Cache hit ratio vs. memory overhead: Caching frequently accessed keys improves performance but increases memory usage and cache invalidation complexity.

Risks and mitigation strategies

Cache stampede: High concurrent requests for the same key can cause cache misses and thundering herd. Mitigation includes probabilistic early expiry, distributed locks, and proper cache warming strategies.
Memory exhaustion: In high-volume systems, keeping all active keys in memory can cause out-of-memory errors. Mitigation includes TTL-based eviction, LRU policies, and monitoring memory usage with alerts.
Race conditions: Concurrent requests with same key can lead to race conditions during validation and processing. Mitigation includes optimistic locking, atomic operations, and proper transaction boundaries.
Data loss: If idempotency key storage fails or is lost, duplicate requests might not be detected. Mitigation includes using durable storage (e.g., Redis with persistence), backup strategies, and monitoring for key store health.

Technical debt acknowledged

Key storage schema evolution: Future changes to the key format or metadata structure may require migration strategies that aren't currently designed.
Cross-service consistency: The current design assumes single-service idempotency, but cross-service scenarios require additional coordination mechanisms that are not yet implemented.
Monitoring gaps: While basic metrics are included, deeper observability for key reuse patterns, duplicate detection rates, and performance bottlenecks is not yet fully implemented.
Idempotency key validation overhead: The current implementation adds latency to every request, which might not be acceptable for high-throughput systems without careful optimization and caching strategies.