Skip to main content
Interview-Mapagam

API Key Management System Design

Comprehensive low-level design for a scalable API key management system with rotation, revocation, and usage tracking capabilities.

  • PublishedMarch 23, 2025
  • Length8,129 words
  • Reading time41 min read
Backend LLD

This document presents a robust low-level architecture for managing API keys at scale, covering key generation, storage, validation, and lifecycle management with security best practices and performance optimizations.

1. Problem Statement & Requirements

Clarifying questions

  1. What scale expectations are there for this API Key Management system?
    • Will it handle millions of keys, or is it more for enterprise customers with hundreds of thousands?
  2. What consistency requirements exist for key operations?
    • Should key creation, revocation, and usage be strongly consistent or eventually consistent?
  3. What are the key entities in this system, and how do they relate?
    • Is it just API keys, or does it include users, applications, permissions, and roles?
  4. What's the read/write ratio for API keys?
    • Are most operations reads (checking key validity) or writes (creating/renewing keys)?
  5. What latency requirements are there for key validation?
    • Should key checks complete in under 50ms for real-time API access?
  6. What authentication model is in use?
    • Is it OAuth2, JWT, or custom API key-based authentication?
  7. Are there any external integrations or dependencies?
    • Does this system integrate with an identity provider (e.g., Auth0, Okta) or external billing system?

Functional requirements

  • API Key Creation: Generate new API keys with metadata (e.g., name, description, permissions).
  • API Key Retrieval: Fetch details of an API key by ID or key value (masked for security).
  • API Key Revocation: Disable or delete an API key.
  • API Key Validation: Verify that a provided key is valid, active, and has appropriate permissions.
  • API Key Rotation: Rotate keys by generating a new key while keeping the old one valid for a grace period.
  • API Key Metadata Management: Update metadata (e.g., name, description) for existing keys.
  • API Key Usage Tracking: Log usage statistics (e.g., last accessed, rate limits).
  • Key Expiry Management: Automatically expire keys after a set time or upon user request.

Non-functional requirements

  • Latency Targets:
    • p50: < 20ms for key validation
    • p95: < 50ms for key validation
    • p99: < 100ms for key validation
  • Throughput:
    • Support at least 50,000 key validations per second
  • Availability:
    • 99.9% uptime (438 minutes of downtime per year)
  • Security:
    • Keys must be securely stored and transmitted
    • Keys must not be exposed in logs or responses

Constraints and assumptions

  • Assumption: API keys are used for authentication and authorization to a protected API.
  • Constraint: Keys must be generated with sufficient entropy to prevent brute-force attacks.
  • Assumption: The system is part of a larger microservices ecosystem with shared authentication infrastructure.
  • Constraint: Keys must support rate limiting and usage tracking for security and billing purposes.

Out of scope

  • User Management: Not part of this system; users are managed elsewhere (e.g., Identity Provider).
  • Key Encryption at Rest: This is handled by the underlying database or storage layer.
  • API Key Distribution: Not part of this system; keys are distributed to clients via secure channels.
  • Key Sharing or Delegation: Keys cannot be shared or delegated to other users or services.
  • Audit Logs for Key Changes: This is handled by a separate audit logging system.
  • Multi-tenant Support: The system assumes a single tenant or namespace for all keys.

2. API Design

API Style Selection

gRPC will be selected for this system due to:

  • Strong typing and auto-generated clients in multiple languages
  • Efficient binary protocol with compression support
  • Built-in authentication and authorization mechanisms
  • Native support for streaming operations (useful for key rotation notifications)
  • Better performance characteristics for high-throughput scenarios

Endpoint Definitions

MethodPathDescriptionAuth
CreateKeyPOST /v1/keysCreates a new API key for a userJWT
GetKeyGET /v1/keys/{keyId}Retrieves details of a specific API keyJWT
ListKeysGET /v1/keysLists all API keys for a userJWT
UpdateKeyPUT /v1/keys/{keyId}Updates metadata of an API keyJWT
DeleteKeyDELETE /v1/keys/{keyId}Soft-deletes an API keyJWT
RotateKeyPOST /v1/keys/{keyId}/rotateRotates the secret of an API keyJWT
ValidateKeyPOST /v1/keys/validateValidates an API key for useNone

Request and Response Schemas

CreateKeyRequest

{
  "name": "string",
  "description": "string",
  "scopes": ["read", "write"],
  "expiresAt": "2024-12-31T23:59:59Z"
}

CreateKeyResponse

{
  "keyId": "uuid",
  "keySecret": "string",
  "name": "string",
  "description": "string",
  "scopes": ["read", "write"],
  "createdAt": "2024-12-31T23:59:59Z",
  "expiresAt": "2024-12-31T23:59:59Z",
  "status": "active"
}

GetKeyResponse

{
  "keyId": "uuid",
  "name": "string",
  "description": "string",
  "scopes": ["read", "write"],
  "createdAt": "2024-12-31T23:59:59Z",
  "expiresAt": "2024-12-31T23:59:59Z",
  "status": "active"
}

ListKeysResponse

{
  "keys": [
    {
      "keyId": "uuid",
      "name": "string",
      "description": "string",
      "scopes": ["read", "write"],
      "createdAt": "2024-12-31T23:59:59Z",
      "expiresAt": "2024-12-31T23:59:59Z",
      "status": "active"
    }
  ],
  "nextPageToken": "string"
}

ValidateKeyRequest

{
  "keyId": "uuid",
  "keySecret": "string"
}

ValidateKeyResponse

{
  "valid": true,
  "keyId": "uuid",
  "name": "string",
  "scopes": ["read", "write"],
  "expiresAt": "2024-12-31T23:59:59Z"
}

HTTP Status Codes and Error Format

Standard Status Codes:

  • 200 OK - Successful GET, PUT, PATCH, DELETE
  • 201 Created - Successful POST (create)
  • 400 Bad Request - Invalid input
  • 401 Unauthorized - Missing or invalid auth token
  • 403 Forbidden - Insufficient permissions
  • 404 Not Found - Resource not found
  • 429 Too Many Requests - Rate limit exceeded
  • 500 Internal Server Error - Unexpected server error

Standardized Error Response Format:

{
  "error": {
    "code": "INVALID_ARGUMENT",
    "message": "The provided key name is too long",
    "details": [
      {
        "field": "name",
        "message": "Name must be between 1 and 50 characters"
      }
    ]
  }
}

API Versioning Strategy

Version will be handled through the path prefix: /v1/keys. This allows for:

  • Clear separation between versions
  • Easy migration path for clients
  • Backward compatibility support during transition

Pagination Design

Cursor-based pagination will be used because:

  • More efficient for large datasets
  • Handles data changes during pagination better
  • Provides consistent results even when items are inserted/deleted

Authentication and Authorization

Authentication:

  • JWT tokens with HS256 signing
  • Token includes user_id and scope claims

Authorization:

  • RBAC model with roles (admin, user)
  • Each API key belongs to a specific user
  • Users can only access their own keys

Rate Limiting Per Endpoint

EndpointLimitPeriodEnforcement
CreateKey100/minute1 minuteIn-memory + Redis
GetKey1000/minute1 minuteIn-memory + Redis
ListKeys500/minute1 minuteIn-memory + Redis
UpdateKey100/minute1 minuteIn-memory + Redis
DeleteKey50/minute1 minuteIn-memory + Redis
RotateKey100/minute1 minuteIn-memory + Redis
ValidateKey5000/minute1 minuteIn-memory + Redis

Idempotency Design

For mutation endpoints, idempotency keys will be supported:

  • Clients send Idempotency-Key header with each request
  • Server stores the operation result for that key
  • If same key is used, return stored result instead of reprocessing
// Example usage in gRPC service
public CreateKeyResponse createKey(CreateKeyRequest request, String idempotencyKey) {
    if (idempotencyKey != null) {
        CreateKeyResponse cached = idempotencyCache.get(idempotencyKey);
        if (cached != null) {
            return cached;
        }
    }
    
    // Process request...
    CreateKeyResponse response = processCreateKey(request);
    
    if (idempotencyKey != null) {
        idempotencyCache.put(idempotencyKey, response);
    }
    
    return response;
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    USER ||--o{ API_KEY : "owns"
    API_KEY ||--o{ API_KEY_USAGE : "records"
    API_KEY ||--o{ API_KEY_PERMISSION : "grants"
    PERMISSION ||--o{ API_KEY_PERMISSION : "defines"
    API_KEY_USAGE ||--o{ EVENT_LOG : "logs"
    
    USER {
        string id PK
        string email
        string name
        timestamp created_at
        timestamp updated_at
    }
    
    API_KEY {
        string id PK
        string user_id FK
        string name
        string key_hash
        string key_prefix
        string status
        timestamp created_at
        timestamp updated_at
        timestamp expires_at
        timestamp last_used_at
    }
    
    PERMISSION {
        string id PK
        string name
        string description
        timestamp created_at
    }
    
    API_KEY_PERMISSION {
        string api_key_id FK
        string permission_id FK
        timestamp created_at
    }
    
    API_KEY_USAGE {
        string id PK
        string api_key_id FK
        string endpoint
        string method
        timestamp accessed_at
        string ip_address
        string user_agent
    }
    
    EVENT_LOG {
        string id PK
        string api_key_id FK
        string event_type
        string details
        timestamp logged_at
    }

Table Schema Definitions

users table

CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email VARCHAR(255) UNIQUE NOT NULL,
    name VARCHAR(255),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

api_keys table

CREATE TABLE api_keys (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    name VARCHAR(255) NOT NULL,
    key_hash BYTEA NOT NULL,
    key_prefix VARCHAR(10) NOT NULL,
    status VARCHAR(20) DEFAULT 'active' CHECK (status IN ('active', 'inactive', 'revoked')),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    expires_at TIMESTAMP,
    last_used_at TIMESTAMP,
    UNIQUE(user_id, key_prefix)
);

permissions table

CREATE TABLE permissions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) UNIQUE NOT NULL,
    description TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

api_key_permissions table

CREATE TABLE api_key_permissions (
    api_key_id UUID REFERENCES api_keys(id) ON DELETE CASCADE,
    permission_id UUID REFERENCES permissions(id) ON DELETE CASCADE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (api_key_id, permission_id)
);

api_key_usage table

CREATE TABLE api_key_usage (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    api_key_id UUID REFERENCES api_keys(id) ON DELETE CASCADE,
    endpoint VARCHAR(255),
    method VARCHAR(10),
    accessed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    ip_address INET,
    user_agent TEXT
);

event_logs table

CREATE TABLE event_logs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    api_key_id UUID REFERENCES api_keys(id) ON DELETE CASCADE,
    event_type VARCHAR(50),
    details JSONB,
    logged_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Primary Key Strategy

UUIDs chosen over auto-increment for the following reasons:

  1. Global uniqueness: UUIDs ensure uniqueness across distributed systems without coordination
  2. Security: Sequential IDs can expose system information to attackers
  3. Scalability: No need for centralized ID generation in multi-region deployments
  4. No hotspots: Avoids write contention on sequential ID generators

Index Design

Index NameFieldsTypePurpose
idx_api_keys_user_iduser_idStandardFast lookup by user
idx_api_keys_key_prefixkey_prefixUniqueFast lookup by key prefix
idx_api_keys_statusstatusStandardFilter active keys
idx_api_keys_expires_atexpires_atPartial (WHERE expires_at > NOW())Efficient expiration checks
idx_api_key_permissions_api_key_idapi_key_idStandardFast permission lookup
idx_api_key_usage_api_key_idapi_key_idStandardFast usage tracking
idx_api_key_usage_accessed_ataccessed_atStandardTime-based queries
idx_event_logs_api_key_idapi_key_idStandardAudit trail lookup

Data Partitioning / Sharding Strategy

Sharding by user_id hash (consistent hashing):

  • Partition key: hash(user_id) % NUMBER_OF_SHARDS
  • Benefits:
    • Even distribution of API keys
    • Predictable sharding for read-heavy workloads
    • Easy to add/remove shards with minimal data movement

Soft Delete vs. Hard Delete Decision

Soft delete chosen for API keys with the following rationale:

  1. Audit compliance: Historical tracking of revoked keys required for security audits
  2. Recovery capability: Ability to restore keys in case of accidental revocation
  3. Usage analytics: Historical usage patterns need to be preserved for analysis
  4. Compliance: GDPR and other regulations may require data retention

Audit Trail Design

Event logging table structure:

-- Event types include: KEY_CREATED, KEY_REVOKED, KEY_EXPIRED, USAGE_ACCESS, PERMISSION_CHANGED
-- Fields: event_type (string), details (JSONB with metadata), logged_at (timestamp)

Audit fields captured:

  • Who (user_id, api_key_id)
  • What (event_type, details)
  • When (logged_at)
  • Where (IP address, user agent)

Data Archival Strategy

Archival policy:

  • Active keys: Retained indefinitely with full audit trail
  • Revoked keys: Retained for 90 days (compliance requirement)
  • Usage logs: Retained for 30 days (usage analytics)
  • Event logs: Retained for 180 days (compliance and security)

Archival process:

  • Daily batch job moves expired data to archive tables
  • Archive tables use different storage (e.g., S3 with lifecycle policies)
  • Metadata maintained in main tables for referential integrity

4. Class Design & Service Layer

classDiagram
    class ApiKeyService {
        +createApiKey(CreateApiKeyRequest) ApiKeyResponse
        +getApiKey(String) ApiKeyResponse
        +revokeApiKey(String) void
        +listApiKeys(ListApiKeysRequest) ApiKeyListResponse
        +rotateApiKey(String) ApiKeyResponse
    }
    
    class ApiKeyRepository {
        +save(ApiKey) void
        +findById(String) Optional~ApiKey~
        +findByUserIdAndName(String, String) Optional~ApiKey~
        +findByUserId(String) List~ApiKey~
        +update(ApiKey) void
        +delete(String) void
    }
    
    class ApiKey {
        -id: String
        -userId: String
        -name: String
        -key: String
        -status: ApiKeyStatus
        -createdAt: Instant
        -expiresAt: Instant
        -lastUsedAt: Instant
        -metadata: Map~String, String~
        +generateKey() String
        +isExpired() boolean
        +isActive() boolean
    }
    
    class ApiKeyValidator {
        +validate(ApiKey) boolean
        +validatePermissions(ApiKey, String) boolean
    }
    
    class EventPublisher {
        +publish(ApiKeyEvent) void
    }
    
    class ApiKeyEvent {
        -eventType: String
        -apiKeyId: String
        -userId: String
        -timestamp: Instant
    }
    
    class ApiKeyGenerator {
        +generateKey() String
    }
    
    class AuditLogger {
        +log(AuditEvent) void
    }
    
    class AuditEvent {
        -userId: String
        -action: String
        -resourceId: String
        -timestamp: Instant
        -details: Map~String, String~
    }
    
    ApiKeyService --> ApiKeyRepository : uses
    ApiKeyService --> ApiKeyValidator : validates
    ApiKeyService --> EventPublisher : publishes events
    ApiKeyService --> AuditLogger : logs audit
    ApiKeyService --> ApiKeyGenerator : generates keys
    ApiKey <-- ApiKeyRepository : returns
    ApiKeyValidator --> ApiKey : validates
    EventPublisher --> ApiKeyEvent : publishes
    AuditLogger --> AuditEvent : logs
stateDiagram-v2
    [*] --> Active
    Active --> Revoked : revoke()
    Active --> Expired : expires_at < now
    Revoked --> Active : rotate()
    Expired --> Active : rotate()
    Active --> Expired : expires_at < now

Service Class Structure and Responsibilities

The API Key Management system is designed with a clean separation of concerns:

  1. ApiKeyService - Core business logic layer handling key lifecycle operations
  2. ApiKeyRepository - Data access abstraction for persistence operations
  3. ApiKeyValidator - Input validation and permission checking
  4. EventPublisher - Event-driven system integration
  5. AuditLogger - Audit trail and compliance logging

Interface / Abstract Class Design

public interface ApiKeyRepository {
    void save(ApiKey apiKey);
    Optional<ApiKey> findById(String id);
    Optional<ApiKey> findByUserIdAndName(String userId, String name);
    List<ApiKey> findByUserId(String userId);
    void update(ApiKey apiKey);
    void delete(String id);
}

public interface EventPublisher {
    void publish(ApiKeyEvent event);
}

public enum ApiKeyStatus {
    ACTIVE, REVOKED, EXPIRED
}

Business Logic Encapsulation Patterns

Strategy Pattern for key generation:

public interface ApiKeyGenerationStrategy {
    String generate();
}

public class SecureRandomApiKeyStrategy implements ApiKeyGenerationStrategy {
    private static final int KEY_LENGTH = 32;
    
    @Override
    public String generate() {
        SecureRandom random = new SecureRandom();
        byte[] bytes = new byte[KEY_LENGTH];
        random.nextBytes(bytes);
        return Base64.getUrlEncoder().withoutPadding().encodeToString(bytes);
    }
}

public class ApiKeyGenerator {
    private final ApiKeyGenerationStrategy strategy;
    
    public ApiKeyGenerator(ApiKeyGenerationStrategy strategy) {
        this.strategy = strategy;
    }
    
    public String generateKey() {
        return strategy.generate();
    }
}

Critical-Path Method Implementation

@Transactional
public ApiKeyResponse createApiKey(CreateApiKeyRequest request) {
    // 1. Validate input and permissions
    if (!apiKeyValidator.validatePermissions(request.getUserId(), "CREATE_API_KEY")) {
        throw new UnauthorizedException("Insufficient permissions to create API key");
    }
    
    // 2. Generate unique key name if not provided
    String keyName = Optional.ofNullable(request.getName())
        .orElse("key-" + System.currentTimeMillis());
    
    // 3. Check for existing key with same name
    if (apiKeyRepository.findByUserIdAndName(request.getUserId(), keyName).isPresent()) {
        throw new ConflictException("API key name already exists for user");
    }
    
    // 4. Generate and persist new API key
    String apiKeyValue = apiKeyGenerator.generateKey();
    ApiKey apiKey = new ApiKey(
        UUID.randomUUID().toString(),
        request.getUserId(),
        keyName,
        apiKeyValue,
        ApiKeyStatus.ACTIVE,
        Instant.now(),
        request.getExpiresAt(),
        null,
        request.getMetadata()
    );
    
    apiKeyRepository.save(apiKey);
    
    // 5. Publish event for audit and monitoring
    ApiKeyEvent event = new ApiKeyEvent(
        "API_KEY_CREATED",
        apiKey.getId(),
        apiKey.getUserId(),
        Instant.now()
    );
    eventPublisher.publish(event);
    
    // 6. Log audit trail
    AuditEvent audit = new AuditEvent(
        apiKey.getUserId(),
        "CREATE_API_KEY",
        apiKey.getId(),
        Instant.now(),
        Map.of("keyName", keyName, "expiresAt", apiKey.getExpiresAt().toString())
    );
    auditLogger.log(audit);
    
    // 7. Return response with key (only public part in response)
    return new ApiKeyResponse(
        apiKey.getId(),
        apiKey.getName(),
        apiKey.getUserId(),
        apiKey.getStatus(),
        apiKey.getCreatedAt(),
        apiKey.getExpiresAt(),
        apiKey.getMetadata()
    );
}

Transaction Boundary Design

A single transaction encompasses:

  1. API key creation validation
  2. Key generation and persistence
  3. Event publishing (if using same DB session)
  4. Audit logging

For distributed systems, we use the outbox pattern to ensure eventual consistency between database writes and event publishing.

Service Interface Contracts

public interface ApiKeyService {
    ApiKeyResponse createApiKey(CreateApiKeyRequest request);
    ApiKeyResponse getApiKey(String apiKeyId);
    void revokeApiKey(String apiKeyId);
    ApiKeyListResponse listApiKeys(ListApiKeysRequest request);
    ApiKeyResponse rotateApiKey(String apiKeyId);
}

Saga / Orchestration Pattern

For complex operations like key rotation involving multiple systems:

  • If external service integrations exist (e.g., IAM sync), use saga pattern
  • For local operations only, single transaction is sufficient

Domain Object / Value Object Design

public class ApiKey {
    private final String id;
    private final String userId;
    private final String name;
    private final String key; // Only stored as hash in DB
    private ApiKeyStatus status;
    private final Instant createdAt;
    private Instant expiresAt;
    private Instant lastUsedAt;
    private final Map<String, String> metadata;
    
    // Constructor, getters, and immutable methods
    
    public boolean isExpired() {
        return expiresAt != null && Instant.now().isAfter(expiresAt);
    }
    
    public boolean isActive() {
        return status == ApiKeyStatus.ACTIVE && !isExpired();
    }
    
    public String getPublicId() {
        return id.substring(0, 8) + "...";
    }
}

The design follows clean architecture principles with clear separation between domain logic, data access, and external integrations. The critical-path method demonstrates proper transaction handling, validation, event publishing, and audit logging in a single coherent operation.

5. Concurrency & Thread Safety

Concurrency Model

The API Key Management system will use a thread-safe, asynchronous processing model with:

  • Java's CompletableFuture for non-blocking operations
  • Thread pools with fixed-size executors for parallel processing of key operations
  • Reactive programming patterns where appropriate (e.g., using Project Reactor for async flows)
  • Single-threaded event loop for critical atomic operations (e.g., key generation, revocation)

Race Condition Prevention Strategy

Race conditions will be prevented through:

  1. Atomic operations using java.util.concurrent.atomic packages for counters and flags
  2. Database-level constraints (unique constraints, foreign key constraints)
  3. Optimistic locking with version fields for concurrent updates
  4. Database transactions with proper isolation levels (READ_COMMITTED or SERIALIZABLE where needed)
  5. Synchronized blocks for critical sections in service layer

Optimistic vs. Pessimistic Locking Selection

  • Optimistic locking is selected for most operations due to:

    • High read-to-write ratio in API key management
    • Low contention scenarios (keys are typically not updated frequently)
    • Better performance under moderate concurrency
    • Reduced lock wait times and improved throughput

  • Pessimistic locking is used for:

    • Key generation operations where uniqueness must be guaranteed
    • Revocation operations that require exclusive access to prevent concurrent modifications

Deadlock Prevention Strategy

  • Lock ordering: All database operations follow a consistent order (e.g., always acquire user lock before key lock)
  • Timeout configuration: Set 5-second timeouts on database locks to prevent indefinite blocking
  • Hierarchical locking: Use a top-down approach where parent entities are locked before child entities
  • Deadlock detection: Leverage database-level deadlock detection and automatic retry mechanisms

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is implemented through:

  1. Idempotency keys in request headers:
// Example header: X-Idempotency-Key: abc123def456
public class IdempotencyService {
    private final Map<String, String> idempotencyCache = new ConcurrentHashMap<>();
    
    public boolean isIdempotentRequest(String key, String operation) {
        return idempotencyCache.putIfAbsent(key, operation) != null;
    }
}
  1. Database-level idempotency tracking:
// Store operation metadata in a separate table with composite primary key (idempotency_key, operation_type)
public class IdempotencyRecord {
    private String idempotencyKey;
    private String operationType;
    private String result;
    private LocalDateTime createdAt;
}
  1. Request deduplication service:
@Service
public class IdempotencyService {
    private final IdempotencyRepository idempotencyRepo;
    
    public ApiResponse<?> processWithIdempotency(String idempotencyKey, 
                                                 Supplier<ApiResponse<?>> operation) {
        IdempotencyRecord record = idempotencyRepo.findByIdempotencyKey(idempotencyKey);
        
        if (record != null) {
            return ApiResponse.success(record.getResult());
        }
        
        ApiResponse<?> result = operation.get();
        idempotencyRepo.save(new IdempotencyRecord(idempotencyKey, operation.toString(), result));
        return result;
    }
}

Thread Safety in Service Layer

All service methods are designed to be thread-safe:

  1. Immutable objects for configuration and metadata
  2. Thread-safe collections (ConcurrentHashMap, CopyOnWriteArrayList)
  3. Synchronized access to shared mutable state
  4. Stateless service methods where possible

Concurrency Control in Key Operations

  • Key generation: Uses atomic counters and database sequences to ensure uniqueness
  • Key retrieval: Read operations are not synchronized, but cached with proper TTL
  • Key revocation: Uses optimistic locking to prevent concurrent revocations
  • Key rotation: Synchronized to ensure atomicity of key replacement operations

Performance Considerations

  • Read-heavy workload: Optimistic locking with caching minimizes lock contention
  • Write operations: Use batch processing where possible to reduce transaction overhead
  • Database connection pooling: Configured with proper pool sizes (e.g., 20-50 connections) to handle concurrent requests
  • Async processing: Non-blocking operations for key validation and audit logging

Testing Concurrency Scenarios

  • Stress testing with concurrent API key creation/retrieval operations
  • Race condition tests using JUnit's @Test with multiple threads
  • Database isolation level testing to ensure proper locking behavior
  • Deadlock detection using database monitoring tools during load testing

This concurrency model ensures high throughput while maintaining data consistency and preventing race conditions in a multi-threaded environment.

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant U as User/Developer
    participant API as API Gateway
    participant AuthS as Auth Service
    participant DB as Database

    U->>API: Request with API Key
    API->>AuthS: Validate API Key
    AuthS->>DB: Lookup API Key
    DB-->>AuthS: Return API Key details
    AuthS-->>API: Validate result (success/failure)
    alt Valid Key
        API->>U: Forward request to target service
    else Invalid Key
        API-->>U: Return 401 Unauthorized
    end
sequenceDiagram
    participant U as User/Developer
    participant AuthS as Auth Service
    participant DB as Database
    participant EventQ as Event Queue

    U->>AuthS: Create API Key (POST /api-keys)
    AuthS->>DB: Generate and store API Key
    DB-->>AuthS: Return stored key details
    AuthS->>EventQ: Publish API_KEY_CREATED event
    EventQ-->>AuthS: Acknowledge event
    AuthS-->>U: Return created API Key with secret
sequenceDiagram
    participant U as User/Developer
    participant AuthS as Auth Service
    participant DB as Database
    participant EventQ as Event Queue

    U->>AuthS: Revoke API Key (DELETE /api-keys/{keyId})
    AuthS->>DB: Mark key as revoked
    DB-->>AuthS: Confirm revocation
    AuthS->>EventQ: Publish API_KEY_REVOKED event
    EventQ-->>AuthS: Acknowledge event
    AuthS-->>U: Return success response

7. Caching Design

Cache Strategy Selection

For API Key Management system, we'll use cache-aside pattern with write-through behavior for critical data and write-behind for audit logs. This provides optimal balance between consistency and performance.

Cache Key Design

Cache keys will follow a structured namespace pattern:

api-key:{keyId}
api-key:by-identifier:{identifier}
api-key:by-user:{userId}
api-key:by-app:{appId}
api-key:active:{keyId}

TTL Strategy

Data TypeTTLReason
Active API Keys1 hourFrequent access, need to refresh periodically
Key Metadata24 hoursLess volatile, infrequent updates
User Permissions15 minutesSecurity sensitive, frequent refresh
Rate Limits1 minuteHigh frequency, short-lived
Audit Logs30 daysCompliance requirements

Cache Eviction Policy

Implement LRU (Least Recently Used) eviction policy with:

  • TTL-based expiration for automatic cleanup
  • Size-based limits (500MB max per cache node)
  • Priority eviction for critical keys

Cache Stampede Prevention

Use probabilistic early expiry:

public class CacheStampedeProtection {
    private static final double EXPIRY_FUDGE_FACTOR = 0.8;
    
    public String getWithStampedeProtection(String key, Supplier<String> loader) {
        String cached = cache.get(key);
        if (cached != null) {
            return cached;
        }
        
        // Try to acquire lock for cache population
        if (lockManager.tryAcquire(key, 500)) {
            try {
                // Double-check pattern
                String fresh = cache.get(key);
                if (fresh != null) {
                    return fresh;
                }
                
                // Load fresh data with early expiry
                String value = loader.get();
                cache.put(key, value, 
                    (long) (getTTL(key) * EXPIRY_FUDGE_FACTOR));
                return value;
            } finally {
                lockManager.release(key);
            }
        }
        
        // Fallback to stale cache or wait
        return cache.get(key);
    }
}

Distributed Cache Architecture

Using Redis Cluster with:

  • 3 master nodes for high availability
  • 3 replica nodes per master for read scaling
  • Redis Sentinel for automatic failover
  • Cluster sharding based on key namespace

Cache Invalidation Strategy

Implement event-driven invalidation:

  1. Immediate invalidation on key creation/update/deletion
  2. Batch invalidation for bulk operations (e.g., app key rotation)
  3. TTL-based cleanup for soft-expired entries
@Component
public class ApiKeyCacheInvalidator {
    private final RedisTemplate<String, String> redisTemplate;
    
    public void invalidateKey(String keyId) {
        // Remove from all relevant cache namespaces
        String[] keysToInvalidate = {
            "api-key:" + keyId,
            "api-key:by-identifier:" + getIdentifier(keyId),
            "api-key:by-user:" + getUserForKey(keyId)
        };
        
        // Batch delete for efficiency
        redisTemplate.delete(Arrays.asList(keysToInvalidate));
        
        // Publish invalidation event for distributed systems
        eventPublisher.publish(new CacheInvalidationEvent("api-key", keyId));
    }
}

Cache Hit/Miss Ratio Optimization

Monitor and optimize:

  • Hit ratio target: >90% for active keys
  • Miss ratio analysis: Identify hot keys that need warming
  • Warm-up strategy: Pre-load frequently accessed keys during startup

Cache Read/Write Separation

Separate cache read and write operations:

public class ApiKeyCacheService {
    private final CacheReadService readService;
    private final CacheWriteService writeService;
    
    public ApiKeyResponse getApiKey(String keyId) {
        // Read from cache first
        return readService.getFromCache(keyId);
    }
    
    public void updateApiKey(ApiKey key) {
        // Write through to cache and database
        writeService.updateCache(key);
        databaseService.save(key);
    }
}

This caching strategy ensures optimal performance while maintaining data consistency and providing resilience against cache failures. The design balances the need for low-latency API key validation with the requirement for strong consistency in security-critical operations.

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain objects and database operations. Each entity has its own repository interface with standardized CRUD operations.

public interface ApiKeyRepository {
    Optional<ApiKey> findById(UUID id);
    Optional<ApiKey> findByKey(String key);
    List<ApiKey> findByOwnerId(UUID ownerId);
    Page<ApiKey> findByOwnerId(UUID ownerId, Pageable pageable);
    ApiKey save(ApiKey apiKey);
    void delete(UUID id);
    boolean existsById(UUID id);
}

public interface ApiKeyAuditRepository {
    void save(AuditLog auditLog);
    List<AuditLog> findByApiKeyId(UUID apiKeyId, Pageable pageable);
}

ORM vs Raw Query Decisions

For the core API key management system, we use JPA/Hibernate as the ORM for its type safety and ease of development. However, for high-volume read operations like key validation, raw SQL queries are preferred to minimize overhead and maximize performance.

@Repository
public class ApiKeyRepositoryImpl implements ApiKeyRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    @Override
    public Optional<ApiKey> findByKey(String key) {
        // Raw SQL for performance-critical validation
        String sql = "SELECT * FROM api_keys WHERE key_hash = :keyHash AND status = 'ACTIVE'";
        TypedQuery<ApiKey> query = entityManager.createNativeQuery(sql, ApiKey.class);
        query.setParameter("keyHash", hashKey(key));
        return Optional.ofNullable(query.getSingleResultOrNull());
    }
    
    @Override
    public ApiKey save(ApiKey apiKey) {
        // JPA for complex domain object persistence
        return entityManager.merge(apiKey);
    }
}

Read vs Write Separation

The system implements CQRS (Command Query Responsibility Segregation) pattern for the API key management to optimize read and write operations:

// Write-side repository - handles mutations
@Repository
public class ApiKeyWriteRepository {
    @PersistenceContext
    private EntityManager entityManager;
    
    public ApiKey save(ApiKey apiKey) {
        return entityManager.merge(apiKey);
    }
    
    public void delete(UUID id) {
        entityManager.remove(entityManager.find(ApiKey.class, id));
    }
}

// Read-side repository - optimized for queries
@Repository
public class ApiKeyReadRepository {
    @PersistenceContext
    private EntityManager entityManager;
    
    public List<ApiKeyView> findAllActiveKeys(UUID ownerId) {
        String jpql = """
            SELECT new com.example.apikey.ApiKeyView(
                ak.id, ak.name, ak.ownerId, ak.status, ak.createdAt
            ) FROM ApiKey ak WHERE ak.ownerId = :ownerId AND ak.status = 'ACTIVE'
            """;
        return entityManager.createQuery(jpql, ApiKeyView.class)
                           .setParameter("ownerId", ownerId)
                           .getResultList();
    }
}

Connection Pool Configuration

Connection pool is configured with appropriate sizing and validation to handle concurrent API key operations efficiently:

# application.yml
spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      validation-timeout: 5000
      idle-timeout: 600000
      max-lifetime: 1800000

N+1 Query Prevention Strategy

To prevent N+1 query problems, eager loading is used for related entities and batch operations are implemented:

@Entity
@Table(name = "api_keys")
@NamedEntityGraph(
    name = "ApiKey.withOwner",
    attributeNodes = @NamedAttributeNode(value = "owner", subgraph = "owner-subgraph"),
    subgraphs = @NamedSubgraph(
        name = "owner-subgraph",
        attributeNodes = @NamedAttributeNode("user")
    )
)
public class ApiKey {
    @ManyToOne(fetch = FetchType.EAGER)
    @JoinColumn(name = "owner_id")
    private User owner;
    
    // ... other fields and methods
}

// In service layer, use fetch joins to prevent N+1:
public List<ApiKey> getActiveKeysWithOwner(UUID ownerId) {
    String jpql = "SELECT ak FROM ApiKey ak JOIN FETCH ak.owner WHERE ak.ownerId = :ownerId AND ak.status = 'ACTIVE'";
    return entityManager.createQuery(jpql, ApiKey.class)
                       .setParameter("ownerId", ownerId)
                       .getResultList();
}

Batch Operation Design

Batch operations are implemented for bulk API key creation and updates:

@Repository
public class BatchApiKeyRepository {
    
    public void batchCreate(List<ApiKey> apiKeys) {
        String sql = """
            INSERT INTO api_keys (id, key_hash, name, owner_id, status, created_at) 
            VALUES (?, ?, ?, ?, ?, ?)
            """;
            
        try (Connection conn = dataSource.getConnection();
             PreparedStatement stmt = conn.prepareStatement(sql)) {
            
            for (ApiKey key : apiKeys) {
                stmt.setObject(1, key.getId());
                stmt.setString(2, key.getKeyHash());
                stmt.setString(3, key.getName());
                stmt.setObject(4, key.getOwnerId());
                stmt.setString(5, key.getStatus().name());
                stmt.setTimestamp(6, Timestamp.from(key.getCreatedAt()));
                stmt.addBatch();
            }
            
            stmt.executeBatch();
        } catch (SQLException e) {
            throw new DataAccessException("Failed to batch create API keys", e);
        }
    }
}

Transaction Management

Transaction boundaries are carefully defined to ensure data consistency:

@Service
@Transactional
public class ApiKeyService {
    
    @Autowired
    private ApiKeyRepository apiKeyRepository;
    
    @Autowired
    private ApiKeyAuditRepository auditRepository;
    
    public ApiKey createApiKey(ApiKeyRequest request) {
        // Single transaction boundary for key creation and audit logging
        ApiKey apiKey = new ApiKey();
        apiKey.setId(UUID.randomUUID());
        apiKey.setKeyHash(hashKey(request.getKey()));
        apiKey.setName(request.getName());
        apiKey.setOwnerId(request.getOwnerId());
        apiKey.setStatus(ApiKeyStatus.ACTIVE);
        apiKey.setCreatedAt(Instant.now());
        
        ApiKey saved = apiKeyRepository.save(apiKey);
        
        // Audit log within same transaction
        AuditLog audit = new AuditLog();
        audit.setApiKeyId(saved.getId());
        audit.setAction("CREATE");
        audit.setActorId(request.getOwnerId());
        audit.setTimestamp(Instant.now());
        auditRepository.save(audit);
        
        return saved;
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The API Key Management system will use asynchronous processing for operations that don't require immediate response to the client and can be safely decoupled from the primary request flow. Key boundaries include:

  1. Key rotation and expiration notifications - When an API key is rotated or expires, the system asynchronously notifies related services (e.g., authentication service, audit logging)
  2. Usage analytics aggregation - Periodic batch processing of key usage metrics for reporting and monitoring
  3. Compliance audit triggers - When a key is revoked or modified, async processing can trigger compliance checks with external systems
  4. Cross-service synchronization - When keys are updated across multiple services (e.g., in a multi-region deployment), async reconciliation ensures consistency

Synchronous operations remain for direct key creation, retrieval, and immediate validation to ensure low-latency responses for authentication flows.

Message schema design

{
  "version": "1.0",
  "event_type": "api_key_rotated",
  "timestamp": "2023-10-15T14:30:00Z",
  "source": "api-key-service",
  "payload": {
    "key_id": "ak-1234567890abcdef",
    "new_key_hash": "sha256:abc123...",
    "previous_key_hash": "sha256:def456...",
    "owner_id": "user-9876543210fedcba",
    "tenant_id": "tenant-1234567890",
    "expires_at": "2024-10-15T14:30:00Z",
    "audit_context": {
      "actor": "system",
      "reason": "automated_rotation"
    }
  },
  "correlation_id": "corr-uuid-1234567890"
}

Dead letter queue strategy and poison message handling

Poison messages will be routed to a dead-letter queue (DLQ) after 3 failed delivery attempts. The system will:

  1. Track failed deliveries with exponential backoff (5s, 10s, 20s)
  2. Move messages to DLQ after max retries (3 attempts)
  3. Log detailed error information including the original payload and error stack trace
  4. Alert on DLQ message accumulation with threshold-based monitoring (e.g., >10 messages in 5 minutes)
  5. Manual intervention process for DLQ message inspection and reprocessing

At-least-once vs. exactly-once delivery

At-least-once delivery is selected for this system because:

  1. API key operations are idempotent - duplicate events won't cause data corruption
  2. The system can handle duplicate notifications gracefully (e.g., key already rotated)
  3. Simpler implementation and better performance characteristics
  4. Most downstream consumers can tolerate duplicate events

Retry policy for consumers

public class RetryPolicy {
    private static final int MAX_RETRIES = 3;
    private static final long INITIAL_DELAY_MS = 5000;
    private static final double BACKOFF_MULTIPLIER = 2.0;
    private static final long MAX_DELAY_MS = 60000;

    public boolean shouldRetry(int attempt, Exception exception) {
        if (attempt >= MAX_RETRIES) return false;
        
        // Don't retry on certain fatal errors
        if (exception instanceof ValidationException || 
            exception instanceof UnauthorizedAccessException) {
            return false;
        }
        
        return true;
    }

    public long getDelay(int attempt) {
        long delay = (long) (INITIAL_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempt));
        return Math.min(delay, MAX_DELAY_MS);
    }
}

Outbox pattern for reliable event publishing

Outbox pattern is implemented to ensure eventual consistency between database writes and message publishing:

  1. When an API key is created or modified, the operation is first persisted to a local outbox table
  2. A background worker polls the outbox table and publishes messages to the message broker
  3. Messages are marked as published once successfully delivered
  4. Failed messages are retried according to the retry policy

This prevents message loss when database transactions fail and ensures that all key state changes are eventually communicated to downstream services.

Sequence diagram for async flows

sequenceDiagram
    participant Client
    participant APIKeyService
    participant DB
    participant Outbox
    participant MessageBroker
    participant Consumer

    Client->>APIKeyService: Create API Key
    APIKeyService->>DB: Save Key Record
    APIKeyService->>Outbox: Insert Event to Outbox
    APIKeyService-->>Client: Return Key ID
    
    loop Poll Outbox
        Outbox->>MessageBroker: Publish Event
        MessageBroker-->>Consumer: Deliver Event
    end
    
    Consumer->>DB: Update Key Usage Metrics
    Consumer->>AuditService: Log Key Event

10. Error Handling Design

Error Classification Hierarchy

The API Key Management system follows a three-tier error classification:

  1. Domain Errors - Business logic violations (e.g., invalid key format, exceeded rate limits)
  2. Infrastructure Errors - System-level issues (e.g., database connection failure, cache unavailability)
  3. Validation Errors - Input validation failures (e.g., missing required fields, malformed JSON)
public class ApiKeyManagementException extends RuntimeException {
    private final ErrorCode errorCode;
    private final ErrorCategory category;

    public ApiKeyManagementException(ErrorCode errorCode, ErrorCategory category, String message) {
        super(message);
        this.errorCode = errorCode;
        this.category = category;
    }
}

public enum ErrorCategory {
    DOMAIN, INFRASTRUCTURE, VALIDATION
}

public enum ErrorCode {
    INVALID_API_KEY_FORMAT,
    API_KEY_NOT_FOUND,
    EXCEEDED_RATE_LIMITS,
    DATABASE_CONNECTION_FAILURE,
    CACHE_UNAVAILABLE,
    INVALID_INPUT,
    INTERNAL_SERVER_ERROR
}

Error Propagation Strategy

Errors are propagated up the call stack with appropriate wrapping to preserve context:

// Example error propagation in service layer
public class ApiKeyService {
    public ApiKey createApiKey(CreateApiKeyRequest request) {
        try {
            validateRequest(request);
            return apiKeyRepository.save(buildApiKey(request));
        } catch (DataAccessException e) {
            throw new ApiKeyManagementException(
                ErrorCode.DATABASE_CONNECTION_FAILURE,
                ErrorCategory.INFRASTRUCTURE,
                "Failed to save API key",
                e
            );
        } catch (ValidationException e) {
            throw new ApiKeyManagementException(
                ErrorCode.INVALID_INPUT,
                ErrorCategory.VALIDATION,
                "Invalid API key request",
                e
            );
        }
    }
}

Retry Strategy with Exponential Backoff and Jitter

For transient infrastructure errors, a retry mechanism is implemented:

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;

    public <T> T executeWithRetry(Supplier<T> operation) {
        Exception lastException = null;
        
        for (int attempt = 0; attempt <= MAX_RETRIES; attempt++) {
            try {
                return operation.get();
            } catch (ApiKeyManagementException e) {
                if (attempt == MAX_RETRIES || !isRetryableError(e)) {
                    throw e;
                }
                lastException = e;
            }
            
            if (attempt < MAX_RETRIES) {
                long delay = calculateBackoffDelay(attempt);
                Thread.sleep(delay);
            }
        }
        
        throw new ApiKeyManagementException(
            ErrorCode.INTERNAL_SERVER_ERROR,
            ErrorCategory.INFRASTRUCTURE,
            "Operation failed after retries",
            lastException
        );
    }

    private long calculateBackoffDelay(int attempt) {
        long baseDelay = (long) (BASE_DELAY_MS * Math.pow(2, attempt));
        long jitter = (long) (baseDelay * JITTER_FACTOR * (Math.random() - 0.5) * 2);
        return baseDelay + jitter;
    }

    private boolean isRetryableError(ApiKeyManagementException e) {
        return e.getErrorCode() == ErrorCode.DATABASE_CONNECTION_FAILURE ||
               e.getErrorCode() == ErrorCode.CACHE_UNAVAILABLE;
    }
}

Circuit Breaker Design

The system implements a circuit breaker pattern for external dependencies:

public class CircuitBreaker {
    private static final int FAILURE_THRESHOLD = 5;
    private static final long TIMEOUT_MS = 60000;
    private static final int RESET_TIMEOUT_MS = 30000;

    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private long lastFailureTime = 0;

    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }

    public <T> T execute(Supplier<T> operation) throws Exception {
        switch (state) {
            case CLOSED:
                return executeClosed(operation);
            case OPEN:
                return executeOpen(operation);
            case HALF_OPEN:
                return executeHalfOpen(operation);
            default:
                throw new IllegalStateException("Unknown circuit state");
        }
    }

    private <T> T executeClosed(Supplier<T> operation) throws Exception {
        try {
            T result = operation.get();
            failureCount = 0; // Reset on success
            return result;
        } catch (Exception e) {
            failureCount++;
            lastFailureTime = System.currentTimeMillis();
            
            if (failureCount >= FAILURE_THRESHOLD) {
                state = CircuitState.OPEN;
            }
            throw e;
        }
    }

    private <T> T executeOpen(Supplier<T> operation) throws Exception {
        if (System.currentTimeMillis() - lastFailureTime > RESET_TIMEOUT_MS) {
            state = CircuitState.HALF_OPEN;
            throw new Exception("Circuit breaker in half-open state");
        }
        
        throw new Exception("Circuit breaker is open, rejecting request");
    }

    private <T> T executeHalfOpen(Supplier<T> operation) throws Exception {
        try {
            T result = operation.get();
            state = CircuitState.CLOSED;
            return result;
        } catch (Exception e) {
            state = CircuitState.OPEN;
            throw e;
        }
    }
}

Fallback Behavior Design

Fallbacks are implemented for critical operations:

public class ApiKeyService {
    private final CircuitBreaker circuitBreaker = new CircuitBreaker();
    
    public ApiKey getApiKey(String keyId) {
        return circuitBreaker.execute(() -> {
            try {
                return apiKeyRepository.findById(keyId);
            } catch (Exception e) {
                // Fallback to cache or default behavior
                return fallbackApiKey(keyId);
            }
        });
    }

    private ApiKey fallbackApiKey(String keyId) {
        // Log the fallback for monitoring
        logger.warn("Falling back to cached or default API key for ID: {}", keyId);
        
        // Return a default or cached version if available
        return new ApiKey(keyId, "fallback", "default");
    }
}

Error Logging Format and Alerting Thresholds

Structured logging is implemented with consistent error fields:

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);

    public void logError(String operation, Exception error, Map<String, Object> context) {
        Map<String, Object> logFields = new HashMap<>();
        logFields.put("timestamp", Instant.now().toString());
        logFields.put("operation", operation);
        logFields.put("error_code", extractErrorCode(error));
        logFields.put("error_message", error.getMessage());
        logFields.put("stack_trace", error.getStackTrace());
        logFields.putAll(context);
        
        logger.error("API Key Management Error: {}", logFields);
    }

    private String extractErrorCode(Exception error) {
        if (error instanceof ApiKeyManagementException) {
            return ((ApiKeyManagementException) error).getErrorCode().name();
        }
        return "UNKNOWN_ERROR";
    }
}

Alerting thresholds are configured for critical errors:

Error TypeAlert ThresholdSeverity
DATABASE_CONNECTION_FAILURE5+ occurrences in 5 minutesCRITICAL
CACHE_UNAVAILABLE3+ occurrences in 1 minuteMAJOR
INVALID_INPUT10+ occurrences in 5 minutesMINOR
INTERNAL_SERVER_ERROR3+ occurrences in 1 minuteCRITICAL

Error Response Format

Standardized JSON error responses are returned to clients:

{
  "error": {
    "code": "DATABASE_CONNECTION_FAILURE",
    "message": "Failed to connect to database",
    "timestamp": "2023-10-15T14:30:00Z",
    "request_id": "req-12345"
  }
}

Error Handling in HTTP Layer

HTTP error responses are standardized:

@RestControllerAdvice
public class ApiKeyExceptionHandler {
    
    @ExceptionHandler(ApiKeyManagementException.class)
    public ResponseEntity<ErrorResponse> handleApiKeyError(
            ApiKeyManagementException ex, WebRequest request) {
        
        ErrorResponse errorResponse = new ErrorResponse(
            ex.getErrorCode().name(),
            ex.getMessage(),
            Instant.now().toString(),
            getRequestId(request)
        );
        
        HttpStatus status = mapErrorCodeToHttpStatus(ex.getErrorCode());
        return ResponseEntity.status(status).body(errorResponse);
    }
    
    private HttpStatus mapErrorCodeToHttpStatus(ErrorCode code) {
        switch (code) {
            case INVALID_API_KEY_FORMAT:
            case INVALID_INPUT:
                return HttpStatus.BAD_REQUEST;
            case API_KEY_NOT_FOUND:
                return HttpStatus.NOT_FOUND;
            case EXCEEDED_RATE_LIMITS:
                return HttpStatus.TOO_MANY_REQUESTS;
            default:
                return HttpStatus.INTERNAL_SERVER_ERROR;
        }
    }
}

11. Security Design

Input validation and sanitization strategy

All API inputs are validated using a combination of:

  • Custom validation annotations for business logic (e.g., @ValidApiKey, @ValidPermission)
  • Apache Commons Validator for standard validations (email, URL, UUID)
  • OWASP Java HTML Sanitizer for any user-provided content that may be rendered in UIs
  • Regex-based validation for API key format, naming conventions, and metadata fields
public class ApiKeyRequest {
    @NotBlank
    @Pattern(regexp = "^[a-zA-Z0-9_-]{1,64}$")
    private String name;
    
    @ValidPermissions
    private List<String> permissions;
    
    @FutureOrPresent
    private LocalDateTime expiration;
}

SQL injection prevention

All database queries are constructed using:

  • Prepared statements with parameterized queries (JDBC/MyBatis)
  • ORM mapping with strict type checking (Hibernate)
  • Whitelisting of column names and table names in dynamic queries
  • No string concatenation for query construction
// Instead of:
String sql = "SELECT * FROM api_keys WHERE name = '" + name + "'";
// Use:
String sql = "SELECT * FROM api_keys WHERE name = ?";
PreparedStatement stmt = connection.prepareStatement(sql);
stmt.setString(1, name);

Authentication middleware design

Authentication is handled via a centralized filter that:

  • Validates JWT tokens using RS256 algorithm
  • Checks token expiration and signature
  • Extracts user context from claims (user_id, roles, tenant_id)
  • Sets security context in thread-local for downstream services
@Component
public class JwtAuthenticationFilter implements Filter {
    @Override
    public void doFilter(ServletRequest request, ServletResponse response, 
                        FilterChain chain) throws IOException, ServletException {
        String token = extractTokenFromHeader(request);
        if (token != null && validateToken(token)) {
            Authentication auth = createAuthentication(token);
            SecurityContextHolder.getContext().setAuthentication(auth);
        }
        chain.doFilter(request, response);
    }
}

Authorization enforcement

RBAC (Role-Based Access Control) is implemented with:

  • Fine-grained permissions stored in database per API key
  • Permission checking at service layer using @PreAuthorize annotations
  • Context-aware authorization that considers user, tenant, and resource scope
@PreAuthorize("@authorizationService.canManageApiKey(principal, #apiKeyId)")
public void deleteApiKey(String apiKeyId) {
    // Implementation
}

Sensitive data encryption at rest and in transit

At Rest:

  • AES-256-GCM encryption for API key secrets using AWS KMS
  • Key rotation every 90 days with automatic re-encryption of existing keys

In Transit:

  • TLS 1.3 enforced for all API endpoints
  • HSTS headers with 31536000 max-age and includeSubDomains
  • Certificate pinning for mobile clients

PII handling and masking strategy

All PII data is:

  • Masked in logs using a consistent pattern (e.g., "API_KEY_XXXXXX")
  • Redacted in responses when returning sensitive fields
  • Stored encrypted with proper key management (AWS KMS)
  • Compliant with GDPR/CCPA - data retention policies enforced
public class ApiKeyResponse {
    @JsonIgnore
    private String secret;
    
    public String getSecret() {
        return "API_KEY_" + secret.substring(secret.length() - 6);
    }
}

Secrets management

  • AWS Secrets Manager for storing API key secrets and rotation policies
  • Automatic rotation every 90 days with 30-day grace period
  • Least privilege access - only services that need secrets can retrieve them
  • Audit logging of all secret access operations

Audit logging design

All security-relevant events are logged with:

  • Structured JSON format including trace_id, user_id, timestamp, action, resource_id
  • Comprehensive event types: key creation, deletion, access, permission changes, rotation
  • Retention policy: 365 days for audit logs, 90 days for operational logs
  • Alerting thresholds: 10 failed authentication attempts within 5 minutes triggers alert
{
  "timestamp": "2023-10-15T14:30:45.123Z",
  "trace_id": "abc123def456",
  "user_id": "user-123",
  "action": "API_KEY_CREATED",
  "resource_id": "key-456",
  "ip_address": "192.168.1.100",
  "permissions": ["read", "write"]
}

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

  • Service Layer: 85% code coverage with focus on business logic, validation, and error handling paths
  • Repository Layer: 90% coverage with emphasis on query correctness and edge case handling
  • Domain Objects: 100% coverage for value objects and immutable entities
  • Exception Handling: 100% coverage for all custom exception paths and fallbacks
  • Security Logic: 95% coverage for authentication, authorization, and input validation logic

Integration Testing Strategy

  • Database Integration: Use Testcontainers with PostgreSQL 13+ for end-to-end database tests
  • External Service Mocking: WireMock for API key validation service integrations and external auth providers
  • Event System Testing: Embedded Kafka or RabbitMQ for async flow validation using embedded message brokers
  • Cache Integration: Redis Testcontainers to validate cache behavior and invalidation logic
  • Authentication Endpoints: Use embedded OAuth2 server (e.g., Spring Security Test) to validate JWT handling and token validation flows

Contract Testing

  • Pact Broker Integration: Use Pact for consumer-driven contract testing between API key service and dependent services (e.g., auth gateway, billing system)
  • Versioned Contracts: Maintain contract versions per API version to prevent breaking changes in dependent systems
  • Consumer Verification: Run contract tests as part of CI pipeline to ensure API key service adheres to defined contracts

Performance Testing Tool and Key Scenarios

  • Tool: k6 (JavaScript-based load testing) for high-concurrency scenarios and latency validation
  • Key Scenarios:
    • Concurrent API key creation (1000 simultaneous requests)
    • High-volume token validation (5000/sec validation rate)
    • Cache warm-up and hit ratio validation under load
    • Database read/write contention under burst traffic
  • Metrics Tracked:
    • p50/p95/p99 response times (target: <20ms for validation, <50ms for creation)
    • Throughput (target: 10k req/sec for validation endpoint)
    • Error rates (<0.1% failure rate expected)

Test Data Management Strategy

  • Fixtures: Use JSON/YAML fixtures for predefined test data (e.g., user roles, client applications)
  • Factories: Implement Factory pattern for generating test entities (API keys, users, apps) with configurable attributes
  • Seeding: Use database migration scripts for initial test data in Testcontainers, with clean state reset between test runs
  • Data Isolation: Each test suite runs in isolated database schema or container to prevent cross-test pollution
  • Test Data Generation: Use libraries like Faker for generating realistic but non-sensitive test data (e.g., fake client names, mock API key values)
// Example unit test for API key validation service
@Test
public void testValidateApiKey_InvalidKey_ReturnsFalse() {
    // Given
    String invalidKey = "invalid-key-format";
    
    // When
    boolean isValid = apiKeyService.validateApiKey(invalidKey);
    
    // Then
    assertFalse(isValid);
}

@Test
public void testValidateApiKey_ValidKey_ReturnsTrue() {
    // Given
    String validKey = "sk_live_abc123xyz";
    when(apiKeyRepository.findByKey(validKey)).thenReturn(Optional.of(createValidApiKey()));
    
    // When
    boolean isValid = apiKeyService.validateApiKey(validKey);
    
    // Then
    assertTrue(isValid);
}

13. Observability Design

Structured Logging Schema

All logs must follow a structured format with consistent fields to enable effective search, filtering, and analysis. The schema includes:

{
  "timestamp": "2023-10-27T14:30:00.000Z",
  "level": "INFO|WARN|ERROR",
  "service": "api-key-mgmt-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user-123",
  "entity_id": "api-key-456",
  "operation": "generateApiKey",
  "request_id": "req-789",
  "message": "API key generated successfully",
  "error_code": "API_KEY_GENERATION_FAILED",
  "error_details": {
    "type": "ValidationException",
    "message": "API key name must be between 3 and 50 characters"
  },
  "duration_ms": 125,
  "tags": ["audit", "security"]
}

PII masking rules:

  • User IDs and API key values are masked in logs
  • Only the first 3 and last 3 characters of sensitive fields are retained (e.g., "api-key-1234567890" → "api-key-123...7890")

Distributed Tracing Instrumentation

Using OpenTelemetry, we instrument key operations with custom spans:

  1. API Key Generation Flow:

    • Span: generate_api_key
    • Attributes: user_id, api_key_name, key_type, request_id

  2. API Key Validation Flow:

    • Span: validate_api_key
    • Attributes: api_key_id, user_id, validation_result

  3. API Key Rotation Flow:

    • Span: rotate_api_key
    • Attributes: old_key_id, new_key_id, user_id

  4. API Key Revocation Flow:

    • Span: revoke_api_key
    • Attributes: api_key_id, user_id, revoked_at

Metrics Design

Service-Level RED Metrics

Metric NameTypeLabelsDescription
api_key_requests_totalCountermethod, endpoint, status_codeTotal API key operations
api_key_request_duration_secondsHistogrammethod, endpointRequest processing time
api_key_errors_totalCountererror_type, endpointFailed API key operations
api_key_active_countGaugekey_type, statusCurrently active keys
api_key_usage_countHistogramkey_id, user_idAPI key usage frequency

Business Metrics

Metric NameTypeLabelsDescription
api_key_creation_rateCounteruser_id, key_typeNumber of keys created per user
api_key_rotation_rateCounteruser_id, key_typeRotation frequency
api_key_deactivation_rateCounteruser_id, reasonKey deactivation reasons

Health Check Endpoint Design

GET /health
GET /ready
GET /live
GET /startup

Response Format:

{
  "status": "healthy|unhealthy",
  "checks": {
    "database": {
      "status": "healthy",
      "latency_ms": 15
    },
    "cache": {
      "status": "healthy",
      "latency_ms": 5
    },
    "auth_service": {
      "status": "healthy",
      "latency_ms": 20
    }
  },
  "timestamp": "2023-10-27T14:30:00.000Z"
}

Alerting Thresholds

MetricWarning LevelCritical LevelEscalation Path
api_key_request_duration_seconds500ms (p99)2s (p99)On-call engineer
api_key_errors_total10 errors/minute50 errors/minuteOn-call engineer + SRE
api_key_requests_total100k requests/minute500k requests/minuteSRE team
database_connection_pool_utilization80%95%SRE team
cache_hit_rate<80%<60%SRE team
api_key_active_count>1M keys>2M keysSRE team + Product

Log Rotation and Retention

  • Log Rotation: Rotate logs daily or when file size exceeds 100MB
  • Retention Policy:
    • 30 days for INFO level logs
    • 7 days for ERROR level logs
    • 1 day for DEBUG level logs

Audit Logging Design

All security-sensitive operations are logged with full audit trail:

{
  "timestamp": "2023-10-27T14:30:00.000Z",
  "operation": "API_KEY_GENERATED",
  "user_id": "user-123",
  "api_key_id": "key-456",
  "details": {
    "name": "Production API Key",
    "type": "production",
    "created_by": "admin@company.com",
    "permissions": ["read", "write"],
    "created_at": "2023-10-27T14:30:00.000Z"
  },
  "ip_address": "192.168.1.100",
  "user_agent": "Mozilla/5.0..."
}

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

  • Key rotation policy: Should keys be rotated automatically or only on explicit request? What's the acceptable window for key rotation without breaking existing integrations?
  • Key lifecycle management: Should API keys be automatically disabled after a certain period of inactivity? What's the grace period before complete deletion?
  • Rate limiting strategy: Should rate limits be applied at key level or account level? How granular should the rate limiting be (per endpoint vs. global)?
  • Key revocation mechanism: Should keys be revoked immediately or marked for soft deletion with a grace period?

Known trade-offs and justifications

  • Storage vs. Performance Trade-off: Storing keys in plaintext in database for fast lookup vs. storing only hashed values for security. We chose to store only hashes with plaintext keys in memory cache for performance, but this introduces risk of key exposure in memory.
  • Consistency vs. Availability Trade-off: Using strong consistency for key state changes vs. eventual consistency. We chose strong consistency for critical operations like key creation/deletion but allow eventual consistency for read-heavy operations.
  • Security vs. Usability Trade-off: Complex key generation vs. simple, user-friendly keys. We opted for UUID-based keys that are secure but also easy to copy/paste.
  • Auditability vs. Performance Trade-off: Comprehensive audit logging vs. minimal overhead. We implement audit logging with async writes to avoid blocking primary operations.

Risks and mitigation strategies

  • Key Exfiltration Risk: If keys are compromised, they can be used indefinitely until revoked. Mitigation: Implement key rotation and monitoring for unusual usage patterns.
  • Denial of Service Risk: If the system becomes unavailable, all API integrations break. Mitigation: Implement circuit breaker patterns and fallback to cached key validation.
  • Data Loss Risk: If the database storing keys fails, all keys become inaccessible. Mitigation: Use replication and backup strategies with cross-region redundancy.
  • Cache Invalidation Risk: If cache becomes stale, systems might accept invalid keys. Mitigation: Implement TTL with proactive cache warming and event-driven invalidation.

Technical debt acknowledged

  • Legacy Key Format Support: The system currently supports both old and new key formats, but this creates maintenance overhead and potential security gaps. This will be deprecated in Q2 2025.
  • Hardcoded Rate Limit Values: Rate limits are currently hardcoded in configuration files rather than being dynamically configurable. This creates operational overhead for tuning.
  • Monolithic Key Storage: All key data is stored in a single database table without proper sharding strategy. This creates a potential scalability bottleneck as key count grows.
  • Inconsistent Error Handling: Some error responses are not standardized across different service methods, creating inconsistency in client-side handling.
  • No Key Metadata Versioning: Key metadata changes are not versioned, which makes it difficult to support backward compatibility when key properties change.

Was this article helpful?

Topics in this article

More from Backend LLD

  • Backend LLD

    JWT Token Management System Design

    Comprehensive low-level design for a secure JWT token management system with refresh token handling, blacklisting, and distributed session management.

    • CategoryBackend LLD
    • UpdatedMarch 9, 2025
    • Reading time39 min read
  • Backend LLD

    Session Management Service Design and Implementation

    Comprehensive low-level design for a scalable session management service with token-based authentication, storage strategies, and security considerations.

    • CategoryBackend LLD
    • UpdatedFebruary 9, 2025
    • Reading time41 min read

Stay in the Loop

Get curated DSA patterns, coding problems, frontend and backend LLD/HLD, and AI/ML System Design insights delivered straight to your inbox. No spam, just focused interview and engineering content.