Skip to main content
Interview-Mapagam

Blocking System Design: Scalable Rate Limiting and Throttling Architecture

Comprehensive low-level design for implementing efficient blocking systems with rate limiting, throttling, and distributed coordination patterns.

  • PublishedJanuary 4, 2026
  • Length8,206 words
  • Reading time42 min read
Backend LLD

A blocking system serves as a critical infrastructure component for controlling resource access and preventing system overload through intelligent rate limiting and throttling mechanisms across distributed environments.

1. Problem Statement & Requirements

Clarifying questions

  1. What is the scope of "blocking"? Is it about blocking users, IP addresses, or content from being accessed? Will this system block based on user behavior, content type, or external threat intelligence?
  2. What is the expected scale? How many concurrent blocking operations are we expecting per second, and how many total blocked entities (users, IPs, content) will there be?
  3. What consistency requirements exist? Should blocking be immediate (strong consistency) or can there be a small delay (eventual consistency)?
  4. What is the read/write ratio? Is this system primarily used for checking if an entity is blocked (read-heavy) or for creating/adding new blocks (write-heavy)?
  5. What are the latency sensitivity requirements? Should blocking decisions happen in <10ms for real-time traffic or is a few hundred milliseconds acceptable?
  6. What authentication and authorization model does the system expect? Will it be used internally by other services or exposed publicly with API keys?
  7. What external systems integrate with this blocking system? Are there integrations with threat intelligence feeds, third-party blocking services, or external APIs?

Functional requirements

  • Block entities: Ability to block users, IP addresses, content IDs, or other identifiers by type.
  • Check blocking status: Query if an entity is currently blocked.
  • Unblock entities: Remove a previously added block.
  • Bulk operations: Support for batch blocking and unblocking of entities.
  • Block expiration: Optionally set TTLs on blocks to auto-unblock after a period.
  • Block metadata: Store additional context with each block (reason, source, timestamp).
  • Audit logs: Track who created/modified blocks and when.
  • Block categories: Support for categorizing blocks (e.g., spam, malicious, content policy).

Non-functional requirements

  • Latency targets:
    • p50: <5ms for blocking checks
    • p95: <20ms for blocking checks
    • p99: <50ms for blocking checks
  • Throughput:
    • 10,000 blocking checks/second (read-heavy)
    • 500 block creation/second (write-heavy)
  • Availability: 99.9% uptime for blocking service
  • Scalability: Support horizontal scaling to handle peak loads up to 10x current capacity
  • Data durability: Blocks must survive system restarts and data center failures

Constraints and assumptions

  • Assumption: The blocking system will be used in a high-throughput, low-latency environment typical of FAANG systems.
  • Constraint: Blocks must be checked in sub-millisecond time for real-time traffic filtering.
  • Assumption: The system will be used by multiple services in a microservices architecture.
  • Constraint: No external dependencies for core blocking functionality (e.g., no reliance on external threat feeds for basic operation).
  • Assumption: The system will be deployed in a multi-region environment with eventual consistency between regions.

Out of scope

  • External threat intelligence integration: No direct integration with external threat feeds or third-party blocking services.
  • Real-time content analysis: The system does not analyze content for blocking criteria, only checks against pre-defined block lists.
  • User interface: No web UI or dashboard for managing blocks; only API access.
  • Rate limiting per user/IP: While the system may block users, it does not implement rate limiting for individual users or IPs.
  • Complex policy engine: No support for complex conditional blocking logic (e.g., "block if user has >5 failed logins in 1 hour and IP is from country X").
  • Multi-tenancy: The system does not support multi-tenant environments or per-tenant block lists.
  • Block propagation: Blocks created in one region do not automatically propagate to other regions (eventual consistency only).

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for the blocking system due to:

  • Strong typing and automatic code generation
  • Efficient binary serialization for high-throughput scenarios
  • Native support for streaming and bidirectional communication
  • Built-in service versioning capabilities
  • Better performance characteristics than REST for internal microservices

Endpoint Definitions

MethodPathDescriptionAuthRate Limit
CreateBlock/v1/blocksCreates a new block on a resourceJWT + RBAC100 req/min
GetBlock/v1/blocks/{id}Retrieves a specific block by IDJWT + RBAC500 req/min
ListBlocks/v1/blocksLists all active blocks with filtersJWT + RBAC200 req/min
ReleaseBlock/v1/blocks/{id}/releaseReleases a specific blockJWT + RBAC100 req/min
ExtendBlock/v1/blocks/{id}/extendExtends the duration of a blockJWT + RBAC50 req/min
ListBlockHistory/v1/blocks/{id}/historyGets historical changes for a blockJWT + RBAC100 req/min

Request and Response Schemas

CreateBlockRequest

message CreateBlockRequest {
  string resource_id = 1;
  string resource_type = 2;
  int64 duration_seconds = 3;
  string reason = 4;
  map<string, string> metadata = 5;
}

CreateBlockResponse

message CreateBlockResponse {
  string block_id = 1;
  string resource_id = 2;
  string resource_type = 3;
  int64 created_at = 4;
  int64 expires_at = 5;
  string reason = 6;
  map<string, string> metadata = 7;
}

Block

message Block {
  string block_id = 1;
  string resource_id = 2;
  string resource_type = 3;
  int64 created_at = 4;
  int64 expires_at = 5;
  string reason = 6;
  map<string, string> metadata = 7;
  bool is_active = 8;
}

HTTP Status Codes and Error Format

Standard HTTP Statuses:

  • 200 OK - Successful GET, PUT, PATCH, DELETE
  • 201 Created - Successful POST
  • 400 Bad Request - Invalid request body or parameters
  • 401 Unauthorized - Missing or invalid authentication
  • 403 Forbidden - Insufficient permissions
  • 404 Not Found - Resource not found
  • 429 Too Many Requests - Rate limit exceeded
  • 500 Internal Server Error - Unexpected server error

Standardized Error Response:

{
  "error": {
    "code": "BLOCK_RESOURCE_CONFLICT",
    "message": "Resource is already blocked",
    "details": {
      "block_id": "b1234567-890a-bcde-f012-34567890abcd",
      "resource_id": "res-123",
      "existing_block_expires_at": 1698765432
    },
    "timestamp": "2023-10-27T14:30:00Z"
  }
}

API Versioning Strategy

Versioned using path prefix: /v1/...

  • Major version changes require migration plan and deprecation timeline
  • Minor versions can be backward compatible
  • Versioning handled at service boundary, not within individual endpoints

Pagination Design

Cursor-based pagination is used because:

  • More consistent and predictable than offset-based pagination
  • Handles data changes during pagination gracefully
  • Better performance for large datasets

Cursor format: cursor=eyJyZXNvdXJjZV9pZCI6InJlc3QtMTIzIiwic2V0X2F0IjoxNjk4NzY1NDMzfQ==

Authentication and Authorization

Authentication: JWT Bearer tokens with audience validation
Authorization: RBAC model with roles:

  • BLOCK_ADMIN: Full CRUD access to all blocks
  • BLOCK_USER: Read access and self-managed blocks only
  • BLOCK_SERVICE: Service-to-service access for internal systems

Rate Limiting

Per-endpoint rate limits:

  • CreateBlock: 100 req/min (token bucket algorithm)
  • GetBlock: 500 req/min (fixed window)
  • ListBlocks: 200 req/min (sliding window)
  • ReleaseBlock: 100 req/min (token bucket)
  • ExtendBlock: 50 req/min (token bucket)

Enforcement: In-memory token buckets with Redis persistence for cross-service consistency.

Idempotency Design

Idempotency Key Strategy:

  • Client provides X-Idempotency-Key header (UUID format)
  • Server stores operation state with key
  • Subsequent identical requests return same result without side effects
  • Key TTL: 5 minutes after last use
// Idempotency key validation in gRPC service
public CreateBlockResponse createBlock(CreateBlockRequest request, String idempotencyKey) {
    if (idempotencyKey != null) {
        String existingResult = idempotencyCache.get(idempotencyKey);
        if (existingResult != null) {
            return parseExistingResponse(existingResult);
        }
    }

    // Process block creation...
    String result = serializeResponse(response);
    
    if (idempotencyKey != null) {
        idempotencyCache.put(idempotencyKey, result, Duration.ofMinutes(5));
    }
    
    return response;
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    USER ||--o{ BLOCKING_REQUEST : "creates"
    BLOCKING_REQUEST ||--o{ BLOCKING_RULE : "matches"
    BLOCKING_RULE ||--o{ BLOCKING_REQUEST : "applies_to"
    BLOCKING_REQUEST ||--o{ BLOCKING_HISTORY : "logs"
    BLOCKING_HISTORY ||--o{ BLOCKING_EVENT : "records"
    BLOCKING_RULE ||--o{ BLOCKING_EVENT : "triggers"
    
    USER {
        string id PK
        string email
        string name
        timestamp created_at
        timestamp updated_at
    }
    
    BLOCKING_REQUEST {
        string id PK
        string user_id FK
        string resource_type
        string resource_id
        string reason
        timestamp requested_at
        timestamp resolved_at
        string status
        string resolution_note
        timestamp created_at
        timestamp updated_at
    }
    
    BLOCKING_RULE {
        string id PK
        string name
        string description
        string resource_type
        string condition_json
        timestamp created_at
        timestamp updated_at
    }
    
    BLOCKING_HISTORY {
        string id PK
        string blocking_request_id FK
        string status
        timestamp timestamp
        string note
        timestamp created_at
    }
    
    BLOCKING_EVENT {
        string id PK
        string blocking_rule_id FK
        string resource_type
        string resource_id
        timestamp triggered_at
        string event_data_json
        timestamp created_at
    }

Table Schema Definitions

users

CREATE TABLE users (
    id VARCHAR(36) PRIMARY KEY,
    email VARCHAR(255) UNIQUE NOT NULL,
    name VARCHAR(255) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    
    INDEX idx_email (email),
    INDEX idx_created_at (created_at)
);

blocking_rules

CREATE TABLE blocking_rules (
    id VARCHAR(36) PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    description TEXT,
    resource_type VARCHAR(100) NOT NULL,
    condition_json JSON NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    
    INDEX idx_resource_type (resource_type),
    INDEX idx_created_at (created_at)
);

blocking_requests

CREATE TABLE blocking_requests (
    id VARCHAR(36) PRIMARY KEY,
    user_id VARCHAR(36) NOT NULL,
    resource_type VARCHAR(100) NOT NULL,
    resource_id VARCHAR(255) NOT NULL,
    reason TEXT NOT NULL,
    requested_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    resolved_at TIMESTAMP NULL,
    status ENUM('PENDING', 'APPROVED', 'REJECTED', 'EXPIRED') NOT NULL DEFAULT 'PENDING',
    resolution_note TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    
    FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE,
    INDEX idx_user_resource (user_id, resource_type, resource_id),
    INDEX idx_status (status),
    INDEX idx_requested_at (requested_at),
    INDEX idx_resolved_at (resolved_at),
    INDEX idx_created_at (created_at)
);

blocking_history

CREATE TABLE blocking_history (
    id VARCHAR(36) PRIMARY KEY,
    blocking_request_id VARCHAR(36) NOT NULL,
    status ENUM('PENDING', 'APPROVED', 'REJECTED', 'EXPIRED') NOT NULL,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    note TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    FOREIGN KEY (blocking_request_id) REFERENCES blocking_requests(id) ON DELETE CASCADE,
    INDEX idx_blocking_request (blocking_request_id),
    INDEX idx_timestamp (timestamp)
);

blocking_events

CREATE TABLE blocking_events (
    id VARCHAR(36) PRIMARY KEY,
    blocking_rule_id VARCHAR(36) NOT NULL,
    resource_type VARCHAR(100) NOT NULL,
    resource_id VARCHAR(255) NOT NULL,
    triggered_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    event_data_json JSON NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    FOREIGN KEY (blocking_rule_id) REFERENCES blocking_rules(id) ON DELETE CASCADE,
    INDEX idx_resource_type (resource_type),
    INDEX idx_resource_id (resource_id),
    INDEX idx_triggered_at (triggered_at),
    INDEX idx_blocking_rule (blocking_rule_id)
);

Primary Key Strategy

UUIDs are used as primary keys for all tables to support:

  • Distributed system compatibility (no coordination needed for ID generation)
  • Audit trail integrity (no sequential patterns that could leak information)
  • Horizontal sharding capability (UUIDs can be distributed across shards without collision)

Index Design

Index NameFieldsTypePurpose
idx_emailemailStandardFast user lookup by email
idx_resource_typeresource_typeStandardFilter blocking rules by resource type
idx_user_resourceuser_id, resource_type, resource_idCompositeFast lookup of blocking requests for a specific user and resource
idx_statusstatusStandardFast filtering of blocking requests by status
idx_requested_atrequested_atStandardFast retrieval of recent blocking requests
idx_resolved_atresolved_atStandardEfficient query for expired or resolved requests
idx_created_atcreated_atStandardAudit trail and time-based queries
idx_blocking_requestblocking_request_idStandardFast history lookup for blocking requests
idx_triggered_attriggered_atStandardEfficient event timeline queries

Data Partitioning / Sharding Strategy

  • Shard by resource_type: Partition blocking_requests and blocking_events by resource type to reduce cross-shard queries
  • Shard by user_id: For high-volume users, consider sharding blocking_requests by user_id hash to distribute load
  • Time-based partitioning: For blocking_history, consider monthly partitions for large-scale historical queries

Soft Delete vs. Hard Delete Decision

Soft delete is used for blocking_requests and blocking_rules to maintain audit trail and allow for recovery in case of accidental deletion. The deleted_at timestamp field can be added to support soft deletes while maintaining referential integrity.

Audit Trail Design

All table modifications are logged with:

  • created_at and updated_at timestamps
  • User who made the change (via user_id in blocking_requests)
  • Audit records in blocking_history table for status changes
  • Event logging in blocking_events for rule triggers

Data Archival Strategy and Retention Policy

-- Archive policy: 
-- 1. blocking_requests older than 2 years → move to archive tables
-- 2. blocking_history older than 5 years → move to archive tables  
-- 3. blocking_events older than 1 year → move to archive tables

CREATE TABLE blocking_requests_archive (
    LIKE blocking_requests INCLUDING ALL
);

CREATE TABLE blocking_history_archive (
    LIKE blocking_history INCLUDING ALL
);

CREATE TABLE blocking_events_archive (
    LIKE blocking_events INCLUDING ALL
);

Retention policy:

  • blocking_requests: 2 years (move to archive)
  • blocking_history: 5 years (move to archive)
  • blocking_events: 1 year (move to archive)
  • blocking_rules: Indefinite (soft-deleted entries can be archived after 1 year)

4. Class Design & Service Layer

classDiagram
    class BlockingSystem {
        +createBlock(BlockRequest) BlockResponse
        +getBlock(String) BlockResponse
        +releaseBlock(String) ReleaseResponse
        +extendBlock(String, Duration) ExtendResponse
        +listActiveBlocks() List~Block~
    }
    
    class Block {
        -id: String
        -resourceId: String
        -holderId: String
        -createdAt: Instant
        -expiresAt: Instant
        -status: BlockStatus
        -metadata: Map~String, String~
        +isExpired() boolean
        +canBeExtended() boolean
    }
    
    class BlockRequest {
        resourceId: String
        holderId: String
        duration: Duration
        metadata: Map~String, String~
    }
    
    class BlockResponse {
        block: Block
        success: boolean
        message: String
    }
    
    class ReleaseResponse {
        success: boolean
        message: String
    }
    
    class ExtendResponse {
        success: boolean
        newExpiry: Instant
        message: String
    }
    
    class BlockStatus {
        PENDING
        ACTIVE
        EXPIRED
        RELEASED
    }
    
    class BlockRepository {
        +save(Block) Block
        +findById(String) Optional~Block~
        +findByResourceIdAndHolderId(String, String) Optional~Block~
        +findActiveBlocksByResource(String) List~Block~
        +update(Block) Block
        +delete(Block) void
    }
    
    class EventPublisher {
        +publish(BlockEvent) void
    }
    
    class BlockEvent {
        eventType: String
        blockId: String
        resourceId: String
        holderId: String
        timestamp: Instant
    }
    
    class BlockValidator {
        +validateCreateRequest(BlockRequest) ValidationResult
        +validateReleaseRequest(String, String) ValidationResult
        +validateExtendRequest(String, Duration) ValidationResult
    }
    
    class BlockService {
        -repository: BlockRepository
        -validator: BlockValidator
        -publisher: EventPublisher
        -maxExtensionDuration: Duration
        -defaultBlockDuration: Duration
    }
    
    class BlockManager {
        -blockService: BlockService
        -lockManager: LockManager
    }
    
    class LockManager {
        +acquireLock(String, String) boolean
        +releaseLock(String, String) void
        +isLocked(String, String) boolean
    }
    
    class BlockState {
        -status: BlockStatus
        -lock: ReentrantLock
        -version: long
    }
    
    class BlockStateTransition {
        +transitionToActive(Block) void
        +transitionToExpired(Block) void
        +transitionToReleased(Block) void
    }
    
    BlockingSystem --> BlockService : uses
    BlockService --> BlockRepository : persists
    BlockService --> BlockValidator : validates
    BlockService --> EventPublisher : publishes
    BlockService --> LockManager : manages concurrency
    BlockService --> BlockStateTransition : handles state
    Block --> BlockState : has state
    BlockValidator --> BlockRequest : validates
    BlockEvent --> Block : references
stateDiagram-v2
    [*] --> PENDING
    PENDING --> ACTIVE : validated and created
    ACTIVE --> EXPIRED : timeout or manual release
    ACTIVE --> RELEASED : explicit release
    EXPIRED --> [*]
    RELEASED --> [*]

Service Layer Overview

The blocking system is designed with a service-oriented architecture to handle concurrent access, state transitions, and distributed consistency. The core service is BlockService which orchestrates all blocking operations.

Service Class Structure and Responsibilities

  1. BlockService: Main orchestration layer handling business logic, validation, and state management
  2. BlockRepository: Data persistence abstraction for block entities
  3. BlockValidator: Input validation and business rule enforcement
  4. EventPublisher: Asynchronous event publishing for audit and monitoring
  5. LockManager: Concurrent access control for resource-level locking
  6. BlockStateTransition: State management and transition logic

Interface / Abstract Class Design

public interface BlockRepository {
    Block save(Block block);
    Optional<Block> findById(String id);
    Optional<Block> findByResourceIdAndHolderId(String resourceId, String holderId);
    List<Block> findActiveBlocksByResource(String resourceId);
    Block update(Block block);
    void delete(Block block);
}

public interface EventPublisher {
    void publish(BlockEvent event);
}

public interface BlockValidator {
    ValidationResult validateCreateRequest(BlockRequest request);
    ValidationResult validateReleaseRequest(String blockId, String holderId);
    ValidationResult validateExtendRequest(String blockId, Duration extension);
}

public enum BlockStatus {
    PENDING,
    ACTIVE,
    EXPIRED,
    RELEASED
}

Business Logic Encapsulation Patterns

The system uses the State Pattern to manage block lifecycle transitions:

public class BlockStateTransition {
    private final Map<BlockStatus, Function<Block, Block>> stateTransitions;
    
    public BlockStateTransition() {
        stateTransitions = new HashMap<>();
        stateTransitions.put(BlockStatus.PENDING, this::transitionToActive);
        stateTransitions.put(BlockStatus.ACTIVE, this::transitionToExpired);
        stateTransitions.put(BlockStatus.EXPIRED, this::transitionToReleased);
    }
    
    public void applyTransition(Block block) {
        var transition = stateTransitions.get(block.getStatus());
        if (transition != null) {
            transition.apply(block);
        }
    }
    
    private Block transitionToActive(Block block) {
        block.setStatus(BlockStatus.ACTIVE);
        return block;
    }
    
    private Block transitionToExpired(Block block) {
        if (block.getExpiresAt().isBefore(Instant.now())) {
            block.setStatus(BlockStatus.EXPIRED);
        }
        return block;
    }
    
    private Block blockToReleased(Block block) {
        block.setStatus(BlockStatus.RELEASED);
        return block;
    }
}

Critical-Path Method Implementation

public BlockResponse createBlock(BlockRequest request) {
    // Transaction boundary - all operations within single DB transaction
    return transactionTemplate.execute(status -> {
        try {
            // Validate request
            ValidationResult validation = validator.validateCreateRequest(request);
            if (!validation.isValid()) {
                throw new ValidationException("Invalid block request: " + validation.getErrorMessage());
            }
            
            // Acquire resource-level lock to prevent concurrent creation
            String lockKey = "block:" + request.getResourceId();
            boolean acquired = lockManager.acquireLock(lockKey, request.getHolderId());
            if (!acquired) {
                throw new ResourceLockedException("Resource " + request.getResourceId() + 
                    " is currently blocked by another process");
            }
            
            // Check for existing active block
            Optional<Block> existingBlock = repository.findByResourceIdAndHolderId(
                request.getResourceId(), request.getHolderId());
            
            if (existingBlock.isPresent() && !existingBlock.get().isExpired()) {
                throw new BlockAlreadyExistsException("Resource " + request.getResourceId() + 
                    " is already blocked by holder " + request.getHolderId());
            }
            
            // Create new block
            Block block = Block.builder()
                .id(UUID.randomUUID().toString())
                .resourceId(request.getResourceId())
                .holderId(request.getHolderId())
                .createdAt(Instant.now())
                .expiresAt(Instant.now().plus(request.getDuration()))
                .status(BlockStatus.PENDING)
                .metadata(request.getMetadata())
                .build();
            
            // Persist block
            Block savedBlock = repository.save(block);
            
            // Transition to active state
            BlockStateTransition transition = new BlockStateTransition();
            transition.applyTransition(savedBlock);
            
            // Publish event
            BlockEvent event = BlockEvent.builder()
                .eventType("BLOCK_CREATED")
                .blockId(savedBlock.getId())
                .resourceId(savedBlock.getResourceId())
                .holderId(savedBlock.getHolderId())
                .timestamp(Instant.now())
                .build();
            
            publisher.publish(event);
            
            // Release lock
            lockManager.releaseLock(lockKey, request.getHolderId());
            
            return BlockResponse.builder()
                .block(savedBlock)
                .success(true)
                .message("Block created successfully")
                .build();
                
        } catch (Exception e) {
            // Release lock on failure
            try {
                lockManager.releaseLock("block:" + request.getResourceId(), request.getHolderId());
            } catch (Exception lockReleaseException) {
                // Log but don't fail the transaction
                logger.warn("Failed to release lock for resource: " + request.getResourceId(), 
                    lockReleaseException);
            }
            throw e;
        }
    });
}

Transaction Boundary Design

All blocking operations are wrapped in database transactions to ensure atomicity:

  • Creation, extension, and release operations must be atomic
  • Lock acquisition is handled within the same transaction scope
  • Events are published after successful persistence to maintain consistency

Service Interface Contracts

public interface BlockingSystem {
    BlockResponse createBlock(BlockRequest request);
    BlockResponse getBlock(String blockId);
    ReleaseResponse releaseBlock(String blockId, String holderId);
    ExtendResponse extendBlock(String blockId, Duration extension, String holderId);
    List<Block> listActiveBlocks(String resourceId);
}

Saga / Orchestration Pattern

The blocking system does not require complex distributed transactions since all operations are within a single service boundary. However, for potential future expansion to distributed systems:

public class BlockSaga {
    private final List<SagaStep> steps;
    
    public void execute(SagaContext context) {
        for (SagaStep step : steps) {
            try {
                step.execute(context);
            } catch (Exception e) {
                // Compensate previous steps
                compensate(context, step);
                throw new SagaExecutionException("Saga failed at step: " + step.getName(), e);
            }
        }
    }
}

Domain Object / Value Object Design

@Value
public class Block {
    String id;
    String resourceId;
    String holderId;
    Instant createdAt;
    Instant expiresAt;
    BlockStatus status;
    Map<String, String> metadata;
    
    public boolean isExpired() {
        return expiresAt.isBefore(Instant.now());
    }
    
    public boolean canBeExtended() {
        return status == BlockStatus.ACTIVE && !isExpired();
    }
}

@Value
public class BlockRequest {
    String resourceId;
    String holderId;
    Duration duration;
    Map<String, String> metadata;
}

@Value
public class BlockEvent {
    String eventType;
    String blockId;
    String resourceId;
    String holderId;
    Instant timestamp;
}

5. Concurrency & Thread Safety

Concurrency Model

The system will use a thread pool-based concurrency model with Java's java.util.concurrent package. Each blocking operation will be handled by a dedicated thread from a fixed-size thread pool to prevent resource exhaustion. For high-concurrency scenarios, we'll leverage CompletableFuture and ExecutorService for asynchronous processing where appropriate.

Race Condition Prevention Strategy

Race conditions will be prevented through:

  1. Synchronized access to shared mutable state using ReentrantLock or synchronized blocks
  2. Atomic operations for counters and flags using java.util.concurrent.atomic package
  3. Immutable data structures where possible to eliminate the need for synchronization
  4. Thread-local storage for per-request state that doesn't need to be shared

Optimistic vs. Pessimistic Locking Selection

Optimistic locking will be used for blocking operations due to:

  • High read-to-write ratio in most use cases
  • Low probability of concurrent conflicts
  • Better performance under moderate contention
  • Reduced lock acquisition overhead

Implementation will use version numbers in database records to detect conflicts during updates.

Deadlock Prevention Strategy

Deadlock prevention will be implemented through:

  1. Lock ordering - All operations acquire locks in a consistent global order
  2. Timeout-based acquisition - Locks will timeout after 5 seconds to prevent indefinite blocking
  3. Deadlock detection - Periodic checks using thread dump analysis for debugging
  4. Hierarchical locking - Locks will be acquired in a strict hierarchy to prevent circular waits

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency will be handled through:

  1. Idempotency keys - Unique identifiers provided by clients for duplicate request detection
  2. Request deduplication cache - In-memory LRU cache storing recent idempotency keys with TTL of 1 hour
  3. Database-level uniqueness constraints - Unique indexes on blocking request identifiers
public class BlockingService {
    private final Map<String, BlockingRequest> idempotencyCache = 
        new ConcurrentHashMap<>();
    private final ScheduledExecutorService cacheCleanupScheduler = 
        Executors.newScheduledThreadPool(1);
    
    public BlockingResult createBlockingRequest(String idempotencyKey, 
                                               BlockingRequest request) {
        // Check for duplicate request
        if (idempotencyCache.containsKey(idempotencyKey)) {
            return idempotencyCache.get(idempotencyKey).getResult();
        }
        
        // Process request with optimistic locking
        BlockingResult result = processBlockingRequest(request);
        
        // Cache the result for idempotency
        idempotencyCache.put(idempotencyKey, 
            new BlockingRequest(request, result));
        
        // Schedule cleanup of cache entry
        cacheCleanupScheduler.schedule(
            () -> idempotencyCache.remove(idempotencyKey), 
            1, TimeUnit.HOURS);
            
        return result;
    }
}

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Auth as Auth Service
    participant BS as Blocking Service
    participant DB as Database
    participant Cache as Redis Cache

    U->>API: Request to block resource
    API->>Auth: Validate JWT token
    Auth-->>API: Authenticated user
    API->>BS: Call blockResource()
    BS->>Cache: Check if resource is already blocked
    Cache-->>BS: Not found
    BS->>DB: Check resource availability
    DB-->>BS: Resource available
    BS->>DB: Insert blocking record
    DB-->>BS: Success
    BS->>Cache: Cache resource block
    Cache-->>BS: Success
    BS-->>API: Return success response
    API-->>U: Block confirmation
sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Auth as Auth Service
    participant BS as Blocking Service
    participant DB as Database
    participant Cache as Redis Cache

    U->>API: Request to block resource
    API->>Auth: Validate JWT token
    Auth-->>API: Authenticated user
    API->>BS: Call blockResource()
    BS->>Cache: Check if resource is already blocked
    Cache-->>BS: Found existing block
    BS->>DB: Check resource availability
    DB-->>BS: Resource not available
    BS-->>API: Return conflict error
    API-->>U: Resource unavailable
sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Auth as Auth Service
    participant BS as Blocking Service
    participant DB as Database
    participant Cache as Redis Cache
    participant MQ as Message Queue

    U->>API: Request to block resource
    API->>Auth: Validate JWT token
    Auth-->>API: Authenticated user
    API->>BS: Call blockResource()
    BS->>Cache: Check if resource is already blocked
    Cache-->>BS: Not found
    BS->>DB: Check resource availability
    DB-->>BS: Resource available
    BS->>DB: Insert blocking record
    DB-->>BS: Success
    BS->>Cache: Cache resource block
    Cache-->>BS: Success
    BS->>MQ: Publish resourceBlocked event
    MQ-->>BS: Event queued
    BS-->>API: Return success response
    API-->>U: Block confirmation
    MQ->>Consumer: Consume resourceBlocked event
    Consumer->>DB: Update related entities
    Consumer-->>MQ: Acknowledge event

7. Caching Design

Cache Strategy Selection

For the blocking system, I'll use a cache-aside pattern with LRU eviction policy. This approach provides the best balance of performance and consistency for a blocking system where data freshness is important but not mission-critical.

The cache-aside pattern allows us to:

  • Maintain control over when data is invalidated
  • Handle cache misses gracefully with database fallback
  • Support multiple cache tiers (application-level + distributed)
  • Implement proper cache warming strategies

Cache Key Design

Cache keys will follow a structured namespace pattern:

{namespace}:{entity_type}:{entity_id}:{version}

Example keys for blocking system:

  • blocking:blocking_rule:12345:v1
  • blocking:blocking_list:67890:v1
  • blocking:user_block:alice:v1
  • blocking:system_config:default:v1

Cache TTL Strategy

Data TypeTTLReason
Blocking Rules5 minutesRules change infrequently, but need to be refreshed periodically
User Blocks1 hourUser blocking is relatively stable, but needs to be refreshed for consistency
System Configs10 minutesConfiguration changes are rare but should be propagated quickly
Active Blocks30 secondsReal-time blocking status needs to be fresh

Cache Eviction Policy

LRU (Least Recently Used) with TTL-based expiration:

  • Primary eviction based on LRU to prevent memory bloat
  • Secondary cleanup via TTL to ensure stale data doesn't persist indefinitely
  • Cache warming for frequently accessed blocking rules

Cache Stampede Prevention

Implement probabilistic early expiry with distributed locks:

public class CacheStampedePrevention {
    private static final double STAMPED_EXPIRY_FACTOR = 0.8;
    
    public String getCachedValue(String key, Supplier<String> cacheMissHandler) {
        String cachedValue = cache.get(key);
        
        if (cachedValue != null && !isExpired(key)) {
            return cachedValue;
        }
        
        // Implement distributed lock to prevent stampede
        String lockKey = "lock:" + key;
        if (distributedLock.tryAcquire(lockKey, 100)) {
            try {
                // Double-check pattern to avoid redundant computation
                String freshValue = cache.get(key);
                if (freshValue == null || isExpired(key)) {
                    freshValue = cacheMissHandler.get();
                    cache.put(key, freshValue, calculateExpiryTime());
                }
                return freshValue;
            } finally {
                distributedLock.release(lockKey);
            }
        } else {
            // Fallback to stale cache or wait briefly
            return cachedValue != null ? cachedValue : fallbackValue();
        }
    }
}

Distributed Cache Architecture

Using Redis Cluster with:

  • 3 master nodes for high availability
  • 3 replica nodes per master for read scaling
  • Redis Sentinel for automatic failover
  • Cache sharding based on entity type to prevent hot keys

Cache Invalidation Strategy

Event-driven invalidation with time-based TTLs:

@Component
public class BlockingCacheInvalidator {
    
    @EventListener
    public void handleBlockingRuleUpdated(BlockingRuleUpdatedEvent event) {
        String cacheKey = "blocking:blocking_rule:" + event.getRuleId() + ":v1";
        cache.invalidate(cacheKey);
        
        // Also invalidate related user blocks that might be affected
        String userBlockKey = "blocking:user_block:" + event.getAffectedUser() + ":v1";
        cache.invalidate(userBlockKey);
    }
    
    @EventListener
    public void handleUserBlocked(UserBlockedEvent event) {
        String cacheKey = "blocking:user_block:" + event.getUserId() + ":v1";
        cache.invalidate(cacheKey);
        
        // Invalidate system-wide blocking status
        String systemCacheKey = "blocking:system_config:default:v1";
        cache.invalidate(systemCacheKey);
    }
}

Cache Warm-up Strategy

Implement background warming for high-traffic blocking rules:

@Component
public class BlockingCacheWarmer {
    
    @Scheduled(fixedRate = 300000) // Every 5 minutes
    public void warmHighPriorityBlockingRules() {
        List<BlockingRule> highPriorityRules = blockingRepository.findHighPriorityRules();
        
        for (BlockingRule rule : highPriorityRules) {
            String cacheKey = "blocking:blocking_rule:" + rule.getId() + ":v1";
            String cachedValue = cache.get(cacheKey);
            
            if (cachedValue == null) {
                // Warm the cache with fresh data
                String ruleData = blockingService.getBlockingRule(rule.getId());
                cache.put(cacheKey, ruleData, calculateTTL(rule));
            }
        }
    }
}

Cache Hit/Miss Ratio Monitoring

@Component
public class CacheMetrics {
    private final MeterRegistry meterRegistry;
    
    public void recordCacheHit(String cacheType) {
        Counter.builder("cache.hits")
            .tag("type", cacheType)
            .register(meterRegistry)
            .increment();
    }
    
    public void recordCacheMiss(String cacheType) {
        Counter.builder("cache.misses")
            .tag("type", cacheType)
            .register(meterRegistry)
            .increment();
    }
}

This caching design provides optimal performance for a blocking system while maintaining data consistency through proper invalidation and stampede prevention mechanisms. The combination of cache-aside pattern, LRU eviction, and event-driven invalidation ensures that the system can handle high throughput while maintaining acceptable consistency levels.

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and database operations. Each entity will have its own repository interface and implementation.

// Repository Interface
public interface BlockingRepository {
    Optional<Blocking> findById(UUID id);
    List<Blocking> findByUserId(UUID userId);
    List<Blocking> findByTargetIdAndType(String targetId, BlockingType type);
    Blocking save(Blocking blocking);
    void delete(UUID id);
    boolean existsByUserIdAndTargetIdAndType(UUID userId, String targetId, BlockingType type);
}

// Repository Implementation
@Repository
public class BlockingRepositoryImpl implements BlockingRepository {
    
    private final JdbcTemplate jdbcTemplate;
    private final ObjectMapper objectMapper;
    
    public BlockingRepositoryImpl(JdbcTemplate jdbcTemplate, ObjectMapper objectMapper) {
        this.jdbcTemplate = jdbcTemplate;
        this.objectMapper = objectMapper;
    }
    
    @Override
    public Optional<Blocking> findById(UUID id) {
        String sql = "SELECT * FROM blockings WHERE id = ?";
        try {
            Blocking blocking = jdbcTemplate.queryForObject(sql, new Object[]{id}, new BlockingRowMapper());
            return Optional.of(blocking);
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }
    
    @Override
    public List<Blocking> findByUserId(UUID userId) {
        String sql = "SELECT * FROM blockings WHERE user_id = ? ORDER BY created_at DESC";
        return jdbcTemplate.query(sql, new Object[]{userId}, new BlockingRowMapper());
    }
    
    @Override
    public List<Blocking> findByTargetIdAndType(String targetId, BlockingType type) {
        String sql = "SELECT * FROM blockings WHERE target_id = ? AND type = ? ORDER BY created_at DESC";
        return jdbcTemplate.query(sql, new Object[]{targetId, type.name()}, new BlockingRowMapper());
    }
    
    @Override
    public Blocking save(Blocking blocking) {
        if (blocking.getId() == null) {
            return create(blocking);
        } else {
            return update(blocking);
        }
    }
    
    private Blocking create(Blocking blocking) {
        String sql = """
            INSERT INTO blockings (id, user_id, target_id, type, reason, created_at, updated_at)
            VALUES (?, ?, ?, ?, ?, ?, ?)
            """;
        
        UUID id = UUID.randomUUID();
        blocking.setId(id);
        blocking.setCreatedAt(Instant.now());
        blocking.setUpdatedAt(Instant.now());
        
        jdbcTemplate.update(sql, 
            id, 
            blocking.getUserId(),
            blocking.getTargetId(),
            blocking.getType().name(),
            blocking.getReason(),
            blocking.getCreatedAt(),
            blocking.getUpdatedAt()
        );
        
        return blocking;
    }
    
    private Blocking update(Blocking blocking) {
        String sql = """
            UPDATE blockings 
            SET target_id = ?, type = ?, reason = ?, updated_at = ?
            WHERE id = ?
            """;
        
        blocking.setUpdatedAt(Instant.now());
        jdbcTemplate.update(sql, 
            blocking.getTargetId(),
            blocking.getType().name(),
            blocking.getReason(),
            blocking.getUpdatedAt(),
            blocking.getId()
        );
        
        return blocking;
    }
    
    @Override
    public void delete(UUID id) {
        String sql = "DELETE FROM blockings WHERE id = ?";
        jdbcTemplate.update(sql, id);
    }
    
    @Override
    public boolean existsByUserIdAndTargetIdAndType(UUID userId, String targetId, BlockingType type) {
        String sql = "SELECT COUNT(*) FROM blockings WHERE user_id = ? AND target_id = ? AND type = ?";
        Integer count = jdbcTemplate.queryForObject(sql, new Object[]{userId, targetId, type.name()}, Integer.class);
        return count != null && count > 0;
    }
}

ORM vs Raw Query Decisions

For the blocking system, raw SQL queries are preferred over ORM for the following reasons:

  1. Performance: Raw queries provide better performance for simple CRUD operations
  2. Control: Direct control over query execution and optimization
  3. Complexity: The blocking system has relatively simple data relationships that don't benefit from ORM complexity

However, for complex joins or aggregations, we would consider using JPA with proper indexing.

Read vs Write Separation

The system uses a read-write separation strategy:

// Read-Only Repository for high-volume reads
@Repository
public class BlockingReadRepository {
    
    private final JdbcTemplate jdbcTemplate;
    
    public BlockingReadRepository(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }
    
    // Optimized for read-heavy operations
    public List<Blocking> findActiveBlockingsForTarget(String targetId, BlockingType type) {
        String sql = """
            SELECT * FROM blockings 
            WHERE target_id = ? AND type = ? AND created_at > ?
            ORDER BY created_at DESC
            """;
        
        Instant cutoff = Instant.now().minus(Duration.ofHours(24));
        return jdbcTemplate.query(sql, new Object[]{targetId, type.name(), cutoff}, new BlockingRowMapper());
    }
}

// Write Repository for mutations
@Repository
public class BlockingWriteRepository {
    
    private final JdbcTemplate jdbcTemplate;
    
    public BlockingWriteRepository(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }
    
    public Blocking createBlocking(Blocking blocking) {
        String sql = """
            INSERT INTO blockings (id, user_id, target_id, type, reason, created_at, updated_at)
            VALUES (?, ?, ?, ?, ?, ?, ?)
            """;
        
        blocking.setId(UUID.randomUUID());
        blocking.setCreatedAt(Instant.now());
        blocking.setUpdatedAt(Instant.now());
        
        jdbcTemplate.update(sql, 
            blocking.getId(),
            blocking.getUserId(),
            blocking.getTargetId(),
            blocking.getType().name(),
            blocking.getReason(),
            blocking.getCreatedAt(),
            blocking.getUpdatedAt()
        );
        
        return blocking;
    }
}

Connection Pool Configuration

# application.yml
spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      leak-detection-threshold: 60000

N+1 Query Prevention Strategy

The system prevents N+1 queries by:

  1. Eager Loading: Fetching all blocking records in a single query
  2. Batch Operations: Using batch inserts/updates where appropriate
// Batch operation for creating multiple blockings
public List<Blocking> createBlockings(List<Blocking> blockings) {
    String sql = """
        INSERT INTO blockings (id, user_id, target_id, type, reason, created_at, updated_at)
        VALUES (?, ?, ?, ?, ?, ?, ?)
        """;
    
    // Prepare batch
    List<Object[]> batchArgs = blockings.stream()
        .map(blocking -> new Object[]{
            UUID.randomUUID(),
            blocking.getUserId(),
            blocking.getTargetId(),
            blocking.getType().name(),
            blocking.getReason(),
            Instant.now(),
            Instant.now()
        })
        .collect(Collectors.toList());
    
    // Execute batch
    jdbcTemplate.batchUpdate(sql, batchArgs);
    
    return blockings;
}

Batch Operation Design

// Batch read operation for performance optimization
public Map<String, List<Blocking>> getBlockingsByTargetIds(List<String> targetIds) {
    if (targetIds.isEmpty()) return Collections.emptyMap();
    
    String placeholders = targetIds.stream()
        .map(id -> "?")
        .collect(Collectors.joining(",", "(", ")"));
    
    String sql = """
        SELECT * FROM blockings 
        WHERE target_id IN """ + placeholders + """
        ORDER BY created_at DESC
        """;
    
    List<Blocking> allBlockings = jdbcTemplate.query(sql, targetIds.toArray(), new BlockingRowMapper());
    
    return allBlockings.stream()
        .collect(Collectors.groupingBy(Blocking::getTargetId));
}

Data Access Layer Design Principles

  1. Single Responsibility: Each repository handles one domain entity
  2. Consistent Interface: All repositories implement the same interface pattern
  3. Transaction Boundaries: Repository methods are designed to work within transaction contexts
  4. Performance Optimization: Read-heavy operations use optimized queries and caching strategies
  5. Type Safety: Strong typing with proper error handling for database operations

The repository layer is designed to be minimal and focused, ensuring that the domain logic in service classes can operate with clean data access patterns while maintaining high performance and scalability for the blocking system.

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The blocking system will use a hybrid synchronous/asynchronous approach. Critical operations like user blocking/unblocking and immediate policy enforcement will be handled synchronously to ensure atomicity and consistency. Non-critical operations such as audit logging, notifications, and analytics will be handled asynchronously to improve system responsiveness.

Key triggers for async processing:

  • User blocking/unblocking operations (publish to blocking events topic)
  • Policy violations detected by monitoring service (publish to policy violation topic)
  • Periodic system health checks and reporting (scheduled async jobs)

Message schema design

{
  "version": "1.0",
  "event_type": "user_blocked",
  "timestamp": "2023-10-15T14:30:00Z",
  "user_id": "user-uuid-12345",
  "blocked_by": "admin-uuid-67890",
  "reason": "violation_of_terms",
  "duration_minutes": 1440,
  "metadata": {
    "source": "policy_engine",
    "ip_address": "192.168.1.100"
  }
}

For versioning, we'll use semantic versioning with a version field in the message schema. Each event type will have its own versioned schema and consumers will be able to handle multiple versions through backward compatibility.

Dead letter queue strategy

All failed message processing will be routed to a dead-letter queue (DLQ) with a retention period of 7 days. Messages in DLQ will be tagged with failure reason and retry count.

Poison message handling:

  • Messages failing more than 5 times will be automatically moved to DLQ
  • DLQ messages will trigger alerts to on-call engineers
  • Manual intervention required for DLQ message reprocessing

At-least-once vs. exactly-once delivery

We will implement at-least-once delivery with idempotency handling. This is chosen because:

  1. The blocking system operations are idempotent (blocking same user twice has same effect)
  2. Simpler to implement than exactly-once semantics
  3. Lower latency and better performance characteristics

Retry policy for consumers

Retry configuration:

  • Maximum attempts: 5
  • Backoff strategy: Exponential with jitter (1s, 2s, 4s, 8s, 16s)
  • Retryable errors: Network timeouts, service unavailability, transient database errors
  • Non-retryable errors: Invalid user ID, malformed message, permanent policy violations

Outbox pattern for reliable event publishing

We will implement the outbox pattern to ensure event delivery reliability:

@Component
public class BlockingEventPublisher {
    
    private final BlockingEventRepository eventRepository;
    private final KafkaTemplate<String, BlockingEvent> kafkaTemplate;
    
    public void publishBlockingEvent(BlockingEvent event) {
        // Store event in outbox table
        BlockingEventOutbox outboxEvent = new BlockingEventOutbox(
            event.getId(),
            event.getEventType(),
            event.toJson(),
            "pending"
        );
        eventRepository.save(outboxEvent);
        
        // Publish to Kafka
        kafkaTemplate.send("blocking-events", event.getId(), event);
    }
    
    @Scheduled(fixedRate = 5000)
    public void processPendingEvents() {
        List<BlockingEventOutbox> pendingEvents = eventRepository.findPendingEvents();
        
        for (BlockingEventOutbox outboxEvent : pendingEvents) {
            try {
                kafkaTemplate.send("blocking-events", outboxEvent.getEventId(), 
                    parseEvent(outboxEvent.getPayload()));
                eventRepository.markAsProcessed(outboxEvent.getId());
            } catch (Exception e) {
                // Log error and continue processing other events
                eventRepository.markAsFailed(outboxEvent.getId(), e.getMessage());
            }
        }
    }
}

Sequence diagram for async flows

sequenceDiagram
    participant User as User Service
    participant Blocking as Blocking Service
    participant Kafka as Message Broker
    participant Audit as Audit Service
    participant Notification as Notification Service
    
    User->>Blocking: Block User Request
    Blocking->>Kafka: Publish UserBlocked Event
    Kafka->>Audit: Consume Event (Async)
    Kafka->>Notification: Consume Event (Async)
    
    Audit-->>Kafka: Acknowledge
    Notification-->>Kafka: Acknowledge
    
    Kafka->>Blocking: Event Processed Successfully

10. Error Handling Design

Error Classification Hierarchy

Errors are classified into three main categories:

  1. Domain Errors - Business logic violations (e.g., insufficient funds, invalid state transitions)
  2. Infrastructure Errors - System-level issues (e.g., database connection failures, timeout exceptions)
  3. Validation Errors - Input validation failures (e.g., missing required fields, invalid format)
public enum ErrorType {
    INSUFFICIENT_FUNDS,
    INVALID_STATE_TRANSITION,
    RESOURCE_NOT_FOUND,
    DATABASE_CONNECTION_FAILURE,
    TIMEOUT_ERROR,
    VALIDATION_ERROR,
    AUTHENTICATION_ERROR,
    AUTHORIZATION_ERROR
}

Error Propagation Strategy

Errors are handled at the appropriate layer with minimal propagation:

  • Service Layer: Handle domain-specific errors and convert to appropriate exceptions
  • Controller Layer: Catch and transform to standardized HTTP responses
  • Infrastructure Layer: Wrap external service errors in domain exceptions

Retry Strategy with Exponential Backoff and Jitter

For transient infrastructure errors, implement retry logic:

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;

    public <T> T executeWithRetry(Supplier<T> operation) {
        int attempts = 0;
        while (attempts < MAX_RETRIES) {
            try {
                return operation.get();
            } catch (TransientException e) {
                attempts++;
                if (attempts >= MAX_RETRIES) {
                    throw new RetryExhaustedException("Operation failed after " + MAX_RETRIES + " attempts", e);
                }
                
                long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempts) * 
                                   (1 + JITTER_FACTOR * (Math.random() - 0.5)));
                try {
                    Thread.sleep(delay);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new RetryExhaustedException("Retry interrupted", ie);
                }
            }
        }
        throw new RetryExhaustedException("Operation failed after "MAX_RETRIES + " attempts");
    }
}

Circuit Breaker Design

Implement circuit breaker for external dependencies:

public class CircuitBreaker {
    private final int failureThreshold;
    private final long timeoutMs;
    private final int successThreshold;
    
    private volatile CircuitState state = CircuitState.CLOSED;
    private volatile int failureCount = 0;
    private volatile long lastFailureTime = 0;
    
    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }
    
    public <T> T execute(Supplier<T> operation) {
        switch (state) {
            case CLOSED:
                return executeClosed(operation);
            case OPEN:
                return executeOpen(operation);
            case HALF_OPEN:
                return executeHalfOpen(operation);
            default:
                throw new IllegalStateException("Unknown circuit state");
        }
    }
    
    private <T> T executeClosed(Supplier<T> operation) {
        try {
            T result = operation.get();
            onSuccess();
            return result;
        } catch (Exception e) {
            onFailure(e);
            throw e;
        }
    }
    
    private <T> T executeOpen(Supplier<T> operation) {
        if (System.currentTimeMillis() - lastFailureTime > timeoutMs) {
            state = CircuitState.HALF_OPEN;
            return executeHalfOpen(operation);
        }
        throw new CircuitBreakerOpenException("Circuit is OPEN");
    }
    
    private <T> T executeHalfOpen(Supplier<T> operation) {
        try {
            T result = operation.get();
            state = CircuitState.CLOSED;
            failureCount = 0;
            return result;
        } catch (Exception e) {
            state = CircuitState.OPEN;
            throw new CircuitBreakerOpenException("Circuit is OPEN after half-open attempt", e);
        }
    }
    
    private void onSuccess() {
        if (state == CircuitState.HALF_OPEN) {
            failureCount = 0;
        }
    }
    
    private void onFailure(Exception e) {
        if (state == CircuitState.CLOSED) {
            failureCount++;
            lastFailureTime = System.currentTimeMillis();
            if (failureCount >= failureThreshold) {
                state = CircuitState.OPEN;
            }
        }
    }
}

Fallback Behavior Design

Implement fallbacks for critical external dependencies:

public class FallbackService {
    private final ExternalPaymentProvider paymentProvider;
    
    public PaymentResult processPaymentWithFallback(PaymentRequest request) {
        try {
            return paymentProvider.processPayment(request);
        } catch (ExternalServiceException e) {
            // Fallback to internal payment processing
            return fallbackToInternalProcessing(request);
        }
    }
    
    private PaymentResult fallbackToInternalProcessing(PaymentRequest request) {
        // Log the failure for monitoring
        logger.warn("Falling back to internal payment processing due to external service failure", 
                   request.getPaymentId());
        
        // Implement fallback logic
        return new PaymentResult("INTERNAL_FALLBACK", "Payment processed via fallback");
    }
}

Error Logging Format and Alerting Thresholds

Structured logging with consistent format:

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);
    
    public void logError(String operation, String errorCode, String errorMessage, 
                        Map<String, Object> context) {
        Map<String, Object> logContext = new HashMap<>(context);
        logContext.put("operation", operation);
        logContext.put("error_code", errorCode);
        logContext.put("message", errorMessage);
        logContext.put("timestamp", Instant.now().toString());
        
        logger.error("{}", JsonUtil.toJson(logContext));
    }
    
    public void alertOnCriticalError(String operation, String errorCode, 
                                   Exception exception) {
        if (isAlertableError(errorCode)) {
            // Send to monitoring system (e.g., PagerDuty, Slack)
            sendAlert(operation, errorCode, exception);
        }
    }
    
    private boolean isAlertableError(String errorCode) {
        return Set.of("DATABASE_CONNECTION_FAILURE", "AUTHENTICATION_ERROR", 
                     "PAYMENT_PROCESSING_FAILURE").contains(errorCode);
    }
}

Alerting Thresholds

Error TypeWarning ThresholdCritical ThresholdAlert Channel
DATABASE_CONNECTION_FAILURE5/min10/minSlack
AUTHENTICATION_ERROR3/min5/minPagerDuty
PAYMENT_PROCESSING_FAILURE2/min5/minPagerDuty
TIMEOUT_ERROR10/min20/minSlack

Error Response Format

Standardized JSON error response:

{
  "error": {
    "code": "INSUFFICIENT_FUNDS",
    "message": "Insufficient funds for payment",
    "details": {
      "required_amount": 100.00,
      "available_balance": 50.00
    },
    "timestamp": "2023-10-15T14:30:00Z",
    "request_id": "req-12345"
  }
}

Error Handling in Service Layer

public class BlockingService {
    private final ErrorLogger errorLogger;
    
    public BlockingResult blockResource(String resourceId, BlockRequest request) {
        try {
            // Business logic validation
            validateBlockRequest(request);
            
            // Check existing blocks
            if (hasActiveBlock(resourceId)) {
                throw new ResourceAlreadyBlockedError("Resource is already blocked");
            }
            
            // Perform blocking operation
            BlockingRecord block = createBlock(resourceId, request);
            
            // Publish event for async processing
            eventPublisher.publish(new ResourceBlockedEvent(resourceId, block));
            
            return new BlockingResult(block);
            
        } catch (ValidationException e) {
            errorLogger.logError("BLOCK_RESOURCE", "VALIDATION_ERROR", 
                               e.getMessage(), Map.of("resource_id", resourceId));
            throw new BadRequestException("Invalid block request", e);
        } catch (ResourceAlreadyBlockedError e) {
            errorLogger.logError("BLOCK_RESOURCE", "RESOURCE_ALREADY_BLOCKED", 
                               e.getMessage(), Map.of("resource_id", resourceId));
            throw new ConflictException("Resource already blocked", e);
        } catch (DatabaseException e) {
            errorLogger.logError("BLOCK_RESOURCE", "DATABASE_ERROR", 
                               "Database operation failed", Map.of("resource_id", resourceId));
            throw new ServiceUnavailableException("Service temporarily unavailable", e);
        } catch (Exception e) {
            errorLogger.logError("BLOCK_RESOURCE", "UNEXPECTED_ERROR", 
                               "Unexpected error occurred", Map.of("resource_id", resourceId));
            throw new InternalServerErrorException("Internal server error", e);
        }
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All inputs undergo strict validation using a combination of:

  • Regex patterns for format validation (e.g., email, phone number)
  • Length constraints to prevent buffer overflow attacks
  • Type checking to ensure expected data types
  • Custom validators for business rule compliance
public class InputValidator {
    private static final Pattern EMAIL_PATTERN = 
        Pattern.compile("^[A-Za-z0-9+_.-]+@([A-Za-z0-9.-]+\\.[A-Za-z]{2,})$");
    
    public static boolean isValidEmail(String email) {
        return email != null && EMAIL_PATTERN.matcher(email).matches();
    }
    
    public static boolean isValidStringLength(String input, int min, int max) {
        return input != null && input.length() >= min && input.length() <= max;
    }
}

SQL Injection Prevention

All database queries use parameterized prepared statements exclusively:

  • Never concatenate user input into SQL strings
  • Use ORM with parameter binding (e.g., JPA/Hibernate)
  • Whitelist allowed characters for specific fields

Authentication Middleware Design

Authentication middleware intercepts all requests and:

  • Verifies JWT tokens for authenticated endpoints
  • Extracts user context from token claims
  • Sets security context in thread-local storage
@Component
public class AuthenticationFilter implements Filter {
    @Override
    public void doFilter(ServletRequest request, ServletResponse response, 
                        FilterChain chain) throws IOException, ServletException {
        HttpServletRequest httpRequest = (HttpServletRequest) request;
        String token = extractToken(httpRequest);
        
        if (token != null && validateToken(token)) {
            SecurityContext context = createSecurityContext(token);
            SecurityContextHolder.setContext(context);
        }
        
        chain.doFilter(request, response);
    }
}

Authorization Enforcement

RBAC-based authorization with role hierarchy:

  • User roles: ADMIN, MODERATOR, USER
  • Resource permissions: READ, WRITE, DELETE
  • Permission checking at service layer level
@PreAuthorize("hasRole('ADMIN') or hasPermission(#userId, 'USER', 'READ')")
public User getUserById(String userId) {
    return userRepository.findById(userId);
}

Sensitive Data Encryption

At Rest: AES-256 encryption with key rotation every 90 days using AWS KMS In Transit: TLS 1.3 with HSTS header enforcement

@Service
public class DataEncryptionService {
    private final AesGcmEncryptor encryptor = new AesGcmEncryptor();
    
    public String encrypt(String plaintext) {
        return encryptor.encrypt(plaintext, getEncryptionKey());
    }
    
    public String decrypt(String ciphertext) {
        return encryptor.decrypt(ciphertext, getEncryptionKey());
    }
}

PII Handling and Masking Strategy

PII fields (SSN, email, phone) are:

  • Masked in logs using format: ***-**-****
  • Redacted from API responses when not required
  • Stored encrypted in database

Secrets Management

Secrets are managed through AWS Secrets Manager:

  • Database credentials rotated every 30 days
  • API keys stored in encrypted format
  • Environment-specific secret paths with IAM role-based access

Audit Logging Design

All security-relevant events are logged with:

  • Required fields: timestamp, user_id, action_type, ip_address, session_id
  • PII masking in logs using redaction rules
  • Log retention: 90 days with archiving to S3
@EventListener
public void handleSecurityEvent(SecurityEvent event) {
    log.info("Security Event - User: {}, Action: {}, IP: {}", 
             event.getUserId(), event.getAction(), event.getIpAddress());
}

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

  • Service Layer: 85% code coverage, focusing on business logic and edge cases
  • Repository Layer: 90% coverage with all CRUD operations tested
  • Domain Objects: 100% coverage for value objects and immutable entities
  • Exception Handling: 100% coverage for all expected error paths
  • Integration Points: 95% coverage for external service mocks and database interactions

Integration Testing Strategy

  • Database Integration: Use Testcontainers with PostgreSQL to test actual database interactions with real schema and constraints
  • External Service Mocks: WireMock for third-party API integrations (payment gateway, notification service)
  • Event System: Embedded Kafka or Redis streams for testing event-driven flows
  • Caching Layer: Use embedded Redis instances for cache interaction testing

Contract Testing

  • API Contracts: Implement Pact-based contract tests between service and consumer teams
  • Event Schemas: Use Avro schema registry with schema validation tests for message format consistency
  • External API Contracts: Mock external service contracts to ensure compatibility and prevent breaking changes

Performance Testing Tool and Key Scenarios

  • Load Testing: Use k6 for load testing with realistic user scenarios (e.g., concurrent blocking requests, bulk operations)
  • Stress Testing: Simulate peak load conditions to identify bottlenecks in blocking logic and database queries
  • Key Scenarios:
    • Concurrent blocking operations (500+ simultaneous requests)
    • Bulk blocking/unblocking operations
    • High-frequency blocking status queries
    • Database connection pool exhaustion under load

Test Data Management Strategy

  • Fixtures: Use factory patterns to generate realistic test data for blocking scenarios (e.g., user, resource, and time-based blocking)
  • Test Data Generators: Implement data generation libraries to create varied test cases (e.g., overlapping blocking periods, edge case time boundaries)
  • Seeding Strategy: Use database migration scripts with test data for consistent baseline state across environments
  • Test Data Isolation: Use database transactional isolation and schema-per-test for parallel test execution
// Example unit test for blocking service with mock dependencies
@Test
public void testBlockResource_Success() {
    // Given
    Resource resource = Resource.builder().id("resource-1").build();
    BlockingRequest request = BlockingRequest.builder()
        .userId("user-1")
        .resourceId("resource-1")
        .startTime(Instant.now())
        .endTime(Instant.now().plus(Duration.ofHours(1)))
        .build();
    
    when(resourceRepository.findById("resource-1")).thenReturn(Optional.of(resource));
    when(blockingRepository.save(any())).thenAnswer(invocation -> invocation.getArgument(0));
    
    // When
    BlockingResult result = blockingService.blockResource(request);
    
    // Then
    assertThat(result.isSuccess()).isTrue();
    verify(blockingRepository).save(any());
}

13. Observability Design

Structured Logging Schema

All logs follow a structured JSON format with consistent fields to enable programmatic parsing and filtering:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO",
  "service": "blocking-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user_12345",
  "entity_id": "block_98765",
  "operation": "createBlock",
  "request_id": "req_abc123",
  "message": "Block created successfully",
  "duration_ms": 45,
  "error_code": null,
  "error_message": null,
  "stack_trace": null,
  "metadata": {
    "block_type": "user",
    "blocked_user_id": "blocked_54321"
  }
}

PII masking rules:

  • User IDs are logged with obfuscation (e.g., user_12345 → user_XXXXX)
  • Email addresses are masked (e.g., john.doe@example.com → j***.d***@example.com)
  • IP addresses are truncated to network level (e.g., 192.168.1.100 → 192.168.1.0)

Distributed Tracing Instrumentation

OpenTelemetry is used for distributed tracing with custom spans:

Key Operations to Trace:

  • blocking.createBlock - Span for creating a new block
  • blocking.checkBlockingStatus - Span for checking if user is blocked
  • blocking.removeBlock - Span for removing a block

Span Attributes Captured:

span.setAttribute("block.type", "user");
span.setAttribute("block.target_id", block.getTargetId());
span.setAttribute("block.created_by", block.getCreatedBy());
span.setAttribute("request.user_id", userId);
span.setAttribute("request.block_type", blockType);

Metrics Design

RED Method Metrics:

Metric NameTypeLabelsDescription
blocking.requests.totalCounteroperation, statusTotal count of blocking operations
blocking.requests.durationHistogramoperation, statusLatency distribution of blocking operations
blocking.errors.totalCounteroperation, error_typeCount of errors by operation and type
blocking.active_blocksGaugeblock_type, statusCurrent count of active blocks

Example Histogram Bucket Configuration:

// For blocking.requests.duration histogram
Histogram.Bucket[] buckets = {
    5.0, 10.0, 25.0, 50.0, 100.0, 250.0, 500.0, 1000.0, 2500.0, 5000.0
};

Health Check Endpoint Design

@RestController
@RequestMapping("/health")
public class HealthController {
    
    @GetMapping("/live")
    public ResponseEntity<Map<String, Object>> live() {
        Map<String, Object> response = new HashMap<>();
        response.put("status", "healthy");
        response.put("timestamp", Instant.now());
        return ResponseEntity.ok(response);
    }
    
    @GetMapping("/ready")
    public ResponseEntity<Map<String, Object>> ready() {
        Map<String, Object> response = new HashMap<>();
        response.put("status", "ready");
        response.put("timestamp", Instant.now());
        response.put("database", databaseHealthCheck());
        response.put("cache", cacheHealthCheck());
        response.put("message_queue", messageQueueHealthCheck());
        return ResponseEntity.ok(response);
    }
    
    @GetMapping("/startup")
    public ResponseEntity<Map<String, Object>> startup() {
        Map<String, Object> response = new HashMap<>();
        response.put("status", "startup_complete");
        response.put("timestamp", Instant.now());
        response.put("dependencies", getDependencyStatuses());
        return ResponseEntity.ok(response);
    }
}

Alerting Thresholds

Critical Alerts (Escalation Path: On-call engineer → Engineering Manager):

  • blocking.requests.duration.p99 > 500ms for 5 minutes
  • blocking.errors.total > 100 in 5 minutes (per operation)
  • blocking.active_blocks > 1M (sudden spike)

Warning Alerts (Escalation Path: On-call engineer):

  • blocking.requests.duration.p99 > 200ms for 15 minutes
  • blocking.errors.total > 50 in 15 minutes (per operation)
  • Database connection pool utilization > 90% for 10 minutes

Service Level Indicators (SLIs):

  • 99.9% of blocking operations complete within 200ms (p99 latency)
  • 99.9% of blocking operations succeed (error rate < 0.1%)

Error Budget:

  • 0.1% error budget (99.9% availability)
  • Alert when error budget consumption exceeds 50% of allocated budget

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

  • Blocking scope definition: Should blocking apply to all user activities or only specific actions (e.g., booking, payment)? This impacts the system's attack surface and complexity.
  • User experience vs. security trade-off: How granular should blocking be? Fine-grained blocking (per action) vs. coarse-grained (per user/session) affects both security and UX.
  • Blocking duration strategy: Should blocking durations be fixed or dynamically calculated based on risk scoring? This affects the system's adaptability to different threat patterns.

Known trade-offs and justifications

Trade-offOption AOption BJustification
Blocking granularityPer-user/sessionPer-action/endpointPer-user provides better performance but less security; per-action offers granular control at cost of increased complexity and potential false positives
Blocking enforcement timingReal-time blockingBatch processingReal-time provides better UX but increases latency; batch processing reduces system load but delays blocking
Blocking state persistenceIn-memory onlyDatabase-backedIn-memory offers better performance but loses state on restart; database persistence ensures durability but adds latency
Blocking policy enforcementCentralized policy engineDistributed rule evaluationCentralized provides consistency but creates single point of failure; distributed allows scaling but increases complexity

Risks and mitigation strategies

  • Risk: False positives leading to legitimate user blocking
    • Mitigation: Implement a review queue with manual override capability and user appeal process
  • Risk: Blocking system becoming a performance bottleneck
    • Mitigation: Use asynchronous processing for non-critical blocking decisions, implement circuit breaker patterns, and cache frequently accessed blocking rules
  • Risk: Blocking bypass through API gateway or direct database access
    • Mitigation: Implement blocking at multiple layers (API gateway, service layer, database level) and enforce strict access control
  • Risk: Blocking rules becoming outdated or ineffective
    • Mitigation: Implement automated rule lifecycle management with periodic review and A/B testing capabilities

Technical debt acknowledged

  • Debt: Hardcoded blocking thresholds in service layer instead of configurable policy engine
    • Impact: Requires code changes for threshold adjustments, increases deployment frequency
    • Mitigation: Plan migration to externalized policy configuration system in next sprint
  • Debt: Single-threaded blocking logic that may not scale to high-concurrency scenarios
    • Impact: Performance degradation under heavy load, potential blocking queue bottlenecks
    • Mitigation: Design for horizontal scaling with sharding strategy and consider async processing patterns
  • Debt: Limited audit trail for blocking decisions (only user ID and timestamp)
    • Impact: Difficult to analyze blocking patterns, debug false positives, or comply with audit requirements
    • Mitigation: Extend audit logging to capture decision context, risk scores, and triggering events

Was this article helpful?

Topics in this article

More from Backend LLD

  • Backend HLD

    API Rate Limiter Design and Implementation

    Comprehensive backend system design for rate limiting APIs with token bucket and sliding window algorithms, including distributed architecture considerations.

    • CategoryBackend HLD
    • UpdatedOctober 5, 2026
    • Reading time52 min read
  • Backend LLD

    API Throttling Service Design and Implementation

    Comprehensive low-level design for rate-limiting and traffic control in high-scale API services with token bucket and sliding window algorithms.

    • CategoryBackend LLD
    • UpdatedApril 27, 2025
    • Reading time41 min read
  • Backend LLD

    Inventory Reservation System Design

    Comprehensive low-level design for a scalable inventory reservation system with distributed locking and eventual consistency patterns.

    • CategoryBackend LLD
    • UpdatedDecember 7, 2025
    • Reading time42 min read

Stay in the Loop

Get curated DSA patterns, coding problems, frontend and backend LLD/HLD, and AI/ML System Design insights delivered straight to your inbox. No spam, just focused interview and engineering content.