Lease Expiration System Design

Comprehensive low-level design for a scalable lease expiration system with automatic cleanup and notification mechanisms.

PublishedJune 29, 2025
Length8,077 words
Reading time41 min read

Backend LLD

A robust lease expiration system requires careful consideration of distributed consistency, automatic cleanup mechanisms, and notification strategies to ensure reliable resource management across scalable infrastructure.

1. Problem Statement & Requirements

Clarifying questions

What is the scale of the system? How many leases are expected to be active concurrently, and what's the peak rate of lease creation/expiration?
What consistency requirements exist? Should lease expiration be strongly consistent (e.g., no race conditions between lease acquisition and expiration) or eventually consistent?
What are the key entities in the system? Are leases tied to specific resources (e.g., vehicles, rooms) or are they generic time-bound locks?
What's the read/write ratio for leases? Is it predominantly read operations (e.g., checking if a resource is leased) or more write-heavy?
What are the latency sensitivity requirements? Should lease expiration be real-time (e.g., <100ms) or can it be batched with some tolerance?
What authentication/authorization model is in place? Will lease creation/expiration be restricted to specific users or roles?
Are there any external systems that integrate with the lease system? For example, payment systems or resource management systems.

Functional requirements

Create leases for specific resources with start and end times
Automatically expire leases that have reached their end time
Allow lease extension by authorized users
Allow manual lease cancellation before expiration
Query active leases for a specific resource or user
Notify stakeholders when leases are about to expire or have expired

Non-functional requirements

Latency targets:
- p50: <50ms for lease creation and querying
- p95: <200ms for lease creation and querying
- p99: <500ms for lease creation and querying
Throughput: 10,000 lease operations/second (creation, extension, cancellation)
Availability: 99.9% uptime for lease service
Durability: Lease state must survive system failures

Constraints and assumptions

System assumes a single lease can be associated with one resource at a time
Lease expiration is handled asynchronously to avoid blocking operations
The system assumes that clocks are synchronized across nodes (e.g., via NTP)
Lease durations are fixed or configurable within a range (e.g., 1 hour to 30 days)
Lease state is not persisted in memory only (must survive restarts)

Out of scope

Lease renewal logic for leases that are about to expire (e.g., auto-renewal)
Integration with external payment systems for lease fees
Multi-tenant support (each tenant has isolated lease data)
Lease transfer between users or resources
Lease state history tracking beyond current status (e.g., no audit trail of lease changes)

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for this system due to:

Strong typing and schema evolution support
Efficient binary serialization reducing payload size
Built-in support for streaming and bidirectional communication
Native support for versioning through proto package management
Better performance characteristics for high-throughput systems

Endpoint Definitions

Method	Path	Description	Auth
CreateLease	/v1/leases	Creates a new lease with specified parameters	JWT + RBAC
GetLease	/v1/leases/{id}	Retrieves lease details by ID	JWT + RBAC
ExtendLease	/v1/leases/{id}/extend	Extends lease duration	JWT + RBAC
CancelLease	/v1/leases/{id}/cancel	Cancels active lease	JWT + RBAC
ListLeases	/v1/leases	Lists leases with filtering and pagination	JWT + RBAC
GetLeaseStatus	/v1/leases/{id}/status	Gets current status of lease	JWT + RBAC
RenewLease	/v1/leases/{id}/renew	Renews lease before expiration	JWT + RBAC

Request and Response Schemas

CreateLeaseRequest

message CreateLeaseRequest {
  string resource_id = 1;
  int64 duration_seconds = 2;
  string tenant_id = 3;
  map<string, string> metadata = 4;
}

CreateLeaseResponse

message CreateLeaseResponse {
  string lease_id = 1;
  string resource_id = 2;
  int64 start_time = 3;
  int64 end_time = 4;
  string status = 5;
}

Lease

message Lease {
  string lease_id = 1;
  string resource_id = 2;
  string tenant_id = 3;
  int64 start_time = 4;
  int64 end_time = 5;
  string status = 6;
  map<string, string> metadata = 7;
  int64 created_at = 8;
  int64 updated_at = 9;
}

HTTP Status Codes and Error Response Format

Status Code	Description
200	Success
201	Created
400	Bad Request
401	Unauthorized
403	Forbidden
404	Not Found
409	Conflict (lease already exists)
429	Rate Limited
500	Internal Server Error

Standardized error response:

{
  "error": {
    "code": "LEASE_NOT_FOUND",
    "message": "Lease with ID abc123 not found",
    "details": {
      "lease_id": "abc123"
    }
  }
}

API Versioning Strategy

Versioned using path prefix: /v1/leases
Supports multiple versions with backward compatibility maintained through:

Protocol buffer package versioning
Service method signature evolution
Migration paths for breaking changes

Pagination Design

Cursor-based pagination selected because:

More stable than offset-based pagination when data changes
Better performance for large datasets
Consistent results across page boundaries

message ListLeasesRequest {
  string cursor = 1;
  int32 page_size = 2;
  string status_filter = 3;
  string resource_id_filter = 4;
}

message ListLeasesResponse {
  repeated Lease leases = 1;
  string next_cursor = 2;
}

Authentication and Authorization

All endpoints require JWT token with RBAC permissions:

lease:read - for read operations
lease:write - for write operations
lease:admin - for administrative actions

Rate Limiting Strategy

Per-endpoint rate limiting:

CreateLease: 100 req/min
GetLease: 500 req/min
ExtendLease: 50 req/min
CancelLease: 25 req/min

Rate limiting enforced using Redis with token bucket algorithm.

Idempotency Design

All mutation endpoints support idempotency keys:

message ExtendLeaseRequest {
  string lease_id = 1;
  int64 extend_duration_seconds = 2;
  string idempotency_key = 3;
}

Idempotency keys are stored in Redis with TTL of 5 minutes to prevent key accumulation.

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    LEASE {
        string id PK
        string tenant_id
        string asset_id
        datetime start_date
        datetime end_date
        datetime expiry_date
        string status
        string lease_type
        string currency
        decimal amount
        datetime created_at
        datetime updated_at
        datetime deleted_at
    }

    ASSET {
        string id PK
        string name
        string description
        string asset_type
        string location
        datetime created_at
        datetime updated_at
    }

    TENANT {
        string id PK
        string name
        string email
        string phone
        datetime created_at
        datetime updated_at
    }

    LEASE_EVENT {
        string id PK
        string lease_id FK
        string event_type
        datetime occurred_at
        string description
        json metadata
        datetime created_at
    }

    LEASE_PAYMENT {
        string id PK
        string lease_id FK
        datetime payment_date
        decimal amount
        string currency
        string payment_method
        string status
        datetime created_at
    }

    LEASE {
        string id PK
        string tenant_id FK
        string asset_id FK
        datetime start_date
        datetime end_date
        datetime expiry_date
        string status
        string lease_type
        decimal amount
        datetime created_at
        datetime updated_at
        datetime deleted_at
    }

    LEASE_EVENT {
        string id PK
        string lease_id FK
        string event_type
        datetime occurred_at
        string description
        json metadata
        datetime created_at
    }

    LEASE_PAYMENT {
        string id PK
        string lease_id FK
        datetime payment_date
        decimal amount
        string currency
        string payment_method
        string status
        datetime created_at
    }

    LEASE ||--o{ LEASE_EVENT : "has"
    LEASE ||--o{ LEASE_PAYMENT : "has"
    LEASE --o{ ASSET : "references"
    LEASE --o{ TENANT : "belongs_to"

Table Schema Definitions

leases table

CREATE TABLE leases (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    asset_id UUID NOT NULL REFERENCES assets(id),
    start_date TIMESTAMP NOT NULL,
    end_date TIMESTAMP NOT NULL,
    expiry_date TIMESTAMP NOT NULL,
    status VARCHAR(50) NOT NULL DEFAULT 'ACTIVE',
    lease_type VARCHAR(50) NOT NULL,
    amount DECIMAL(15,2) NOT NULL,
    currency VARCHAR(3) NOT NULL DEFAULT 'USD',
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    deleted_at TIMESTAMP,
    
    -- Indexes
    INDEX idx_lease_tenant_id (tenant_id),
    INDEX idx_lease_asset_id (asset_id),
    INDEX idx_lease_expiry_date (expiry_date),
    INDEX idx_lease_status (status),
    INDEX idx_lease_created_at (created_at),
    INDEX idx_lease_deleted_at (deleted_at)
);

lease_events table

CREATE TABLE lease_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    lease_id UUID NOT NULL REFERENCES leases(id),
    event_type VARCHAR(100) NOT NULL,
    occurred_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    description TEXT,
    metadata JSONB,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_lease_event_lease_id (lease_id),
    INDEX idx_lease_event_occurred_at (occurred_at)
);

lease_payments table

CREATE TABLE lease_payments (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    lease_id UUID NOT NULL REFERENCES leases(id),
    payment_date TIMESTAMP NOT NULL,
    amount DECIMAL(15,2) NOT NULL,
    currency VARCHAR(3) NOT NULL DEFAULT 'USD',
    payment_method VARCHAR(50),
    status VARCHAR(50) NOT NULL DEFAULT 'PENDING',
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_lease_payment_lease_id (lease_id),
    INDEX idx_lease_payment_status (status),
    INDEX idx_lease_payment_date (payment_date)
);

Primary Key Strategy

UUID as Primary Key

Justification:
- Enables distributed system design without coordination
- Provides better security by not exposing sequential IDs
- Allows for data sharding based on UUID ranges
- Supports cross-system integration without ID conflicts

Index Design

Index Name	Fields	Type	Purpose
idx_lease_tenant_id	tenant_id	Standard	Tenant-based queries
idx_lease_asset_id	asset_id	Standard	Asset-based queries
idx_lease_expiry_date	expiry_date	Standard	Expiration query optimization
idx_lease_status	status	Standard	Status-based filtering
idx_lease_created_at	created_at	Standard	Time-range queries
idx_lease_deleted_at	deleted_at	Standard	Soft-delete filtering
idx_lease_payment_lease_id	lease_id	Standard	Payment lookup by lease
idx_lease_event_lease_id	lease_id	Standard	Event lookup by lease

Data Partitioning / Sharding Strategy

Sharding by Tenant ID

Strategy: Hash-based sharding on tenant_id
Rationale:
- Tenant isolation is critical for compliance and security
- Reduces cross-tenant query complexity
- Supports horizontal scaling with predictable data distribution

Soft Delete vs. Hard Delete Decision

Soft Delete

Justification:
- Legal and audit compliance requirements
- Data recovery capability for business operations
- Historical data analysis for lease patterns
- Support for data retention policies

Audit Trail Design

Audit Fields

All tables include:

created_at - Timestamp of creation
updated_at - Timestamp of last modification
deleted_at - Timestamp of soft deletion (if applicable)
created_by - User or system that created the record (optional)

Data Archival Strategy and Retention Policy

Archival Strategy

Active Leases: Keep in primary database with real-time access
Expired/Cancelled Leases: Move to archival storage after 30 days
Retention Policy:
- Active leases: Keep forever with full audit trail
- Historical leases: Retain for 7 years per regulatory requirements
- Payment data: Retain for 10 years for tax and audit purposes

Archival Implementation:

-- Archive expired leases to separate table or storage system
CREATE TABLE lease_archives (
    -- Same structure as leases but with additional archive metadata
    archived_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    archive_reason VARCHAR(100),
    -- ... other fields from leases table
);

4. Class Design & Service Layer

classDiagram
    class LeaseService {
        +createLease(LeaseRequest) Lease
        +renewLease(String, Duration) Lease
        +terminateLease(String) boolean
        +getActiveLeases(String) List~Lease~
        +extendLease(String, Duration) Lease
        +checkLeaseStatus(String) LeaseStatus
        +processExpiredLeases() List~String~
    }
    
    class LeaseRepository {
        +save(Lease) Lease
        +findById(String) Optional~Lease~
        +findByStatusAndExpiryTimeBefore(LeaseStatus, Instant) List~Lease~
        +updateStatus(String, LeaseStatus) boolean
        +findAllByOwnerId(String) List~Lease~
    }
    
    class Lease {
        -id: String
        -ownerId: String
        -assetId: String
        -status: LeaseStatus
        -createdAt: Instant
        -expiresAt: Instant
        -renewalCount: int
        +extend(Duration)
        +isExpired() boolean
        +canBeRenewed() boolean
    }
    
    class LeaseStatus {
        PENDING
        ACTIVE
        RENEWED
        EXPIRED
        TERMINATED
    }
    
    class LeaseEventPublisher {
        +publishLeaseExpired(Lease) void
        +publishLeaseRenewed(Lease) void
        +publishLeaseTerminated(Lease) void
    }
    
    class LeaseValidator {
        +validateCreateRequest(LeaseRequest) ValidationResult
        +validateRenewalRequest(String, Duration) ValidationResult
    }
    
    class LeaseProcessor {
        +processExpiredLeases() List~String~
        +handleLeaseExpiry(Lease) void
    }
    
    class LeaseEvent {
        -id: String
        -leaseId: String
        -eventType: LeaseEventType
        -timestamp: Instant
        -payload: Map~String, Object~
    }
    
    class LeaseEventType {
        EXPIRED
        RENEWED
        TERMINATED
    }
    
    LeaseService --> LeaseRepository : uses
    LeaseService --> LeaseEventPublisher : publishes
    LeaseService --> LeaseValidator : validates
    LeaseService --> LeaseProcessor : delegates
    Lease --> LeaseStatus : has
    LeaseEventPublisher --> LeaseEvent : creates

stateDiagram-v2
    [*] --> PENDING
    PENDING --> ACTIVE : lease created and validated
    ACTIVE --> RENEWED : renewal requested
    ACTIVE --> EXPIRED : expiry time reached
    ACTIVE --> TERMINATED : manual termination
    RENEWED --> ACTIVE : renewal completed
    EXPIRED --> [*] : cleanup or notification
    TERMINATED --> [*] : end state

Service Layer Architecture

The LeaseService is the core business logic layer that orchestrates lease operations. It implements a strategy pattern for handling different lease statuses and includes validation, event publishing, and processing logic.

Business Logic Patterns

Strategy Pattern Implementation:

public interface LeaseStatusHandler {
    void handle(Lease lease);
}

@Component
public class ActiveLeaseHandler implements LeaseStatusHandler {
    private final LeaseEventPublisher eventPublisher;
    
    @Override
    public void handle(Lease lease) {
        if (lease.isExpired()) {
            eventPublisher.publishLeaseExpired(lease);
            lease.updateStatus(LeaseStatus.EXPIRED);
        }
    }
}

@Component
public class LeaseProcessor {
    private final Map<LeaseStatus, LeaseStatusHandler> handlers;
    
    public LeaseProcessor(List<LeaseStatusHandler> handlerList) {
        this.handlers = handlerList.stream()
            .collect(Collectors.toMap(
                h -> getHandlerStatus(h),
                Function.identity()
            ));
    }
    
    public void processLease(Lease lease) {
        LeaseStatusHandler handler = handlers.get(lease.getStatus());
        if (handler != null) {
            handler.handle(lease);
        }
    }
}

Critical-Path Method Implementation

@Transactional
public Lease extendLease(String leaseId, Duration extension) {
    // Transaction boundary: entire operation is atomic
    Lease lease = leaseRepository.findById(leaseId)
        .orElseThrow(() -> new LeaseNotFoundException("Lease not found: " + leaseId));
    
    // Optimistic locking - check version or use database constraints
    if (!lease.canBeRenewed()) {
        throw new LeaseInvalidStateException("Cannot renew lease in status: " + lease.getStatus());
    }
    
    // Validate extension duration
    if (extension.isNegative() || extension.isZero()) {
        throw new IllegalArgumentException("Extension must be positive");
    }
    
    // Update lease expiration time
    Instant newExpiry = lease.getExpiresAt().plus(extension);
    lease.setExpiresAt(newExpiry);
    lease.incrementRenewalCount();
    
    // Update status if needed
    if (lease.getStatus() == LeaseStatus.EXPIRED) {
        lease.updateStatus(LeaseStatus.RENEWED);
    }
    
    // Save updated lease
    Lease updatedLease = leaseRepository.save(lease);
    
    // Publish event for monitoring and external systems
    eventPublisher.publishLeaseRenewed(updatedLease);
    
    return updatedLease;
}

Service Interface Contracts

public interface LeaseService {
    Lease createLease(LeaseRequest request);
    Lease renewLease(String leaseId, Duration extension);
    boolean terminateLease(String leaseId);
    List<Lease> getActiveLeases(String ownerId);
    Lease extendLease(String leaseId, Duration extension);
    LeaseStatus checkLeaseStatus(String leaseId);
    List<String> processExpiredLeases();
}

Domain Object Design

@Entity
@Table(name = "leases")
public class Lease {
    @Id
    @GeneratedValue(strategy = GenerationType.UUID)
    private String id;
    
    @Column(name = "owner_id")
    private String ownerId;
    
    @Column(name = "asset_id")
    private String assetId;
    
    @Enumerated(EnumType.STRING)
    @Column(name = "status")
    private LeaseStatus status;
    
    @Column(name = "created_at")
    private Instant createdAt;
    
    @Column(name = "expires_at")
    private Instant expiresAt;
    
    @Column(name = "renewal_count")
    private int renewalCount;
    
    public boolean isExpired() {
        return Instant.now().isAfter(expiresAt);
    }
    
    public boolean canBeRenewed() {
        return status == LeaseStatus.ACTIVE || 
               status == LeaseStatus.RENEWED ||
               status == LeaseStatus.EXPIRED;
    }
    
    public void extend(Duration duration) {
        this.expiresAt = this.expiresAt.plus(duration);
        this.renewalCount++;
    }
    
    public void updateStatus(LeaseStatus newStatus) {
        this.status = newStatus;
    }
}

Transaction Boundary Design

The transaction boundary encompasses:

Lease validation and creation
Status transitions within a single lease
Event publishing with database consistency
Atomic updates to lease state and metadata

Saga Pattern for Distributed Transactions

For complex operations involving external systems (e.g., asset management, billing), a saga pattern is implemented:

public class LeaseSaga {
    private final List<SagaStep> steps;
    
    public void executeLeaseCreation(LeaseRequest request) {
        // Step 1: Create lease record
        // Step 2: Reserve asset (external system)
        // Step 3: Create billing record (external system)
        // If any step fails, execute compensation steps
        
        for (SagaStep step : steps) {
            try {
                step.execute();
            } catch (Exception e) {
                compensateFailedStep(step);
                throw new SagaExecutionException("Saga failed at step: " + step.getName(), e);
            }
        }
    }
}

Event Publishing Strategy

@Component
public class LeaseEventPublisher {
    private final ApplicationEventPublisher eventPublisher;
    
    public void publishLeaseExpired(Lease lease) {
        LeaseEvent event = new LeaseEvent(
            UUID.randomUUID().toString(),
            lease.getId(),
            LeaseEventType.EXPIRED,
            Instant.now(),
            Map.of("leaseId", lease.getId(), "ownerId", lease.getOwnerId())
        );
        
        eventPublisher.publishEvent(event);
    }
}

5. Concurrency & Thread Safety

Concurrency Model

The system will use a thread pool model with Java's ExecutorService for managing concurrent lease expiration operations. Each lease expiration task will be executed in a separate thread from a fixed-size thread pool to prevent resource exhaustion. For high-concurrency scenarios, we'll leverage CompletableFuture with custom ForkJoinPool to handle asynchronous lease validation and cleanup operations.

Race Condition Prevention Strategy

Race conditions will be mitigated through:

Database-level atomic operations - Using SELECT FOR UPDATE in transactions to lock lease records during expiration checks
Optimistic locking - Using version fields in lease entities to detect concurrent modifications
Idempotency keys - For external system calls to prevent duplicate processing of the same expiration event

Optimistic vs. Pessimistic Locking Selection

We'll use optimistic locking for most operations as it provides better performance under moderate contention and avoids blocking. The system will:

Use version fields in the Lease entity to detect concurrent modifications
Implement retry logic with exponential backoff for optimistic lock failures
Use pessimistic locking only in critical paths like immediate lease renewal or cancellation

Deadlock Prevention Strategy

Deadlock prevention will be implemented through:

Consistent lock ordering - All operations acquire locks in a predefined order (e.g., by lease ID)
Timeout-based lock acquisition - Set 500ms timeouts for database locks to prevent indefinite waits
Lock hierarchy - Lease entities will always be locked before related entities (e.g., lease history, payments)
Transaction isolation level - Use READ COMMITTED to prevent phantom reads

Idempotency Implementation for Concurrent Duplicate Requests

For concurrent duplicate lease expiration requests, we'll implement the following:

public class LeaseExpirationService {
    private static final String EXPIRATION_LOCK_PREFIX = "lease_exp_";
    
    public boolean handleLeaseExpiration(String leaseId, String idempotencyKey) {
        // Check if this expiration request has already been processed
        String cacheKey = EXPIRATION_LOCK_PREFIX + leaseId + "_" + idempotencyKey;
        if (cacheService.exists(cacheKey)) {
            return true; // Already processed, idempotent success
        }
        
        try {
            // Attempt to acquire exclusive lock for this lease expiration
            String lockKey = "lease_lock:" + leaseId;
            boolean lockAcquired = distributedLockService.acquire(lockKey, 30000); // 30s timeout
            
            if (!lockAcquired) {
                // Another process is handling this lease, wait and retry or return failure
                return false;
            }
            
            // Proceed with expiration logic
            boolean result = processLeaseExpiration(leaseId);
            
            // Mark as processed in cache for idempotency
            cacheService.set(cacheKey, "processed", Duration.ofHours(24));
            
            return result;
            
        } finally {
            // Release the lock
            distributedLockService.release("lease_lock:" + leaseId);
        }
    }
    
    private boolean processLeaseExpiration(String leaseId) {
        // Transaction boundary - all operations in single DB transaction
        return leaseRepository.withTransaction(() -> {
            Lease lease = leaseRepository.findByIdForUpdate(leaseId);
            
            if (lease == null || !lease.isExpired()) {
                return false;
            }
            
            // Perform expiration actions
            lease.markAsExpired();
            leaseRepository.save(lease);
            
            // Publish expiration event
            eventPublisher.publish(new LeaseExpiredEvent(leaseId));
            
            return true;
        });
    }
}

The idempotency strategy uses:

Cache-based tracking - Store processed expiration keys with TTL to prevent reprocessing
Distributed locking - Prevents concurrent processing of same lease by different instances
Transaction boundary - Ensures atomicity of the entire expiration operation

This approach prevents race conditions while maintaining system responsiveness under concurrent load. The system will handle up to 10,000 concurrent lease expirations with proper resource management and graceful degradation when system load exceeds capacity.

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant U as User
    participant API as LeaseServiceAPI
    participant DB as Database
    participant Cache as CacheLayer
    participant ExpiredLeaseProcessor as ExpiredLeaseProcessor

    U->>API: CreateLease(leaseDetails)
    API->>DB: Save(lease)
    DB-->>API: leaseId
    API->>Cache: Put(leaseId, lease)
    API-->>U: LeaseCreatedResponse(leaseId)

    U->>API: GetLease(leaseId)
    API->>Cache: Get(leaseId)
    Cache-->>API: lease
    API-->>U: LeaseResponse(lease)

    U->>API: RenewLease(leaseId, newExpiry)
    API->>DB: Update(leaseId, newExpiry)
    DB-->>API: updatedLease
    API->>Cache: Update(leaseId, updatedLease)
    API-->>U: LeaseRenewedResponse(leaseId)

    U->>API: CancelLease(leaseId)
    API->>DB: Update(status, cancelled)
    DB-->>API: updatedLease
    API->>Cache: Invalidate(leaseId)
    API-->>U: LeaseCancelledResponse(leaseId)

    Note over API,ExpiredLeaseProcessor: Periodic check for expired leases
    loop Every 5 minutes
        API->>DB: FindExpiredLeases(currentTime)
        DB-->>API: expiredLeaseIds
        API->>ExpiredLeaseProcessor: ProcessExpiredLeases(expiredLeaseIds)
        ExpiredLeaseProcessor->>DB: Update(status, expired)
        ExpiredLeaseProcessor->>Cache: Invalidate(expiredLeaseIds)
        ExpiredLeaseProcessor->>EventPublisher: Publish(LeaseExpiredEvent)
    end

sequenceDiagram
    participant U as User
    participant API as LeaseServiceAPI
    participant DB as Database
    participant Cache as CacheLayer
    participant EventPublisher as EventPublisher

    U->>API: CreateLease(leaseDetails)
    API->>DB: Save(lease)
    DB-->>API: leaseId
    API->>Cache: Put(leaseId, lease)
    API-->>U: LeaseCreatedResponse(leaseId)

    U->>API: GetLease(leaseId)
    API->>Cache: Get(leaseId)
    Cache-->>API: lease
    API-->>U: LeaseResponse(lease)

    U->>API: RenewLease(leaseId, newExpiry)
    API->>DB: Update(leaseId, newExpiry)
    DB-->>API: updatedLease
    API->>Cache: Update(leaseId, updatedLease)
    API-->>U: LeaseRenewedResponse(leaseId)

    U->>API: CancelLease(leaseId)
    API->>DB: Update(status, cancelled)
    DB-->>API: updatedLease
    API->>Cache: Invalidate(leaseId)
    API-->>U: LeaseCancelledResponse(leaseId)

    Note over API,EventPublisher: Async processing for expired leases
    API->>DB: FindExpiredLeases(currentTime)
    DB-->>API: expiredLeaseIds
    API->>EventPublisher: Publish(LeaseExpiredEvent)
    EventPublisher-->>API: Ack
    API->>DB: Update(status, expired)
    API->>Cache: Invalidate(expiredLeaseIds)

sequenceDiagram
    participant User as User
    participant LeaseService as LeaseServiceAPI
    participant DB as Database
    participant Cache as CacheLayer
    participant EventPublisher as EventPublisher
    participant NotificationService as NotificationService

    User->>LeaseService: CreateLease(leaseDetails)
    LeaseService->>DB: Save(lease)
    DB-->>LeaseService: leaseId
    LeaseService->>Cache: Put(leaseId, lease)
    LeaseService-->>User: LeaseCreatedResponse(leaseId)

    User->>LeaseService: GetLease(leaseId)
    LeaseService->>Cache: Get(leaseId)
    Cache-->>LeaseService: lease
    LeaseService-->>User: LeaseResponse(lease)

    User->>LeaseService: RenewLease(leaseId, newExpiry)
    LeaseService->>DB: Update(leaseId, newExpiry)
    DB-->>LeaseService: updatedLease
    LeaseService->>Cache: Update(leaseId, updatedLease)
    LeaseService-->>User: LeaseRenedResponse(leaseId)

    User->>LeaseService: CancelLease(leaseId)
    LeaseService->>DB: Update(status, cancelled)
    DB-->>LeaseService: updatedLease
    LeaseService->>Cache: Invalidate(leaseId)
    LeaseService-->>User: LeaseCancelledResponse(leaseId)

    LeaseService->>DB: FindExpiredLeases(currentTime)
    DB-->>LeaseService: expiredLeaseIds
    LeaseService->>EventPublisher: Publish(LeaseExpiredEvent)
    EventPublisher-->>LeaseService: Ack
    LeaseService->>DB: Update(status, expired)
    LeaseService->>Cache: Invalidate(expiredLeaseIds)

    EventPublisher->>NotificationService: SendNotification(leaseId, expired)
    NotificationService-->>EventPublisher: Ack

7. Caching Design

Cache Strategy Selection

For lease expiration system, I'll use cache-aside pattern with write-through for critical data and write-behind for non-critical audit data. This provides the best balance between consistency and performance.

Cache Key Design

Cache keys will follow a namespace strategy:

lease:{leaseId}:metadata
lease:{leaseId}:status
lease:{leaseId}:expiration
lease:{leaseId}:tenant:{tenantId}

TTL Strategy

Data Type	TTL	Reason
Lease metadata	5 minutes	Frequent reads, short-lived
Lease status	1 minute	High update frequency
Lease expiration	30 seconds	Critical for real-time checks
Tenant-specific leases	10 minutes	Tenant-level caching for performance

Eviction Policy

Using LRU (Least Recently Used) with TTL-based expiration as primary policy. For high-traffic scenarios, I'll implement LFU (Least Frequently Used) for the most accessed lease data.

Cache Stampede Prevention

Implement probabilistic early expiry with 15% random jitter:

public class CacheStampedePrevention {
    private static final double JITTER_PERCENTAGE = 0.15;
    
    public String getWithStampedeProtection(String key, Supplier<String> loader) {
        String cached = cache.get(key);
        if (cached != null) {
            return cached;
        }
        
        // Add jitter to expiration time
        int ttlWithJitter = (int) (ttl * (1 - JITTER_PERCENTAGE + 
            Math.random() * JITTER_PERCENTAGE * 2));
        
        // Use distributed lock for cache population
        if (distributedLock.acquire(key, ttlWithJitter)) {
            try {
                String freshData = loader.get();
                cache.put(key, freshData, ttlWithJitter);
                return freshData;
            } finally {
                distributedLock.release(key);
            }
        }
        
        // Fallback to stale data with warning
        return cache.get(key);
    }
}

Distributed Cache Architecture

Using Redis Cluster with:

3 master nodes for high availability
3 replica nodes per master for read scaling
Redis Sentinel for automatic failover
Cache sharding by tenant ID to reduce hotspots

Cache Invalidation Strategy

Implement event-driven invalidation using the following approach:

@Component
public class LeaseCacheInvalidator {
    
    @EventListener
    public void handleLeaseExpiryEvent(LeaseExpiryEvent event) {
        String leaseId = event.getLeaseId();
        
        // Invalidate all related cache entries
        cache.evict("lease:" + leaseId + ":metadata");
        cache.evict("lease:" + leaseId + ":status");
        cache.evict("lease:" + leaseId + ":expiration");
        
        // Invalidate tenant-specific caches
        cache.evict("lease:" + leaseId + ":tenant:" + event.getTenantId());
        
        // Publish cache invalidation event for distributed systems
        eventPublisher.publish(new CacheInvalidationEvent("lease", leaseId));
    }
}

Cache Read-Through Pattern

@Service
public class LeaseCacheService {
    
    private final Cache<String, Lease> leaseCache;
    private final LeaseRepository leaseRepository;
    
    public Lease getLeaseWithCaching(String leaseId) {
        String cacheKey = "lease:" + leaseId + ":metadata";
        
        // Try cache first
        Lease cachedLease = leaseCache.getIfPresent(cacheKey);
        if (cachedLease != null) {
            return cachedLease;
        }
        
        // Cache miss - fetch from database
        Lease lease = leaseRepository.findById(leaseId);
        if (lease != null) {
            // Cache with TTL
            leaseCache.put(cacheKey, lease, Duration.ofMinutes(5));
        }
        
        return lease;
    }
}

Cache Performance Considerations

Warm-up strategy: Preload frequently accessed leases during system startup
Cache warming: Use background jobs to warm cache for high-traffic tenants
Memory allocation: Reserve 20% of total memory for cache with automatic eviction
Cache hit ratio monitoring: Track and alert when below 85% threshold

Cache Tiering Strategy

Implement multi-tier caching:

In-memory (Local) - Fastest access for most frequently used leases
Redis Cluster - Distributed cache for cross-service consistency
Database - Fallback storage with eventual consistency

This design ensures optimal performance while maintaining data consistency and preventing cache stampede scenarios that could overwhelm the system during peak load periods.

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and persistence concerns. Each entity has its dedicated repository interface and implementation.

public interface LeaseRepository {
    Optional<Lease> findById(UUID id);
    List<Lease> findByTenantId(UUID tenantId);
    List<Lease> findByPropertyId(UUID propertyId);
    List<Lease> findActiveLeases();
    List<Lease> findExpiringSoon(Duration window);
    Lease save(Lease lease);
    void delete(UUID id);
    boolean existsById(UUID id);
}

@Repository
public class LeaseRepositoryImpl implements LeaseRepository {
    private final JdbcTemplate jdbcTemplate;
    private final ObjectMapper objectMapper;
    
    public LeaseRepositoryImpl(JdbcTemplate jdbcTemplate, ObjectMapper objectMapper) {
        this.jdbcTemplate = jdbcTemplate;
        this.objectMapper = objectMapper;
    }
    
    @Override
    public Optional<Lease> findById(UUID id) {
        String sql = "SELECT * FROM leases WHERE id = ?";
        try {
            return Optional.of(jdbcTemplate.queryForObject(sql, new LeaseRowMapper(), id));
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }
    
    @Override
    public List<Lease> findByTenantId(UUID tenantId) {
        String sql = "SELECT * FROM leases WHERE tenant_id = ? ORDER BY created_at DESC";
        return jdbcTemplate.query(sql, new LeaseRowMapper(), tenantId);
    }
    
    @Override
    public List<Lease> findByPropertyId(UUID propertyId) {
        String sql = "SELECT * FROM leases WHERE property_id = ? ORDER BY created_at DESC";
        return jdbcTemplate.query(sql, new LeaseRowMapper(), propertyId);
    }
    
    @Override
    public List<Lease> findActiveLeases() {
        String sql = "SELECT * FROM leases WHERE status = 'ACTIVE' AND end_date > NOW()";
        return jdbcTemplate.query(sql, new LeaseRowMapper());
    }
    
    @Override
    public List<Lease> findExpiringSoon(Duration window) {
        String sql = "SELECT * FROM leases WHERE status = 'ACTIVE' AND end_date BETWEEN NOW() AND DATE_ADD(NOW(), INTERVAL ? SECOND)";
        return jdbcTemplate.query(sql, new LeaseRowMapper(), window.getSeconds());
    }
    
    @Override
    public Lease save(Lease lease) {
        if (lease.getId() == null) {
            return createLease(lease);
        } else {
            return updateLease(lease);
        }
    }
    
    private Lease createLease(Lease lease) {
        String sql = """
            INSERT INTO leases (id, tenant_id, property_id, start_date, end_date, status, lease_terms, created_at, updated_at)
            VALUES (?, ?, ?, ?, ?, ?, ?, NOW(), NOW())
            """;
        jdbcTemplate.update(sql, 
            lease.getId(),
            lease.getTenantId(),
            lease.getPropertyId(),
            lease.getStartDate(),
            lease.getEndDate(),
            lease.getStatus().name(),
            objectMapper.writeValueAsString(lease.getLeaseTerms())
        );
        return findById(lease.getId()).orElseThrow();
    }
    
    private Lease updateLease(Lease lease) {
        String sql = """
            UPDATE leases SET 
                tenant_id = ?, 
                property_id = ?, 
                start_date = ?, 
                end_date = ?, 
                status = ?, 
                lease_terms = ?,
                updated_at = NOW()
            WHERE id = ?
            """;
        jdbcTemplate.update(sql,
            lease.getTenantId(),
            lease.getPropertyId(),
            lease.getStartDate(),
            lease.getEndDate(),
            lease.getStatus().name(),
            objectMapper.writeValueAsString(lease.getLeaseTerms()),
            lease.getId()
        );
        return findById(lease.getId()).orElseThrow();
    }
    
    @Override
    public void delete(UUID id) {
        String sql = "DELETE FROM leases WHERE id = ?";
        jdbcTemplate.update(sql, id);
    }
    
    @Override
    public boolean existsById(UUID id) {
        String sql = "SELECT COUNT(*) FROM leases WHERE id = ?";
        return jdbcTemplate.queryForObject(sql, Integer.class, id) > 0;
    }
}

ORM vs. Raw Query Decisions

For the lease system, raw SQL queries are preferred over ORM for several reasons:

Performance: Raw queries provide better control over query execution and can be optimized for specific use cases
Complexity: Lease expiration logic involves complex date/time operations that are better expressed in SQL
Maintainability: Explicit queries make it easier to understand and debug performance issues

The system uses JdbcTemplate for raw SQL operations with a custom RowMapper to convert database results to domain objects.

Read vs. Write Separation

The system implements read-write separation using a CQRS-like approach:

public interface LeaseReadRepository {
    Optional<Lease> findById(UUID id);
    List<Lease> findByTenantId(UUID tenantId);
    List<Lease> findByPropertyId(UUID propertyId);
    List<Lease> findActiveLeases();
    List<Lease> findExpiringSoon(Duration window);
}

public interface LeaseWriteRepository {
    Lease save(Lease lease);
    void delete(UUID id);
    boolean existsById(UUID id);
}

Connection Pool Configuration

# application.yml
spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      validation-timeout: 5000
      leak-detection-threshold: 60000

N+1 Query Prevention Strategy

To prevent N+1 query problems, the system uses eager loading for related entities:

public class LeaseRowMapper implements RowMapper<Lease> {
    @Override
    public Lease mapRow(ResultSet rs, int rowNum) throws SQLException {
        UUID id = rs.getObject("id", UUID.class);
        UUID tenantId = rs.getObject("tenant_id", UUID.class);
        UUID propertyId = rs.getObject("property_id", UUID.class);
        
        // Eagerly load related entities to prevent N+1
        LeaseTerms terms = objectMapper.readValue(
            rs.getString("lease_terms"), 
            LeaseTerms.class
        );
        
        return Lease.builder()
            .id(id)
            .tenantId(tenantId)
            .propertyId(propertyId)
            .startDate(rs.getTimestamp("start_date").toLocalDateTime())
            .endDate(rs.getTimestamp("end_date").toLocalDateTime())
            .status(LeaseStatus.valueOf(rs.getString("status")))
            .leaseTerms(terms)
            .createdAt(rs.getTimestamp("created_at").toLocalDateTime())
            .updatedAt(rs.getTimestamp("updated_at").toLocalDateTime())
            .build();
    }
}

Batch Operation Design

For bulk operations like lease expiration processing:

@Repository
public class LeaseBatchRepository {
    private final JdbcTemplate jdbcTemplate;
    
    public void updateLeaseStatuses(List<UUID> leaseIds, LeaseStatus newStatus) {
        String sql = "UPDATE leases SET status = ?, updated_at = NOW() WHERE id IN (" +
            String.join(",", Collections.nCopies(leaseIds.size(), "?")) + ")";
        
        // Use batch update for better performance
        jdbcTemplate.batchUpdate(sql, new BatchPreparedStatementSetter() {
            @Override
            public void setValues(PreparedStatement ps, int i) throws SQLException {
                ps.setString(1, newStatus.name());
                ps.setObject(2, leaseIds.get(i));
            }
            
            @Override
            public int getBatchSize() {
                return leaseIds.size();
            }
        });
    }
}

Query Optimization

The system implements several query optimizations:

Index usage: All frequently queried fields are indexed
Query caching: Frequently executed queries use prepared statements
Partitioning: Large tables are partitioned by date ranges for better query performance

-- Indexes for optimal query performance
CREATE INDEX idx_leases_tenant_id ON leases(tenant_id);
CREATE INDEX idx_leases_property_id ON leases(property_id);
CREATE INDEX idx_leases_status_end_date ON leases(status, end_date);
CREATE INDEX idx_leases_created_at ON leases(created_at);

This repository design balances performance, maintainability, and scalability while supporting the core lease expiration functionality.

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The lease expiration system requires asynchronous processing for several key reasons:

Lease renewal notifications - When a lease is about to expire, we need to notify stakeholders (owners, tenants) without blocking the main transaction flow
Automated lease extension - For auto-renewable leases, we need to process renewal operations without user intervention
Grace period management - Handling grace periods and late payment notifications asynchronously to avoid blocking primary operations
Audit trail updates - Logging lease expiration events and status changes in a separate process to maintain system responsiveness

The system will use a hybrid synchronous/asynchronous approach:

Lease creation/update operations remain synchronous for immediate consistency
Expiration notifications and renewal processing are handled asynchronously via message queue
The core lease lifecycle management (creation, renewal, termination) stays synchronous to ensure data integrity

Message schema design with sample payload

{
  "version": "1.0",
  "eventId": "lease-expiration-1234567890",
  "eventType": "LEASE_EXPIRED",
  "timestamp": "2023-12-01T10:30:00Z",
  "payload": {
    "leaseId": "l-abc123def456",
    "propertyId": "p-xyz789uvw012",
    "tenantId": "t-tenant123",
    "ownerId": "o-owner456",
    "leaseStatus": "EXPIRED",
    "expirationDate": "2023-12-01T10:30:00Z",
    "gracePeriodEnd": "2023-12-08T10:30:00Z",
    "notificationType": "EXPIRATION_ALERT",
    "previousStatus": "ACTIVE"
  },
  "source": "lease-service",
  "correlationId": "corr-lease-expiration-123"
}

Dead letter queue strategy and poison message handling

The system implements a robust DLQ strategy with:

Dead Letter Queue (DLQ): Messages that fail more than 3 times are moved to a DLQ
Poison Message Detection: Messages that consistently fail with same error are flagged for manual intervention
Retry Configuration:
- Max retries: 3 attempts with exponential backoff (5s, 10s, 20s)
- Retry delay: 5s base with exponential growth and jitter (±1s)
Manual Intervention: DLQ messages are logged with detailed error context and require operator review

At-least-once vs. exactly-once delivery

At-least-once delivery is selected for this system because:

Idempotency is built into the system: Lease expiration events are designed to be idempotent - multiple deliveries of the same event won't cause duplicate actions
Business logic tolerance: Re-sending expiration notifications is acceptable and better than missing them entirely
Simplified implementation: Achieving exactly-once semantics would require complex distributed transaction handling and additional infrastructure overhead

The system ensures idempotency through:

Unique event IDs per lease expiration
State tracking in the system to prevent duplicate processing

Retry policy for consumers

public class LeaseEventRetryPolicy {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 5000;
    private static final double BACKOFF_MULTIPLIER = 2.0;
    private static final long MAX_DELAY_MS = 60000; // 1 minute
    private static final double JITTER_RANGE = 0.5; // ±50% jitter
    
    public RetryResult executeWithRetry(Supplier<CompletableFuture<Void>> operation) {
        int attempts = 0;
        Exception lastException = null;
        
        while (attempts < MAX_RETRIES) {
            try {
                operation.get().join();
                return RetryResult.SUCCESS;
            } catch (Exception e) {
                lastException = e;
                attempts++;
                
                if (attempts >= MAX_RETRIES) {
                    return RetryResult.FAILED_PERMANENTLY;
                }
                
                // Calculate exponential backoff with jitter
                long delay = calculateBackoffDelay(attempts);
                Thread.sleep(delay);
            }
        }
        
        return RetryResult.FAILED_PERMANENTLY;
    }
    
    private long calculateBackoffDelay(int attempt) {
        double delay = BASE_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempt - 1);
        delay = Math.min(delay, MAX_DELAY_MS);
        
        // Add jitter (±50%)
        double jitter = (Math.random() - 0.5) * JITTER_RANGE;
        delay = delay * (1 + jitter);
        
        return (long) delay;
    }
}

Outbox pattern for reliable event publishing

The outbox pattern is implemented in the lease expiration system:

public class LeaseEventOutbox {
    private final EventRepository eventRepository;
    private final MessagingService messagingService;
    
    public void publishLeaseEvent(LeaseEvent event) {
        // 1. Store event in outbox table
        LeaseEventEntity outboxEvent = new LeaseEventEntity(
            event.getEventId(),
            event.getEventType(),
            event.getPayload(),
            "PENDING"
        );
        
        eventRepository.save(outboxEvent);
        
        // 2. Publish to message queue (with transactional consistency)
        try {
            messagingService.publish(event);
            
            // 3. Mark as published
            outboxEvent.setStatus("PUBLISHED");
            eventRepository.save(outboxEvent);
        } catch (Exception e) {
            // 4. If publish fails, keep in PENDING state for retry
            outboxEvent.setStatus("FAILED");
            eventRepository.save(outboxEvent);
            throw new EventPublishingException("Failed to publish lease event", e);
        }
    }
}

Sequence diagram for async flows

sequenceDiagram
    participant LS as Lease Service
    participant DB as Database
    participant MQ as Message Queue
    participant Notif as Notification Service
    participant Audit as Audit Service
    
    LS->>DB: Create Lease (synchronous)
    LS->>DB: Update Lease Status to EXPIRED
    LS->>DB: Save Expiration Event (outbox pattern)
    DB-->>LS: Confirm Status Update
    LS->>MQ: Publish LEASE_EXPIRED event
    MQ-->>Notif: Deliver to Notification Service
    MQ-->>Audit: Deliver to Audit Service
    
    Note over Notif,LS: Async processing - no blocking
    Notif->>Notif: Send Email/SMS notifications
    Audit->>Audit: Log expiration event
    
    opt Retry Logic
        MQ-->>Notif: Delivery failure
        Notif->>MQ: Retry with exponential backoff
    end

10. Error Handling Design

Error Classification Hierarchy

The system implements a three-level error classification hierarchy:

Domain Errors - Business logic violations (e.g., lease not found, invalid renewal)
Infrastructure Errors - System-level issues (e.g., database connection failure, cache timeout)
Validation Errors - Input validation failures (e.g., malformed date, missing required field)

public abstract class BaseException extends RuntimeException {
    private final String errorCode;
    private final int httpStatus;

    public BaseException(String errorCode, int httpStatus, String message) {
        super(message);
        this.errorCode = errorCode;
        this.httpStatus = httpStatus;
    }

    public String getErrorCode() { return errorCode; }
    public int getHttpStatus() { return httpStatus; }
}

public class DomainException extends BaseException {
    public DomainException(String errorCode, String message) {
        super(errorCode, 400, message);
    }
}

public class InfrastructureException extends BaseException {
    public InfrastructureException(String errorCode, String message) {
        super(errorCode, 500, message);
    }
}

public class ValidationException extends BaseException {
    public ValidationException(String errorCode, String message) {
        super(errorCode, 400, message);
    }
}

Error Propagation Strategy

Errors are handled at their point of occurrence but propagated up the call stack when appropriate:

Validation errors are immediately thrown and handled at API boundary
Infrastructure errors are wrapped in domain exceptions or re-thrown as infrastructure errors
Business logic errors are wrapped and re-thrown to maintain context

Retry Strategy with Exponential Backoff and Jitter

Retry logic is implemented using exponential backoff with jitter to prevent thundering herd:

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_RANGE = 0.5;

    public <T> T executeWithRetry(Supplier<T> operation) {
        Exception lastException = null;
        
        for (int attempt = 0; attempt <= MAX_RETRIES; attempt++) {
            try {
                return operation.get();
            } catch (InfrastructureException e) {
                lastException = e;
                if (attempt >= MAX_RETRIES) break;
                
                long delay = calculateBackoffDelay(attempt);
                try {
                    Thread.sleep(delay);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new InfrastructureException("Retry interrupted", "RETRY_INTERRUPTED");
                }
            }
        }
        
        throw new InfrastructureException("Operation failed after retries", "OPERATION_FAILED");
    }

    private long calculateBackoffDelay(int attempt) {
        long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempt));
        double jitter = (Math.random() * 2 - 1) * JITTER_RANGE;
        return Math.max(100, delay + (long) (delay * jitter));
    }
}

Circuit Breaker Design

The circuit breaker pattern is implemented to prevent cascading failures:

public class CircuitBreaker {
    private final int failureThreshold;
    private final long timeoutMs;
    private final int successThreshold;
    
    private volatile CircuitState state = CircuitState.CLOSED;
    private volatile int failureCount = 0;
    private volatile long lastFailureTime = 0;

    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }

    public <T> T execute(Supplier<T> operation) {
        switch (state) {
            case CLOSED:
                return executeClosed(operation);
            case OPEN:
                return executeOpen(operation);
            case HALF_OPEN:
                return executeHalfOpen(operation);
            default:
                throw new IllegalStateException("Unknown circuit state");
        }
    }

    private <T> T executeClosed(Supplier<T> operation) {
        try {
            T result = operation.get();
            onSuccess();
            return result;
        } catch (Exception e) {
            onFailure(e);
            throw e;
        }
    }

    private <T> T executeOpen(Supplier<T> operation) {
        if (System.currentTimeMillis() - lastFailureTime > timeoutMs) {
            state = CircuitState.HALF_OPEN;
            return executeHalfOpen(operation);
        }
        throw new InfrastructureException("Circuit breaker is OPEN", "CIRCUIT_OPEN");
    }

    private <T> T executeHalfOpen(Supplier<T> operation) {
        try {
            T result = operation.get();
            state = CircuitState.CLOSED;
            failureCount = 0;
            return result;
        } catch (Exception e) {
            state = CircuitState.OPEN;
            lastFailureTime = System.currentTimeMillis();
            throw e;
        }
    }

    private void onSuccess() {
        if (state == CircuitState.HALF_OPEN) {
            failureCount = 0;
        }
    }

    private void onFailure(Exception e) {
        if (state == CircuitState.CLOSED) {
            failureCount++;
            lastFailureTime = System.currentTimeMillis();
            if (failureCount >= failureThreshold) {
                state = CircuitState.OPEN;
            }
        } else if (state == CircuitState.HALF_OPEN) {
            state = CircuitState.OPEN;
        }
    }
}

Fallback Behavior Design

Fallbacks are implemented for critical dependencies:

public class LeaseService {
    private final CircuitBreaker circuitBreaker = new CircuitBreaker(5, 30000, 3);
    private final Cache<String, Lease> cache = new LoadingCache<>(this::fetchLeaseFromDB);

    public Lease getLease(String leaseId) {
        try {
            return circuitBreaker.execute(() -> cache.get(leaseId));
        } catch (InfrastructureException e) {
            // Fallback to direct DB access with retry
            return fetchLeaseFromDB(leaseId);
        }
    }

    private Lease fetchLeaseFromDB(String leaseId) {
        // Implementation details
        return null;
    }
}

Error Logging Format and Alerting Thresholds

Structured logging is implemented with consistent error fields:

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);

    public void logError(String operation, Exception error, Map<String, Object> context) {
        Map<String, Object> logFields = new HashMap<>();
        logFields.put("timestamp", Instant.now().toString());
        logFields.put("operation", operation);
        logFields.put("error_code", getErrorCode(error));
        logFields.put("error_message", error.getMessage());
        logFields.put("stack_trace", getStackTrace(error));
        logFields.putAll(context);

        logger.error("Lease system error: {}", logFields);
    }

    private String getErrorCode(Exception error) {
        if (error instanceof BaseException) {
            return ((BaseException) error).getErrorCode();
        }
        return "UNKNOWN_ERROR";
    }

    private String getStackTrace(Exception error) {
        StringWriter sw = new StringWriter();
        PrintWriter pw = new PrintWriter(sw);
        error.printStackTrace(pw);
        return sw.toString();
    }
}

Alerting thresholds:

Error Type	Warning Threshold	Critical Threshold	Alert Channel
Database connection failures	5/min	20/min	Slack
Cache timeouts	10/min	50/min	PagerDuty
Lease expiration processing failures	2/min	10/min	Email + Slack

Error Response Format

Standardized JSON error response:

{
  "error": {
    "code": "LEASE_NOT_FOUND",
    "message": "Lease with ID '12345' was not found",
    "timestamp": "2023-10-15T14:30:00Z",
    "request_id": "req-abc-123"
  }
}

Error Handling in API Layer

@RestController
@RequestMapping("/leases")
public class LeaseController {
    
    @GetMapping("/{leaseId}")
    public ResponseEntity<LeaseResponse> getLease(
            @PathVariable String leaseId,
            HttpServletRequest request) {
        try {
            Lease lease = leaseService.getLease(leaseId);
            return ResponseEntity.ok(new LeaseResponse(lease));
        } catch (DomainException e) {
            ErrorLogger.logError("getLease", e, Map.of("lease_id", leaseId));
            return ResponseEntity.status(HttpStatus.NOT_FOUND)
                .body(new ErrorResponse(e.getErrorCode(), e.getMessage()));
        } catch (InfrastructureException e) {
            ErrorLogger.logError("getLease", e, Map.of("lease_id", leaseId));
            return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
                .body(new ErrorResponse("SYSTEM_ERROR", "Service temporarily unavailable"));
        }
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using a combination of:

Apache Commons Validator for standard validation (email, URL, phone number)
Custom regex patterns for domain-specific constraints (lease ID format, tenant ID)
Length and character set restrictions to prevent injection attacks
Type coercion with strict parsing to ensure data integrity

SQL Injection Prevention

Prepared statements are used exclusively for all database queries
Parameterized queries with named parameters (e.g., :leaseId, :tenantId)
No dynamic query construction from user input
Input sanitization for any field that might be used in query building

Authentication Middleware Design

JWT-based authentication with 2-hour expiration and refresh token rotation
OAuth2 integration for external identity providers (Azure AD, Google Workspace)
Authentication filter that validates JWT tokens and extracts user claims
API key authentication for service-to-service communication with HMAC signing

Authorization Enforcement

Role-Based Access Control (RBAC) with the following roles:
- LEASE_ADMIN: Full lease management permissions
- LEASE_MANAGER: Lease creation, update, and read access
- LEASE_VIEWER: Read-only lease access
Permission checking at service layer using a centralized authorization service
Resource-level permissions: Tenant-specific access control enforced via tenant ID in all queries

Sensitive Data Encryption

At-rest encryption: AES-256-GCM using AWS KMS for all sensitive fields (tenant contact info, lease terms)
In-transit encryption: TLS 1.3 for all API communications
Key rotation policy: 90-day automatic key rotation with seamless transition

PII Handling and Masking Strategy

PII fields: Tenant name, email, phone number, address
Masking policy:
- Email: j***@example.com
- Phone: +1 (555) ***-****
- Address: First two characters masked
Data classification: PII marked with metadata tags for audit and compliance

Secrets Management

AWS Secrets Manager for storing:
- Database credentials (rotated every 90 days)
- Third-party API keys (Stripe, Google Maps)
- JWT signing keys
Secrets rotation: Automated rotation with 7-day warning window
Access control: IAM policies restricting secret access to specific services only

Audit Logging Design

Log format includes:

{
  "timestamp": "2023-10-15T14:30:00Z",
  "trace_id": "abc123def456",
  "user_id": "user-123",
  "action": "LEASE_EXPIRED",
  "resource_type": "LEASE",
  "resource_id": "lease-456",
  "tenant_id": "tenant-789",
  "ip_address": "10.0.0.1",
  "user_agent": "lease-service/2.1.0"
}

Log retention: 90 days for compliance, 30 days for operational logs
Compliance tracking: Audit trail maintained for lease expiration events with full context of changes made

12. Testing Strategy

Unit Testing Scope and Coverage Targets

Target Coverage: 85% code coverage for business logic, 95% for critical paths and error handling. Unit tests will focus on:

Service layer business logic (90% of test suite)
Repository method boundary conditions
Value object validation and transformation logic
Error handling paths in service methods
Idempotency and concurrency edge cases

Test Structure:

// Example: Service layer unit test structure
@Test
void shouldExtendLeaseWhenValidRequest() {
    // Given
    Lease lease = createActiveLease();
    ExtendLeaseRequest request = new ExtendLeaseRequest("lease-123", 30);
    
    // When
    ExtendLeaseResponse response = leaseService.extendLease(request);
    
    // Then
    assertThat(response.getNewExpiry()).isEqualTo(expectedExpiry);
    verify(leaseRepository).save(any(Lease.class));
}

Integration Testing Strategy

Database Integration Tests:

Use testcontainers with PostgreSQL 13+ for real database integration
Test transaction boundaries and concurrent access patterns
Validate data consistency under stress scenarios
Mock external dependencies (payment gateway, notification service)

Test Data Management:

// Factory pattern for test data
public class LeaseTestDataFactory {
    public static Lease createActiveLease() {
        return Lease.builder()
            .id(UUID.randomUUID().toString())
            .status(LeaseStatus.ACTIVE)
            .startDate(Instant.now().minus(30, ChronoUnit.DAYS))
            .expiryDate(Instant.now().plus(30, ChronoUnit.DAYS))
            .tenantId("tenant-123")
            .propertyId("property-456")
            .build();
    }
}

Contract Testing

API Contract Tests:

Use Pact for consumer-driven contract testing between Lease service and dependent services
Define contracts for lease creation, extension, and expiration events
Validate message format consistency with external systems (e.g., property management system)

Performance Testing Tool and Scenarios

Tool: Gatling for load testing and performance characterization.

Key Performance Scenarios:

Peak Load: 10,000 concurrent lease extension requests per minute
High Concurrency: Simultaneous lease expiration checks for 10,000 active leases
Stress Test: 50,000 concurrent requests with 20% failure rate simulation
Latency Targets: p50 < 100ms, p95 < 500ms, p99 < 2000ms for core operations

Test Data Management Strategy

Fixtures and Factories:

Use factory pattern with configurable parameters for test data generation
Implement fixture-based setup for integration tests (e.g., 1000 active leases, 500 expired leases)
Seed data with realistic distribution patterns (lease durations, tenant types)

Test Data Isolation:

Use separate test database schemas per environment
Implement data cleanup strategy after each test suite run
Use UUIDs for test entities to prevent naming conflicts

Data Generation Strategy:

// Sample data generation for performance testing
public class LeasePerformanceTestData {
    public static List<Lease> generateActiveLeases(int count) {
        return IntStream.range(0, count)
            .mapToObj(i -> Lease.builder()
                .id(UUID.randomUUID().toString())
                .status(LeaseStatus.ACTIVE)
                .startDate(Instant.now().minus(30, ChronoUnit.DAYS))
                .expiryDate(Instant.now().plus(30, ChronoUnit.DAYS))
                .tenantId("tenant-" + i)
                .propertyId("property-" + (i % 100))
                .build())
            .collect(Collectors.toList());
    }
}

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective search, filtering, and analysis:

{
  "timestamp": "2023-10-27T14:30:00.000Z",
  "level": "INFO|WARN|ERROR",
  "service": "lease-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user-123",
  "entity_id": "lease-456",
  "operation": "lease.expiration.check",
  "message": "Checking lease expiration for entity",
  "error_code": "LEASE_EXPIRED",
  "error_details": {
    "message": "Lease has expired at 2023-10-26T14:30:00Z",
    "stack_trace": "..."
  },
  "duration_ms": 15,
  "tags": {
    "lease_status": "EXPIRED",
    "tenant_id": "tenant-789"
  }
}

PII masking rules:

User IDs and entity IDs are logged as hashed values when not required for debugging
Sensitive fields like lease terms or financial data are redacted in logs

Distributed Tracing Instrumentation

Using OpenTelemetry, we instrument key operations with custom spans:

Key Operations to Trace:

lease.expiration.check - Main lease expiration check operation
lease.update.status - Status update of expired leases
lease.notification.send - Sending expiration notifications

Span Attributes:

// Example custom span attributes for lease expiration check
span.setAttribute("lease.id", leaseId);
span.setAttribute("lease.status", currentStatus.name());
span.setAttribute("tenant.id", tenantId);
span.setAttribute("expected.expiry.time", expiryTime.toString());

Metrics Design

RED Method Metrics (Rate, Errors, Duration)

Metric Name	Type	Labels	Description
`lease_expiration_check_count`	Counter	`status`, `tenant_id`	Total lease checks
`lease_expiration_errors_total`	Counter	`error_type`, `tenant_id`	Lease expiration errors
`lease_expiration_duration_seconds`	Histogram	`status`, `tenant_id`	Duration of lease checks
`lease_status_updates_total`	Counter	`new_status`, `tenant_id`	Lease status updates
`lease_notifications_sent_total`	Counter	`notification_type`, `tenant_id`	Sent expiration notifications

Additional Gauges:

lease_active_count - Gauge, tenant_id - Active leases count
lease_expired_count - Gauge, tenant_id - Expired leases count

Health Check Endpoint Design

@RestController
public class HealthCheckController {
    
    @GetMapping("/health")
    public ResponseEntity<HealthStatus> health() {
        return ResponseEntity.ok(new HealthStatus(
            "healthy",
            System.currentTimeMillis(),
            Map.of("database", "healthy", "cache", "healthy")
        ));
    }
    
    @GetMapping("/ready")
    public ResponseEntity<ReadyStatus> ready() {
        // Check if service can accept traffic
        return ResponseEntity.ok(new ReadyStatus(
            "ready",
            System.currentTimeMillis(),
            Map.of("database", "connected", "cache", "available")
        ));
    }
    
    @GetMapping("/live")
    public ResponseEntity<LiveStatus> live() {
        // Basic liveness check
        return ResponseEntity.ok(new LiveStatus(
            "alive",
            System.currentTimeMillis(),
            "Service is running"
        ));
    }
}

Alerting Thresholds

Metric	Warning Level	Critical Level	Alerting Path
`lease_expiration_errors_total`	> 10 per minute	> 50 per minute	On-call SRE
`lease_expiration_duration_seconds`	> 500ms p95	> 2s p95	On-call SRE
`lease_expiration_check_count`	< 100 per minute	< 50 per minute	On-call SRE
`lease_status_updates_total`	> 1000 per minute	> 5000 per minute	On-call SRE
`lease_notifications_sent_total`	> 5000 per minute	> 10000 per minute	On-call SRE

Alerting Logic:

Use sliding window for error rate thresholds (5-minute window)
Implement auto-recovery alerting when service returns to normal
Include tenant-specific alerts for multi-tenant environments

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

Lease expiration vs. renewal behavior: Should leases automatically renew or require explicit action? This affects the system's operational complexity and user experience.
Grace period policy: How long should a grace period be before lease expiration is enforced? This impacts user expectations and system behavior during maintenance windows.
Notification strategy: What channels and timing should be used for lease expiration notifications? This affects user engagement and system load.
Recovery from lease expiration: Should system allow recovery of expired leases or require complete reinitialization? This impacts data consistency and user experience.

Known trade-offs and justifications

Real-time vs. batch lease expiration:
- Trade-off: Real-time processing (via async events) vs. batch processing (cron jobs)
- Justification: Real-time provides better user experience but increases system complexity and cost. Batch processing is more cost-effective but introduces latency in lease enforcement.
Eventual consistency vs. strong consistency:
- Trade-off: Immediate lease state updates vs. eventual consistency in distributed systems
- Justification: Eventual consistency allows better scalability but requires careful handling of race conditions and state synchronization.
Cache invalidation vs. cache freshness:
- Trade-off: Immediate cache invalidation vs. TTL-based cache refresh
- Justification: Immediate invalidation provides better consistency but increases cache miss rates and system load.

Risks and mitigation strategies

Race condition in lease renewal: Concurrent renewal requests may result in inconsistent state
- Risk: Multiple concurrent requests could cause lease extension to be lost or double-counted
- Mitigation: Use optimistic locking with version numbers and retry logic for lease renewal operations
System clock drift: Time synchronization issues across distributed systems
- Risk: Lease expiration times may be inconsistent across services due to clock drift
- Mitigation: Implement NTP synchronization and use monotonic time sources for internal operations
High load during lease expiration: Massive concurrent lease expirations during peak hours
- Risk: System overload when large numbers of leases expire simultaneously
- Mitigation: Implement rate limiting and staggered expiration processing with exponential backoff
Data loss in async processing: Failure to process lease expiration events
- Risk: Unprocessed lease expiration events leading to stale leases in system
- Mitigation: Implement dead letter queue with retry logic and manual intervention capability

Technical debt acknowledged

Monolithic lease management: Centralized lease handling may become a bottleneck as system scales
- Debt: Single point of failure and performance bottleneck for lease operations
- Future consideration: Consider sharding or distributed lease management patterns
Hardcoded expiration policies: Configuration-driven approaches not yet implemented
- Debt: Lease policies (duration, grace period) are hardcoded, requiring code changes for policy updates
- Future consideration: Implement configuration service or feature flags for lease policy management
Limited audit trail granularity: Current design lacks detailed event logging for compliance scenarios
- Debt: Basic audit logging insufficient for comprehensive compliance and forensic analysis
- Future consideration: Enhance audit logging with detailed event metadata and timeline capabilities
No rollback capability: Once a lease expires, there's no built-in mechanism to revert state
- Debt: System lacks ability to recover from incorrect lease expiration decisions
- Future consideration: Implement state versioning or snapshot capabilities for rollback scenarios