Skip to main content
Interview-Mapagam

Tenant Isolation Framework Design and Implementation

Comprehensive low-level design for a multi-tenant backend system with robust isolation, resource management, and security controls.

  • PublishedAugust 2, 2026
  • Length9,225 words
  • Reading time47 min read
Backend LLD

A tenant isolation framework ensures secure multi-tenant environments by providing isolated resource access, preventing data leakage, and maintaining strict boundaries between tenants in shared infrastructure.

1. Problem Statement & Requirements

Clarifying questions

  1. What scale expectations are we targeting? Are we designing for hundreds of tenants or tens of thousands?
  2. What consistency requirements does each tenant need? Do they require strong consistency or eventual consistency?
  3. What are the key entities and their relationships? Is this a multi-tenant SaaS platform or a shared infrastructure?
  4. What's the read/write ratio? Is this primarily write-heavy (e.g., real-time analytics) or read-heavy (e.g., content delivery)?
  5. What latency sensitivity do we have? Are sub-100ms SLAs required for all operations or can some be batched?
  6. What authentication model is in place? Is it OAuth2, JWT, or API keys with tenant context?
  7. Are there any external integrations or data sources we must consider? Do tenants have external dependencies (e.g., SSO, third-party APIs)?

Functional requirements

  • Tenant creation and lifecycle management - Support for creating, updating, and deleting tenants with proper validation.
  • Isolated data access - Ensure data of one tenant is not accessible by another tenant.
  • Tenant context propagation - All operations must be associated with a specific tenant context.
  • Multi-tenancy-aware queries - All database queries must filter by tenant ID to ensure data isolation.
  • Tenant-specific configuration and settings - Allow tenants to have custom configurations without affecting others.

Non-functional requirements

  • Latency targets:
    • p50: <50ms for read operations
    • p95: <100ms for read operations
    • p99: <200ms for read operations
  • Throughput: Support 10,000 concurrent requests across all tenants with no single tenant causing bottlenecks.
  • Availability: 99.9% uptime (5-minute SLA) with automatic failover capability.
  • Scalability: Support horizontal scaling across multiple database shards or instances.

Constraints and assumptions

  • Assumption: Tenant IDs are globally unique and stable across the system.
  • Constraint: No cross-tenant data sharing is allowed under any circumstances.
  • Assumption: All tenant data will be stored in a single database or database cluster for simplicity.
  • Constraint: Tenant context must be provided with every request (e.g., via header or JWT claim).
  • Assumption: The system will not support multi-tenancy across different regions or data centers without additional infrastructure.

Out of scope

  • Tenant migration - Moving tenant data between databases or clusters.
  • Cross-tenant analytics or reporting - Aggregating or analyzing data across tenants.
  • Tenant-to-tenant communication - Direct interaction between tenants.
  • Multi-region or cross-cloud deployments - Isolation is only within a single data center or cloud region.
  • Tenant-specific SLAs - All tenants are treated equally in terms of performance and availability.

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for the Tenant Isolation Framework due to:

  • Strong typing and schema validation at compile time
  • Efficient binary serialization for high-throughput scenarios
  • Native support for streaming and bidirectional communication
  • Built-in tooling for code generation across multiple languages
  • Better performance characteristics than REST for microservices communication

Endpoint Definitions

MethodPathDescriptionAuthRate Limit
CreateTenant/tenant.v1.TenantService/CreateTenantCreates a new tenant with configurationJWT + Admin Role10 req/min
GetTenant/tenant.v1.TenantService/GetTenantRetrieves tenant details by IDJWT + Tenant Access100 req/min
UpdateTenant/tenant.v1.TenantService/UpdateTenantUpdates tenant configurationJWT + Admin Role5 req/min
DeleteTenant/tenant.v1.TenantService/DeleteTenantSoft deletes tenant and marks for archivalJWT + Admin Role2 req/min
ListTenants/tenant.v1.TenantService/ListTenantsLists tenants with paginationJWT + Admin Role20 req/min
GetTenantStatus/tenant.v1.TenantService/GetTenantStatusGets tenant operational statusJWT + Tenant Access50 req/min

Request and Response Schemas

CreateTenantRequest

message CreateTenantRequest {
  string name = 1;
  string description = 2;
  map<string, string> metadata = 3;
  TenantConfig config = 4;
}

message TenantConfig {
  int32 max_users = 1;
  int32 max_apps = 2;
  string storage_quota_gb = 3;
  repeated string allowed_regions = 4;
}

CreateTenantResponse

message CreateTenantResponse {
  string tenant_id = 1;
  string name = 2;
  string status = 3; // ACTIVE, PENDING, SUSPENDED
  int64 created_at = 4;
}

GetTenantRequest

message GetTenantRequest {
  string tenant_id = 1;
}

GetTenantResponse

message GetTenantResponse {
  string tenant_id = 1;
  string name = 2;
  string description = 3;
  map<string, string> metadata = 4;
  TenantConfig config = 5;
  string status = 6;
  int64 created_at = 7;
  int64 updated_at = 8;
}

HTTP Status Codes and Error Response Format

Standard gRPC error codes are used:

  • OK (0) - Success
  • INVALID_ARGUMENT (3) - Malformed request
  • UNAUTHENTICATED (16) - Invalid authentication token
  • PERMISSION_DENIED (7) - Insufficient permissions
  • NOT_FOUND (5) - Tenant not found
  • ALREADY_EXISTS (6) - Tenant already exists
  • RESOURCE_EXHAUSTED (8) - Rate limit exceeded

Standardized error response:

message ErrorDetail {
  string code = 1;
  string message = 2;
  string details = 3;
  repeated string field_violations = 4;
}

API Versioning Strategy

Semantic versioning with gRPC service names:

  • tenant.v1.TenantService - Current stable version
  • Future: tenant.v2.TenantService for breaking changes

Pagination Design

Cursor-based pagination is used because:

  • More efficient than offset-based for large datasets
  • Consistent results even when data changes during pagination
  • Better performance characteristics with large offsets
message Pagination {
  string cursor = 1;
  int32 page_size = 2; // max 1000
}

Authentication and Authorization

Authentication:

  • JWT tokens with audience claim matching service name
  • Token validation via OAuth2 introspection endpoint

Authorization:

  • Admin role required for tenant creation/update/deletion
  • Tenant access token required for tenant-specific operations
  • RBAC model with role-based permissions:
    • TENANT_ADMIN - Full tenant control
    • TENANT_USER - Read-only access to own tenant data

Rate Limiting Per Endpoint

EndpointLimitEnforcement
CreateTenant10/minIn-memory counter with Redis fallback
GetTenant100/minToken bucket algorithm
UpdateTenant5/minFixed window with sliding counter
DeleteTenant2/minRate limiter with burst support
ListTenants20/minDistributed rate limiting via Redis

Idempotency Design

Idempotency key strategy:

  • X-Idempotency-Key header in requests
  • For create operations, same key returns existing resource if already created
  • Stored in Redis with TTL of 24 hours
// Example idempotency check logic in gRPC interceptor
public class IdempotencyInterceptor implements ServerInterceptor {
    private final RedisTemplate<String, String> redisTemplate;
    
    @Override
    public <ReqT, RespT> ServerCall.Listener<ReqT> interceptCall(
            ServerCall<ReqT, RespT> call, Metadata headers, 
            ServerCallHandler<ReqT, RespT> next) {
        
        String idempotencyKey = headers.get(Metadata.Key.of("X-Idempotency-Key", 
            Metadata.ASCII_STRING_MARSHALLER));
        
        if (idempotencyKey != null) {
            String existingResponse = redisTemplate.opsForValue().get(
                "idempotency:" + idempotencyKey);
            if (existingResponse != null) {
                // Return cached response
                return new ServerCall.Listener<ReqT>() {};
            }
        }
        
        return next.startCall(call, headers);
    }
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    TENANT ||--o{ USER : "has"
    TENANT ||--o{ ORGANIZATION : "owns"
    TENANT ||--o{ APPLICATION : "manages"
    USER ||--o{ USER_ROLE : "has"
    USER_ROLE ||--o{ ROLE : "is"
    ROLE ||--o{ PERMISSION : "grants"
    ORGANIZATION ||--o{ DEPARTMENT : "has"
    DEPARTMENT ||--o{ EMPLOYEE : "employs"
    APPLICATION ||--o{ FEATURE_FLAG : "uses"
    APPLICATION ||--o{ CONFIGURATION : "has"
    APPLICATION ||--o{ EVENT_LOG : "logs"
    
    TENANT {
        string id PK
        string name
        string description
        string status
        timestamp created_at
        timestamp updated_at
        timestamp deleted_at
    }
    
    USER {
        string id PK
        string tenant_id FK
        string email
        string first_name
        string last_name
        string status
        timestamp created_at
        timestamp updated_at
        timestamp deleted_at
    }
    
    ORGANIZATION {
        string id PK
        string tenant_id FK
        string name
        string description
        timestamp created_at
        timestamp updated_at
        timestamp deleted_at
    }
    
    DEPARTMENT {
        string id PK
        string organization_id FK
        string name
        timestamp created_at
        timestamp updated_at
        timestamp deleted_at
    }
    
    EMPLOYEE {
        string id PK
        string department_id FK
        string user_id FK
        timestamp created_at
        timestamp updated_at
    }
    
    APPLICATION {
        string id PK
        string tenant_id FK
        string name
        string description
        timestamp created_at
        timestamp updated_at
        timestamp deleted_at
    }
    
    FEATURE_FLAG {
        string id PK
        string application_id FK
        string key
        boolean enabled
        timestamp created_at
        timestamp updated_at
    }
    
    CONFIGURATION {
        string id PK
        string application_id FK
        string key
        string value
        timestamp created_at
        timestamp updated_at
    }
    
    EVENT_LOG {
        string id PK
        string application_id FK
        string event_type
        string payload
        timestamp created_at
    }
    
    USER_ROLE {
        string user_id FK
        string role_id FK
        timestamp created_at
    }
    
    ROLE {
        string id PK
        string name
        string description
        timestamp created_at
        timestamp updated_at
    }
    
    PERMISSION {
        string id PK
        string role_id FK
        string resource
        string action
        timestamp created_at
    }

Table Schema Definitions

Tenant Table

CREATE TABLE tenants (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL UNIQUE,
    description TEXT,
    status VARCHAR(50) NOT NULL DEFAULT 'active',
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    deleted_at TIMESTAMP WITH TIME ZONE
);

CREATE INDEX idx_tenants_status ON tenants(status);
CREATE INDEX idx_tenants_deleted_at ON tenants(deleted_at);

User Table

CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
    email VARCHAR(255) NOT NULL UNIQUE,
    first_name VARCHAR(100),
    last_name VARCHAR(100),
    status VARCHAR(50) NOT NULL DEFAULT 'active',
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    deleted_at TIMESTAMP WITH TIME ZONE,
    
    CONSTRAINT chk_email_format CHECK (email ~* '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$')
);

CREATE INDEX idx_users_tenant_id ON users(tenant_id);
CREATE INDEX idx_users_email ON users(email);
CREATE INDEX idx_users_status ON users(status);
CREATE INDEX idx_users_deleted_at ON users(deleted_at);

Organization Table

CREATE TABLE organizations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
    name VARCHAR(255) NOT NULL,
    description TEXT,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    deleted_at TIMESTAMP WITH TIME ZONE,
    
    CONSTRAINT chk_org_name_length CHECK (LENGTH(name) > 0)
);

CREATE INDEX idx_organizations_tenant_id ON organizations(tenant_id);
CREATE INDEX idx_organizations_deleted_at ON organizations(deleted_at);

Department Table

CREATE TABLE departments (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    name VARCHAR(255) NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    deleted_at TIMESTAMP WITH TIME ZONE,
    
    CONSTRAINT chk_dept_name_length CHECK (LENGTH(name) > 0)
);

CREATE INDEX idx_departments_organization_id ON departments(organization_id);
CREATE INDEX idx_departments_deleted_at ON departments(deleted_at);

Employee Table

CREATE TABLE employees (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    department_id UUID NOT NULL REFERENCES departments(id) ON DELETE CASCADE,
    user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    
    CONSTRAINT uk_employee_dept_user UNIQUE (department_id, user_id)
);

CREATE INDEX idx_employees_department_id ON employees(department_id);
CREATE INDEX idx_employees_user_id ON employees(user_id);

Application Table

CREATE TABLE applications (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
    name VARCHAR(255) NOT NULL,
    description TEXT,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    deleted_at TIMESTAMP WITH TIME ZONE,
    
    CONSTRAINT chk_app_name_length CHECK (LENGTH(name) > 0)
);

CREATE INDEX idx_applications_tenant_id ON applications(tenant_id);
CREATE INDEX idx_applications_deleted_at ON applications(deleted_at);

Feature Flag Table

CREATE TABLE feature_flags (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    application_id UUID NOT NULL REFERENCES applications(id) ON DELETE CASCADE,
    key VARCHAR(255) NOT NULL,
    enabled BOOLEAN NOT NULL DEFAULT FALSE,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    
    CONSTRAINT uk_feature_flag_app_key UNIQUE (application_id, key)
);

CREATE INDEX idx_feature_flags_application_id ON feature_flags(application_id);

Configuration Table

CREATE TABLE configurations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    application_id UUID NOT NULL REFERENCES applications(id) ON DELETE CASCADE,
    key VARCHAR(255) NOT NULL,
    value TEXT NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    
    CONSTRAINT uk_configuration_app_key UNIQUE (application_id, key)
);

CREATE INDEX idx_configurations_application_id ON configurations(application_id);

Event Log Table

CREATE TABLE event_logs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    application_id UUID NOT NULL REFERENCES applications(id) ON DELETE CASCADE,
    event_type VARCHAR(255) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_event_logs_application_id ON event_logs(application_id);
CREATE INDEX idx_event_logs_event_type ON event_logs(event_type);
CREATE INDEX idx_event_logs_created_at ON event_logs(created_at);

Role Table

CREATE TABLE roles (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL UNIQUE,
    description TEXT,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_roles_name ON roles(name);

Permission Table

CREATE TABLE permissions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    role_id UUID NOT NULL REFERENCES roles(id) ON DELETE CASCADE,
    resource VARCHAR(255) NOT NULL,
    action VARCHAR(50) NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    
    CONSTRAINT chk_permission_resource_action CHECK (LENGTH(resource) > 0 AND LENGTH(action) > 0)
);

CREATE INDEX idx_permissions_role_id ON permissions(role_id);

User Role Table

CREATE TABLE user_roles (
    user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    role_id UUID NOT NULL REFERENCES roles(id) ON DELETE CASCADE,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    
    CONSTRAINT pk_user_roles PRIMARY KEY (user_id, role_id)
);

CREATE INDEX idx_user_roles_role_id ON user_roles(role_id);

Primary Key Strategy

UUIDs are used as primary keys for all tables to ensure:

  1. Global uniqueness across distributed systems
  2. No sequential patterns that could expose system structure
  3. Better performance in distributed environments where auto-increment can cause contention
  4. Tenant isolation without requiring separate sequences per tenant

Index Design

Composite Indices

  1. idx_users_tenant_id_email - For fast user lookups by tenant and email
  2. idx_employees_department_user - For efficient employee-to-user lookups

Partial Indices

  1. idx_tenants_active - WHERE status = 'active' for tenant status filtering
  2. idx_users_active - WHERE status = 'active' AND deleted_at IS NULL for active user filtering

Covering Indices

  1. idx_applications_tenant_active - Covers tenant and status queries for applications
  2. idx_feature_flags_app_enabled - Covers application and enabled flag lookups

Data Partitioning / Sharding Strategy

Tenant-based sharding is implemented:

  1. Shard key: tenant_id in all tables
  2. Partitioning strategy:
    • Horizontal partitioning by tenant ID using consistent hashing or range-based sharding
    • Each tenant gets its own database shard or table space in multi-tenant DBs
  3. Cross-tenant queries: Prohibited by design - all queries must include tenant context

Soft Delete vs. Hard Delete Decision

Soft delete is used for all entities to:

  1. Maintain audit trail of deleted records
  2. Support data recovery in case of accidental deletions
  3. Compliance requirements for data retention policies
  4. Historical analysis capabilities

Audit Trail Design

All tables include audit fields:

  • created_at - Timestamp when record was created
  • updated_at - Timestamp when record was last modified
  • deleted_at - Timestamp when record was soft-deleted (if applicable)
  • created_by - User who created the record (not included in current design but could be added)
  • updated_by - User who last updated the record (not included in current design but could be added)

Data Archival Strategy and Retention Policy

  1. Archival trigger: Records with deleted_at older than 90 days are archived
  2. Archive storage: Separate schema or database for archived data with reduced indexing
  3. Retention policy: 365 days for compliance, 90 days for general cleanup
  4. Archival process: Scheduled batch job that moves records to archive tables with minimal metadata retention

4. Class Design & Service Layer

UML Class Diagram

classDiagram
    class TenantIsolationFramework {
        +isolateTenant(TenantContext)
        +validateTenantAccess(TenantContext, Resource)
        +getActiveTenant()
    }
    
    class TenantContext {
        -tenantId: String
        -userId: String
        -permissions: Set~String~
        +getTenantId()
        +getUserId()
        +hasPermission(String)
    }
    
    class TenantResource {
        -id: String
        -tenantId: String
        -createdAt: Instant
        -updatedAt: Instant
    }
    
    class TenantAwareRepository {
        <<Interface>>
        +findByTenantId(String)
        +save(TenantResource)
        +deleteById(String, String)
    }
    
    class TenantAwareService {
        <<Interface>>
        +createResource(TenantContext, Resource)
        +getResourceById(TenantContext, String)
        +updateResource(TenantContext, String, Resource)
        +deleteResource(TenantContext, String)
    }
    
    class DefaultTenantAwareService {
        -repository: TenantAwareRepository
        -eventPublisher: EventPublisher
        -auditLogger: AuditLogger
        +createResource(TenantContext, Resource)
        +getResourceById(TenantContext, String)
        +updateResource(TenantContext, String, Resource)
        +deleteResource(TenantContext, String)
    }
    
    class TenantIsolationInterceptor {
        -tenantValidator: TenantValidator
        +preHandle(HttpServletRequest, HttpServletResponse, HandlerMethod)
    }
    
    class TenantValidator {
        -tenantContextProvider: TenantContextProvider
        +validateTenantAccess(TenantContext, Resource)
    }
    
    class TenantContextProvider {
        <<Interface>>
        +getCurrentTenantContext()
    }
    
    class DefaultTenantContextProvider {
        -tenantHeaderExtractor: TenantHeaderExtractor
        +getCurrentTenantContext()
    }
    
    class TenantHeaderExtractor {
        -tenantIdHeader: String
        -userIdHeader: String
        +extractTenantContext(HttpServletRequest)
    }
    
    class AuditLogger {
        -auditEventPublisher: EventPublisher
        +logAuditEvent(AuditEvent)
    }
    
    class AuditEvent {
        -tenantId: String
        -userId: String
        -action: String
        -resourceType: String
        -resourceId: String
        -timestamp: Instant
    }
    
    class EventPublisher {
        <<Interface>>
        +publish(Event)
    }
    
    class DefaultEventPublisher {
        -eventBus: EventBus
        +publish(Event)
    }
    
    class Event {
        -type: String
        -payload: Object
        -timestamp: Instant
    }
    
    TenantIsolationFramework --> TenantContext : uses
    TenantIsolationFramework --> TenantValidator : uses
    TenantValidator --> TenantContextProvider : uses
    TenantContextProvider --> DefaultTenantContextProvider : implements
    DefaultTenantContextProvider --> TenantHeaderExtractor : uses
    DefaultTenantAwareService --> TenantAwareRepository : uses
    DefaultTenantAwareService --> EventPublisher : publishes
    DefaultTenantAwareService --> AuditLogger : logs
    TenantAwareService <|-- DefaultTenantAwareService : implements
    TenantContextProvider <|-- DefaultTenantContextProvider : implements
    TenantValidator --> TenantContext : validates
    AuditLogger --> EventPublisher : publishes
    EventPublisher <|-- DefaultEventPublisher : implements

State Machine Diagram for TenantResource

stateDiagram-v2
    [*] --> Active
    Active --> Deleted : deleteResource()
    Deleted --> Active : restoreResource()

Service Class Structure and Responsibilities

The service layer implements tenant isolation logic with clear separation of concerns:

  1. TenantAwareService - Interface defining the contract for tenant-aware operations
  2. DefaultTenantAwareService - Primary implementation handling business logic and tenant validation
  3. TenantIsolationFramework - Central coordinator managing isolation context and validation
  4. TenantValidator - Validates tenant access permissions for operations
  5. AuditLogger - Logs all tenant-accessible operations for compliance and monitoring

Interface / Abstract Class Design

public interface TenantAwareRepository<T extends TenantResource> {
    Optional<T> findById(String id, String tenantId);
    List<T> findByTenantId(String tenantId);
    T save(T resource);
    void deleteById(String id, String tenantId);
}

public interface TenantAwareService<T extends TenantResource> {
    T createResource(TenantContext context, T resource);
    Optional<T> getResourceById(TenantContext context, String id);
    T updateResource(TenantContext context, String id, T resource);
    void deleteResource(TenantContext context, String id);
}

public interface TenantContextProvider {
    TenantContext getCurrentTenantContext();
}

Business Logic Encapsulation Patterns

Strategy Pattern for tenant validation strategies:

public interface TenantValidationStrategy {
    boolean validate(TenantContext context, TenantResource resource);
}

public class RBACTenantValidationStrategy implements TenantValidationStrategy {
    private final RoleBasedAccessControl rbac;
    
    public RBACTenantValidationStrategy(RoleBasedAccessControl rbac) {
        this.rbac = rbac;
    }
    
    @Override
    public boolean validate(TenantContext context, TenantResource resource) {
        return rbac.hasPermission(context.getUserId(), resource.getTenantId(), "RESOURCE_ACCESS");
    }
}

public class TenantValidationStrategyFactory {
    public static TenantValidationStrategy createStrategy(String type) {
        return switch (type.toLowerCase()) {
            case "rbac" -> new RBACTenantValidationStrategy(new RoleBasedAccessControl());
            default -> new DefaultTenantValidationStrategy();
        };
    }
}

Critical-Path Method Implementation

@Transactional
public User createResource(TenantContext context, User user) {
    // Validate tenant access before proceeding
    if (!tenantValidator.validateTenantAccess(context, user)) {
        throw new TenantAccessDeniedException("User does not have access to tenant: " + context.getTenantId());
    }
    
    // Set tenant ID on resource
    user.setTenantId(context.getTenantId());
    
    // Ensure unique constraint across tenants (if applicable)
    String userKey = generateUserKey(user.getEmail(), context.getTenantId());
    if (userRepository.existsByKey(userKey)) {
        throw new DuplicateResourceException("User with email " + user.getEmail() + " already exists in tenant");
    }
    
    // Save to database
    User savedUser = userRepository.save(user);
    
    // Publish audit event
    AuditEvent auditEvent = new AuditEvent(
        context.getTenantId(),
        context.getUserId(),
        "CREATE_USER",
        "User",
        savedUser.getId(),
        Instant.now()
    );
    
    auditLogger.logAuditEvent(auditEvent);
    
    // Publish domain event for async processing
    UserCreatedEvent event = new UserCreatedEvent(
        savedUser.getId(),
        context.getTenantId(),
        savedUser.getEmail(),
        Instant.now()
    );
    
    eventPublisher.publish(event);
    
    return savedUser;
}

Transaction Boundary Design

Each tenant-aware operation is wrapped in a single database transaction:

  1. Create/Update/Delete operations - Single transaction per operation
  2. Cross-tenant operations - Split into separate transactions with proper error handling
  3. Event publishing - Part of same transaction to ensure consistency

Service Interface Contracts

public interface TenantAwareService<T extends TenantResource> {
    /**
     * Creates a new resource within the tenant scope
     * @param context Tenant context for isolation
     * @param resource Resource to create
     * @return Created resource with tenant context applied
     * @throws TenantAccessDeniedException when tenant access is not authorized
     * @throws DuplicateResourceException when resource already exists
     */
    T createResource(TenantContext context, T resource);
    
    /**
     * Retrieves a resource by ID within tenant scope
     * @param context Tenant context for isolation
     * @param id Resource identifier
     * @return Optional resource if found and accessible
     * @throws TenantAccessDeniedException when tenant access is not authorized
     */
    Optional<T> getResourceById(TenantContext context, String id);
    
    /**
     * Updates a resource within tenant scope
     * @param context Tenant context for isolation
     * @param id Resource identifier
     * @param resource Updated resource data
     * @return Updated resource
     * @throws TenantAccessDeniedException when tenant access is not authorized
     * @throws ResourceNotFoundException when resource doesn't exist
     */
    T updateResource(TenantContext context, String id, T resource);
    
    /**
     * Deletes a resource within tenant scope
     * @param context Tenant context for isolation
     * @param id Resource identifier to delete
     * @throws TenantAccessDeniedException when tenant access is not authorized
     * @throws ResourceNotFoundException when resource doesn't exist
     */
    void deleteResource(TenantContext context, String id);
}

Saga / Orchestration Pattern

For multi-step tenant operations, a saga pattern is implemented:

public class TenantResourceSaga {
    private final List<SagaStep> steps;
    
    public void execute(TenantContext context, String operationId) {
        for (SagaStep step : steps) {
            try {
                step.execute(context, operationId);
            } catch (Exception e) {
                // Compensate previous steps
                compensateSteps(steps, step);
                throw new SagaExecutionException("Saga failed at step: " + step.getName(), e);
            }
        }
    }
    
    private void compensateSteps(List<SagaStep> steps, SagaStep failedStep) {
        // Reverse the order and execute compensation logic
        for (int i = steps.indexOf(failedStep) - 1; i >= 0; i--) {
            steps.get(i).compensate();
        }
    }
}

Domain Object / Value Object Design

public class TenantContext {
    private final String tenantId;
    private final String userId;
    private final Set<String> permissions;
    
    public TenantContext(String tenantId, String userId, Set<String> permissions) {
        this.tenantId = Objects.requireNonNull(tenantId);
        this.userId = Objects.requireNonNull(userId);
        this.permissions = Set.copyOf(permissions);
    }
    
    public String getTenantId() {
        return tenantId;
    }
    
    public String getUserId() {
        return userId;
    }
    
    public boolean hasPermission(String permission) {
        return permissions.contains(permission);
    }
}

public abstract class TenantResource {
    protected String id;
    protected String tenantId;
    protected Instant createdAt;
    protected Instant updatedAt;
    
    // Getters and setters
}

Transaction Boundary Design

Each tenant-aware operation is wrapped in a single database transaction:

  1. Create/Update/Delete operations - Single transaction per operation
  2. Cross-tenant operations - Split into separate transactions with proper error handling
  3. Event publishing - Part of same transaction to ensure consistency

Service Interface Contracts

public interface TenantAwareService<T extends TenantResource> {
    /**
     * Creates a new resource within the tenant scope
     * @param context Tenant context for isolation
     * @param resource Resource to create
     * @return Created resource with tenant context applied
     * @throws TenantAccessDeniedException when tenant access is not authorized
     * @throws DuplicateResourceException when resource already exists
     */
    T createResource(TenantContext context, T resource);
    
    /**
     * Retrieves a resource by ID within tenant scope
     * @param context Tenant context for isolation
     * @param id Resource identifier
     * @return Optional resource if found and accessible
     * @throws TenantAccessDeniedException when tenant access is not authorized
     */
    Optional<T> getResourceById(TenantContext context, String id);
    
    /**
     * Updates a resource within tenant scope
     * @param context Tenant context for isolation
     * @param id Resource identifier
     * @param resource Updated resource data
     * @return Updated resource
     * @throws TenantAccessDeniedException when tenant access is not authorized
     * @throws ResourceNotFoundException when resource doesn't exist
     */
    T updateResource(TenantContext context, String id, T resource);
    
    /**
     * Deletes a resource within tenant scope
     * @param context Tenant context for isolation
     * @param id Resource identifier to delete
     * @throws TenantAccessDeniedException when tenant access is not authorized
     * @throws ResourceNotFoundException when resource doesn't exist
     */
    void deleteResource(TenantContext context, String id);
}

5. Concurrency & Thread Safety

Concurrency Model

The system uses a thread-per-request model with a thread pool for handling concurrent tenant requests. Each HTTP request is processed in its own thread, ensuring that tenant-specific operations don't block each other. For asynchronous processing (e.g., event handling), a separate thread pool is used with bounded queues to prevent resource exhaustion.

Race Condition Prevention Strategy

Race conditions are mitigated through:

  1. Tenant-aware locking: All operations that access shared resources use tenant-specific locks to prevent cross-tenant interference
  2. Database-level atomicity: Critical operations are wrapped in database transactions with proper isolation levels
  3. Version-based concurrency control: Optimistic locking using version fields to detect and handle concurrent updates
  4. Idempotency keys: For mutation operations, idempotency keys prevent duplicate processing in concurrent scenarios

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected for the following reasons:

  • High read-to-write ratio: Most operations are reads, so pessimistic locking would cause unnecessary contention
  • Tenant isolation: Each tenant's data is isolated, reducing conflict likelihood between tenants
  • Scalability: Optimistic locking scales better with concurrent tenants
  • Performance: Eliminates lock wait times for read-heavy workloads

Implementation approach:

// Example optimistic locking in a tenant-aware service
public class TenantAwareService {
    private final EntityManager entityManager;
    
    @Transactional
    public void updateTenantResource(String tenantId, Long resourceId, ResourceUpdateRequest request) {
        // Fetch resource with version
        Resource resource = entityManager.find(Resource.class, resourceId);
        
        if (resource == null) {
            throw new ResourceNotFoundException("Resource not found");
        }
        
        // Validate tenant ownership
        if (!resource.getTenantId().equals(tenantId)) {
            throw new TenantMismatchException("Resource does not belong to tenant");
        }
        
        // Validate version for optimistic locking
        if (resource.getVersion() != request.getExpectedVersion()) {
            throw new ConcurrencyException("Resource has been modified by another process");
        }
        
        // Apply updates
        resource.setName(request.getName());
        resource.setVersion(resource.getVersion() + 1);
        
        // Persist changes
        entityManager.merge(resource);
    }
}

Deadlock Prevention Strategy

Deadlock prevention is implemented through:

  1. Lock ordering: All tenant operations acquire locks in a consistent order (e.g., by tenant ID, then resource ID)
  2. Timeout configuration: Database transaction timeouts to prevent indefinite lock waits
  3. Lock hierarchy: Tenant-level locks are acquired before resource-level locks to avoid circular dependencies

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is enforced using a distributed idempotency key store (Redis or similar) with TTL:

@Component
public class IdempotencyService {
    private final RedisTemplate<String, String> redisTemplate;
    
    public boolean isIdempotent(String tenantId, String idempotencyKey) {
        String key = "idempotency:" + tenantId + ":" + idempotencyKey;
        String existing = redisTemplate.opsForValue().get(key);
        
        if (existing != null) {
            // Request was already processed
            return true;
        }
        
        // Store the key with TTL (e.g., 5 minutes)
        redisTemplate.opsForValue().set(key, "processed", 5, TimeUnit.MINUTES);
        return false;
    }
    
    public void processWithIdempotency(String tenantId, String idempotencyKey, 
                                     Supplier<Boolean> operation) {
        if (isIdempotent(tenantId, idempotencyKey)) {
            // Already processed - return success or cached result
            return;
        }
        
        try {
            operation.get();
        } catch (Exception e) {
            // If operation fails, remove the key so it can be retried
            String key = "idempotency:" + tenantId + ":" + idempotencyKey;
            redisTemplate.delete(key);
            throw e;
        }
    }
}

Thread Safety in Tenant Context

Tenant context is maintained thread-local to ensure proper isolation:

public class TenantContext {
    private static final ThreadLocal<String> currentTenant = new ThreadLocal<>();
    
    public static void setCurrentTenant(String tenantId) {
        currentTenant.set(tenantId);
    }
    
    public static String getCurrentTenant() {
        return currentTenant.get();
    }
    
    public static void clear() {
        currentTenant.remove();
    }
}

Concurrency Control for Multi-Tenant Operations

When operations span multiple tenants, the system uses:

  1. Tenant-aware transaction boundaries: Each tenant's operations are isolated within their own transaction scope
  2. Batch processing with tenant grouping: Operations are batched by tenant to minimize cross-tenant contention
  3. Async processing with tenant queues: Tenant-specific message queues ensure proper ordering and isolation

Performance Considerations

  • Lock contention monitoring: Track lock wait times to identify potential bottlenecks
  • Tenant-specific thread pools: Separate thread pools per tenant for critical operations to prevent one tenant from starving others
  • Connection pool sizing: Configure database connection pools with tenant-aware sizing to prevent resource exhaustion

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant Client
    participant AuthN
    participant TenantMiddleware
    participant Service
    participant DB

    Client->>AuthN: Request with tenant header (X-Tenant-ID)
    AuthN-->>Client: JWT token with tenant info
    Client->>TenantMiddleware: Request with JWT token
    TenantMiddleware->>Service: Forward request with tenant context
    Service->>DB: Query with tenant filter (tenant_id = 'tenant-123')
    DB-->>Service: Query results
    Service-->>Client: Response with data for tenant-123
sequenceDiagram
    participant Client
    participant AuthN
    participant TenantMiddleware
    participant Service
    participant DB

    Client->>AuthN: Request with invalid tenant ID (X-Tenant-ID: 'tenant-999')
    AuthN-->>Client: JWT token with tenant info
    Client->>TenantMiddleware: Request with JWT token and invalid tenant ID
    TenantMiddleware->>Service: Forward request with tenant context
    Service->>DB: Query with tenant filter (tenant_id = 'tenant-999')
    DB-->>Service: Empty results
    Service-->>Client: 404 Not Found - Tenant does not exist or no data
sequenceDiagram
    participant Client
    participant AuthN
    participant TenantMiddleware
    participant Service
    participant DB
    participant EventPublisher

    Client->>AuthN: Create tenant request (X-Tenant-ID: 'tenant-123')
    AuthN-->>Client: JWT token with tenant info
    Client->>TenantMiddleware: Request with JWT token and create tenant payload
    TenantMiddleware->>Service: Forward request with tenant context
    Service->>DB: Insert new tenant record (tenant_id = 'tenant-123')
    Service->>EventPublisher: Publish tenant.created event
    EventPublisher-->>Service: Acknowledgement
    Service-->>Client: 201 Created - Tenant created successfully

7. Caching Design

Cache Strategy Selection

For tenant isolation, we'll use cache-aside pattern with tenant-aware cache keys. This approach provides the most control over cache invalidation and allows for fine-grained tenant-level cache management.

Cache Key Design

Cache keys will follow the namespace pattern:

{cacheNamespace}:{tenantId}:{entityType}:{entityId}

Example:

user:12345:profile:67890
booking:12345:seat:54321

TTL Strategy

Data TypeTTLReason
User Profile5 minutesFrequent updates, low cache hit ratio
Booking Details1 hourHigh consistency requirements
Seat Availability2 minutesReal-time updates needed
Tenant Metadata1 dayInfrequently changing
Session Tokens30 minutesSecurity sensitive

Cache Eviction Policy

We'll use LRU (Least Recently Used) eviction policy with TTL-based soft expiration to handle cache stampede prevention. Cache entries will be marked as "expired" but remain accessible for a grace period to prevent thundering herd.

Cache Stampede Prevention

  1. Probabilistic Early Expiry: 20% of TTL is used for early refresh
  2. Distributed Locks: For high-traffic entities, use Redis distributed locks to serialize cache rebuilds
  3. Cache Entry Versioning: Each cache entry includes a version number that's incremented on data changes

Distributed Cache Architecture

We'll use Redis Cluster with:

  • 3 master nodes for data distribution
  • 3 replica nodes per master for high availability
  • Redis Sentinel for automatic failover
  • Cache sharding based on tenant ID hash

Cache Invalidation Strategy

Event-driven invalidation: Cache entries are invalidated via events published to a message broker when data changes. This ensures immediate consistency across tenants.

Cache warming: On application startup, warm tenant caches for frequently accessed entities using batch operations.

Cache Implementation Example

@Component
public class TenantAwareCache {
    private final RedisTemplate<String, Object> redisTemplate;
    private final ObjectMapper objectMapper;
    
    public <T> T get(String tenantId, String entityType, String entityId, 
                     Class<T> type, Supplier<T> loader) {
        String cacheKey = buildCacheKey(tenantId, entityType, entityId);
        
        // Try to get from cache
        String cachedValue = redisTemplate.opsForValue().get(cacheKey);
        if (cachedValue != null) {
            try {
                return objectMapper.readValue(cachedValue, type);
            } catch (Exception e) {
                // Log and proceed to reload
                log.warn("Failed to deserialize cache value for key: {}", cacheKey);
            }
        }
        
        // Cache miss - load and store
        T value = loader.get();
        if (value != null) {
            String serializedValue = objectMapper.writeValueAsString(value);
            redisTemplate.opsForValue().set(cacheKey, serializedValue, 
                                          Duration.ofMinutes(5));
        }
        
        return value;
    }
    
    public void invalidate(String tenantId, String entityType, String entityId) {
        String cacheKey = buildCacheKey(tenantId, entityType, entityId);
        redisTemplate.delete(cacheKey);
    }
    
    private String buildCacheKey(String tenantId, String entityType, String entityId) {
        return String.format("%s:%s:%s:%s", 
                           "tenant", tenantId, entityType, entityId);
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and persistence concerns. Each entity has its own repository interface with standardized CRUD operations.

public interface UserRepository {
    User findById(UUID id);
    User findByEmail(String email);
    List<User> findAllByTenantId(UUID tenantId);
    User save(User user);
    void delete(UUID id);
}

public interface TenantRepository {
    Tenant findById(UUID id);
    Tenant findByDomain(String domain);
    List<Tenant> findAll();
    Tenant save(Tenant tenant);
}

ORM vs Raw Query Decisions

For the core tenant-isolation framework, raw SQL queries are preferred over ORM for:

  • Fine-grained control over tenant filtering
  • Performance optimization for high-throughput operations
  • Explicit handling of tenant context in queries
@Repository
public class UserRepositoryImpl implements UserRepository {
    
    private final JdbcTemplate jdbcTemplate;
    private final TenantContext tenantContext;
    
    public UserRepositoryImpl(JdbcTemplate jdbcTemplate, TenantContext tenantContext) {
        this.jdbcTemplate = jdbcTemplate;
        this.tenantContext = tenantContext;
    }
    
    @Override
    public User findById(UUID id) {
        String sql = "SELECT * FROM users WHERE id = ? AND tenant_id = ?";
        return jdbcTemplate.queryForObject(sql, new Object[]{id, tenantContext.getCurrentTenantId()}, userRowMapper);
    }
    
    @Override
    public List<User> findAllByTenantId(UUID tenantId) {
        String sql = "SELECT * FROM users WHERE tenant_id = ? ORDER BY created_at DESC";
        return jdbcTemplate.query(sql, new Object[]{tenantId}, userRowMapper);
    }
    
    @Override
    public User save(User user) {
        if (user.getId() == null) {
            String sql = "INSERT INTO users (id, email, name, tenant_id, created_at) VALUES (?, ?, ?, ?, ?)";
            UUID id = UUID.randomUUID();
            jdbcTemplate.update(sql, id, user.getEmail(), user.getName(), 
                              tenantContext.getCurrentTenantId(), LocalDateTime.now());
            return findById(id);
        } else {
            String sql = "UPDATE users SET email = ?, name = ? WHERE id = ? AND tenant_id = ?";
            jdbcTemplate.update(sql, user.getEmail(), user.getName(), 
                              user.getId(), tenantContext.getCurrentTenantId());
            return findById(user.getId());
        }
    }
}

Read vs Write Separation

The CQRS pattern is implemented to separate read and write operations:

public interface UserReadRepository {
    User findByEmail(String email);
    List<User> findAllByTenantId(UUID tenantId);
    long countByTenantId(UUID tenantId);
}

public interface UserWriteRepository {
    User save(User user);
    void delete(UUID id);
    void updateLastLogin(UUID userId);
}

Connection Pool Configuration

Connection pool is configured with:

  • Max pool size: 20
  • Min idle: 5
  • Connection timeout: 30 seconds
  • Validation query: SELECT 1
@Configuration
public class DatabaseConfig {
    
    @Bean
    public DataSource dataSource() {
        HikariConfig config = new HikariConfig();
        config.setJdbcUrl("jdbc:postgresql://localhost:5432/tenantdb");
        config.setUsername("user");
        config.setPassword("password");
        config.setMaximumPoolSize(20);
        config.setMinimumIdle(5);
        config.setConnectionTimeout(30000);
        config.setValidationTimeout(5000);
        config.setLeakDetectionThreshold(60000);
        return new HikariDataSource(config);
    }
}

N+1 Query Prevention Strategy

Eager loading is implemented for related entities, and batch operations are used where appropriate:

@Repository
public class TenantRepositoryImpl implements TenantRepository {
    
    private final JdbcTemplate jdbcTemplate;
    
    @Override
    public List<Tenant> findAllWithUsers() {
        String sql = "SELECT t.id as tenant_id, t.name as tenant_name, u.id as user_id, u.email as user_email " +
                    "FROM tenants t LEFT JOIN users u ON t.id = u.tenant_id";
        
        return jdbcTemplate.query(sql, (rs, rowNum) -> {
            Tenant tenant = new Tenant();
            tenant.setId(UUID.fromString(rs.getString("tenant_id")));
            tenant.setName(rs.getString("tenant_name"));
            
            // Handle multiple users per tenant
            if (rs.getObject("user_id") != null) {
                User user = new User();
                user.setId(UUID.fromString(rs.getString("user_id")));
                user.setEmail(rs.getString("user_email"));
                tenant.addUser(user);
            }
            
            return tenant;
        });
    }
    
    public List<User> findUsersByTenantIds(List<UUID> tenantIds) {
        String placeholders = String.join(",", Collections.nCopies(tenantIds.size(), "?"));
        String sql = "SELECT * FROM users WHERE tenant_id IN (" + placeholders + ")";
        
        return jdbcTemplate.query(sql, tenantIds.toArray(), userRowMapper);
    }
}

Batch Operation Design

Batch operations are implemented for bulk data processing:

@Repository
public class UserBatchRepository {
    
    private final JdbcTemplate jdbcTemplate;
    
    public void batchCreateUsers(List<User> users) {
        String sql = "INSERT INTO users (id, email, name, tenant_id, created_at) VALUES (?, ?, ?, ?, ?)";
        
        List<Object[]> batchArgs = users.stream()
            .map(user -> new Object[]{
                user.getId(),
                user.getEmail(),
                user.getName(),
                tenantContext.getCurrentTenantId(),
                LocalDateTime.now()
            })
            .collect(Collectors.toList());
            
        jdbcTemplate.batchUpdate(sql, batchArgs);
    }
    
    public void batchUpdateUserEmails(List<UUID> userIds, String newEmail) {
        String sql = "UPDATE users SET email = ? WHERE id IN (" + 
            String.join(",", Collections.nCopies(userIds.size(), "?")) + ")";
        
        List<Object[]> batchArgs = userIds.stream()
            .map(id -> new Object[]{newEmail, id})
            .collect(Collectors.toList());
            
        jdbcTemplate.batchUpdate(sql, batchArgs);
    }
}

Transaction Management

Transaction boundaries are carefully defined to ensure tenant isolation:

@Service
@Transactional
public class UserService {
    
    private final UserWriteRepository userWriteRepository;
    private final TenantContext tenantContext;
    
    public User createUser(User user) {
        // Transaction boundary includes tenant context validation
        validateTenantAccess();
        
        User savedUser = userWriteRepository.save(user);
        
        // Publish event within same transaction
        eventPublisher.publish(new UserCreatedEvent(savedUser));
        
        return savedUser;
    }
    
    private void validateTenantAccess() {
        if (tenantContext.getCurrentTenantId() == null) {
            throw new TenantNotFoundException("No tenant context found");
        }
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature — what triggers async vs. what stays synchronous, and why

In a tenant isolation framework, the primary use case for asynchronous processing is around tenant lifecycle operations and audit logging, where immediate response to the client isn't required but eventual consistency is acceptable. For example:

  • Tenant creation: The system needs to provision resources (databases, storage buckets, etc.) which can take time. These operations should be async to avoid blocking the API response.
  • Tenant deletion: Cleanup of resources across multiple systems (databases, storage, etc.) should be handled asynchronously to ensure the operation doesn't timeout or fail due to external dependencies.
  • Audit logging: All tenant operations should be logged for compliance and monitoring purposes, but these logs can be published asynchronously to avoid impacting the performance of the primary operation.

Synchronous operations include:

  • Tenant lookup: A direct query to find a tenant by ID or name, which must return immediately.
  • Tenant update: Changes like metadata updates or role assignments should be synchronous to ensure immediate consistency.

Message schema design with a sample payload (Avro / Protobuf / JSON Schema — with versioning strategy)

We use JSON Schema for simplicity and readability, but in production, Avro or Protocol Buffers would be preferred for strong typing and schema evolution.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "TenantEvent",
  "type": "object",
  "version": "1.0",
  "properties": {
    "eventId": {
      "type": "string",
      "description": "Unique identifier for the event"
    },
    "tenantId": {
      "type": "string",
      "description": "Tenant identifier"
    },
    "eventType": {
      "type": "string",
      "enum": ["TENANT_CREATED", "TENANT_DELETED", "TENANT_UPDATED"]
    },
    "timestamp": {
      "type": "string",
      "format": "date-time"
    },
    "data": {
      "type": "object",
      "description": "Event-specific data"
    },
    "source": {
      "type": "string",
      "description": "Service that published the event"
    }
  },
  "required": ["eventId", "tenantId", "eventType", "timestamp", "source"]
}

Versioning strategy:

  • Semantic versioning (MAJOR.MINOR.PATCH) for breaking changes, feature additions, and bug fixes.
  • Schema registry to manage versions and enforce compatibility.
  • Backward compatibility is maintained for minor version changes.

Dead letter queue strategy and poison message handling

Poison messages are events that repeatedly fail to be processed. Strategy:

  • DLQ topic: Each event type has a corresponding DLQ topic (e.g., tenant.created.dlq).
  • Retry logic: Events are retried up to 3 times with exponential backoff (1s, 2s, 4s).
  • Dead letter handling: If retries exceed the limit, messages are sent to DLQ.
  • Manual intervention: DLQ messages are logged with metadata (last error, retry count) and can be manually reprocessed by an operator.

At-least-once vs. exactly-once delivery — decision with justification for this feature

We choose at-least-once delivery because:

  • Tenant operations are generally idempotent (e.g., creating a tenant twice is safe).
  • The system uses idempotency keys for deduplication.
  • Eventual consistency is acceptable for audit logging and resource provisioning.

Retry policy for consumers (max attempts, backoff formula, non-retryable error classification)

Retry policy:

  • Max retries: 3 attempts
  • Backoff formula: delay = baseDelay * (2^retryCount) + random(0, 1000)
  • Non-retryable errors: 400 Bad Request, 401 Unauthorized, 403 Forbidden, 500 Internal Server Error (if the error is not transient)

Outbox pattern for reliable event publishing — include or explicitly justify omission

We do not use the outbox pattern in this design because:

  • The system assumes that event publishing is part of a single transaction (e.g., within the same database).
  • If we were to use an external message broker, the outbox pattern would be necessary to ensure that events are published only after the database transaction is committed.

Sequence diagram for all async flows (Mermaid)

sequenceDiagram
    participant Client
    participant APIGateway
    participant TenantService
    participant EventPublisher
    participant MessageBroker

    Client->>APIGateway: POST /tenants
    APIGateway->>TenantService: CreateTenantRequest
    TenantService->>TenantService: Validate and persist tenant
    TenantService->>EventPublisher: Publish TENANT_CREATED event
    EventPublisher->>MessageBroker: Send to tenant.created topic
    MessageBroker-->>EventPublisher: Acknowledge
    TenantService-->>APIGateway: 201 Created
    APIGateway-->>Client: Tenant created successfully

10. Error Handling Design

Error Classification Hierarchy

The error handling framework follows a hierarchical classification to enable proper routing and response generation:

  1. Domain Errors - Business logic violations (e.g., TenantNotFound, InsufficientPermissions)
  2. Validation Errors - Input validation failures (e.g., InvalidTenantId, MissingRequiredField)
  3. Infrastructure Errors - System-level issues (e.g., DatabaseConnectionFailed, CacheTimeout)
  4. External Service Errors - Third-party integration failures (e.g., PaymentGatewayTimeout, AuthServiceUnavailable)

Error Propagation Strategy

Errors are handled at their point of occurrence and propagated upward through the service layer. The framework follows a "bubble up" strategy for domain and validation errors, while infrastructure errors are caught at the service boundary and wrapped with appropriate context.

public class ErrorHandlingStrategy {
    public void handleDomainError(DomainException ex) {
        // Log with context and rethrow for higher-level handling
        logger.error("Domain error occurred: {}", ex.getMessage(), ex);
        throw new ServiceException("Business logic error", ex);
    }
    
    public void handleInfrastructureError(InfrastructureException ex) {
        // Log and wrap with service-level context
        logger.error("Infrastructure error: {}", ex.getMessage(), ex);
        throw new ServiceUnavailableException("Service temporarily unavailable", ex);
    }
}

Retry Strategy with Exponential Backoff and Jitter

For transient infrastructure errors, a retry mechanism is implemented using exponential backoff with jitter to prevent thundering herd:

public class RetryStrategy {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;

    public <T> T executeWithRetry(Supplier<T> operation) {
        int attempts = 0;
        while (attempts < MAX_RETRIES) {
            try {
                return operation.get();
            } catch (TransientException e) {
                if (attempts == MAX_RETRIES - 1) throw e;
                
                long delay = calculateDelay(attempts);
                logger.warn("Transient error on attempt {}: retrying in {}ms", 
                           attempts + 1, delay, e);
                Thread.sleep(delay);
                attempts++;
            }
        }
        throw new RuntimeException("Retry limit exceeded");
    }

    private long calculateDelay(int attempt) {
        long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempt));
        long jitter = (long) (delay * JITTER_FACTOR * (Math.random() - 0.5) * 2);
        return delay + jitter;
    }
}

Circuit Breaker Design

For external service calls, a circuit breaker pattern is implemented to prevent cascading failures:

public class CircuitBreaker {
    private final int failureThreshold = 5;
    private final long timeoutMs = 30000; // 30 seconds
    private final int resetTimeoutMs = 60000; // 1 minute
    
    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private long lastFailureTime = 0;

    public <T> T execute(Supplier<T> operation) {
        switch (state) {
            case CLOSED:
                try {
                    return operation.get();
                } catch (Exception e) {
                    handleFailure();
                    throw e;
                }
            case OPEN:
                if (System.currentTimeMillis() - lastFailureTime > resetTimeoutMs) {
                    state = CircuitState.HALF_OPEN;
                    return operation.get();
                }
                throw new CircuitBreakerOpenException("Circuit breaker is open");
            case HALF_OPEN:
                try {
                    T result = operation.get();
                    state = CircuitState.CLOSED;
                    failureCount = 0;
                    return result;
                } catch (Exception e) {
                    state = CircuitState.OPEN;
                    throw e;
                }
        }
        throw new IllegalStateException("Invalid circuit state");
    }

    private void handleFailure() {
        failureCount++;
        lastFailureTime = System.currentTimeMillis();
        
        if (failureCount >= failureThreshold) {
            state = CircuitState.OPEN;
        }
    }
}

Fallback Behavior Design

Fallbacks are implemented for critical external dependencies:

public class FallbackHandler {
    public List<Tenant> getTenantsWithFallback(String userId) {
        try {
            return tenantRepository.findAll();
        } catch (DatabaseException e) {
            logger.warn("Database unavailable, using cached tenants");
            return getCachedTenants();
        }
    }

    public Tenant getTenantWithFallback(String tenantId) {
        try {
            return tenantRepository.findById(tenantId);
        } catch (DatabaseException e) {
            logger.warn("Database unavailable, using default tenant");
            return createDefaultTenant();
        }
    }
}

Error Logging Format and Alerting Thresholds

Structured logging includes trace context and error classification:

public class ErrorLogger {
    private static final ObjectMapper objectMapper = new ObjectMapper();
    
    public void logError(String operation, String userId, String tenantId, 
                        Exception exception, ErrorType errorType) {
        Map<String, Object> logEntry = new HashMap<>();
        logEntry.put("timestamp", Instant.now());
        logEntry.put("operation", operation);
        logEntry.put("userId", userId);
        logEntry.put("tenantId", tenantId);
        logEntry.put("errorType", errorType.name());
        logEntry.put("errorCode", getErrorCode(exception));
        logEntry.put("errorMessage", exception.getMessage());
        logEntry.put("stackTrace", getStackTrace(exception));
        
        logger.error("{}", objectMapper.writeValueAsString(logEntry));
    }
    
    private String getErrorCode(Exception ex) {
        if (ex instanceof TenantNotFoundException) return "TENANT_NOT_FOUND";
        if (ex instanceof InsufficientPermissionsException) return "INSUFFICIENT_PERMISSIONS";
        return "INTERNAL_ERROR";
    }
}

Alerting Thresholds

Critical error thresholds trigger alerts:

Error TypeAlert ThresholdSeverity
Database Connection Failures> 5 errors in 5 minutesCRITICAL
Authentication Failures> 100 errors in 1 hourWARNING
Tenant Isolation Violations> 1 occurrenceCRITICAL
Cache Timeout Errors> 5 errors in 10 minutesWARNING

Error Response Format

Standardized JSON error responses:

{
  "error": {
    "code": "TENANT_NOT_FOUND",
    "message": "Tenant with ID 'tenant-123' was not found",
    "details": {
      "tenantId": "tenant-123",
      "timestamp": "2023-10-15T14:30:00Z"
    },
    "httpStatus": 404
  }
}

Error Handling in Service Layer

@Service
public class TenantService {
    public Tenant getTenant(String tenantId, String userId) {
        try {
            // Validate input
            if (tenantId == null || tenantId.trim().isEmpty()) {
                throw new ValidationException("Tenant ID is required");
            }
            
            // Check tenant isolation
            if (!tenantAuthorizationService.isAuthorized(userId, tenantId)) {
                throw new InsufficientPermissionsException(
                    "User does not have access to tenant: " + tenantId);
            }
            
            // Fetch tenant with retry logic
            return retryStrategy.executeWithRetry(() -> {
                Tenant tenant = tenantRepository.findById(tenantId);
                if (tenant == null) {
                    throw new TenantNotFoundException("Tenant not found: " + tenantId);
                }
                return tenant;
            });
            
        } catch (TenantNotFoundException e) {
            logger.warn("Tenant not found: {}", tenantId, e);
            throw new TenantNotFoundException("Tenant not found", e);
        } catch (InsufficientPermissionsException e) {
            logger.warn("Permission denied for user {} accessing tenant {}", userId, tenantId, e);
            throw new InsufficientPermissionsException("Access denied", e);
        } catch (DatabaseException e) {
            logger.error("Database error while fetching tenant: {}", tenantId, e);
            throw new ServiceUnavailableException("Service temporarily unavailable", e);
        }
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using a combination of:

  • Custom validation annotations (e.g., @NotBlank, @Size) for basic constraints
  • Apache Commons Validator for complex patterns (email, phone numbers, UUIDs)
  • OWASP Java HTML Sanitizer for any user-provided content that gets rendered
  • Custom regex patterns for domain-specific constraints (e.g., tenant ID format)

SQL Injection Prevention

  • Prepared statements used exclusively for all database queries
  • Parameterized queries with no string concatenation in SQL generation
  • ORM framework (Hibernate) with strict mapping configuration to prevent dynamic query building
  • Query whitelisting for any dynamic SQL components (e.g., column names in ORDER BY clauses)

Authentication Middleware Design

  • JWT-based authentication with RS256 signing algorithm using asymmetric keys
  • Authentication filter intercepts all requests and validates JWT tokens
  • Tenant context extraction from JWT claims to establish current tenant scope
  • Token refresh mechanism with sliding window (30-minute TTL, 15-minute refresh window)
  • Token revocation via Redis-based blacklisting for immediate logout scenarios

Authorization Enforcement

  • Role-Based Access Control (RBAC) model with:
    • TENANT_ADMIN - Full access to tenant resources
    • TENANT_USER - Read/write access within tenant scope
    • SYSTEM_ADMIN - Cross-tenant admin capabilities (system-wide)
  • Permission checks implemented at service layer using @PreAuthorize annotations
  • Tenant-aware resource access - All queries include tenant ID filter in WHERE clause
  • Resource ownership validation - Verifies that user has access to requested tenant resource

Sensitive Data Encryption

  • At-rest encryption using AES-256 with GCM mode for database fields containing sensitive data (user passwords, PII)
  • In-transit encryption enforced via TLS 1.3 for all API communications
  • Key management through AWS KMS (Customer Managed Keys) with automatic rotation every 90 days
  • Data encryption keys are stored in encrypted format within the application's secure configuration store

PII Handling and Masking Strategy

  • PII fields (email, phone, SSN) are encrypted at rest using AES-256-GCM
  • Masking in logs - PII fields are truncated or replaced with asterisks in log output
  • PII in API responses - Only masked versions shown to non-admin users
  • Data retention policy - PII data is automatically deleted after 3 years of inactivity

Secrets Management

  • AWS Secrets Manager for storing:
    • Database connection credentials (rotated every 90 days)
    • Third-party API keys (Stripe, Twilio)
    • JWT signing keys (rotated every 6 months)
  • Secrets rotation handled via Lambda functions with CloudWatch alarms for failure detection
  • Access control - IAM roles restrict access to secrets to only necessary services

Audit Logging Design

  • Audit log schema includes:
    • timestamp (UTC)
    • tenant_id
    • user_id
    • action_type (CREATE, UPDATE, DELETE, READ)
    • resource_type
    • resource_id
    • ip_address
    • user_agent
    • success (boolean)
  • Audit logging implemented via interceptor pattern that wraps all service methods
  • Log retention policy - 2 years for audit logs, archived to S3 after 1 year
  • Compliance reporting - Audit logs exported in structured format (JSON) for SOC2 and GDPR compliance checks

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

  • Service Layer: 85% code coverage with focus on business logic, validation, and error paths
  • Repository Layer: 90% coverage with all CRUD operations and query edge cases
  • Domain Objects: 100% coverage for value objects and immutable entities
  • Exception Handling: 100% coverage for all custom exception paths
  • Integration Points: 95% coverage for external service mocks and database interactions

Integration Testing Strategy

  • Test Containers: Use Docker-based test containers for PostgreSQL, Redis, and Kafka to ensure realistic environment behavior
  • Embedded Databases: H2/PostgreSQL in-memory databases for fast unit test execution with schema compatibility
  • WireMock: For external HTTP service stubbing (e.g., payment gateways, notification services)
  • Test Data Isolation: Each test suite runs in isolated database schemas with automatic cleanup

Contract Testing

  • Pact Framework: Consumer-driven contract testing between tenant isolation service and dependent services (e.g., user management, billing)
  • Versioned Contracts: Maintain contract versions with semantic versioning to support backward compatibility
  • Contract Validation: Run contract tests in CI pipeline as part of pre-deployment validation

Performance Testing Tool and Key Scenarios

  • k6: Load testing tool for simulating concurrent tenant access patterns
  • Key Scenarios:
    • Concurrent tenant creation under load (1000 concurrent requests)
    • Multi-tenant query performance with large datasets (10k+ records per tenant)
    • Tenant data isolation under high contention
    • Cache hit/miss ratio validation under realistic traffic patterns

Test Data Management Strategy

  • Fixtures: Pre-defined tenant configurations and user data for integration tests
  • Factories: Programmatic test data generation using Factory Pattern to create valid tenant and user objects
  • Seeding: Automated database seeding scripts for test environments with consistent baseline data
  • Data Cleanup: Transactional test cleanup using database snapshots or schema-level truncation to maintain test isolation
// Example unit test for tenant validation logic
@Test
public void testCreateTenantWithValidData() {
    TenantRequest request = new TenantRequest.Builder()
        .withName("ValidTenant")
        .withDomain("valid.tenant.com")
        .withContactEmail("admin@valid.com")
        .build();
    
    Tenant tenant = tenantService.createTenant(request);
    
    assertThat(tenant.getId()).isNotNull();
    assertThat(tenant.getName()).isEqualTo("ValidTenant");
    assertThat(tenant.getDomain()).isEqualTo("valid.tenant.com");
}

@Test
public void testCreateTenantWithInvalidDomain() {
    TenantRequest request = new TenantRequest.Builder()
        .withName("InvalidTenant")
        .withDomain("invalid-domain")
        .withContactEmail("admin@valid.com")
        .build();
    
    assertThatThrownBy(() -> tenantService.createTenant(request))
        .isInstanceOf(InvalidTenantDomainException.class)
        .hasMessageContaining("Invalid domain format");
}

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective querying and analysis:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO|WARN|ERROR",
  "service": "tenant-isolation-service",
  "trace_id": "123e4567-e89b-12d3-a456-426614174000",
  "span_id": "abcdef1234567890",
  "user_id": "user-123",
  "tenant_id": "tenant-abc",
  "operation": "create_tenant",
  "request_id": "req-xyz",
  "duration_ms": 150,
  "error_code": "TENANT_NOT_FOUND",
  "error_message": "Tenant with ID tenant-abc does not exist",
  "stack_trace": "...",
  "payload": {
    "input_data": "{...}",
    "output_data": "{...}"
  }
}

PII masking rules:

  • User IDs and tenant IDs are logged in masked format (e.g., user-123 → user-***)
  • Full email addresses are masked to user@***.com
  • Any sensitive fields in payloads are redacted or replaced with ***

Distributed Tracing Instrumentation

Using OpenTelemetry, we instrument key operations:

Custom Spans to Add:

  1. tenant_isolation_check - wrapping tenant validation logic
  2. data_access_tenant_filter - filtering data by tenant context
  3. tenant_context_extraction - extracting tenant from request headers

Attributes to Capture:

Span currentSpan = Span.current();
currentSpan.setAttribute("tenant.id", tenantId);
currentSpan.setAttribute("user.id", userId);
currentSpan.setAttribute("operation.name", "create_tenant");
currentSpan.setAttribute("isolation.level", "multi-tenant");

Metrics Design

Service-Level RED Metrics:

Metric NameTypeDescriptionLabels
tenant_isolation_requests_totalCounterTotal requests processedoperation, status
tenant_isolation_errors_totalCounterRequest errors by typeerror_type, tenant_id
tenant_isolation_duration_msHistogramRequest processing durationoperation, status
tenant_isolation_active_tenant_countGaugeActive tenants in systemenvironment
tenant_isolation_data_access_countCounterData access operationstenant_id, operation_type

Labels Design:

  • operation: create_tenant, validate_access, filter_data
  • status: success, error, timeout
  • error_type: invalid_tenant, access_denied, internal_error
  • environment: production, staging, development

Health Check Endpoint Design

/health:
  - Status: 200 OK (healthy) or 503 Service Unavailable (unhealthy)
  - Response: {"status": "healthy", "timestamp": "...", "checks": {...}}
  - Checks: 
    - Database connectivity (tenant_isolation_db)
    - Cache availability (redis_tenant_cache)
    - External dependency health (auth_service)

/ready:
  - Status: 200 OK (ready) or 503 Service Unavailable (not ready)
  - Response: {"status": "ready", "timestamp": "..."}
  - Checks:
    - All required services are reachable
    - Cache is initialized
    - Database pool has available connections

/live:
  - Status: 200 OK (live) or 503 Service Unavailable (unhealthy)
  - Response: {"status": "live", "timestamp": "..."}
  - Checks:
    - Service process is running
    - Memory usage below threshold (e.g., 80%)
    - CPU utilization is normal

Alerting Thresholds

MetricWarning LevelCritical LevelEscalation Path
tenant_isolation_errors_total (rate over 5m)> 10 errors/min> 50 errors/minOn-call engineer, SRE team
tenant_isolation_duration_ms (p95)> 500ms> 2sOn-call engineer, SRE team
tenant_isolation_requests_total (rate)> 10k req/sec> 50k req/secSRE team, product team
tenant_isolation_data_access_count (rate)> 100k access/sec> 500k access/secSRE team, security team
tenant_isolation_active_tenant_count (change rate)> 100% in 5m> 300% in 5mSRE team, product team

Alerts will be configured with:

  • Slack notifications for warning level
  • PagerDuty escalation for critical level
  • Email alerts to engineering leadership for critical incidents

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

  • Tenant scope granularity: Should tenants be isolated at database level, schema level, or table level? This decision impacts the complexity of cross-tenant queries and data migration strategies.
  • Cross-tenant data access policy: Will tenants be allowed to access shared data or other tenants' data? If so, what's the scope and how is it controlled?
  • Multi-tenancy model: Is this a single-tenant-per-database or multi-tenant-per-database model? This affects the complexity of tenant provisioning and migration strategies.
  • Tenant lifecycle management: How are tenants created, migrated, or deleted? What's the process for handling tenant data archival and compliance requirements?

Known trade-offs and justifications

  • Performance vs. Isolation: Stronger tenant isolation (e.g., separate databases) provides better security and performance isolation but increases operational complexity, cost, and management overhead. Weighed against the need for shared resources and reduced infrastructure costs.
  • Flexibility vs. Simplicity: Supporting multiple isolation strategies (database, schema, table) provides flexibility but increases complexity in the framework and requires more sophisticated tenant management logic.
  • Consistency vs. Availability: Enforcing strict consistency across all tenant data operations (e.g., using distributed transactions) provides strong guarantees but can impact availability and performance under high load or network issues.
  • Security vs. Performance: Implementing strict tenant data access controls and validation at every layer adds security but can introduce latency and complexity in the request handling pipeline.

Risks and mitigation strategies

  • Cross-Tenant Data Leakage: Risk of data leakage between tenants due to improper filtering or configuration errors. Mitigation includes: comprehensive unit and integration testing, strict access control validation, regular security audits, and automated compliance checks.
  • Performance Degradation: Risk of one tenant's heavy usage impacting others. Mitigation includes: implementing resource quotas and limits, using database-level query optimization, and monitoring tenant-specific metrics for anomaly detection.
  • Tenant Data Migration Failures: Risk of data loss or corruption during tenant migration. Mitigation includes: implementing robust backup and rollback strategies, using atomic operations where possible, and maintaining detailed audit trails for recovery.
  • Scalability Bottlenecks: Risk of framework becoming a bottleneck as tenant count increases. Mitigation includes: implementing efficient caching strategies, using asynchronous processing where appropriate, and designing for horizontal scaling.
  • Configuration Drift: Risk of inconsistent tenant configurations leading to security or performance issues. Mitigation includes: centralized configuration management, automated validation checks, and enforced configuration versioning.

Technical debt acknowledged

  • Framework Extensibility: The current design assumes a fixed set of tenant isolation strategies, which may not accommodate future requirements without significant refactoring. This is acceptable for MVP but needs to be revisited for long-term maintainability.
  • Tenant Metadata Management: The current implementation stores tenant metadata in a centralized location, which can become a single point of failure and performance bottleneck at scale. This is a known limitation with mitigation through eventual consistency patterns and distributed metadata stores.
  • Cross-Tenant Query Complexity: Supporting cross-tenant queries introduces complexity in query planning and execution that is not fully addressed in the current design. This is a trade-off for the flexibility of multi-tenant architecture that will need to be revisited for high-volume cross-tenant operations.
  • Tenant-Specific Configuration: The framework currently assumes all tenants follow the same configuration patterns, which may not scale well for highly customized tenant needs. This is a design limitation that can be addressed through configuration inheritance and override mechanisms in future iterations.

Was this article helpful?

Topics in this article

More from Backend LLD

Stay in the Loop

Get curated DSA patterns, coding problems, frontend and backend LLD/HLD, and AI/ML System Design insights delivered straight to your inbox. No spam, just focused interview and engineering content.