Multi-Tenant Backend Architecture Design: Scalable Cloud Services

Multi-tenant backend architectures enable efficient resource sharing while maintaining tenant security and performance isolation through careful design of data models, resource allocation, and access control mechanisms.

1. Problem Statement & Requirements

Clarifying questions

1. What scale expectations are there for tenants and users?

   • Are we targeting hundreds of tenants with thousands of users each, or millions of users across a few hundred tenants? This impacts sharding strategy and caching.

2. What consistency requirements exist for multi-tenant data?

   • Do tenants require strict isolation (e.g., no cross-tenant reads/writes), or can some data sharing be allowed for better performance?

3. What are the key entities and their relationships in this multi-tenant system?

   • Is it a SaaS product with users, organizations, subscriptions, or is it more focused on specific use cases like booking systems or content management?

4. What’s the read/write ratio for tenant data?

   • If reads dominate (e.g., analytics dashboard), caching and read replicas will be critical.

5. What latency sensitivity is expected for tenant-specific operations?

   • Is sub-100ms SLA required, or is 500ms acceptable for less critical operations?

6. What authentication and authorization model will be used?

   • Will it be OAuth2 with JWTs, API keys, or a custom RBAC system? This affects how tenants are identified and isolated.

7. Are there any external integrations or third-party dependencies?

   • Will the system need to integrate with payment providers, email services, or other SaaS platforms that also must be multi-tenant?

Functional requirements

• Tenant isolation: Each tenant's data must be completely isolated from others.
• Multi-tenancy support: The system must support multiple tenants with independent configurations and policies.
• Tenant onboarding: Ability to onboard new tenants with unique identifiers, configuration, and initial setup.
• Tenant lifecycle management: Support for tenant deactivation, suspension, and deletion.
• Tenant-specific configuration: Each tenant can have custom settings (e.g., branding, features, limits).
• User management per tenant: Users must be scoped to specific tenants and have role-based access control.
• Audit logging per tenant: Track all actions performed within a tenant’s context.

Non-functional requirements

Requirement Type	Target
Latency (p50/p95/p99)	p50: < 50ms, p95: < 200ms, p99: < 500ms
Throughput	10,000 RPS (reads) + 1,000 RPS (writes)
Availability	99.9% uptime
Scalability	Support for 10,000+ tenants with 1M+ users
Data durability	Data persistence and recovery within 5 minutes
Security compliance	GDPR, SOC2, HIPAA (if applicable)

Constraints and assumptions

• Assumption: Tenants will have varying levels of data volume and usage patterns.
• Assumption: Tenant configuration and policies can be updated without downtime.
• Constraint: Tenant data must never be leaked or accessible by other tenants.
• Constraint: Multi-tenant system must support horizontal scaling across regions or availability zones.
• Assumption: Tenant onboarding and configuration will be handled via admin APIs or UI.

Out of scope

• Tenant migration: Moving tenants between systems or regions.
• Cross-tenant data sharing: No feature to share data across tenants, even with explicit permissions.
• Tenant-to-tenant communication: No direct inter-tenant messaging or data exchange.
• Multi-region replication: Multi-region support is out of scope for initial design but can be added later.
• Tenant billing: Billing logic or integration with payment providers is not part of this design.

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for the multi-tenant backend architecture due to:

• Strong typing and schema evolution support for versioning
• Efficient binary serialization reducing payload size and network overhead
• Built-in support for streaming and bidirectional communication
• Native support for authentication, authorization, and encryption
• Better performance characteristics for high-throughput multi-tenant workloads

API Endpoints

Method	Path	Description	Auth	Rate Limit
GET	/v1/tenants	List all tenants	JWT + RBAC	100 req/min
POST	/v1/tenants	Create new tenant	JWT + Admin RBAC	5 req/min
GET	/v1/tenants/{id}	Get tenant details	JWT + RBAC	100 req/min
PUT	/v1/tenants/{id}	Update tenant metadata	JWT + Admin RBAC	50 req/min
DELETE	/v1/tenants/{id}	Delete tenant (soft)	JWT + Admin RBAC	5 req/min
GET	/v1/tenants/{id}/users	List tenant users	JWT + Tenant RBAC	100 req/min
POST	/v1/tenants/{id}/users	Add user to tenant	JWT + Admin RBAC	50 req/min
GET	/v1/tenants/{id}/users/{userId}	Get user in tenant	JWT + Tenant RBAC	100 req/min
PUT	/v1/tenants/{id}/users/{userId}	Update user in tenant	JWT + Admin RBAC	50 req/min
DELETE	/v1/tenants/{id}/users/{userId}	Remove user from tenant	JWT + Admin RBAC	50 req/min
GET	/v1/tenants/{id}/resources	List tenant resources	JWT + Tenant RBAC	100 req/min
POST	/v1/tenants/{id}/resources	Create resource in tenant	JWT + Admin RBAC	50 req/min

Request and Response Schemas

Create Tenant Request

{
  "name": "string",
  "description": "string",
  "metadata": {
    "key": "value"
  }
}

Create Tenant Response

{
  "id": "uuid",
  "name": "string",
  "description": "string",
  "metadata": {
    "key": "value"
  },
  "createdAt": "timestamp",
  "updatedAt": "timestamp"
}

List Tenants Response

{
  "tenants": [
    {
      "id": "uuid",
      "name": "string",
      "description": "string",
      "metadata": {
        "key": "value"
      },
      "createdAt": "timestamp",
      "updatedAt": "timestamp"
    }
  ],
  "pagination": {
    "cursor": "string",
    "hasNext": true,
    "total": 100
  }
}

HTTP Status Codes and Error Response Format

Standard HTTP status codes:

• 200 OK - Successful GET, PUT, DELETE
• 201 Created - Successful POST
• 400 Bad Request - Invalid input
• 401 Unauthorized - Missing or invalid auth token
• 403 Forbidden - Insufficient permissions
• 404 Not Found - Resource not found
• 429 Too Many Requests - Rate limit exceeded
• 500 Internal Server Error - Unexpected server error

Standardized error response:

{
  "error": {
    "code": "string",
    "message": "string",
    "details": [
      {
        "field": "string",
        "message": "string"
      }
    ],
    "timestamp": "timestamp",
    "requestId": "uuid"
  }
}

API Versioning Strategy

gRPC versioning via package names and protobuf schema evolution:

• tenant.v1 - Initial version
• tenant.v2 - Future breaking changes (e.g., new fields)
• Schema evolution using field numbers and optional fields to maintain backward compatibility

Pagination Design

Cursor-based pagination is selected because:

• Consistent performance regardless of page size
• Handles data changes during pagination gracefully
• Provides better support for real-time data scenarios

Authentication and Authorization

Authentication: JWT tokens with OAuth2 integration via OpenID Connect for tenant users.

Authorization: Role-Based Access Control (RBAC) with:

• TENANT_ADMIN - Full access to tenant resources
• TENANT_USER - Read-only access to tenant resources
• SYSTEM_ADMIN - Full system access

Rate Limiting Strategy

Per-tenant rate limiting using Redis with:

• Token bucket algorithm for burst handling
• Tenant ID as the key in rate limiting logic
• Configurable limits per endpoint (5-100 req/min)

Idempotency Design

Idempotency key strategy:

• Clients provide X-Idempotency-Key header on mutation requests
• Server stores operation result for the key in Redis with TTL of 1 hour
• If same key is used, return cached result instead of reprocessing

Sample gRPC Method Definition (Proto)

syntax = "proto3";

package tenant;

option java_multiple_files = true;
option java_package = "com.example.tenant.grpc";
option java_outer_classname = "TenantServiceProto";

service TenantService {
  rpc CreateTenant(CreateTenantRequest) returns (CreateTenantResponse);
  rpc GetTenant(GetTenantRequest) returns (GetTenantResponse);
  rpc ListTenants(ListTenantsRequest) returns (ListTenantsResponse);
  rpc UpdateTenant(UpdateTenantRequest) returns (UpdateTenantResponse);
  rpc DeleteTenant(DeleteTenantRequest) returns (DeleteTenantResponse);
}

message CreateTenantRequest {
  string name = 1;
  string description = 2;
  map<string, string> metadata = 3;
}

message CreateTenantResponse {
  Tenant tenant = 1;
}

message GetTenantRequest {
  string id = 1;
}

message GetTenantResponse {
  Tenant tenant = 1;
}

message ListTenantsRequest {
  int32 page_size = 1;
  string page_token = 2;
}

message ListTenantsResponse {
  repeated Tenant tenants = 1;
  string next_page_token = 2;
}

message UpdateTenantRequest {
  string id = 1;
  string name = 2;
  string description = 3;
  map<string, string> metadata = 4;
}

message UpdateTenantResponse {
  Tenant tenant = 1;
}

message DeleteTenantRequest {
  string id = 1;
}

message DeleteTenantResponse {
  bool deleted = 1;
}

message Tenant {
  string id = 1;
  string name = 2;
  string description = 3;
  map<string, string> metadata = 4;
  int64 created_at = 5;
  int64 updated_at = 6;
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    TENANT ||--o{ USERS : "has"
    TENANT ||--o{ ORGANIZATIONS : "owns"
    USERS ||--o{ USER_ROLES : "has"
    ORGANIZATIONS ||--o{ USER_ROLES : "has"
    USER_ROLES ||--o{ ROLES : "grants"
    ROLES ||--o{ PERMISSIONS : "has"
    ORGANIZATIONS ||--o{ PRODUCTS : "manages"
    ORGANIZATIONS ||--o{ SUBSCRIPTIONS : "has"
    PRODUCTS ||--o{ PRODUCT_FEATURES : "has"
    SUBSCRIPTIONS ||--o{ SUBSCRIPTION_PLANS : "uses"
    SUBSCRIPTIONS ||--o{ INVOICES : "generates"
    ORGANIZATIONS ||--o{ AUDIT_LOGS : "logs"

    TENANT {
        string id PK
        string name
        string status
        timestamp created_at
        timestamp updated_at
    }

    USERS {
        string id PK
        string tenant_id FK
        string email
        string status
        timestamp created_at
        timestamp updated_at
    }

    ORGANIZATIONS {
        string id PK
        string tenant_id FK
        string name
        string status
        timestamp created_at
        timestamp updated_at
    }

    USER_ROLES {
        string user_id FK
        string role_id FK
        string organization_id FK
        timestamp created_at
    }

    ROLES {
        string id PK
        string name
        string description
        timestamp created_at
        timestamp updated_at
    }

    PERMISSIONS {
        string id PK
        string role_id FK
        string resource
        string action
        timestamp created_at
    }

    PRODUCTS {
        string id PK
        string organization_id FK
        string name
        string status
        timestamp created_at
        timestamp updated_at
    }

    SUBSCRIPTIONS {
        string id PK
        string organization_id FK
        string plan_id FK
        string status
        timestamp created_at
        timestamp updated_at
    }

    SUBSCRIPTION_PLANS {
        string id PK
        string organization_id FK
        string name
        string description
        decimal price
        timestamp created_at
        timestamp updated_at
    }

    INVOICES {
        string id PK
        string subscription_id FK
        decimal amount
        timestamp due_date
        timestamp created_at
        timestamp updated_at
    }

    PRODUCT_FEATURES {
        string id PK
        string product_id FK
        string name
        string description
        timestamp created_at
        timestamp updated_at
    }

    AUDIT_LOGS {
        string id PK
        string tenant_id FK
        string user_id FK
        string action
        string resource_type
        string resource_id
        jsonb details
        timestamp created_at
    }

Table Schema Definitions

Tenants Table

CREATE TABLE tenants (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL UNIQUE,
    status VARCHAR(50) NOT NULL DEFAULT 'active',
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

Users Table

CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
    email VARCHAR(255) NOT NULL UNIQUE,
    status VARCHAR(50) NOT NULL DEFAULT 'active',
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

Organizations Table

CREATE TABLE organizations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
    name VARCHAR(255) NOT NULL,
    status VARCHAR(50) NOT NULL DEFAULT 'active',
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

User Roles Table

CREATE TABLE user_roles (
    user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    role_id UUID NOT NULL REFERENCES roles(id) ON DELETE CASCADE,
    organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    PRIMARY KEY (user_id, role_id, organization_id)
);

Roles Table

CREATE TABLE roles (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL,
    description TEXT,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

Permissions Table

CREATE TABLE permissions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    role_id UUID NOT NULL REFERENCES roles(id) ON DELETE CASCADE,
    resource VARCHAR(255) NOT NULL,
    action VARCHAR(100) NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

Products Table

CREATE TABLE products (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    name VARCHAR(255) NOT NULL,
    status VARCHAR(50) NOT NULL DEFAULT 'active',
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

Subscriptions Table

CREATE TABLE subscriptions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    plan_id UUID NOT NULL REFERENCES subscription_plans(id),
    status VARCHAR(50) NOT NULL DEFAULT 'active',
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

Subscription Plans Table

CREATE TABLE subscription_plans (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    name VARCHAR(255) NOT NULL,
    description TEXT,
    price DECIMAL(10, 2) NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

Invoices Table

CREATE TABLE invoices (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    subscription_id UUID NOT NULL REFERENCES subscriptions(id) ON DELETE CASCADE,
    amount DECIMAL(10, 2) NOT NULL,
    due_date TIMESTAMP WITH TIME ZONE NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

Product Features Table

CREATE TABLE product_features (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    product_id UUID NOT NULL REFERENCES products(id) ON DELETE CASCADE,
    name VARCHAR(255) NOT NULL,
    description TEXT,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

Audit Logs Table

CREATE TABLE audit_logs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
    user_id UUID NOT NULL REFERENCES users(id),
    action VARCHAR(255) NOT NULL,
    resource_type VARCHAR(255) NOT NULL,
    resource_id UUID NOT NULL,
    details JSONB,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

Primary Key Strategy

UUIDs are used as primary keys for all tables to ensure:

1. Global Uniqueness: UUIDs provide globally unique identifiers across tenants and systems
2. No Cross-Tenant Collisions: Each tenant's data remains isolated even if identifiers collide
3. Scalability: UUIDs can be generated without coordination across distributed systems
4. Security: UUIDs don't reveal sequential patterns that could be exploited

Index Design

Table	Index Name	Fields	Type	Purpose
tenants	idx_tenants_name	name	UNIQUE	Fast tenant lookup by name
users	idx_users_tenant_id	tenant_id	B-tree	Tenant-based queries
users	idx_users_email	email	UNIQUE	Fast user lookup by email
organizations	idx_organizations_tenant_id	tenant_id	B-tree	Tenant-based organization queries
user_roles	idx_user_roles_user_id	user_id	B-tree	Fast role lookup for users
user_roles	idx_user_roles_role_id	role_id	B-tree	Fast role-based queries
permissions	idx_permissions_role_id	role_id	B-tree	Fast permission lookup by role
products	idx_products_organization_id	organization_id	B-tree	Organization-based product queries
subscriptions	idx_subscriptions_organization_id	organization_id	B-tree	Organization-based subscription queries
subscriptions	idx_subscriptions_plan_id	plan_id	B-tree	Fast plan-based subscription queries
invoices	idx_invoices_subscription_id	subscription_id	B-tree	Fast invoice lookup by subscription
audit_logs	idx_audit_logs_tenant_id_created_at	tenant_id, created_at	B-tree	Fast audit log queries by tenant
audit_logs	idx_audit_logs_user_id	user_id	B-tree	Fast audit log queries by user

Data Partitioning / Sharding Strategy

Tenant-based Sharding is implemented:

1. Shard Key: tenant_id in all tables
2. Partitioning Strategy:
   • Single database with multiple schemas per tenant (PostgreSQL schema isolation)
   • Alternatively, separate databases per tenant for maximum isolation
3. Sharding Logic:
   • All queries include tenant_id in WHERE clause
   • Tenant ID is used to route requests to appropriate database/schema

Soft Delete vs. Hard Delete Decision

Soft Delete is used for the following reasons:

1. Compliance: Regulatory requirements often mandate data retention
2. Auditability: Historical context is preserved for compliance and debugging
3. Recovery: Data can be recovered if deleted accidentally
4. Referential Integrity: Foreign key constraints are maintained during soft deletes

Audit Trail Design

Audit Logs Table captures:

1. Who: user_id from authentication context
2. What: action (CREATE, UPDATE, DELETE) and resource_type
3. When: created_at timestamp
4. Details: JSONB field containing specific changes made

Data Archival Strategy and Retention Policy

Archival Strategy:

1. Daily Archives: Daily snapshots of audit logs and user activity
2. Monthly Archives: Monthly aggregates for analytics and reporting
3. Quarterly Archives: Quarterly compliance reports

Retention Policy:

• Audit Logs: 5 years (compliance requirement)
• User Activity: 2 years (analytics and debugging)
• Subscription Data: 3 years (billing and compliance)
• Product Data: Indefinite (business critical)

public class AuditLog {
    private String id;
    private String tenantId;
    private String userId;
    private String action;
    private String resourceType;
    private String resourceId;
    private Map<String, Object> details;
    private LocalDateTime createdAt;
}

4. Class Design & Service Layer

UML Class Diagram

classDiagram
    class Tenant {
        -String id
        -String name
        -String status
        -Instant createdAt
        -Instant updatedAt
        +create()
        +update()
        +delete()
    }

    class TenantContext {
        -String tenantId
        -Map~String, Object~ attributes
        +getTenantId()
        +getAttribute(String key)
        +setAttribute(String key, Object value)
    }

    class TenantService {
        -TenantRepository tenantRepository
        -EventPublisher eventPublisher
        -AuditLogger auditLogger
        +createTenant(CreateTenantRequest request)
        +getTenant(String tenantId)
        +updateTenant(UpdateTenantRequest request)
        +deleteTenant(String tenantId)
        +listTenants(ListTenantsQuery query)
    }

    class TenantRepository {
        -DataSource dataSource
        -JdbcTemplate jdbcTemplate
        +save(Tenant tenant)
        +findById(String id)
        +findAll(ListTenantsQuery query)
        +deleteById(String id)
    }

    class EventPublisher {
        -EventBus eventBus
        +publish(Event event)
    }

    class AuditLogger {
        -AuditRepository auditRepository
        +log(AuditEvent event)
    }

    class TenantValidator {
        -TenantService tenantService
        +validateCreateRequest(CreateTenantRequest request)
        +validateUpdateRequest(UpdateTenantRequest request)
    }

    class TenantLifecycleManager {
        -TenantService tenantService
        -EventPublisher eventPublisher
        +activateTenant(String tenantId)
        +deactivateTenant(String tenantId)
    }

    class CreateTenantRequest {
        -String name
        -Map~String, String~ metadata
    }

    class UpdateTenantRequest {
        -String id
        -String name
        -Map~String, String~ metadata
    }

    class ListTenantsQuery {
        -String status
        -Integer limit
        -String cursor
    }

    class TenantEvent {
        -String tenantId
        -String eventType
        -Instant timestamp
    }

    class AuditEvent {
        -String tenantId
        -String userId
        -String action
        -Instant timestamp
    }

    Tenant --> TenantContext : uses
    TenantService --> TenantRepository : delegates to
    TenantService --> EventPublisher : publishes events
    TenantService --> AuditLogger : logs audit
    TenantService --> TenantValidator : validates requests
    TenantService --> TenantLifecycleManager : manages lifecycle
    TenantValidator --> TenantService : validates
    TenantLifecycleManager --> TenantService : updates state
    CreateTenantRequest --> Tenant : creates
    UpdateTenantRequest --> Tenant : updates
    ListTenantsQuery --> TenantRepository : queries with
    EventPublisher --> EventBus : publishes to
    AuditLogger --> AuditRepository : logs to

State Machine Diagram for Tenant

stateDiagram-v2
    [*] --> Active
    Active --> Inactive : Deactivate
    Active --> Deleted : Delete
    Inactive --> Active : Activate
    Inactive --> Deleted : Delete
    Deleted --> [*]

Service Class Structure and Responsibilities

The backend service layer follows a clean architecture pattern with the following responsibilities:

1. TenantService: Main orchestrator for tenant operations, handles business logic and coordination between components
2. TenantValidator: Validates incoming requests according to business rules and constraints
3. TenantLifecycleManager: Manages tenant state transitions (activate/deactivate)
4. EventPublisher: Handles asynchronous event publishing for cross-service communication
5. AuditLogger: Logs all tenant operations for compliance and audit purposes

Interface / Abstract Class Design

// Service interfaces
public interface TenantService {
    Tenant createTenant(CreateTenantRequest request);
    Tenant getTenant(String tenantId);
    Tenant updateTenant(UpdateTenantRequest request);
    void deleteTenant(String tenantId);
    List<Tenant> listTenants(ListTenantsQuery query);
}

public interface TenantRepository {
    Tenant save(Tenant tenant);
    Optional<Tenant> findById(String id);
    List<Tenant> findAll(ListTenantsQuery query);
    void deleteById(String id);
}

public interface EventPublisher {
    void publish(Event event);
}

// Abstract base classes
public abstract class BaseTenantService implements TenantService {
    protected final TenantRepository tenantRepository;
    protected final EventPublisher eventPublisher;
    protected final AuditLogger auditLogger;
    protected final TenantValidator validator;

    public BaseTenantService(TenantRepository tenantRepository, 
                           EventPublisher eventPublisher,
                           AuditLogger auditLogger,
                           TenantValidator validator) {
        this.tenantRepository = tenantRepository;
        this.eventPublisher = eventPublisher;
        this.auditLogger = auditLogger;
        this.validator = validator;
    }
}

Business Logic Encapsulation Patterns

Strategy Pattern for Tenant Creation Validation

public interface TenantValidationStrategy {
    void validate(CreateTenantRequest request);
}

@Component
public class DefaultTenantValidationStrategy implements TenantValidationStrategy {
    private final TenantService tenantService;

    public DefaultTenantValidationStrategy(TenantService tenantService) {
        this.tenantService = tenantService;
    }

    @Override
    public void validate(CreateTenantRequest request) {
        if (request.getName() == null || request.getName().trim().isEmpty()) {
            throw new ValidationException("Tenant name is required");
        }
        
        if (request.getName().length() > 100) {
            throw new ValidationException("Tenant name must be less than 100 characters");
        }
        
        if (tenantService.getTenantByName(request.getName()).isPresent()) {
            throw new DuplicateTenantException("Tenant with name " + request.getName() + " already exists");
        }
    }
}

@Component
public class TenantValidator {
    private final List<TenantValidationStrategy> strategies;

    public TenantValidator(List<TenantValidationStrategy> strategies) {
        this.strategies = strategies;
    }

    public void validateCreateRequest(CreateTenantRequest request) {
        strategies.forEach(strategy -> strategy.validate(request));
    }
}

Critical-Path Method Implementation

public class DefaultTenantService extends BaseTenantService {
    
    @Override
    @Transactional
    public Tenant createTenant(CreateTenantRequest request) {
        // Validate request
        validator.validateCreateRequest(request);
        
        // Create tenant object with generated ID
        Tenant tenant = Tenant.builder()
                .id(UUID.randomUUID().toString())
                .name(request.getName())
                .status(TenantStatus.ACTIVE)
                .metadata(request.getMetadata())
                .createdAt(Instant.now())
                .updatedAt(Instant.now())
                .build();
        
        // Save tenant to database
        Tenant savedTenant = tenantRepository.save(tenant);
        
        // Publish event for cross-service communication
        TenantEvent event = TenantEvent.builder()
                .tenantId(savedTenant.getId())
                .eventType("TENANT_CREATED")
                .timestamp(Instant.now())
                .build();
        eventPublisher.publish(event);
        
        // Log audit trail
        AuditEvent auditEvent = AuditEvent.builder()
                .tenantId(savedTenant.getId())
                .userId("system") // In real implementation, this would come from security context
                .action("CREATE_TENANT")
                .timestamp(Instant.now())
                .build();
        auditLogger.log(auditEvent);
        
        return savedTenant;
    }
    
    // Additional methods...
}

Transaction Boundary Design

Transaction boundaries are defined per business operation:

• Create Tenant: Single database transaction with event publishing in same scope
• Update Tenant: Single database transaction with audit logging and event publishing
• Delete Tenant: Single database transaction with soft-delete and event publishing

Service Interface Contracts

public interface TenantService {
    /**
     * Creates a new tenant with provided details.
     * @param request Details for creating the tenant
     * @return Created tenant object
     * @throws ValidationException when request validation fails
     * @throws DuplicateTenantException when tenant name already exists
     */
    Tenant createTenant(CreateTenantRequest request);
    
    /**
     * Retrieves tenant by ID.
     * @param tenantId Unique identifier for the tenant
     * @return Tenant object if found, null otherwise
     */
    Tenant getTenant(String tenantId);
    
    /**
     * Updates existing tenant details.
     * @param request Updated tenant information
     * @return Updated tenant object
     * @throws ValidationException when request validation fails
     */
    Tenant updateTenant(UpdateTenantRequest request);
    
    /**
     * Deletes tenant by ID (soft delete).
     * @param tenantId Unique identifier for the tenant
     */
    void deleteTenant(String tenantId);
    
    /**
     * Lists tenants with optional filtering.
     * @param query Query parameters for filtering and pagination
     * @return List of tenants matching criteria
     */
    List<Tenant> listTenants(ListTenantsQuery query);
}

Saga / Orchestration Pattern

For distributed operations involving multiple services, a saga pattern is implemented:

public class TenantCreationSaga {
    private final List<SagaStep> steps;
    
    public void execute(CreateTenantRequest request) {
        try {
            for (SagaStep step : steps) {
                step.execute();
            }
        } catch (Exception e) {
            // Compensate for failed steps
            compensate();
            throw new SagaExecutionException("Tenant creation saga failed", e);
        }
    }
    
    private void compensate() {
        // Reverse all completed steps
        for (int i = steps.size() - 1; i >= 0; i--) {
            steps.get(i).compensate();
        }
    }
}

Domain Object / Value Object Design

@Value
@Builder
public class Tenant {
    String id;
    String name;
    TenantStatus status;
    Map<String, String> metadata;
    Instant createdAt;
    Instant updatedAt;
    
    public boolean isActive() {
        return TenantStatus.ACTIVE.equals(status);
    }
}

public enum TenantStatus {
    ACTIVE, INACTIVE, DELETED
}

@Value
public class CreateTenantRequest {
    String name;
    Map<String, String> metadata;
    
    public boolean hasMetadata() {
        return metadata != null && !metadata.isEmpty();
    }
}

5. Concurrency & Thread Safety

Concurrency Model

The system will leverage a thread pool-based concurrency model using Java's ExecutorService with CompletableFuture for asynchronous operations. This approach provides:

• Efficient thread reuse to minimize overhead
• Non-blocking I/O handling for better throughput
• Built-in support for async composition and error handling

Race Condition Prevention Strategy

Race conditions will be mitigated through:

1. Database-level atomic operations using optimistic locking with version fields
2. Application-level mutexes for critical sections requiring coordination across multiple database operations
3. Transaction isolation levels set to READ COMMITTED or SERIALIZABLE where necessary

Optimistic vs. Pessimistic Locking

Optimistic locking will be used for most operations due to:

• Lower contention in multi-tenant environments
• Better performance under moderate concurrency
• Natural fit for read-heavy workloads typical in SaaS systems

Pessimistic locking will be applied only for critical operations like:

• Tenant resource allocation (e.g., quota enforcement)
• Multi-step atomic operations requiring exclusive access

Deadlock Prevention Strategy

Deadlock prevention will be implemented through:

1. Consistent lock ordering - all operations acquire locks in a predefined order (tenant_id, resource_id)
2. Timeout-based acquisition - 500ms timeout for lock acquisition with retry logic
3. Deadlock detection - periodic checks in background threads to identify and break cycles

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency keys will be implemented using a distributed cache (Redis) with TTL of 5 minutes:

public class IdempotencyService {
    private final RedisTemplate<String, String> redisTemplate;
    
    public boolean isIdempotent(String idempotencyKey) {
        String existing = redisTemplate.opsForValue().get(idempotencyKey);
        if (existing != null) {
            return true; // Already processed
        }
        
        // Store key with TTL to prevent duplicate processing
        redisTemplate.opsForValue().set(idempotencyKey, "processed", 5, TimeUnit.MINUTES);
        return false;
    }
}

Thread Safety in Multi-Tenant Context

Thread safety will be ensured through:

1. Immutable objects for configuration and metadata
2. Thread-local storage for tenant context (tenantId, userContext)
3. Synchronized collections where required for caching operations
4. Immutable domain objects with proper encapsulation to prevent concurrent modification

Concurrency Control for Tenant Isolation

Each tenant's operations will be isolated through:

1. Tenant-aware thread pool - separate executors per tenant for critical operations
2. Isolated database connections with tenant-specific connection pools
3. Tenant-scoped caching using tenant-aware cache keys

Locking Strategy for Multi-Tenant Operations

public class TenantLockManager {
    private final Map<String, ReentrantReadWriteLock> tenantLocks = new ConcurrentHashMap<>();
    
    public Lock getTenantReadLock(String tenantId) {
        return tenantLocks.computeIfAbsent(tenantId, k -> new ReentrantReadWriteLock()).readLock();
    }
    
    public Lock getTenantWriteLock(String tenantId) {
        return tenantLocks.computeIfAbsent(tenantId, k -> new ReentrantReadWriteLock()).writeLock();
    }
}

Performance Considerations

• Thread pool sizing: Based on CPU cores + I/O bound operations (typically 2x CPU cores)
• Queue depth: Configurable bounded queues to prevent memory exhaustion
• Monitoring: Track thread pool metrics (queue size, active threads, rejected tasks)
• Graceful degradation: Fallback to single-threaded processing when contention is high

Implementation Patterns

1. Read-Write Locks for tenant configuration data that's read frequently but written rarely
2. StampedLock for high-performance concurrent operations where version information is needed
3. AtomicReference for managing shared state in multi-tenant contexts with minimal locking overhead

This concurrency model ensures that multi-tenant systems can handle high throughput while maintaining data consistency and tenant isolation. The design balances performance with safety, particularly important in SaaS environments where tenant data must remain completely isolated.

6. Sequence Diagrams for Key Flows

Happy Path Sequence Diagram

sequenceDiagram
    participant Client
    participant AuthN
    participant APIGateway
    participant TenantService
    participant UserService
    participant DB as Database

    Client->>AuthN: Authenticate with JWT
    AuthN-->>Client: JWT Token
    Client->>APIGateway: Request with JWT
    APIGateway->>TenantService: Validate tenant access
    TenantService->>DB: Fetch tenant details
    DB-->>TenantService: Tenant data
    TenantService->>UserService: Fetch user details
    UserService->>DB: Query user by ID
    DB-->>UserService: User data
    UserService->>APIGateway: Return user context
    APIGateway->>TenantService: Process request with tenant context
    TenantService->>DB: Execute operation within tenant scope
    DB-->>TenantService: Operation result
    TenantService-->>APIGateway: Response
    APIGateway-->>Client: Final response

Error / Edge Case Sequence Diagram

sequenceDiagram
    participant Client
    participant AuthN
    participant APIGateway
    participant TenantService
    participant DB as Database

    Client->>AuthN: Authenticate with invalid token
    AuthN-->>Client: 401 Unauthorized
    Client->>APIGateway: Request with invalid token
    APIGateway-->>Client: 401 Unauthorized
    Client->>APIGateway: Request with valid token but invalid tenant ID
    APIGateway->>TenantService: Validate tenant access
    TenantService->>DB: Fetch tenant details
    DB-->>TenantService: Tenant not found
    TenantService-->>APIGateway: 404 Not Found
    APIGateway-->>Client: 404 Tenant not found
    Client->>APIGateway: Request with valid token but insufficient permissions
    APIGateway->>TenantService: Validate tenant access
    TenantService-->>APIGateway: 403 Forbidden
    APIGateway-->>Client: 403 Forbidden

Async Flow Sequence Diagram

sequenceDiagram
    participant Client
    participant APIGateway
    participant OrderService
    participant EventPublisher
    participant MessageQueue
    participant NotificationService

    Client->>APIGateway: Create Order Request
    APIGateway->>OrderService: Process order creation
    OrderService->>DB: Save order in tenant context
    DB-->>OrderService: Order created successfully
    OrderService->>EventPublisher: Publish OrderCreated event
    EventPublisher->>MessageQueue: Queue event
    MessageQueue-->>NotificationService: Deliver event
    NotificationService->>DB: Log notification status
    DB-->>NotificationService: Status updated
    OrderService-->>APIGateway: Return order ID
    APIGateway-->>Client: 201 Created with Order ID

7. Caching Design

Cache Strategy Selection

For a multi-tenant backend, we'll use cache-aside pattern with write-through strategy for critical data and write-behind for non-critical operations. This approach provides the best balance between consistency, performance, and complexity.

Cache Key Design

Cache keys will follow a structured namespace pattern:

{tenantId}:{entityType}:{entityId}

Example key structure:

tenant-123:booking:456
tenant-123:user:789
tenant-123:seat:101

TTL Strategy

Data Type	TTL	Reason
User Profile	5 minutes	Frequent updates, acceptable staleness
Booking Details	1 hour	High read frequency, moderate update rate
Seat Availability	2 minutes	Real-time, high volatility
Tenant Configuration	1 day	Rare updates, critical for performance
Session Tokens	30 minutes	Security-sensitive, short-lived

Cache Eviction Policy

We'll use LRU (Least Recently Used) with TTL-based expiration as the primary eviction policy. This ensures:

• Most frequently accessed data stays in cache
• Memory is reclaimed for stale data
• Predictable performance characteristics

Cache Stampede Prevention

To prevent cache stampedes:

1. Probabilistic Early Expiry: Cache entries expire 20% earlier than TTL for hot keys
2. Distributed Locks: Use Redis distributed locks when cache misses occur to prevent multiple simultaneous database hits
3. Cache warming: Pre-warm hot keys during low-traffic periods

Distributed Cache Architecture

We'll use Redis Cluster with:

• 3 master nodes for data distribution
• 3 replica nodes per master for high availability
• Automatic failover capability
• Cross-zone replication for disaster recovery

Cache Invalidation Strategy

We'll implement event-driven invalidation:

1. Write Operations: Invalidate cache entries on data modification
2. Batch Updates: Use cache tags for bulk invalidation
3. Scheduled Cleanup: Periodic cache cleanup for expired entries

@Component
public class MultiTenantCacheManager {
    
    private final RedisTemplate<String, Object> redisTemplate;
    private final ObjectMapper objectMapper;
    
    public void invalidateTenantCache(String tenantId, String entityType, String entityId) {
        String key = buildCacheKey(tenantId, entityType, entityId);
        redisTemplate.delete(key);
        
        // For bulk invalidation of user sessions
        if ("user".equals(entityType)) {
            String sessionKeyPattern = tenantId + ":session:*";
            Set<String> sessionKeys = redisTemplate.keys(sessionKeyPattern);
            redisTemplate.delete(sessionKeys);
        }
    }
    
    public void invalidateTenantCacheByTag(String tenantId, String tag) {
        String tagKey = "tag:" + tenantId + ":" + tag;
        Set<String> keysToInvalidate = redisTemplate.smembers(tagKey);
        redisTemplate.delete(keysToInvalidate);
        redisTemplate.delete(tagKey);
    }
    
    private String buildCacheKey(String tenantId, String entityType, String entityId) {
        return String.format("%s:%s:%s", tenantId, entityType, entityId);
    }
}

Cache Read-Through Implementation

@Service
public class CachedBookingService {
    
    private final BookingRepository bookingRepository;
    private final MultiTenantCacheManager cacheManager;
    private static final String BOOKING_CACHE_TTL = "3600"; // 1 hour
    
    public Booking getBooking(String tenantId, String bookingId) {
        String cacheKey = buildCacheKey(tenantId, "booking", bookingId);
        
        // Try to get from cache first
        Booking cachedBooking = (Booking) cacheManager.get(cacheKey);
        if (cachedBooking != null) {
            return cachedBooking;
        }
        
        // Cache miss - acquire distributed lock to prevent stampede
        String lockKey = "lock:" + cacheKey;
        if (cacheManager.acquireLock(lockKey, 5000)) {
            try {
                // Double-check cache after acquiring lock
                Booking doubleChecked = (Booking) cacheManager.get(cacheKey);
                if (doubleChecked != null) {
                    return doubleChecked;
                }
                
                // Fetch from database
                Booking booking = bookingRepository.findById(bookingId);
                if (booking != null) {
                    // Cache with TTL
                    cacheManager.put(cacheKey, booking, BOOKING_CACHE_TTL);
                }
                
                return booking;
            } finally {
                cacheManager.releaseLock(lockKey);
            }
        }
        
        // If we couldn't acquire lock, wait and retry
        return bookingRepository.findById(bookingId);
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain objects and database operations. Each entity has its own repository interface and implementation.

// Repository interface
public interface TenantRepository {
    Optional<Tenant> findById(UUID id);
    List<Tenant> findAll();
    Tenant save(Tenant tenant);
    void delete(UUID id);
    boolean existsById(UUID id);
}

// Repository implementation
@Repository
public class TenantRepositoryImpl implements TenantRepository {
    private final EntityManager entityManager;
    private final QueryFactory queryFactory;
    
    public TenantRepositoryImpl(EntityManager entityManager, QueryFactory queryFactory) {
        this.entityManager = entityManager;
        this.queryFactory = queryFactory;
    }
    
    @Override
    public Optional<Tenant> findById(UUID id) {
        return Optional.ofNullable(entityManager.find(Tenant.class, id));
    }
    
    @Override
    public List<Tenant> findAll() {
        return entityManager.createQuery("SELECT t FROM Tenant t", Tenant.class)
                           .getResultList();
    }
    
    @Override
    public Tenant save(Tenant tenant) {
        if (tenant.getId() == null) {
            entityManager.persist(tenant);
        } else {
            entityManager.merge(tenant);
        }
        return tenant;
    }
    
    @Override
    public void delete(UUID id) {
        Tenant tenant = entityManager.find(Tenant.class, id);
        if (tenant != null) {
            entityManager.remove(tenant);
        }
    }
    
    @Override
    public boolean existsById(UUID id) {
        return entityManager.createQuery(
            "SELECT COUNT(t) FROM Tenant t WHERE t.id = :id", Long.class)
            .setParameter("id", id)
            .getSingleResult() > 0;
    }
}

ORM vs. Raw Query Decisions

For multi-tenant systems, we use JPA/Hibernate as the ORM for most entities due to:

1. Type safety and compile-time checking
2. Support for complex object mapping with inheritance
3. Built-in lazy loading and caching capabilities
4. Easy integration with Spring Data JPA

Raw queries are used for:

1. Complex analytical queries that don't map cleanly to entities
2. Performance-critical operations where ORM overhead is unacceptable
3. Multi-tenant partitioning logic that requires custom SQL

// Example of raw query for tenant-specific analytics
@Query(value = """
    SELECT t.name, COUNT(*) as user_count 
    FROM users u 
    JOIN tenants t ON u.tenant_id = t.id 
    WHERE t.id IN (:tenantIds) 
    GROUP BY t.name
    """, nativeQuery = true)
List<TenantUserCount> getTenantUserCounts(@Param("tenantIds") List<UUID> tenantIds);

Read vs. Write Separation

Given the read-heavy nature of multi-tenant systems, we implement CQRS pattern for critical entities:

// Read model repository (for queries)
@Repository
public class TenantReadRepository {
    private final JdbcTemplate jdbcTemplate;
    
    public List<TenantSummary> findAllSummaries() {
        return jdbcTemplate.query(
            "SELECT id, name, created_at FROM tenants WHERE deleted_at IS NULL",
            (rs, rowNum) -> new TenantSummary(
                rs.getObject("id", UUID.class),
                rs.getString("name"),
                rs.getTimestamp("created_at").toInstant()
            )
        );
    }
}

// Write model repository (for mutations)
@Repository
public class TenantWriteRepository {
    private final EntityManager entityManager;
    
    public Tenant save(Tenant tenant) {
        if (tenant.getId() == null) {
            entityManager.persist(tenant);
        } else {
            entityManager.merge(tenant);
        }
        return tenant;
    }
}

Connection Pool Configuration

Connection pool is configured with:

• Max pool size: 50 (based on concurrent request capacity)
• Min idle connections: 10
• Connection timeout: 30 seconds
• Validation query: SELECT 1
• Idle timeout: 600 seconds

# application.yml
spring:
  datasource:
    hikari:
      maximum-pool-size: 50
      minimum-idle: 10
      connection-timeout: 30000
      validation-timeout: 5000
      idle-timeout: 600000
      pool-name: MultiTenantPool

N+1 Query Prevention Strategy

We implement eager loading and batch fetching for related entities:

// Entity with proper fetch strategies
@Entity
@Table(name = "tenants")
@Inheritance(strategy = InheritanceType.SINGLE_TABLE)
public class Tenant {
    @Id
    private UUID id;
    
    @OneToMany(mappedBy = "tenant", fetch = FetchType.LAZY)
    @BatchSize(size = 20) // Prevents N+1 for user loading
    private List<User> users;
    
    @OneToMany(mappedBy = "tenant", fetch = FetchType.LAZY)
    @Fetch(FetchMode.SUBSELECT) // Reduces N+1 for permissions
    private List<Permission> permissions;
    
    @ManyToOne(fetch = FetchType.EAGER)
    @JoinColumn(name = "organization_id")
    private Organization organization;
}

Batch Operation Design

Batch operations are implemented for bulk data processing:

@Service
@Transactional
public class TenantBatchService {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    public void batchCreateTenants(List<Tenant> tenants) {
        int batchSize = 50;
        for (int i = 0; i < tenants.size(); i++) {
            entityManager.persist(tenants.get(i));
            
            if (i % batchSize == 0) {
                entityManager.flush();
                entityManager.clear();
            }
        }
        
        // Flush remaining entities
        entityManager.flush();
    }
    
    public void batchUpdateTenantUsers(List<TenantUserUpdate> updates) {
        String sql = "UPDATE users SET tenant_id = :tenantId WHERE id IN (:userIds)";
        
        // Process in batches to avoid memory issues
        for (int i = 0; i < updates.size(); i += 100) {
            var batch = updates.subList(i, Math.min(i + 100, updates.size()));
            
            // Use native query for better performance
            entityManager.createNativeQuery(sql)
                .setParameter("tenantId", batch.get(0).getTenantId())
                .setParameter("userIds", batch.stream()
                    .map(TenantUserUpdate::getUserId)
                    .collect(Collectors.toList()))
                .executeUpdate();
        }
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature — what triggers async vs. what stays synchronous, and why

In a multi-tenant backend architecture, the decision between synchronous and asynchronous processing is crucial for scalability and responsiveness. For multi-tenant systems, we must balance the need for immediate feedback with the requirement to handle potentially large volumes of concurrent operations across tenants.

Synchronous flows (should remain synchronous):

• Tenant onboarding and configuration validation
• Real-time user interactions (e.g., dashboard queries, real-time analytics)
• Immediate business rule validations (e.g., tenant limits, quotas)

Asynchronous flows (should be async):

• Tenant provisioning and resource allocation (e.g., database schema creation, storage setup)
• Batch data processing (e.g., nightly analytics, report generation)
• Cross-tenant notifications and alerts
• Audit logging and compliance reporting

The key driver for async processing is the need to scale tenant operations independently. For example, when a new tenant is onboarded, we don't want to block the entire system while setting up their database and configurations. These operations can be queued and processed in the background, allowing the system to return a quick acknowledgment to the user.

Message schema design with a sample payload (Avro / Protobuf / JSON Schema — with versioning strategy)

We'll use JSON Schema for its simplicity and wide adoption in microservices. For versioning, we'll use a major.minor.patch semantic versioning scheme with the version field in the message header.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "TenantProvisioningEvent",
  "version": "1.0.0",
  "type": "object",
  "properties": {
    "eventId": { "type": "string" },
    "tenantId": { "type": "string" },
    "eventType": { "type": "string", "enum": ["TENANT_PROVISIONING_STARTED", "TENANT_PROVISIONING_COMPLETED"] },
    "timestamp": { "type": "string", "format": "date-time" },
    "payload": {
      "type": "object",
      "properties": {
        "tenantName": { "type": "string" },
        "planType": { "type": "string", "enum": ["FREE", "STANDARD", "PREMIUM"] },
        "resources": {
          "type": "object",
          "properties": {
            "databaseSizeMB": { "type": "integer" },
            "storageQuotaGB": { "type": "integer" }
          }
        }
      },
      "required": ["tenantName", "planType"]
    }
  },
  "required": ["eventId", "tenantId", "eventType", "timestamp"]
}

Dead letter queue strategy and poison message handling

We implement a DLQ with exponential backoff retry pattern:

1. Retry Strategy: Messages are retried up to 5 times with exponential backoff (1s, 2s, 4s, 8s, 16s) before being moved to DLQ.
2. DLQ Handling: Messages in DLQ are manually inspected and either:
   • Re-processed after fixing root cause
   • Logged for operator review
3. Poison Message Detection: If a message fails more than 3 times in 1 hour, it's flagged for operator review.

At-least-once vs. exactly-once delivery — decision with justification for this feature

We choose at-least-once delivery for most events in a multi-tenant system because:

1. Tenant safety: Multi-tenant systems often deal with financial or operational data where idempotency is hard to guarantee.
2. Simpler implementation: At-least-once is easier to implement and debug than exactly-once.
3. Scalability: The system can scale better with at-least-once semantics.

For critical operations like billing or quota enforcement, we implement idempotency keys to ensure that duplicate events don't cause unintended side effects.

Retry policy for consumers (max attempts, backoff formula, non-retryable error classification)

public class RetryPolicy {
    private static final int MAX_RETRIES = 5;
    private static final long BASE_DELAY_MS = 1000;
    private static final double BACKOFF_MULTIPLIER = 2.0;

    public boolean shouldRetry(Exception exception) {
        // Non-retryable errors (e.g., validation, authorization)
        if (exception instanceof ValidationException || 
            exception instanceof AuthorizationException) {
            return false;
        }
        
        // Retryable errors (e.g., network, timeout)
        return exception instanceof NetworkException || 
               exception instanceof TimeoutException ||
               exception instanceof RetryableException;
    }

    public long getDelay(int attempt) {
        return (long) (BASE_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempt));
    }
}

Outbox pattern for reliable event publishing — include or explicitly justify omission

We do implement the outbox pattern in multi-tenant systems for the following reasons:

1. Consistency guarantee: Ensures that events are published only after the database transaction commits.
2. Tenant isolation: Prevents partial tenant data updates from being published to other tenants.
3. Auditability: Events are stored in the same database as business data, making it easier to audit and debug.

public class EventPublisher {
    private final EventRepository eventRepository;
    private final MessageBroker messageBroker;

    public void publishEvent(TenantEvent event) {
        // 1. Store event in outbox table (within same transaction)
        EventOutbox outboxEvent = new EventOutbox(event);
        eventRepository.save(outboxEvent);

        // 2. Publish to message broker (after transaction commits)
        try {
            messageBroker.publish(outboxEvent);
        } catch (Exception e) {
            // Log error and retry via DLQ mechanism
            log.error("Failed to publish event to broker", e);
        }
    }
}

Sequence diagram for all async flows (Mermaid)

sequenceDiagram
    participant User
    participant APIGateway
    participant TenantService
    participant EventPublisher
    participant MessageBroker
    participant TenantProvisioningConsumer

    User->>APIGateway: POST /tenants
    APIGateway->>TenantService: CreateTenantRequest
    TenantService->>TenantService: Validate and create tenant record
    TenantService->>EventPublisher: Publish TenantProvisioningStarted
    EventPublisher->>MessageBroker: Send to tenant-provisioning-topic
    MessageBroker-->>TenantProvisioningConsumer: Event
    TenantProvisioningConsumer->>TenantProvisioningConsumer: Provision resources (DB, storage)
    TenantProvisioningConsumer->>EventPublisher: Publish TenantProvisioningCompleted
    EventPublisher->>MessageBroker: Send to tenant-provisioning-topic
    MessageBroker-->>TenantService: Event (completed)
    TenantService->>TenantService: Update tenant status to ACTIVE
    APIGateway-->>User: 201 Created with tenant details

10. Error Handling Design

Error Classification Hierarchy

The error handling strategy follows a three-level hierarchy to enable proper routing and response generation:

1. Domain Errors - Business logic violations (e.g., insufficient funds, invalid input)
2. Infrastructure Errors - System-level issues (e.g., database connection failure, timeout)
3. Validation Errors - Input validation failures (e.g., missing required fields, malformed data)

Each error type is encapsulated in a dedicated exception class hierarchy:

// Base exception for all custom errors
public abstract class BaseException extends RuntimeException {
    private final String errorCode;
    private final int httpStatus;

    public BaseException(String errorCode, int httpStatus, String message) {
        super(message);
        this.errorCode = errorCode;
        this.httpStatus = httpStatus;
    }

    // Getters
}

// Domain-specific exceptions
public class InsufficientFundsException extends BaseException {
    public InsufficientFundsException(String message) {
        super("INSUFFICIENT_FUNDS", 400, message);
    }
}

public class TenantNotFoundException extends BaseException {
    public TenantNotFoundException(String message) {
        super("TENANT_NOT_FOUND", 404, message);
    }
}

// Infrastructure exceptions
public class DatabaseConnectionException extends BaseException {
    public DatabaseConnectionException(String message) {
        super("DB_CONNECTION_ERROR", 503, message);
    }
}

// Validation exceptions
public class ValidationError extends BaseException {
    public ValidationError(String message) {
        super("VALIDATION_ERROR", 400, message);
    }
}

Error Propagation Strategy

Errors are handled at the appropriate layer:

• Service Layer: Business logic errors are thrown as domain exceptions
• Controller Layer: Infrastructure and validation errors are caught and mapped to HTTP responses
• Global Exception Handler: All unhandled exceptions are logged and converted to generic internal server errors

Retry Strategy with Exponential Backoff and Jitter

Retry logic is implemented using exponential backoff with jitter to prevent thundering herd:

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;

    public <T> T executeWithRetry(Supplier<T> operation) {
        int attempts = 0;
        while (attempts < MAX_RETRIES) {
            try {
                return operation.get();
            } catch (DatabaseConnectionException e) {
                attempts++;
                if (attempts >= MAX_RETRIES) throw e;
                
                long delay = calculateBackoffDelay(attempts);
                Thread.sleep(delay);
            }
        }
        throw new RuntimeException("Max retries exceeded");
    }

    private long calculateBackoffDelay(int attempt) {
        long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempt));
        long jitter = (long) (delay * JITTER_FACTOR * (Math.random() - 0.5) * 2);
        return delay + jitter;
    }
}

Circuit Breaker Design

The circuit breaker pattern prevents cascading failures in downstream services:

public class CircuitBreaker {
    private static final int FAILURE_THRESHOLD = 5;
    private static final long TIMEOUT_MS = 60000;
    private static final int RESET_TIMEOUT_MS = 30000;

    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private long lastFailureTime = 0;

    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }

    public <T> T execute(Supplier<T> operation) {
        switch (state) {
            case CLOSED:
                return executeWhenClosed(operation);
            case OPEN:
                if (System.currentTimeMillis() - lastFailureTime > RESET_TIMEOUT_MS) {
                    state = CircuitState.HALF_OPEN;
                    return executeWhenHalfOpen(operation);
                } else {
                    throw new CircuitBreakerOpenException();
                }
            case HALF_OPEN:
                return executeWhenHalfOpen(operation);
        }
        throw new IllegalStateException("Invalid circuit state");
    }

    private <T> T executeWhenClosed(Supplier<T> operation) {
        try {
            T result = operation.get();
            failureCount = 0; // Reset on success
            return result;
        } catch (Exception e) {
            failureCount++;
            lastFailureTime = System.currentTimeMillis();
            if (failureCount >= FAILURE_THRESHOLD) {
                state = CircuitState.OPEN;
            }
            throw e;
        }
    }

    private <T> T executeWhenHalfOpen(Supplier<T> operation) {
        try {
            T result = operation.get();
            state = CircuitState.CLOSED;
            return result;
        } catch (Exception e) {
            state = CircuitState.OPEN;
            throw e;
        }
    }
}

Fallback Behavior Design

Fallbacks are implemented for critical external dependencies:

public class ExternalServiceFallback {
    private final CircuitBreaker circuitBreaker;

    public String getTenantConfig(String tenantId) {
        return circuitBreaker.execute(() -> {
            // Primary operation
            return externalConfigService.fetchTenantConfig(tenantId);
        });
    }

    public String fallbackGetTenantConfig(String tenantId) {
        // Fallback logic - return default config or throw specific error
        return "default-config";
    }
}

Error Logging Format and Alerting Thresholds

Structured logging ensures consistent error tracking:

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);

    public void logError(String operation, Exception error, Map<String, Object> context) {
        logger.error(
            "Operation failed: {}, Error Code: {}, Message: {}, Context: {}",
            operation,
            error.getClass().getSimpleName(),
            error.getMessage(),
            context,
            error
        );
    }

    public void logErrorWithTrace(String operation, Exception error, String traceId) {
        logger.error(
            "Trace ID: {}, Operation: {}, Error Code: {}, Message: {}",
            traceId,
            operation,
            error.getClass().getSimpleName(),
            error.getMessage(),
            error
        );
    }
}

Alerting thresholds:

• Critical: >500 errors per minute for any endpoint
• Warning: >100 errors per minute for any endpoint
• Info: >50 errors per minute for any endpoint

Key Error Handling Patterns

1. Idempotency-aware error handling - Retryable errors are handled with proper idempotency keys
2. Graceful degradation - Fallbacks are provided for non-critical operations
3. Comprehensive logging - All errors include trace context and relevant metadata
4. Fail fast principle - Critical validation errors are caught early with clear messages

Error Response Format

All API responses include standardized error structures:

{
  "error": {
    "code": "TENANT_NOT_FOUND",
    "message": "Tenant with ID 'tenant-123' does not exist",
    "timestamp": "2023-10-15T14:30:00Z",
    "traceId": "abc123def456"
  }
}

This design ensures robust error handling that maintains system reliability while providing clear feedback to both internal systems and external consumers.

11. Security Design

Input Validation and Sanitization Strategy

All inputs are validated using a combination of:

• Library-based validation: Apache Commons Validator for email, phone, and URL formats
• Custom regex patterns for business-specific constraints (e.g., tenant ID format)
• Sanitization: HTML escaping using OWASP Java HTML Sanitizer for all user inputs to prevent XSS

SQL Injection Prevention

• Prepared statements: All database queries use parameterized prepared statements with JDBC (no string concatenation)
• ORM usage: Hibernate is used for all data access to ensure query parameterization
• Whitelisting: Query parameters are validated against a whitelist of allowed values where applicable

Authentication Middleware Design

• JWT-based authentication: JWT tokens are issued by Identity Provider (e.g., Auth0 or custom OAuth2 server)
• Token validation: Middleware validates token signature, expiration, and audience claims before each request
• Tenant context extraction: Tenant ID is extracted from JWT claims and stored in a ThreadLocal context for all subsequent operations

Authorization Enforcement

• RBAC model: Role-based access control with roles like TENANT_ADMIN, USER, ANALYST
• Permission checking: Service layer checks permissions before executing business logic using a centralized AuthorizationService
• Tenant isolation: All operations enforce tenant context to prevent cross-tenant access

Sensitive Data Encryption

• At rest: AES-256 encryption using AWS KMS for sensitive data fields (e.g., user PII, payment info)
• In transit: TLS 1.3 enforced for all external communications; internal services use mTLS (mutual TLS)
• Key rotation: Automatic key rotation every 90 days with audit trail

PII Handling and Masking Strategy

• Masking: Credit card numbers are masked (last 4 digits shown) in logs and responses
• Redaction: Full PII fields are redacted from logs, only hash values or IDs are logged
• Data retention: PII is retained for 3 years after user deactivation, then securely deleted

Secrets Management

• Vault integration: All secrets (API keys, database passwords) are stored in HashiCorp Vault
• Dynamic credentials: Database credentials are generated dynamically with short TTLs (1 hour)
• Access control: Vault policies enforce least privilege access per service and tenant

Audit Logging Design

• Event types: Login, data access, modification, deletion, configuration changes
• Log format:

{
  "timestamp": "2023-10-15T14:30:00Z",
  "tenantId": "tenant-123",
  "userId": "user-456",
  "action": "DATA_ACCESS",
  "resource": "/api/v1/users/123",
  "ipAddress": "192.168.1.100",
  "userAgent": "Mozilla/5.0...",
  "status": "SUCCESS"
}

• Retention: Audit logs retained for 2 years with daily backups to secure S3 bucket
• Alerting: Security events (failed logins, unauthorized access) trigger immediate alerts to security team

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

• Service Layer: 85% code coverage, focusing on business logic and edge cases
• Repository Layer: 90% coverage with mocked database interactions
• Domain Objects: 100% coverage for value objects and immutable entities
• Exception Handling: 100% coverage for all handled exceptions and error paths
• Validation Logic: 100% coverage for all validation rules and constraints

Integration Testing Strategy

• Database Integration: Use Testcontainers with PostgreSQL to test actual database interactions and schema compliance
• External Service Mocks: WireMock for third-party API integrations (e.g., payment gateways, notification services)
• Tenant Isolation: Validate multi-tenant data separation using embedded H2 databases with separate schemas per tenant
• Transaction Boundaries: Test transaction rollback and commit behaviors using embedded database transactions

Contract Testing

• API Contracts: Implement Pact-based contract testing between services to ensure API compatibility
• Event Schemas: Use Avro schema registry with contract testing for message format consistency across services
• Consumer-Provider Pairs: Define contracts for each external service integration (e.g., billing, analytics)

Performance Testing Tool and Key Scenarios

• Load Testing: Use k6 with 10,000 concurrent users simulating multi-tenant workflows
• Key Scenarios:
  • Tenant creation and data isolation under load
  • Concurrent user access to shared resources (e.g., global configuration)
  • Multi-tenant query performance (100 tenants, 1M records each)
  • Cache hit/miss ratio under high concurrent load
• Metrics Tracked:
  • p50/p95/p99 response times (target: <200ms for 95th percentile)
  • Throughput (1,000 RPS minimum)
  • Error rates (<0.1% failure rate)

Test Data Management Strategy

• Fixtures: Use JSON/YAML fixtures for static tenant configurations and global settings
• Factories: Implement factory classes (e.g., TenantFactory, UserFactory) for generating test data with realistic constraints
• Seeding: Use database migration scripts (e.g., Flyway) to seed test environments with multi-tenant data sets
• Isolation: Each test suite runs in isolated database schema or container to prevent cross-contamination between tests

// Example test data factory for multi-tenant user creation
public class TenantUserFactory {
    public static User createTenantUser(String tenantId, String email) {
        return User.builder()
            .id(UUID.randomUUID())
            .tenantId(tenantId)
            .email(email)
            .role("USER")
            .createdAt(Instant.now())
            .build();
    }
}

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective search, filtering, and analysis. The logging structure includes:

{
  "timestamp": "2023-10-15T14:30:00.000Z",
  "level": "INFO|WARN|ERROR",
  "service": "tenant-service",
  "trace_id": "1234567890abcdef1234567890abcdef",
  "span_id": "abcdef1234567890abcdef1234567890",
  "user_id": "user-12345",
  "tenant_id": "tenant-abcde",
  "request_id": "req-98765",
  "operation": "createTenant",
  "entity_id": "tenant-abcde",
  "message": "Tenant created successfully",
  "error_code": "TENANT_ALREADY_EXISTS",
  "error_stack": "...",
  "duration_ms": 123,
  "http_method": "POST",
  "http_path": "/v1/tenants",
  "http_status_code": 201,
  "payload_size_bytes": 1024
}

PII masking rules:

• User IDs and emails are masked in logs when not required for debugging.
• IP addresses are truncated to network prefix (e.g., 192.168.x.x).
• Sensitive fields like passwords or tokens are completely omitted.

Distributed Tracing Instrumentation

Using OpenTelemetry, we instrument key operations to trace end-to-end flows:

Custom Spans:

• tenant.create - from API entry to DB write
• tenant.update - for tenant metadata updates
• tenant.delete - for soft-delete and cleanup
• tenant.validate - for tenant validation logic
• tenant.permission.check - RBAC checks

Attributes Captured:

Span currentSpan = Span.current();
currentSpan.setAttribute("tenant.id", tenantId);
currentSpan.setAttribute("user.id", userId);
currentSpan.setAttribute("operation.type", "create");
currentSpan.setAttribute("is.multi.tenant", true);

Metrics Design

RED Method Metrics:

Metric Name	Type	Labels	Description
tenant.requests.total	Counter	{method, path, status}	Total request count
tenant.requests.duration	Histogram	{method, path}	Request latency in ms
tenant.requests.errors	Counter	{method, path, error_code}	Failed request count
tenant.active.tenants	Gauge	{environment}	Current active tenants
tenant.db.query.duration	Histogram	{query_type, tenant_id}	DB query time
tenant.cache.hit	Counter	{cache_type, operation}	Cache hit count
tenant.cache.miss	Counter	{cache_type, operation}	Cache miss count

Health Check Endpoint Design

@GetMapping("/health")
public ResponseEntity<HealthResponse> health() {
    HealthResponse response = new HealthResponse();
    response.setStatus("healthy");
    response.setTimestamp(Instant.now());
    response.setServices(Map.of(
        "database", databaseHealth(),
        "cache", cacheHealth(),
        "auth", authHealth()
    ));
    return ResponseEntity.ok(response);
}

public static class HealthResponse {
    private String status;
    private Instant timestamp;
    private Map<String, Object> services;
}

Endpoints:

• /health - overall service health (liveness)
• /ready - readiness check (dependencies ready)
• /live - service is running
• /startup - startup completion status

Alerting Thresholds

Metric	Warning Level	Critical Level	Escalation Path
tenant.requests.errors	> 5/minute	> 20/minute	On-call engineer
tenant.requests.duration.p95	> 500ms	> 2s	SRE team
tenant.db.query.duration.p95	> 100ms	> 500ms	DBA team
tenant.cache.hit.rate	< 70%	< 50%	Platform team
tenant.active.tenants	> 100k	> 200k	Platform team

Alerts are sent via PagerDuty with detailed context including trace IDs and error logs.

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• Tenant isolation strategy: Should tenants be completely isolated at database level, or use shared database with strict schema/permission boundaries? This decision impacts infrastructure costs, complexity, and security posture.
• Multi-tenancy scope: Should the system support cross-tenant operations (e.g., shared resources, cross-tenant analytics)? This affects data model design and API surface.
• Tenant onboarding workflow: Should tenants be provisioned automatically or require manual approval? This impacts system scalability and operational overhead.
• Tenant resource limits: Should we implement soft/hard limits per tenant (e.g., API rate limits, storage quotas)? This affects system fairness and cost control.

Known trade-offs and justifications

Trade-off	Option A	Option B	Justification
Database Isolation	Shared database with tenant IDs	Dedicated database per tenant	Shared database reduces infrastructure costs but increases risk of data leakage. Dedicated databases provide stronger isolation at higher cost.
Schema Design	Single schema with tenant_id	Separate schemas per tenant	Single schema simplifies maintenance but may impact performance and data privacy. Separate schemas provide better isolation but increase operational complexity.
Caching Strategy	Shared cache across tenants	Tenant-specific cache	Shared cache improves resource utilization but may cause cross-tenant cache pollution. Tenant-specific caches provide better isolation but increase memory usage.
Authentication	JWT-based with tenant context	OAuth2 with tenant-specific clients	JWTs simplify stateless authentication but require careful token management. OAuth2 provides better integration with external identity providers but increases complexity.

Risks and mitigation strategies

1. Data Leakage Risk

   • Risk: Cross-tenant data access due to improper filtering or shared resources
   • Mitigation: Implement strict tenant ID validation at every data access point, enforce RBAC policies, conduct regular security audits

2. Performance Degradation

   • Risk: High tenant density causing resource contention and slow response times
   • Mitigation: Implement rate limiting per tenant, use read replicas for heavy query loads, monitor per-tenant resource usage

3. Tenant Burst Capacity

   • Risk: Sudden spike in tenant activity causing system overload
   • Mitigation: Implement circuit breaker patterns, use auto-scaling groups, provision burst capacity with reserved instances

4. Audit Trail Integrity

   • Risk: Incomplete or tampered audit logs for multi-tenant environments
   • Mitigation: Use immutable logging systems, implement audit log signing, enforce write-once policies for logs

5. API Rate Limiting Bypass

   • Risk: Malicious tenants bypass rate limits through API misuse
   • Mitigation: Implement distributed rate limiting with Redis, enforce per-tenant quotas with circuit breakers

Technical debt acknowledged

1. Hardcoded tenant ID in queries: Temporary solution for rapid prototyping that requires refactoring to use tenant-aware repositories and service layers.

2. Shared infrastructure for sensitive tenants: Initial design uses shared infrastructure for all tenants, but sensitive tenant workloads may require dedicated resources.

3. Inconsistent error handling: Early implementation uses generic error responses that don't provide tenant-specific context for debugging.

4. Limited tenant metadata support: Current design doesn't support rich tenant metadata or custom attributes, which may be needed for advanced multi-tenancy features.

5. No tenant migration path: System doesn't support tenant data migration between different isolation strategies, requiring careful planning for future architecture evolution.