Feature Flag Platform Design: Scalable A/B Testing and Rollout Management

Comprehensive low-level design for a scalable feature flag platform supporting real-time flag management, A/B testing, and controlled rollouts across distributed systems.

PublishedAugust 9, 2026
Length8,634 words
Reading time44 min read

Backend LLD

A feature flag platform enables controlled feature deployments, A/B testing, and gradual rollouts while maintaining system stability and observability across distributed microservices architectures.

1. Problem Statement & Requirements

Clarifying questions

What scale and traffic expectations are we targeting?
- How many feature flags are expected to be managed (e.g., 10K, 100K, 1M+)?
- What is the expected concurrent evaluation rate (e.g., 10K requests/sec, 100K+)?
What consistency guarantees are required for feature flag evaluation?
- Should flags be evaluated with strong consistency or eventual consistency?
- Are there specific use cases where latency is more important than immediate consistency (e.g., A/B testing)?
What are the key entities and their relationships?
- Are feature flags tied to user IDs, groups, or environments (e.g., dev, staging, prod)?
- Do flags support targeting rules (e.g., by user ID, cohort, or percentage of users)?
What is the read/write ratio for feature flags?
- Is flag evaluation more frequent than flag creation/modification (e.g., 100:1 ratio)?
- How often are flags updated vs. evaluated in real-time?
What latency sensitivity and SLA requirements exist?
- What is the maximum acceptable latency for flag evaluation (e.g., <5ms p99)?
- Are there specific SLAs for flag availability (e.g., 99.9% uptime)?
What authentication and authorization model is expected?
- Will the platform support OAuth2, JWTs, or API keys for admin access?
- Should flag evaluation be authenticated and authorized by user context?
Are there external integrations or dependencies (e.g., analytics, A/B testing tools)?
- Is there a need to integrate with external systems like Segment or Amplitude for flag usage tracking?
- Should the platform support webhooks or event streams for real-time flag updates?

Functional requirements

Flag Management
- Create, read, update, and delete feature flags with metadata (name, description, status).
Flag Targeting
- Support targeting flags to specific user IDs, groups, or percentage of users.
Flag Evaluation
- Evaluate flags in real-time for a given user or context with low latency.
Environment Management
- Support multiple environments (dev, staging, prod) with isolated flag configurations.
Flag Versioning
- Maintain version history for flags to support rollbacks and audit trails.
Flag Metrics
- Track usage of flags for analytics and A/B testing purposes.
Flag Rollout Strategy
- Support gradual rollouts (e.g., 10% of users, then 50%, etc.).
User Context Support
- Evaluate flags based on user attributes (e.g., user ID, email, region).

Non-functional requirements

Requirement	Target
P50 latency for flag evaluation	< 2ms
P95 latency for flag evaluation	< 5ms
P99 latency for flag evaluation	< 10ms
Throughput (flag evaluations/sec)	> 50,000
Availability	99.9% (2.5 min downtime/month)
Flag update propagation delay	< 100ms (for real-time environments)
Concurrent flag evaluations	Support 100,000+ concurrent users
Data durability	Flags must be persisted with no data loss
Audit logging	All flag changes must be logged with user, timestamp, and diff

Constraints and assumptions

Assumption: Feature flags are evaluated in high-volume, low-latency environments (e.g., web/mobile app backends).
Assumption: Flag evaluation logic is deterministic and fast.
Constraint: The system must support real-time flag updates with minimal propagation delay.
Constraint: Flag data must be highly available and durable, with no single point of failure.
Assumption: Flag rules are not expected to change more than once per second for any given flag.
Constraint: Flag evaluation must not block or significantly impact the performance of the calling application.

Out of scope

Flag deployment automation (e.g., CI/CD integration)
UI dashboard for flag management (only API is in scope)
Multi-region replication or cross-cloud support
Advanced analytics or A/B testing tooling (e.g., statistical significance)
Flag rollout scheduling (e.g., scheduled rollouts or gradual rollouts with time-based triggers)
Flag health monitoring or auto-healing (e.g., flag evaluation failure detection)
Support for complex targeting logic beyond simple user ID or percentage (e.g., regex matching, custom attributes)

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for the Feature Flag platform due to:

Strong typing and schema evolution support
Efficient binary serialization with smaller payload sizes
Built-in support for streaming and bidirectional communication
Native support for versioning through protobuf packages
Better performance characteristics for high-throughput use cases

Endpoint Definitions

Method	Path	Description	Auth
CreateFeatureFlag	`POST /v1/feature-flags`	Creates a new feature flag with configuration	JWT + RBAC
GetFeatureFlag	`GET /v1/feature-flags/{id}`	Retrieves a specific feature flag by ID	JWT + RBAC
UpdateFeatureFlag	`PUT /v1/feature-flags/{id}`	Updates an existing feature flag configuration	JWT + RBAC
DeleteFeatureFlag	`DELETE /v1/feature-flags/{id}`	Soft-deletes a feature flag	JWT + RBAC
ListFeatureFlags	`GET /v1/feature-flags`	Lists feature flags with optional filtering	JWT + RBAC
EvaluateFlag	`POST /v1/feature-flags/evaluate`	Evaluates whether a feature flag is enabled for a user	JWT + RBAC
GetFlagEvaluations	`GET /v1/feature-flags/{id}/evaluations`	Retrieves evaluation history for a flag	JWT + RBAC
BulkEvaluateFlags	`POST /v1/feature-flags/bulk-evaluate`	Evaluates multiple flags for a user	JWT + RBAC

Request/Response Schemas

CreateFeatureFlagRequest

{
  "name": "experiment-2024",
  "description": "New experiment for 2024",
  "enabled": true,
  "rolloutPercentage": 50,
  "targetingRules": [
    {
      "userIds": ["user-1", "user-2"],
      "groups": ["beta-users"],
      "country": "US",
      "region": "CA"
    }
  ],
  "variants": [
    {
      "name": "control",
      "weight": 50,
      "payload": {"message": "Hello World"}
    },
    {
      "name": "variant-a",
      "weight": 50,
      "payload": {"message": "Hello Variant A"}
    }
  ],
  "tags": ["marketing", "experiment"],
  "owner": "user@example.com"
}

CreateFeatureFlagResponse

{
  "id": "ff-1234567890",
  "name": "experiment-2024",
  "description": "New experiment for 2024",
  "enabled": true,
  "rolloutPercentage": 50,
  "targetingRules": [...],
  "variants": [...],
  "tags": ["marketing", "experiment"],
  "owner": "user@example.com",
  "createdAt": "2024-01-15T10:30:00Z",
  "updatedAt": "2024-01-15T10:30:00Z"
}

EvaluateFlagRequest

{
  "flagId": "ff-1234567890",
  "userId": "user-123",
  "context": {
    "country": "US",
    "region": "CA",
    "deviceType": "mobile"
  }
}

EvaluateFlagResponse

{
  "flagId": "ff-1234567890",
  "enabled": true,
  "variant": {
    "name": "control",
    "weight": 50,
    "payload": {"message": "Hello World"}
  },
  "evaluationTimestamp": "2024-01-15T10:30:00Z"
}

HTTP Status Codes and Error Response Format

Code	Description
200	Success
201	Created
400	Bad Request - Invalid input
401	Unauthorized
403	Forbidden
404	Not Found
429	Rate Limited
500	Internal Server Error

Standardized Error Response Format

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid input provided",
    "details": [
      {
        "field": "name",
        "message": "Name is required"
      }
    ],
    "timestamp": "2024-01-15T10:30:00Z",
    "requestId": "req-1234567890"
  }
}

API Versioning Strategy

Versioned using path prefix: /v1/feature-flags
Future versions will use /v2/feature-flags with backward compatibility guarantees for major version changes.

Pagination Design

Cursor-based pagination is used because:

More stable and predictable than offset-based pagination
Better performance with large datasets
Handles data changes during pagination gracefully

Response includes nextCursor field for continuation.

Authentication and Authorization

Authentication: JWT tokens with OAuth2 integration
Authorization: Role-based access control (RBAC) with scopes:

feature-flag:read - Read access to feature flags
feature-flag:write - Write access to feature flags
feature-flag:evaluate - Evaluate feature flags for users

Rate Limiting

Per Endpoint Rate Limits:

CreateFeatureFlag: 100 req/min
EvaluateFlag: 1000 req/sec (per user)
BulkEvaluateFlags: 50 req/sec (per user)

Enforcement Strategy:

Token bucket algorithm with sliding window
Distributed rate limiting using Redis

Idempotency Design

Idempotency Key Strategy:

Clients provide X-Idempotency-Key header
Server stores operation result for key duration (5 minutes)
Duplicate requests with same key return cached result
Used primarily for create/update operations

@PostMapping("/v1/feature-flags")
public ResponseEntity<FeatureFlag> createFeatureFlag(
    @RequestHeader("X-Idempotency-Key") String idempotencyKey,
    @RequestBody CreateFeatureFlagRequest request) {
    
    // Use idempotency key to check for existing operation
    FeatureFlag result = featureFlagService.createFeatureFlag(request, idempotencyKey);
    
    return ResponseEntity.status(HttpStatus.CREATED).body(result);
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    FEATURE_FLAG {
        string id PK
        string name
        string description
        boolean enabled
        string owner
        timestamp created_at
        timestamp updated_at
        string version
    }

    FEATURE_FLAG_VARIATION {
        string id PK
        string flag_id FK
        string name
        string description
        float percentage
        json config
        timestamp created_at
        timestamp updated_at
    }

    FEATURE_FLAG_TARGET {
        string id PK
        string flag_id FK
        string user_id
        string variation_id FK
        timestamp created_at
        timestamp updated_at
    }

    FEATURE_FLAG_SEGMENT {
        string id PK
        string flag_id FK
        string segment_name
        timestamp created_at
        timestamp updated_at
    }

    FEATURE_FLAG_SEGMENT_RULE {
        string id PK
        string segment_id FK
        string attribute
        string operator
        string value
        timestamp created_at
        timestamp updated_at
    }

    FEATURE_FLAG_SEGMENT_RULE_CONDITION {
        string id PK
        string rule_id FK
        string attribute
        string operator
        string value
        timestamp created_at
        timestamp updated_at
    }

    FEATURE_FLAG_ENVIRONMENT {
        string id PK
        string flag_id FK
        string environment_name
        boolean enabled
        timestamp created_at
        timestamp updated_at
    }

    FEATURE_FLAG_ENVIRONMENT_VARIATION {
        string id PK
        string environment_id FK
        string variation_id FK
        timestamp created_at
        timestamp updated_at
    }

    FEATURE_FLAG_ENVIRONMENT_TARGET {
        string id PK
        string environment_id FK
        string user_id
        string variation_id FK
        timestamp created_at
        timestamp updated_at
    }

    FEATURE_FLAG_HISTORY {
        string id PK
        string flag_id FK
        string action
        json changes
        string user_id
        timestamp created_at
    }

    FEATURE_FLAG {
        id ||--o{ FEATURE_FLAG_VARIATION : "has"
        id ||--o{ FEATURE_FLAG_TARGET : "has"
        id ||--o{ FEATURE_FLAG_SEGMENT : "has"
        id ||--o{ FEATURE_FLAG_ENVIRONMENT : "has"
        id ||--o{ FEATURE_FLAG_HISTORY : "has"
    }

    FEATURE_FLAG_SEGMENT {
        id ||--o{ FEATURE_FLAG_SEGMENT_RULE : "has"
    }

    FEATURE_FLAG_SEGMENT_RULE {
        id ||--o{ FEATURE_FLAG_SEGMENT_RULE_CONDITION : "has"
    }

    FEATURE_FLAG_ENVIRONMENT {
        id ||--o{ FEATURE_FLAG_ENVIRONMENT_VARIATION : "has"
        id ||--o{ FEATURE_FLAG_ENVIRONMENT_TARGET : "has"
    }

Table Schema Definitions

-- Feature Flag table
CREATE TABLE feature_flags (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL UNIQUE,
    description TEXT,
    enabled BOOLEAN DEFAULT FALSE,
    owner VARCHAR(255),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    version VARCHAR(50) DEFAULT '1.0'
);

-- Feature Flag Variations table
CREATE TABLE feature_flag_variations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    flag_id UUID REFERENCES feature_flags(id) ON DELETE CASCADE,
    name VARCHAR(255) NOT NULL,
    description TEXT,
    percentage FLOAT CHECK (percentage >= 0 AND percentage <= 100),
    config JSONB DEFAULT '{}',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Feature Flag Targeting table (per-user targeting)
CREATE TABLE feature_flag_targets (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    flag_id UUID REFERENCES feature_flags(id) ON DELETE CASCADE,
    user_id VARCHAR(255) NOT NULL,
    variation_id UUID REFERENCES feature_flag_variations(id),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(flag_id, user_id)
);

-- Feature Flag Segments table
CREATE TABLE feature_flag_segments (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    flag_id UUID REFERENCES feature_flags(id) ON DELETE CASCADE,
    segment_name VARCHAR(255) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Feature Flag Segment Rules table
CREATE TABLE feature_flag_segment_rules (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    segment_id UUID REFERENCES feature_flag_segments(id) ON DELETE CASCADE,
    attribute VARCHAR(255) NOT NULL,
    operator VARCHAR(50) NOT NULL, -- 'equals', 'contains', 'startsWith', etc.
    value TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Feature Flag Segment Rule Conditions table (for complex conditions)
CREATE TABLE feature_flag_segment_rule_conditions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    rule_id UUID REFERENCES feature_flag_segment_rules(id) ON DELETE CASCADE,
    attribute VARCHAR(255) NOT NULL,
    operator VARCHAR(50) NOT NULL, -- 'equals', 'contains', etc.
    value TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Feature Flag Environments table (per environment config)
CREATE TABLE feature_flag_environments (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    flag_id UUID REFERENCES feature_flags(id) ON DELETE CASCADE,
    environment_name VARCHAR(255) NOT NULL,
    enabled BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(flag_id, environment_name)
);

-- Feature Flag Environment Variations table (per env variation config)
CREATE TABLE feature_flag_environment_variations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    environment_id UUID REFERENCES feature_flag_environments(id) ON DELETE CASCADE,
    variation_id UUID REFERENCES feature_flag_variations(id),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Feature Flag Environment Targets table (per env user targeting)
CREATE TABLE feature_flag_environment_targets (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    environment_id UUID REFERENCES feature_flag_environments(id) ON DELETE CASCADE,
    user_id VARCHAR(255) NOT NULL,
    variation_id UUID REFERENCES feature_flag_variations(id),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(environment_id, user_id)
);

-- Feature Flag History table (audit trail)
CREATE TABLE feature_flag_history (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    flag_id UUID REFERENCES feature_flags(id) ON DELETE CASCADE,
    action VARCHAR(50) NOT NULL, -- 'created', 'updated', 'deleted'
    changes JSONB,
    user_id VARCHAR(255),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Primary Key Strategy

For primary keys, we use UUIDs for the following reasons:

Distributed system compatibility: UUIDs are globally unique across services and databases, avoiding conflicts in distributed environments.
No sequential exposure: Unlike auto-incrementing IDs, UUIDs don't expose system information like creation order or internal counters.
Scalability: UUIDs support horizontal scaling without coordination between systems.

Index Design

-- Feature Flags table indexes
CREATE INDEX idx_feature_flags_name ON feature_flags(name);
CREATE INDEX idx_feature_flags_owner ON feature_flags(owner);
CREATE INDEX idx_feature_flags_created_at ON feature_flags(created_at DESC);

-- Feature Flag Variations table indexes
CREATE INDEX idx_feature_flag_variations_flag_id ON feature_flag_variations(flag_id);
CREATE INDEX idx_feature_flag_variations_flag_id_name ON feature_flag_variations(flag_id, name);

-- Feature Flag Targets table indexes
CREATE INDEX idx_feature_flag_targets_flag_id ON feature_flag_targets(flag_id);
CREATE INDEX idx_feature_flag_targets_user_id ON feature_flag_targets(user_id);
CREATE INDEX idx_feature_flag_targets_flag_user ON feature_flag_targets(flag_id, user_id);

-- Feature Flag Segments table indexes
CREATE INDEX idx_feature_flag_segments_flag_id ON feature_flag_segments(flag_id);
CREATE INDEX idx_feature_flag_segments_segment_name ON feature_flag_segments(segment_name);

-- Feature Flag Segment Rules table indexes
CREATE INDEX idx_feature_flag_segment_rules_segment_id ON feature_flag_segment_rules(segment_id);
CREATE INDEX idx_feature_flag_segment_rules_attribute_operator ON feature_flag_segment_rules(attribute, operator);

-- Feature Flag Environment table indexes
CREATE INDEX idx_feature_flag_environments_flag_id ON feature_flag_environments(flag_id);
CREATE INDEX idx_feature_flag_environments_environment_name ON feature_flag_environments(environment_name);

-- Feature Flag Environment Variations table indexes
CREATE INDEX idx_feature_flag_env_variations_env_id ON feature_flag_environment_variations(environment_id);
CREATE INDEX idx_feature_flag_env_variations_variation_id ON feature_flag_environment_variations(variation_id);

-- Feature Flag History table indexes
CREATE INDEX idx_feature_flag_history_flag_id ON feature_flag_history(flag_id);
CREATE INDEX idx_feature_flag_history_created_at ON feature_flag_history(created_at DESC);

Data Partitioning / Sharding Strategy

For large-scale deployments, we implement horizontal sharding:

Shard by flag name prefix: Distribute feature flags across shards based on the first character of their name (e.g., A-K, L-Z).
Shard by environment: Separate environments into different database clusters or schemas.
Shard by user ID hash: For targeting tables, we can shard by a hash of the user ID to distribute load.

Soft Delete vs. Hard Delete

We implement soft deletes for feature flags and their associated entities to:

Preserve audit trails: Historical changes are maintained for compliance.
Support recovery: Allow restoration of deleted flags and their configurations.
Maintain referential integrity: Prevent orphaned records in related tables.

-- Add soft delete columns to tables
ALTER TABLE feature_flags ADD COLUMN deleted_at TIMESTAMP DEFAULT NULL;
ALTER TABLE feature_flag_variations ADD COLUMN deleted_at TIMESTAMP DEFAULT NULL;
ALTER TABLE feature_flag_targets ADD COLUMN deleted_at TIMESTAMP DEFAULT NULL;
ALTER TABLE feature_flag_segments ADD COLUMN deleted_at TIMESTAMP DEFAULT NULL;
ALTER TABLE feature_flag_segment_rules ADD COLUMN deleted_at TIMESTAMP DEFAULT NULL;
ALTER TABLE feature_flag_segment_rule_conditions ADD COLUMN deleted_at TIMESTAMP DEFAULT NULL;
ALTER TABLE feature_flag_environments ADD COLUMN deleted_at TIMESTAMP DEFAULT NULL;
ALTER TABLE feature_flag_environment_variations ADD COLUMN deleted_at TIMESTAMP DEFAULT NULL;
ALTER TABLE feature_flag_environment_targets ADD COLUMN deleted_at TIMESTAMP DEFAULT NULL;

Audit Trail Design

Each modification to a feature flag is logged in the feature_flag_history table with:

Action: Type of operation (create, update, delete)
Changes: JSON containing the diff between old and new values
User ID: Identifier of who made the change (from auth context)
Timestamp: When the operation occurred

Data Archival Strategy and Retention Policy

Archival: Flags that are inactive for > 90 days can be archived to a separate storage (e.g., S3 or cold database).
Retention: Keep full history for 1 year, then archive to long-term storage.
Purge: Completely remove flags and their data after 5 years of inactivity.

-- Example retention policy trigger (pseudo-code)
CREATE OR REPLACE FUNCTION archive_inactive_flags()
RETURNS TRIGGER AS $$
BEGIN
    IF OLD.updated_at < CURRENT_DATE - INTERVAL '90 days' THEN
        -- Move to archive table or S3 storage
        INSERT INTO feature_flag_archives SELECT * FROM feature_flags WHERE id = OLD.id;
        DELETE FROM feature_flags WHERE id = OLD.id;
    END IF;
    RETURN NULL;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trigger_archive_inactive_flags
    AFTER UPDATE ON feature_flags
    FOR EACH ROW EXECUTE FUNCTION archive_inactive_flags();

4. Class Design & Service Layer

UML Class Diagram

classDiagram
    class FeatureFlag {
        -String id
        -String name
        -String description
        -Boolean enabled
        -Map~String, Object~ configuration
        -DateTime createdAt
        -DateTime updatedAt
        -String createdBy
        -String updatedBy
        +isEnabled(userContext) Boolean
        +evaluate(userContext) EvaluationResult
    }

    class FeatureFlagVariant {
        -String id
        -String flagId
        -String name
        -Double weight
        -Map~String, Object~ configuration
        -Boolean enabled
    }

    class UserContext {
        -String userId
        -Map~String, String~ attributes
        -String environment
    }

    class EvaluationResult {
        -Boolean enabled
        -String variantId
        -Map~String, Object~ payload
    }

    class FeatureFlagService {
        +createFlag(flag) FeatureFlag
        +updateFlag(flagId, update) FeatureFlag
        +deleteFlag(flagId) void
        +getFlag(flagId) FeatureFlag
        +getAllFlags() List~FeatureFlag~
        +evaluateFlag(flagId, userContext) EvaluationResult
        +isEnabled(flagId, userContext) Boolean
        +getFlagVariants(flagId) List~FeatureFlagVariant~
        +createVariant(variant) FeatureFlagVariant
        +updateVariant(variantId, update) FeatureFlagVariant
        +deleteVariant(variantId) void
    }

    class FeatureFlagRepository {
        +save(flag) FeatureFlag
        +findById(id) Optional~FeatureFlag~
        +findAll() List~FeatureFlag~
        +deleteById(id) void
        +findVariantsByFlagId(flagId) List~FeatureFlagVariant~
    }

    class FeatureFlagEvaluator {
        +evaluate(flag, userContext) EvaluationResult
    }

    class Strategy {
        +evaluate(flag, userContext) EvaluationResult
    }

    class UserAttributeStrategy {
        +evaluate(flag, userContext) EvaluationResult
    }

    class PercentageStrategy {
        +evaluate(flag, userContext) EvaluationResult
    }

    class RolloutStrategy {
        -String strategyType
        -Map~String, Object~ config
    }

    class FeatureFlagEventPublisher {
        +publish(event) void
    }

    class Event {
        -String eventType
        -String flagId
        -DateTime timestamp
        -Map~String, Object~ payload
    }

    FeatureFlagService --> FeatureFlagRepository : uses
    FeatureFlagService --> FeatureFlagEvaluator : uses
    FeatureFlagService --> FeatureFlagEventPublisher : publishes
    FeatureFlagEvaluator --> Strategy : delegates to
    Strategy <|-- UserAttributeStrategy : implements
    Strategy <|-- PercentageStrategy : implements
    FeatureFlagService --> RolloutStrategy : uses

State Machine Diagram for Feature Flag

stateDiagram-v2
    [*] --> Disabled
    Disabled --> Enabled : enable_flag
    Enabled --> Disabled : disable_flag
    Enabled --> Enabled : update_configuration
    Disabled --> Disabled : update_configuration

Service Class Structure and Responsibilities

The service layer is organized around the core domain entities with clear separation of concerns:

FeatureFlagService - Main orchestrator for feature flag operations, handles business logic and validation
FeatureFlagRepository - Data access abstraction for feature flags and variants
FeatureFlagEvaluator - Core evaluation logic with strategy pattern implementation
RolloutStrategy - Strategy interface for different flag evaluation methods
FeatureFlagEventPublisher - Event publishing for audit and monitoring

Interface / Abstract Class Design

public interface Strategy {
    EvaluationResult evaluate(FeatureFlag flag, UserContext userContext);
}

public abstract class RolloutStrategy {
    protected String strategyType;
    protected Map<String, Object> config;

    public RolloutStrategy(String strategyType, Map<String, Object> config) {
        this.strategyType = strategyType;
        this.config = config;
    }

    public abstract EvaluationResult evaluate(FeatureFlag flag, UserContext userContext);
}

Business Logic Encapsulation Patterns

The Strategy pattern is used for implementing different evaluation methods:

public class UserAttributeStrategy extends RolloutStrategy implements Strategy {
    public UserAttributeStrategy(Map<String, Object> config) {
        super("user_attribute", config);
    }

    @Override
    public EvaluationResult evaluate(FeatureFlag flag, UserContext userContext) {
        String attributeKey = (String) config.get("attribute");
        String attributeValue = (String) config.get("value");
        
        if (userContext.getAttributes().containsKey(attributeKey)) {
            String userAttributeValue = userContext.getAttributes().get(attributeKey);
            if (userAttributeValue.equals(attributeValue)) {
                return new EvaluationResult(true, null, null);
            }
        }
        
        return new EvaluationResult(false, null, null);
    }
}

public class PercentageStrategy extends RolloutStrategy implements Strategy {
    public PercentageStrategy(Map<String, Object> config) {
        super("percentage", config);
    }

    @Override
    public EvaluationResult evaluate(FeatureFlag flag, UserContext userContext) {
        Integer percentage = (Integer) config.get("percentage");
        String userId = userContext.getUserId();
        
        if (userId == null) {
            return new EvaluationResult(false, null, null);
        }
        
        int hash = userId.hashCode();
        int normalizedHash = Math.abs(hash) % 100;
        
        if (normalizedHash < percentage) {
            return new EvaluationResult(true, null, null);
        }
        
        return new EvaluationResult(false, null, null);
    }
}

Critical-Path Method Implementation

@Transactional
public EvaluationResult evaluateFlag(String flagId, UserContext userContext) {
    FeatureFlag flag = featureFlagRepository.findById(flagId)
        .orElseThrow(() -> new FeatureFlagNotFoundException("Flag not found: " + flagId));
    
    // Check if flag is enabled globally
    if (!flag.isEnabled()) {
        return new EvaluationResult(false, null, null);
    }
    
    // Evaluate flag variants with their strategies
    List<FeatureFlagVariant> variants = featureFlagRepository.findVariantsByFlagId(flagId);
    
    // Apply strategies in order of precedence
    EvaluationResult result = null;
    for (FeatureFlagVariant variant : variants) {
        if (variant.isEnabled()) {
            // Strategy evaluation logic here
            String strategyType = variant.getStrategyType();
            Map<String, Object> strategyConfig = variant.getStrategyConfig();
            
            Strategy strategy = strategyFactory.createStrategy(strategyType, strategyConfig);
            EvaluationResult variantResult = strategy.evaluate(flag, userContext);
            
            if (variantResult.isEnabled()) {
                result = new EvaluationResult(true, variant.getId(), variant.getConfiguration());
                break;
            }
        }
    }
    
    // If no specific variant matches, check default behavior
    if (result == null) {
        result = new EvaluationResult(true, null, null);
    }
    
    // Publish evaluation event
    FeatureFlagEvent event = new FeatureFlagEvent(
        "flag_evaluated",
        flagId,
        userContext.getUserId(),
        result
    );
    
    eventPublisher.publish(event);
    
    return result;
}

Transaction Boundary Design

Transaction boundaries are defined at the service layer level:

Create/Update/Delete operations - Single transaction to ensure atomicity
Evaluation operations - Read-only, no transaction required (can be made read-only)
Batch operations - Individual transactions per operation with appropriate error handling

Service Interface Contracts

public interface FeatureFlagService {
    FeatureFlag createFlag(FeatureFlag flag);
    FeatureFlag updateFlag(String flagId, FeatureFlagUpdateRequest update);
    void deleteFlag(String flagId);
    FeatureFlag getFlag(String flagId);
    List<FeatureFlag> getAllFlags();
    EvaluationResult evaluateFlag(String flagId, UserContext userContext);
    Boolean isEnabled(String flagId, UserContext userContext);
    List<FeatureFlagVariant> getFlagVariants(String flagId);
    FeatureFlagVariant createVariant(FeatureFlagVariant variant);
    FeatureFlagVariant updateVariant(String variantId, FeatureFlagVariantUpdateRequest update);
    void deleteVariant(String variantId);
}

Saga / Orchestration Pattern

For distributed operations involving external systems:

Flag creation with external sync: Use saga pattern for coordination across multiple services
Event-driven propagation: Use event sourcing for audit trail and cross-service consistency

Domain Object / Value Object Design

public class FeatureFlag {
    private String id;
    private String name;
    private String description;
    private Boolean enabled = false;
    private Map<String, Object> configuration;
    private LocalDateTime createdAt;
    private LocalDateTime updatedAt;
    private String createdBy;
    private String updatedBy;

    // Getters and setters
}

public class FeatureFlagVariant {
    private String id;
    private String flagId;
    private String name;
    private Double weight;
    private Map<String, Object> configuration;
    private Boolean enabled = false;
    private String strategyType;
    private Map<String, Object> strategyConfig;
}

public class UserContext {
    private String userId;
    private Map<String, String> attributes;
    private String environment;

    // Getters and setters
}

public class EvaluationResult {
    private Boolean enabled;
    private String variantId;
    private Map<String, Object> payload;

    // Getters and setters
}

5. Concurrency & Thread Safety

Concurrency Model

The feature flag platform will use a thread-safe, asynchronous processing model with Java's CompletableFuture and ExecutorService for handling concurrent flag evaluations. The system will leverage a fixed thread pool to manage concurrent requests efficiently without overwhelming the underlying infrastructure.

Race Condition Prevention Strategy

Race conditions in feature flag evaluation are mitigated through:

Immutable flag state: All flag configurations are immutable once created, preventing concurrent modification issues
Atomic flag evaluation: Each flag evaluation operation is atomic and does not modify shared state
Read-Write Lock pattern: For flag configuration updates, a ReadWriteLock ensures that concurrent reads don't block each other while maintaining consistency during writes

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected for flag configuration updates because:

Feature flags are evaluated frequently (high read ratio)
Write operations are infrequent and typically involve small data changes
Reduces contention and improves throughput for concurrent flag evaluations
Implements version-based conflict detection using ETags or version numbers

Deadlock Prevention Strategy

Deadlock prevention is achieved through:

Fixed ordering of operations: All flag evaluation and configuration updates follow a consistent order
Timeout-based acquisition: Locks are acquired with timeouts to prevent indefinite blocking
Hierarchical lock management: Configuration locks are acquired before evaluation locks when needed

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is implemented for flag configuration operations using:

Idempotency keys: Generated from client request metadata (user ID + feature flag key + timestamp)
Cache-based deduplication: Concurrent requests with same idempotency key are cached and return the same result
Database-level uniqueness constraints: Prevents duplicate configuration creation

public class FeatureFlagService {
    private final ReadWriteLock configLock = new ReentrantReadWriteLock();
    private final Map<String, String> idempotencyCache = new ConcurrentHashMap<>();
    
    public FeatureFlagConfig updateFlag(String flagKey, FeatureFlagConfig config) {
        String idempotencyKey = generateIdempotencyKey(flagKey, config);
        
        // Check if this operation was already processed
        if (idempotencyCache.containsKey(idempotencyKey)) {
            return idempotencyCache.get(idempotencyKey);
        }
        
        // Acquire write lock for configuration update
        configLock.writeLock().lock();
        try {
            // Validate and update flag configuration
            FeatureFlagConfig updated = configRepository.save(config);
            
            // Cache the result for idempotency
            idempotencyCache.put(idempotencyKey, updated);
            
            return updated;
        } finally {
            configLock.writeLock().unlock();
        }
    }
    
    private String generateIdempotencyKey(String flagKey, FeatureFlagConfig config) {
        return "flag:" + flagKey + ":" + 
               config.getVersion() + ":" + 
               System.currentTimeMillis();
    }
}

Thread Safety in Flag Evaluation

Flag evaluation operations are designed to be thread-safe:

Immutable flag data structures: All flag configurations and targeting rules are immutable
Thread-local evaluation context: Each evaluation thread maintains its own context without sharing state
Synchronized access to volatile data: For any runtime flag state that may change, synchronized access is used

Lock Granularity and Performance Trade-offs

The system uses fine-grained locking:

Per-flag configuration locks for updates to minimize contention
Read locks allow concurrent reads while maintaining consistency
Batch operations for bulk flag updates to reduce lock contention

This approach balances high throughput with consistency guarantees, ensuring that concurrent flag evaluations don't block each other while maintaining data integrity for configuration changes.

6. Sequence Diagrams for Key Flows

Happy Path Sequence Diagram

sequenceDiagram
    participant Client as Feature Flag Client SDK
    participant Gateway as Feature Flag Gateway/Service
    participant Cache as Redis Cache
    participant DB as Database (PostgreSQL)
    
    Client->>Gateway: Evaluate flag with key "feature-x"
    Gateway->>Cache: Check if flag exists in cache
    Cache-->>Gateway: Cache miss
    Gateway->>DB: Fetch flag configuration
    DB-->>Gateway: Return flag config
    Gateway->>Cache: Store flag in cache
    Cache-->>Gateway: Cache set
    Gateway-->>Client: Return evaluated flag value (true/false)

Error / Edge Case Sequence Diagram

sequenceDiagram
    participant Client as Feature Flag Client SDK
    participant Gateway as Feature Flag Gateway/Service
    participant Cache as Redis Cache
    participant DB as Database (PostgreSQL)
    
    Client->>Gateway: Evaluate flag with key "non-existent-flag"
    Gateway->>Cache: Check if flag exists in cache
    Cache-->>Gateway: Cache miss
    Gateway->>DB: Fetch flag configuration
    DB-->>Gateway: Return error (flag not found)
    Gateway->>Client: Return default value or error response

Async Flow Sequence Diagram

sequenceDiagram
    participant Client as Feature Flag Client SDK
    participant Gateway as Feature Flag Gateway/Service
    participant EventBus as Message Broker (Kafka)
    participant DB as Database (PostgreSQL)
    
    Client->>Gateway: Update flag configuration
    Gateway->>DB: Persist updated flag config
    DB-->>Gateway: Acknowledge write
    Gateway->>EventBus: Publish flag updated event
    EventBus-->>Gateway: Event acknowledged
    Gateway-->>Client: Return success response

7. Caching Design

Cache Strategy Selection

For feature flags, we'll use a cache-aside pattern with a write-through strategy for the primary cache layer. This ensures that all reads go through the cache and that writes update both the database and cache in a consistent manner.

Cache Key Design

Cache keys will follow this namespace structure:

feature_flag:{flag_key}:{user_id}:{context_hash}

For global flags (no user/context specific):

feature_flag:{flag_key}:global

For environment-specific flags:

feature_flag:{flag_key}:{environment_id}:global

TTL Strategy

Data Type	TTL	Reason
Feature Flag Configurations	5 minutes	Frequent updates during feature releases
User Flag Evaluations	1 minute	High volatility in user-specific flags
Environment Flags	10 minutes	Less frequent environment changes
Flag Metadata	30 minutes	Stable metadata, infrequent updates

Cache Eviction Policy

We'll use LRU (Least Recently Used) eviction policy with a maximum cache size of 100,000 entries. This ensures that the most frequently accessed feature flags remain in cache while evicting rarely used ones.

Cache Stampede Prevention

To prevent cache stampedes:

Probabilistic Early Expiry: Cache entries will be invalidated 20% before their TTL
Distributed Locks: For high-traffic flags, we'll use Redis distributed locks during cache rebuilds
Cache warming: Pre-warm popular flags during low-traffic periods

Distributed Cache Architecture

We'll use Redis Cluster with:

3 master nodes for high availability
3 replica nodes per master for read scaling
Automatic failover with Redis Sentinel monitoring
Cache sharding based on flag key hash

Cache Invalidation Strategy

We'll implement event-driven invalidation:

When a feature flag is updated, publish an event to the flag.updated topic
All service instances listen to this topic and invalidate relevant cache entries
For immediate consistency, we'll use a cache invalidation queue with retry logic

@Component
public class FeatureFlagCacheInvalidator {
    private final RedisTemplate<String, Object> redisTemplate;
    private final ObjectMapper objectMapper;

    @EventListener
    public void handleFlagUpdated(FeatureFlagUpdatedEvent event) {
        // Invalidate all cache entries for this flag
        String globalKey = "feature_flag:" + event.getFlagKey() + ":global";
        String userKeyPattern = "feature_flag:" + event.getFlagKey() + ":*";
        
        // Use Redis SCAN to find and invalidate matching keys
        Set<String> keysToInvalidate = new HashSet<>();
        try (Cursor<Map.Entry<String, Object>> cursor = redisTemplate.scan(
                userKeyPattern.getBytes(), 100)) {
            while (cursor.hasNext()) {
                Map.Entry<String, Object> entry = cursor.next();
                keysToInvalidate.add((String) entry.getKey());
            }
        }
        
        // Remove all invalidated keys
        if (!keysToInvalidate.isEmpty()) {
            redisTemplate.delete(keysToInvalidate);
        }
        
        // Also invalidate global flag cache
        redisTemplate.delete(globalKey);
    }
}

Cache Read-Through Pattern

@Service
public class FeatureFlagService {
    private final RedisTemplate<String, Object> redisTemplate;
    private final FeatureFlagRepository featureFlagRepository;

    public boolean isFeatureEnabled(String flagKey, String userId, Map<String, Object> context) {
        String cacheKey = buildCacheKey(flagKey, userId, context);
        
        // Try cache first
        Object cachedValue = redisTemplate.opsForValue().get(cacheKey);
        if (cachedValue != null) {
            return (Boolean) cachedValue;
        }
        
        // Cache miss - fetch from database
        FeatureFlag flag = featureFlagRepository.findByKey(flagKey);
        boolean isEnabled = evaluateFlag(flag, userId, context);
        
        // Cache the result with TTL
        redisTemplate.opsForValue().set(
            cacheKey, 
            isEnabled, 
            60, TimeUnit.SECONDS
        );
        
        return isEnabled;
    }
    
    private String buildCacheKey(String flagKey, String userId, Map<String, Object> context) {
        if (userId == null) {
            return "feature_flag:" + flagKey + ":global";
        }
        
        String contextHash = hashContext(context);
        return "feature_flag:" + flagKey + ":" + userId + ":" + contextHash;
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain objects and database operations. Each entity has its own repository interface with standardized CRUD operations.

public interface FeatureFlagRepository {
    FeatureFlag save(FeatureFlag featureFlag);
    Optional<FeatureFlag> findById(UUID id);
    List<FeatureFlag> findAll();
    List<FeatureFlag> findByEnvironmentAndStatus(String environment, FeatureFlagStatus status);
    void deleteById(UUID id);
    boolean existsById(UUID id);
}

public interface FeatureFlagVariantRepository {
    List<FeatureFlagVariant> findByFlagId(UUID flagId);
    FeatureFlagVariant save(FeatureFlagVariant variant);
    void deleteByFlagId(UUID flagId);
}

ORM vs Raw Query Decisions

We use JPA/Hibernate as the ORM for most operations due to its type safety, query optimization capabilities, and integration with Spring Data. However, for high-volume read operations like feature flag evaluation, we use raw SQL queries to minimize overhead and maximize performance.

@Repository
public class FeatureFlagRepositoryImpl implements FeatureFlagRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    @Override
    public List<FeatureFlag> findByEnvironmentAndStatus(String environment, FeatureFlagStatus status) {
        // Raw SQL for high-performance read operations
        String sql = """
            SELECT f.* FROM feature_flags f 
            JOIN feature_flag_environments e ON f.id = e.flag_id
            WHERE e.environment_name = :environment AND f.status = :status
            """;
        
        return entityManager.createNativeQuery(sql, FeatureFlag.class)
                .setParameter("environment", environment)
                .setParameter("status", status.name())
                .getResultList();
    }
    
    @Override
    public FeatureFlag save(FeatureFlag featureFlag) {
        // JPA for write operations to leverage entity lifecycle management
        return entityManager.merge(featureFlag);
    }
}

Read vs Write Separation

We implement CQRS (Command Query Responsibility Segregation) for the feature flag system to optimize read and write performance:

// Write model - handles mutations
@Repository
public class FeatureFlagWriteRepository {
    @PersistenceContext
    private EntityManager entityManager;
    
    public FeatureFlag save(FeatureFlag flag) {
        return entityManager.merge(flag);
    }
}

// Read model - optimized for querying
@Repository
public class FeatureFlagReadRepository {
    @PersistenceContext
    private EntityManager entityManager;
    
    public List<FeatureFlagProjection> findAllActiveFlagsForEnvironment(String environment) {
        String sql = """
            SELECT f.id, f.name, f.description, f.status, 
                   fv.variant_name, fv.percentage, fv.value
            FROM feature_flags f
            JOIN feature_flag_environments e ON f.id = e.flag_id
            LEFT JOIN feature_flag_variants fv ON f.id = fv.flag_id
            WHERE e.environment_name = :environment AND f.status = 'ACTIVE'
            """;
            
        return entityManager.createNativeQuery(sql, "FeatureFlagProjection")
                .setParameter("environment", environment)
                .getResultList();
    }
}

Connection Pool Configuration

Connection pool is configured with optimal settings for high throughput:

# application.yml
spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      validation-timeout: 5000
      leak-detection-threshold: 60000

N+1 Query Prevention Strategy

We use eager loading and batch fetching to prevent N+1 queries:

@Entity
@Table(name = "feature_flags")
@NamedEntityGraph(
    name = "FeatureFlag.withVariants",
    attributeNodes = @NamedAttributeNode(value = "variants", fetchType = FetchType.EAGER)
)
public class FeatureFlag {
    @Id
    private UUID id;
    
    @OneToMany(mappedBy = "flag", fetch = FetchType.LAZY, cascade = CascadeType.ALL)
    @BatchSize(size = 20)
    private List<FeatureFlagVariant> variants = new ArrayList<>();
    
    // getters and setters
}

Batch Operation Design

Batch operations are implemented for bulk updates:

@Repository
public class FeatureFlagBatchRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    public void updateFlagsStatus(List<UUID> flagIds, FeatureFlagStatus status) {
        String sql = """
            UPDATE feature_flags 
            SET status = :status, updated_at = CURRENT_TIMESTAMP
            WHERE id IN (:flagIds)
            """;
            
        entityManager.createNativeQuery(sql)
                .setParameter("status", status.name())
                .setParameter("flagIds", flagIds)
                .executeUpdate();
    }
    
    public List<FeatureFlag> batchLoadFlags(List<UUID> flagIds) {
        String sql = """
            SELECT * FROM feature_flags 
            WHERE id IN (:flagIds)
            """;
            
        return entityManager.createNativeQuery(sql, FeatureFlag.class)
                .setParameter("flagIds", flagIds)
                .getResultList();
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The Feature Flag platform will use a hybrid synchronous/asynchronous approach. Core flag evaluation and updates will be handled synchronously to ensure immediate consistency for clients. Asynchronous processing is used for:

Flag change notifications - When flags are updated, we publish events to notify all connected clients
Audit logging - All flag operations are logged asynchronously for compliance and debugging
Analytics aggregation - Usage statistics and rollup metrics are computed asynchronously to avoid impacting client response times
Flag state propagation - When flags change in one region/datacenter, the change is propagated asynchronously to other instances

The decision to use async processing for these operations is based on the principle that client-facing operations should be fast and consistent, while background operations can tolerate latency for better system scalability.

Message schema design

{
  "version": "1.0",
  "event_type": "flag_updated",
  "timestamp": "2023-10-27T14:30:00Z",
  "flag_id": "ff-12345",
  "flag_key": "new-feature-flag",
  "flag_value": true,
  "environment": "production",
  "tenant_id": "tenant-abc",
  "user_context": {
    "user_id": "user-xyz",
    "ip_address": "192.168.1.100"
  },
  "metadata": {
    "source": "api",
    "change_reason": "feature-launch"
  }
}

Dead letter queue strategy and poison message handling

For dead letter queue (DLQ) handling, we implement a 3-tier strategy:

Immediate retry - First 2 attempts use exponential backoff (1s, 2s, 4s)
DLQ routing - Messages that fail after 3 attempts are routed to a dedicated DLQ topic
Manual intervention - Messages in DLQ are flagged for manual review and can be replayed after fixes

Poison messages (messages that consistently fail) are identified by tracking the failure count and error type. When a message exceeds 5 consecutive failures, it's automatically moved to the DLQ with a notification to the operations team.

At-least-once vs. exactly-once delivery

We use at-least-once delivery for feature flag events because:

Idempotency is built into the system - Flag updates are idempotent operations that can safely be processed multiple times
Client-side deduplication - Clients can handle duplicate events gracefully by checking timestamp and event content
Simpler architecture - At-least-once is more straightforward to implement and debug than exactly-once

The system uses message IDs and content hashing to detect duplicates at the consumer level, ensuring that even if a message is delivered multiple times, it's only processed once.

Retry policy for consumers

Retry policy configuration:

Component	Max Attempts	Backoff Strategy	Jitter	Non-Retryable Errors
Flag Change Notifier	3	Exponential (1s, 2s, 4s)	Yes	Invalid flag format, auth errors
Audit Logger	5	Linear (2s)	Yes	Invalid log format, storage errors
Analytics Aggregator	3	Exponential (5s, 10s, 20s)	Yes	Invalid metric format, timeout

Retry logic includes jitter to prevent thundering herd issues when multiple consumers retry simultaneously.

Outbox pattern for reliable event publishing

We implement the outbox pattern to ensure reliable event publishing:

When a flag is updated, the operation first writes to an event_outbox table
A background worker periodically polls this table and publishes events to the message broker
Events are marked as published once successfully delivered

This pattern ensures that even if the database transaction commits but message publishing fails, the event will be retried.

// OutboxEvent.java
@Entity
@Table(name = "event_outbox")
public class OutboxEvent {
    @Id
    private UUID id;
    
    private String eventType;
    private String payload;
    private String status; // PENDING, PROCESSED, FAILED
    private LocalDateTime createdAt;
    private LocalDateTime processedAt;
    
    // Getters and setters
}

Sequence diagram for async flows

sequenceDiagram
    participant Client
    participant API
    participant DB
    participant EventPublisher
    participant MessageBroker
    participant Consumer
    
    Client->>API: Update flag (flag-123)
    API->>DB: Save flag update
    DB-->>API: Transaction committed
    API->>EventPublisher: Store event in outbox
    EventPublisher->>MessageBroker: Publish flag_updated event
    MessageBroker->>Consumer: Deliver to subscribers
    
    Note over Consumer: Async processing of flag updates

10. Error Handling Design

Error Classification Hierarchy

The system follows a three-level error classification hierarchy to enable proper handling and alerting:

Validation Errors - Client input issues (400 Bad Request)
Domain Errors - Business logic violations (409 Conflict, 422 Unprocessable Entity)
Infrastructure Errors - System availability issues (500 Internal Server Error, 503 Service Unavailable)

public class FeatureFlagError extends RuntimeException {
    private final ErrorType errorType;
    private final String errorCode;
    
    public FeatureFlagError(ErrorType errorType, String errorCode, String message) {
        super(message);
        this.errorType = errorType;
        this.errorCode = errorCode;
    }
    
    public enum ErrorType {
        VALIDATION, DOMAIN, INFRASTRUCTURE
    }
}

Error Propagation Strategy

Errors are handled at the appropriate layer:

Validation errors are caught and converted to HTTP 400 responses at the API layer
Domain errors are handled at service layer and may bubble up to API layer for HTTP 409/422 responses
Infrastructure errors are logged and may trigger circuit breaker or retry logic at service layer

Retry Strategy with Exponential Backoff and Jitter

For infrastructure failures, a retry mechanism is implemented using exponential backoff with jitter:

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;

    public <T> T executeWithRetry(Supplier<T> operation) {
        int attempts = 0;
        while (attempts < MAX_RETRIES) {
            try {
                return operation.get();
            } catch (InfrastructureException e) {
                attempts++;
                if (attempts >= MAX_RETRIES) throw e;
                
                long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempts) 
                    * (1 + JITTER_FACTOR * (Math.random() - 0.5)));
                try {
                    Thread.sleep(delay);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new InfrastructureException("Retry interrupted", ie);
                }
            }
        }
        throw new IllegalStateException("Unexpected retry loop exit");
    }
}

Circuit Breaker Design

The circuit breaker pattern prevents cascading failures by monitoring the health of downstream dependencies:

public class CircuitBreaker {
    private final int failureThreshold;
    private final long timeoutMs;
    private final AtomicInteger failureCount = new AtomicInteger(0);
    private final AtomicLong lastFailureTime = new AtomicLong(0);
    private volatile CircuitState state = CircuitState.CLOSED;

    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }

    public <T> T execute(Supplier<T> operation) {
        switch (state) {
            case CLOSED:
                return executeClosed(operation);
            case OPEN:
                if (isTimeoutElapsed()) {
                    state = CircuitState.HALF_OPEN;
                    return executeHalfOpen(operation);
                }
                throw new CircuitBreakerOpenException("Circuit is open");
            case HALF_OPEN:
                return executeHalfOpen(operation);
        }
        throw new IllegalStateException("Invalid circuit state");
    }

    private <T> T executeClosed(Supplier<T> operation) {
        try {
            T result = operation.get();
            failureCount.set(0);
            return result;
        } catch (Exception e) {
            int failures = failureCount.incrementAndGet();
            lastFailureTime.set(System.currentTimeMillis());
            if (failures >= failureThreshold) {
                state = CircuitState.OPEN;
            }
            throw e;
        }
    }

    private boolean isTimeoutElapsed() {
        return System.currentTimeMillis() - lastFailureTime.get() > timeoutMs;
    }
}

Fallback Behavior Design

Fallbacks are implemented for external dependencies like database or cache operations:

public class FeatureFlagService {
    private final CircuitBreaker circuitBreaker = new CircuitBreaker(5, 30000);
    
    public FeatureFlag getFeatureFlag(String flagKey) {
        return circuitBreaker.execute(() -> {
            try {
                return featureFlagRepository.findByKey(flagKey);
            } catch (InfrastructureException e) {
                // Fallback to default flag or throw error
                return createDefaultFeatureFlag(flagKey);
            }
        });
    }
}

Error Logging Format and Alerting Thresholds

Structured logging follows a consistent format with trace context:

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);
    
    public void logError(String operation, String errorCode, Exception e, 
                        Map<String, Object> context) {
        Map<String, Object> logContext = new HashMap<>(context);
        logContext.put("operation", operation);
        logContext.put("errorCode", errorCode);
        logContext.put("errorType", e.getClass().getSimpleName());
        logContext.put("errorMessage", e.getMessage());
        
        logger.error("Feature flag operation failed", 
            new StructuredLogEvent(logContext));
    }
}

Key metrics that trigger alerts:

Infrastructure Error Rate: > 1% error rate over 5-minute window → Critical alert
Circuit Breaker Open Events: > 3 occurrences in 1 hour → Warning alert
Validation Errors: > 100 occurrences per hour → Warning alert

Error Response Format

Standardized JSON error response format:

{
  "error": {
    "code": "FLAG_NOT_FOUND",
    "message": "Feature flag with key 'new-feature' not found",
    "details": {
      "flagKey": "new-feature",
      "timestamp": "2023-10-15T14:30:00Z"
    }
  }
}

public class ErrorResponse {
    private final Error error;
    
    public ErrorResponse(String code, String message, Map<String, Object> details) {
        this.error = new Error(code, message, details);
    }
    
    public static class Error {
        private final String code;
        private final String message;
        private final Map<String, Object> details;
        
        public Error(String code, String message, Map<String, Object> details) {
            this.code = code;
            this.message = message;
            this.details = details;
        }
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using a combination of:

Custom validation annotations (e.g., @NotBlank, @Size) for basic constraints
Apache Commons Validator for complex patterns (email, URL, UUID)
OWASP Java HTML Sanitizer for any user-provided content that might be rendered in UIs
Custom regex patterns for feature flag keys and values to prevent injection

public class FeatureFlagRequest {
    @NotBlank
    @Pattern(regexp = "^[a-zA-Z0-9_-]+$", message = "Invalid flag key format")
    private String key;
    
    @NotBlank
    @Size(max = 1000)
    private String description;
    
    @ValidFeatureFlagValue
    private String value;
}

SQL Injection Prevention

All database queries use parameterized prepared statements
No dynamic query building from user input
ORM framework (Hibernate) is configured with strict mode to prevent raw SQL injection

Authentication Middleware Design

JWT-based authentication with 24-hour refresh tokens
OAuth2 integration for enterprise SSO (Okta, Azure AD)
Custom JWT filter that validates token signature and expiration
Role-based access control enforced at the authentication layer

Authorization Enforcement

RBAC model with roles: ADMIN, EDITOR, VIEWER
Permission checking at service layer using @PreAuthorize annotations
Feature flag ownership model: Users can only modify flags they own or belong to teams they're part of
Resource-based authorization where each feature flag has an associated team and permission set

@PreAuthorize("@featureFlagPermissionService.canAccessFlag(#flagId, authentication)")
public FeatureFlag getFeatureFlag(String flagId) {
    return featureFlagRepository.findById(flagId);
}

Sensitive Data Encryption

At rest: AES-256 encryption using AWS KMS for feature flag values and metadata
In transit: TLS 1.3 with HSTS header enforcement
Secrets management: All sensitive values (feature flag values, tokens) are encrypted using the same KMS key

PII Handling and Masking Strategy

No PII in feature flag names or descriptions - all user identifiers are mapped to internal IDs
Audit logs mask any sensitive fields using a 3-character retention policy (e.g., "user123" becomes "use***")
PII is never stored in logs or monitoring systems - only hashed identifiers are used

Secrets Management

AWS Secrets Manager for managing:
- Admin API keys
- OAuth2 client secrets
- Database credentials (rotated every 90 days)
Vault integration for feature flag values that contain sensitive configuration (e.g., API keys, passwords)
Automated secret rotation with 30-day lifecycle for non-sensitive secrets

Audit Logging Design

All CRUD operations on feature flags are logged with:
- userId, userEmail, actionType, flagId, oldValue, newValue
- Timestamps in UTC with microsecond precision
- Session ID for correlation across operations

{
  "timestamp": "2023-10-15T14:30:45.123Z",
  "userId": "user-123",
  "userEmail": "john.doe@example.com",
  "actionType": "UPDATE_FLAG",
  "flagId": "feature-abc123",
  "oldValue": "false",
  "newValue": "true",
  "sessionToken": "session-xyz789"
}

Log retention: 90 days for audit logs, 30 days for operational logs
Log format: Structured JSON with mandatory fields for compliance and monitoring systems
Compliance: Logs are stored in secure, encrypted S3 buckets with access control policies

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

Service Layer: 85% code coverage, focusing on business logic and edge cases
Repository Layer: 90% coverage with all data access paths tested
Domain Objects: 100% coverage for value objects and immutable entities
API Layer: 75% coverage for request/response validation and error handling
Security Layer: 100% coverage for authentication, authorization, and input validation logic

Integration Testing Strategy

Database Integration: Test with embedded H2 in-memory database for fast execution and test isolation
External Service Integration: Use WireMock or Testcontainers for mocking external dependencies like Redis cache and event brokers
Feature Flag Evaluation: Integration tests validate flag evaluation logic against real database state and cache behavior
Event Processing: Validate event publishing and consumption using embedded Kafka or RabbitMQ test containers

Contract Testing

API Contract Testing: Use Pact to define consumer-driven contracts for API endpoints between frontend services and feature flag service
Event Contract Testing: Define Avro schema contracts for event payloads with versioning support and backward compatibility checks
Consumer Verification: Regular contract verification runs in CI pipeline to prevent breaking changes

Performance Testing Tool and Key Scenarios

Tool: k6 (JavaScript-based load testing) for realistic concurrent user simulation
Key Scenarios:
- High concurrent flag evaluation (10,000 concurrent requests)
- Flag evaluation under burst load (500 concurrent requests for 10 seconds)
- Cache hit ratio validation under sustained load
- Database query performance under concurrent access (500 concurrent reads)

Test Data Management Strategy

Fixtures: Use factory pattern to generate test data with realistic flag configurations and user segments
Data Seeding: Initialize database with pre-defined test data sets using Liquibase or Flyway migration scripts
Test Data Isolation: Use database transaction rollback for each test method to ensure clean state
Data Generation: Implement data factories for feature flags, user contexts, and targeting rules with configurable parameters

// Example test data factory for feature flags
public class FeatureFlagFactory {
    public static FeatureFlag createActiveFlag(String name, boolean defaultValue) {
        return FeatureFlag.builder()
                .id(UUID.randomUUID())
                .name(name)
                .enabled(true)
                .defaultValue(defaultValue)
                .createdAt(Instant.now())
                .updatedAt(Instant.now())
                .build();
    }
    
    public static UserContext createTestUserContext(String userId, String environment) {
        return UserContext.builder()
                .userId(userId)
                .environment(environment)
                .userAttributes(Map.of("country", "US", "tier", "premium"))
                .build();
    }
}

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective querying and analysis:

{
  "timestamp": "2023-10-27T14:30:00.000Z",
  "level": "INFO|WARN|ERROR",
  "service": "feature-flag-service",
  "trace_id": "1234567890abcdef1234567890abcdef",
  "span_id": "abcdef1234567890",
  "user_id": "user-123",
  "feature_flag_id": "ff-abc123",
  "request_id": "req-xyz789",
  "message": "Feature flag evaluated successfully",
  "attributes": {
    "flag_key": "new-feature-toggle",
    "variant": "control",
    "evaluation_context": {
      "user_id": "user-123",
      "environment": "production"
    }
  },
  "error_code": "EVALUATION_FAILED",
  "stack_trace": "..."
}

PII masking rules:

User IDs are masked in logs after first 4 characters (e.g., user-123 → user-****)
IP addresses are masked to first octet only (e.g., 192.168.1.100 → 192.***.***.***)
Email addresses are masked to domain only (e.g., user@example.com → user@***.com)

Distributed Tracing Instrumentation

Using OpenTelemetry for distributed tracing:

Custom Spans:

feature_flag_evaluation - measures time to evaluate a feature flag
flag_configuration_fetch - tracks fetching configuration from database/cache
variant_assignment - measures variant assignment logic execution

Key Attributes Captured:

flag_key: The feature flag key being evaluated
user_id: Identifier for the user requesting evaluation
environment: Environment context (production, staging, etc.)
variant: Assigned variant (if any)
evaluation_result: Boolean or variant name
evaluation_context_size: Size of context object

Metrics Design

RED Method Metrics:

Metric Name	Type	Labels	Description
`feature_flag_evaluations_total`	Counter	flag_key, variant, environment, success	Total feature flag evaluations
`feature_flag_evaluation_duration_seconds`	Histogram	flag_key, environment	Time to evaluate a feature flag
`feature_flag_evaluation_errors_total`	Counter	flag_key, error_type, environment	Failed evaluations
`feature_flag_cache_hits_total`	Counter	flag_key, environment	Cache hits for flag lookups
`feature_flag_cache_misses_total`	Counter	flag_key, environment	Cache misses for flag lookups
`feature_flag_config_updates_total`	Counter	environment, source	Flag configuration updates

Gauge Metrics:

feature_flag_active_count - Current count of active feature flags
feature_flag_cache_size - Current size of in-memory cache

Health Check Endpoint Design

GET /health
Accept: application/json

Response Format:

{
  "status": "healthy|unhealthy",
  "timestamp": "2023-10-27T14:30:00.000Z",
  "components": {
    "database": {
      "status": "healthy|unhealthy",
      "latency_ms": 5,
      "error": null
    },
    "cache": {
      "status": "healthy|unhealthy",
      "latency_ms": 2,
      "error": null
    },
    "external_services": {
      "status": "healthy|unhealthy",
      "error": null
    }
  },
  "uptime_seconds": 3600,
  "version": "1.2.3"
}

Health Check Endpoints:

/health - Overall service health (status, uptime)
/ready - Service ready to serve traffic (DB, cache connections)
/live - Process is alive (memory usage, CPU)
/startup - Service startup completion status

Alerting Thresholds

Metric	Warning Level	Critical Level	Alerting Channel
`feature_flag_evaluation_errors_total`	> 10 errors/minute	> 50 errors/minute	Slack/PagerDuty
`feature_flag_evaluation_duration_seconds`	p95 > 200ms	p95 > 500ms	Slack/PagerDuty
`feature_flag_cache_misses_total`	> 500 misses/minute	> 1000 misses/minute	Slack/PagerDuty
`database_connection_pool_utilization`	> 80%	> 95%	Slack/PagerDuty
`cache_hit_ratio`	< 80%	< 60%	Slack/PagerDuty
`feature_flag_active_count`	> 10000 flags	> 20000 flags	Slack/PagerDuty

On-call Escalation Path:

Primary engineer (5 min response)
Secondary engineer (15 min response)
Team lead (30 min response)
Engineering manager (60 min response)

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

Feature flag evaluation strategy: Should we support A/B testing, gradual rollouts, or just simple on/off flags? This decision impacts the complexity of the evaluation engine and the data model.
Flag configuration persistence: Should feature flags be persisted in a database or managed through an external system like AWS Systems Manager Parameter Store or Consul? This affects the platform's operational complexity and scalability.
User targeting strategy: How granular should user targeting be? Should we support targeting by user ID, email, organization, or custom attributes? This impacts the data model and performance of flag evaluation.
Flag inheritance and hierarchy: Should flags support hierarchical inheritance where child flags inherit properties from parent flags? This adds complexity to the data model and evaluation logic.
Flag versioning strategy: Should we implement a versioned flag system where changes to flags are tracked and can be rolled back? This affects the audit trail and rollback capabilities.

Known trade-offs and justifications

Consistency vs. Performance: Using eventual consistency for flag evaluation to achieve better performance and lower latency at the cost of temporary inconsistency during flag updates. This is acceptable for most use cases where slight delays in flag propagation are tolerable.
Database vs. Cache trade-off: Using Redis for caching flags to reduce database load and improve response times, but this introduces a potential single point of failure and requires careful cache invalidation strategies.
Simplicity vs. Feature Richness: Prioritizing a simple, core feature flag system over complex features like multi-dimensional targeting to ensure faster development and easier maintenance.
Real-time vs. Batch Updates: Using real-time flag updates for immediate propagation but accepting that this might increase system load compared to batched updates.
Single vs. Multi-tenant Architecture: Supporting multi-tenancy for better resource isolation but increasing system complexity in terms of data separation and security.

Risks and mitigation strategies

Cache Invalidation Risk: Risk of stale flags being served due to cache invalidation issues. Mitigation: Implement a robust cache invalidation strategy with TTLs and event-driven invalidation using Redis pub/sub.
Performance Degradation: Risk of performance degradation under high load due to flag evaluation overhead. Mitigation: Implement caching, database read replicas, and efficient query patterns with proper indexing.
Data Consistency Risk: Risk of inconsistent flag states across different environments or services. Mitigation: Use distributed locking mechanisms and event-driven architecture to ensure consistency.
Security Risk: Risk of unauthorized flag access or modification. Mitigation: Implement comprehensive RBAC, audit logging, and secure API endpoints with proper authentication and authorization.
Scalability Risk: Risk of system not scaling to handle large numbers of flags and users. Mitigation: Design with horizontal scalability in mind, use sharding strategies, and implement proper load balancing.

Technical debt acknowledged

Flag Evaluation Logic Complexity: The current implementation assumes simple flag evaluation logic, but as more complex targeting strategies are added (e.g., percentage rollouts, user attribute matching), the evaluation logic will become more complex and harder to maintain.
Database Schema Evolution: The current schema design is simplified and may not support all future requirements for flag targeting, such as complex attribute-based filtering or multi-dimensional rollouts.
Caching Strategy Limitations: The current caching strategy is basic and may not handle complex scenarios like cache warming or advanced cache invalidation strategies for different flag types.
Audit Trail Limitations: The current audit trail only logs basic information, but more detailed tracking of flag changes and user actions may be needed in the future.
Testing Coverage Gap: The current design assumes adequate test coverage but does not provide specific testing strategies or coverage targets, which could lead to gaps in quality assurance.
Monitoring and Alerting Gaps: The current design has basic observability but may not include comprehensive alerting for all critical system metrics and error conditions.