Dynamic Configuration Service Low-Level Design

A dynamic configuration service enables applications to receive real-time configuration updates without restarts, supporting multiple environments and versioned configurations for robust deployment strategies.

1. Problem Statement & Requirements

Clarifying questions

1. What scale expectations are there for the configuration service?

   • How many clients (services) will use it?
   • How many configuration items or keys are expected to be stored?
   • What is the expected rate of configuration updates?

2. What consistency guarantees are required?

   • Should all clients see the same configuration value at any given time?
   • Is eventual consistency acceptable for some use cases?

3. What are the key entities and their relationships?

   • Is configuration tied to environments (dev, staging, prod)?
   • Are there hierarchical or dependency relationships between configurations?

4. What is the read/write ratio and latency sensitivity?

   • Is configuration accessed more frequently than updated?
   • Are there strict SLAs for configuration fetch times?

5. How is authentication and authorization handled?

   • Will configuration access be controlled via OAuth2, JWTs, or API keys?
   • Do different services or users have different levels of access?

6. Are there any external integrations or dependencies?

   • Will the service integrate with a configuration management system like Consul, etcd, or AWS Systems Manager?
   • Are there any external services that depend on configuration changes?

7. What is out of scope for this design?

   • Should the service handle configuration versioning or rollback?
   • Is there a requirement for configuration drift detection or compliance?

Functional requirements

• Configuration Management: Store, update, and retrieve configuration values for services.
• Environment-based Configuration: Support configuration per environment (e.g., dev, staging, prod).
• Real-time Updates: Notify clients of configuration changes in near real-time.
• Configuration Validation: Validate configuration values before storing them.
• Audit Trail: Track who made changes and when.
• Multi-tenancy Support: Allow multiple tenants to manage their own configuration sets.

Non-functional requirements

• Latency Targets:

  • p50: <5ms for configuration fetch
  • p95: <10ms for configuration fetch
  • p99: <25ms for configuration fetch

• Throughput:

  • 10,000+ concurrent clients fetching configuration
  • 1,000+ updates/sec

• Availability:

  • 99.9% uptime (43.2 minutes of downtime/month)
  • Auto-recovery from failures

Constraints and assumptions

• Assumptions:

  • Configuration values are small (max 1KB per key)
  • Configuration updates are infrequent compared to reads
  • Clients will use a configuration client SDK for efficient caching and change notifications

• Constraints:

  • Service must support horizontal scaling (stateless design)
  • No external dependencies for core configuration storage
  • Configuration updates must be atomic per key

Out of scope

• Versioning and Rollback: Configuration versioning or rollback to previous values is out of scope.
• Configuration Validation Rules: Complex validation logic (e.g., regex or schema validation) is out of scope.
• Configuration Encryption: Encryption of configuration values at rest or in transit is out of scope.
• Multi-region Support: Multi-region replication or disaster recovery is out of scope.
• Configuration Diffing: Comparing configuration changes over time is out of scope.

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for the Dynamic Configuration service due to:

• Strong typing and schema evolution support
• Efficient binary serialization reducing payload size
• Built-in streaming capabilities for real-time configuration updates
• First-class support for versioning and backward compatibility
• Native support for authentication and authorization in gRPC

Endpoint Definitions

Method	Path	Description	Auth
GetConfig	/config.v1.ConfigService/GetConfig	Fetch configuration by key	JWT
UpdateConfig	/config.v1.ConfigService/UpdateConfig	Update configuration value	JWT
DeleteConfig	/config.v1.ConfigService/DeleteConfig	Delete configuration by key	JWT
ListConfigs	/config.v1.ConfigService/ListConfigs	List all configurations with pagination	JWT
WatchConfig	/config.v1.ConfigService/WatchConfig	Stream configuration updates	JWT
GetConfigHistory	/config.v1.ConfigService/GetConfigHistory	Fetch version history of a config	JWT

Request and Response Schemas

GetConfigRequest

message GetConfigRequest {
  string key = 1;
  string environment = 2;
  string version = 3;
}

GetConfigResponse

message GetConfigResponse {
  string key = 1;
  string value = 2;
  string environment = 3;
  string version = 4;
  int64 created_at = 5;
  int64 updated_at = 6;
}

UpdateConfigRequest

message UpdateConfigRequest {
  string key = 1;
  string value = 2;
  string environment = 3;
  string description = 4;
}

UpdateConfigResponse

message UpdateConfigResponse {
  string key = 1;
  string value = 2;
  string environment = 3;
  string version = 4;
  int64 updated_at = 5;
}

ListConfigsRequest

message ListConfigsRequest {
  string environment = 1;
  int32 page_size = 2;
  string page_token = 3;
}

ListConfigsResponse

message ListConfigsResponse {
  repeated Config configs = 1;
  string next_page_token = 2;
}

message Config {
  string key = 1;
  string value = 2;
  string environment = 3;
  string version = 4;
  int64 created_at = 5;
  int64 updated_at = 6;
  string description = 7;
}

WatchConfigRequest

message WatchConfigRequest {
  string key = 1;
  string environment = 2;
}

WatchConfigResponse

message WatchConfigResponse {
  string key = 1;
  string value = 2;
  string environment = 3;
  string version = 4;
  int64 updated_at = 5;
}

HTTP Status Codes and Error Response Format

Status Code	Description
200 OK	Successful operation
400 Bad Request	Invalid request parameters
401 Unauthorized	Authentication required
403 Forbidden	Insufficient permissions
404 Not Found	Config not found
429 Too Many Requests	Rate limit exceeded
500 Internal Server Error	Server error

Standardized error response:

{
  "error": {
    "code": "CONFIG_NOT_FOUND",
    "message": "Configuration with key 'feature_flag' not found in environment 'production'",
    "details": {
      "key": "feature_flag",
      "environment": "production"
    }
  }
}

API Versioning Strategy

Versioned using semantic versioning (v1, v2) in the service name:

• config.v1.ConfigService for current version
• Future versions will use config.v2.ConfigService

Versioning strategy:

1. Backward compatible changes in same major version
2. Breaking changes require new major version
3. Deprecation period of 6 months for old versions

Pagination Design

Cursor-based pagination is used because:

• More efficient than offset-based for large datasets
• Consistent results even when data changes during pagination
• Better performance characteristics

Cursor format: base64_encode(environment:config_key)

Authentication and Authorization

Authentication:

• JWT tokens with audience config-service
• Token validation using public key from JWKS endpoint
• Token expiration of 1 hour

Authorization:

• RBAC model with roles: admin, developer, viewer
• Admins can read/write all configs
• Developers can read/write configs in their environment
• Viewers can only read configs

Rate Limiting

Rate limits per endpoint:

Endpoint	Limit	Period
GetConfig	1000 req/min	1 minute
UpdateConfig	100 req/min	1 minute
ListConfigs	500 req/min	1 minute
WatchConfig	1000 req/min	1 minute

Implementation:

• Token bucket algorithm with sliding window
• Distributed rate limiting using Redis
• Per-user and per-environment limits

Idempotency Design

Idempotency key strategy:

• Client-provided idempotency_key in request headers
• Server stores operation state for 5 minutes
• If same key is used within 5 minutes, return cached result
• For update operations, use optimistic concurrency control

// Example idempotency key usage in gRPC service
public UpdateConfigResponse updateConfig(UpdateConfigRequest request, String idempotencyKey) {
    if (idempotencyKey != null) {
        String cacheKey = "idempotency:" + idempotencyKey;
        if (redisTemplate.hasKey(cacheKey)) {
            return redisTemplate.opsForValue().get(cacheKey);
        }
    }
    
    // Process update...
    UpdateConfigResponse response = processUpdate(request);
    
    if (idempotencyKey != null) {
        redisTemplate.opsForValue().set(cacheKey, response, 5, TimeUnit.MINUTES);
    }
    
    return response;
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    CONFIGURATION {
        string id PK
        string name
        string description
        jsonb properties
        string environment
        string owner
        boolean is_active
        timestamp created_at
        timestamp updated_at
        timestamp deleted_at
    }

    CONFIGURATION_HISTORY {
        string id PK
        string config_id FK
        jsonb properties
        timestamp versioned_at
        string changed_by
        string change_reason
    }

    CONFIGURATION_AUDIT {
        string id PK
        string config_id FK
        string action
        string user_id
        timestamp occurred_at
        jsonb before_properties
        jsonb after_properties
    }

    CONFIGURATION {
        string id PK
        string name
        string description
        jsonb properties
        string environment
        string owner
        boolean is_active
        timestamp created_at
        timestamp updated_at
        timestamp deleted_at
    }

    CONFIGURATION_HISTORY {
        string id PK
        string config_id FK
        jsonb properties
        timestamp versioned_at
        string changed_by
        string change_reason
    }

    CONFIGURATION_AUDIT {
        string id PK
        string config_id FK
        string action
        string user_id
        timestamp occurred_at
        jsonb before_properties
        jsonb after_properties
    }

    CONFIGURATION ||--o{ CONFIGURATION_HISTORY : "has"
    CONFIGURATION ||--o{ CONFIGURATION_AUDIT : "audits"

Table Schema Definitions

-- Main configuration table
CREATE TABLE configurations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL,
    description TEXT,
    properties JSONB NOT NULL,
    environment VARCHAR(64) NOT NULL,
    owner VARCHAR(255) NOT NULL,
    is_active BOOLEAN DEFAULT TRUE,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    deleted_at TIMESTAMP WITH TIME ZONE NULL,
    
    -- Indexes
    INDEX idx_configurations_environment (environment),
    INDEX idx_configurations_owner (owner),
    INDEX idx_configurations_name (name),
    INDEX idx_configurations_active (is_active),
    INDEX idx_configurations_updated_at (updated_at)
);

-- Configuration history table for versioning
CREATE TABLE configuration_history (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    config_id UUID NOT NULL REFERENCES configurations(id) ON DELETE CASCADE,
    properties JSONB NOT NULL,
    versioned_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    changed_by VARCHAR(255) NOT NULL,
    change_reason TEXT,
    
    INDEX idx_configuration_history_config_id (config_id),
    INDEX idx_configuration_history_versioned_at (versioned_at)
);

-- Audit trail table
CREATE TABLE configuration_audit (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    config_id UUID NOT NULL REFERENCES configurations(id) ON DELETE CASCADE,
    action VARCHAR(32) NOT NULL, -- 'CREATE', 'UPDATE', 'DELETE'
    user_id VARCHAR(255) NOT NULL,
    occurred_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    before_properties JSONB,
    after_properties JSONB,
    
    INDEX idx_configuration_audit_config_id (config_id),
    INDEX idx_configuration_audit_occurred_at (occurred_at),
    INDEX idx_configuration_audit_user_id (user_id)
);

Primary Key Strategy

UUID is selected as the primary key for all tables due to:

1. Scalability: UUIDs provide globally unique identifiers without requiring coordination across distributed systems
2. Partitioning: UUIDs distribute well across database shards and can be used in sharded environments
3. Security: UUIDs don't reveal sequential information that could be exploited for enumeration attacks
4. Cross-system compatibility: UUIDs can be generated in distributed systems without coordination

Index Design

Index Name	Columns	Type	Purpose
idx_configurations_environment	environment	Standard	Fast filtering by environment
idx_configurations_owner	owner	Standard	Efficient ownership queries
idx_configurations_name	name	Standard	Fast lookup by configuration name
idx_configurations_active	is_active	Standard	Quick filtering of active configs
idx_configurations_updated_at	updated_at	Standard	Efficient change tracking
idx_configuration_history_config_id	config_id	Standard	Fast history lookups
idx_configuration_history_versioned_at	versioned_at	Standard	Version timeline queries
idx_configuration_audit_config_id	config_id	Standard	Audit trail by config
idx_configuration_audit_occurred_at	occurred_at	Standard	Time-based audit queries

Data Partitioning / Sharding Strategy

Environment-based sharding is implemented:

1. Partition Key: environment field (e.g., "dev", "staging", "prod")
2. Shard Strategy:
   • Each environment gets its own database or tablespace
   • Cross-environment queries are handled through application-level routing
3. Benefits:
   • Isolation of environments
   • Reduced contention between environments
   • Easier compliance and audit per environment

Soft Delete vs. Hard Delete

Soft delete is implemented for the following reasons:

1. Auditability: Historical changes can be tracked through audit logs
2. Recovery capability: Configuration can be restored if deleted accidentally
3. Compliance: Regulatory requirements often mandate data retention
4. Versioning support: History tracking requires keeping deleted records in history tables

Audit Trail Design

Configuration audit trail includes:

• Who: user_id field from authentication context
• What: action (CREATE, UPDATE, DELETE) and property changes
• When: occurred_at timestamp with timezone
• Changes: Full before/after JSONB snapshots for diff analysis

Data Archival Strategy and Retention Policy

Archival Strategy:

1. Active configurations: Keep in primary storage with 30-day retention
2. Historical versions: Retain all version history for 1 year (with compression)
3. Audit logs: Maintain full audit trail for 5 years

Retention Policy:

• Active configs: 30 days (soft delete)
• History versions: 1 year
• Audit logs: 5 years
• Archival process: Daily batch job to move old data to cold storage (S3/Cloud Storage)

// Example Java POJO for configuration entity
public class Configuration {
    private String id;
    private String name;
    private String description;
    private Map<String, Object> properties;
    private String environment;
    private String owner;
    private boolean isActive;
    private Instant createdAt;
    private Instant updatedAt;
    private Instant deletedAt;
}

4. Class Design & Service Layer

classDiagram
    class ConfigService {
        +getConfig(String key) ConfigValue
        +updateConfig(String key, ConfigValue value) void
        +deleteConfig(String key) void
        +listConfigs() List~ConfigValue~
        +watchConfig(String key, ConfigChangeListener listener) void
    }
    
    class ConfigRepository {
        +save(ConfigValue config) void
        +findById(String id) Optional~ConfigValue~
        +findAll() List~ConfigValue~
        +deleteById(String id) void
        +findByKey(String key) Optional~ConfigValue~
    }
    
    class ConfigValue {
        -id: String
        -key: String
        -value: String
        -version: Long
        -lastModified: Instant
        -tags: Set~String~
        -isDeleted: Boolean
    }
    
    class ConfigChangeListener {
        +onChange(ConfigValue config) void
    }
    
    class ConfigValidator {
        +validate(ConfigValue config) ValidationResult
    }
    
    class ConfigEventPublisher {
        +publish(ConfigChangeEvent event) void
    }
    
    class ConfigChangeEvent {
        -configId: String
        -key: String
        -eventType: EventType
        -timestamp: Instant
    }
    
    class ConfigCache {
        +get(String key) Optional~ConfigValue~
        +put(String key, ConfigValue value) void
        +invalidate(String key) void
    }
    
    class ConfigUpdateStrategy {
        +apply(ConfigValue current, ConfigValue updated) ConfigValue
    }
    
    class DefaultConfigUpdateStrategy {
        +apply(ConfigValue current, ConfigValue updated) ConfigValue
    }
    
    class ConfigManager {
        -configRepository: ConfigRepository
        -configCache: ConfigCache
        -configValidator: ConfigValidator
        -eventPublisher: ConfigEventPublisher
        -updateStrategy: ConfigUpdateStrategy
        +getConfig(String key) ConfigValue
        +updateConfig(String key, ConfigValue value) void
        +deleteConfig(String key) void
    }
    
    class ConfigVersionManager {
        -version: AtomicLong
        +getNextVersion() Long
    }
    
    ConfigService --> ConfigManager : delegates to
    ConfigManager --> ConfigRepository : persists configs
    ConfigManager --> ConfigCache : caches configs
    ConfigManager --> ConfigValidator : validates inputs
    ConfigManager --> ConfigEventPublisher : publishes events
    ConfigManager --> ConfigUpdateStrategy : applies updates
    ConfigUpdateStrategy <|-- DefaultConfigUpdateStrategy : implements
    ConfigManager --> ConfigVersionManager : manages versioning
    
    class ConfigValue {
        +getId(): String
        +getKey(): String
        +getValue(): String
        +getVersion(): Long
        +getLastModified(): Instant
        +getTags(): Set~String~
        +isDeleted(): Boolean
    }
    
    class ConfigChangeEvent {
        +getConfigId(): String
        +getKey(): String
        +getEventType(): EventType
        +getTimestamp(): Instant
    }
    
    class ConfigUpdateStrategy {
        +apply(ConfigValue current, ConfigValue updated) ConfigValue
    }
    
    enum EventType {
        CREATED
        UPDATED
        DELETED
    }

stateDiagram-v2
    [*] --> Active
    Active --> Inactive : deleteConfig()
    Active --> Active : updateConfig()
    Active --> Active : getConfig()
    Inactive --> Active : createConfig()
    Active --> Inactive : deleteConfig()

Service Class Structure and Responsibilities

The core service layer is built around the ConfigManager which orchestrates all configuration operations. It encapsulates:

• Persistence: Delegates to ConfigRepository for data storage
• Caching: Manages ConfigCache for fast retrieval
• Validation: Uses ConfigValidator to ensure data integrity
• Eventing: Publishes changes via ConfigEventPublisher
• Versioning: Maintains version consistency with ConfigVersionManager

Interface and Abstract Class Design

public interface ConfigRepository {
    void save(ConfigValue config);
    Optional<ConfigValue> findById(String id);
    List<ConfigValue> findAll();
    void deleteById(String id);
    Optional<ConfigValue> findByKey(String key);
}

public interface ConfigChangeListener {
    void onChange(ConfigValue config);
}

public interface ConfigUpdateStrategy {
    ConfigValue apply(ConfigValue current, ConfigValue updated);
}

Business Logic Encapsulation Patterns

We implement the Strategy Pattern for configuration updates to support different update policies:

public class DefaultConfigUpdateStrategy implements ConfigUpdateStrategy {
    @Override
    public ConfigValue apply(ConfigValue current, ConfigValue updated) {
        return new ConfigValue(
            current.getId(),
            updated.getKey(),
            updated.getValue(),
            updated.getVersion() + 1,
            Instant.now(),
            updated.getTags(),
            false
        );
    }
}

Critical-Path Method Implementation

public void updateConfig(String key, ConfigValue updatedConfig) {
    // Transaction boundary - all operations within single transaction
    try (Transaction tx = transactionManager.begin()) {
        // Pessimistic locking - acquire write lock on config key
        ConfigValue existing = configRepository.findByKey(key)
            .orElseThrow(() -> new ConfigNotFoundException("Config not found: " + key));
        
        // Validate input
        ValidationResult validation = configValidator.validate(updatedConfig);
        if (!validation.isValid()) {
            throw new ValidationException("Invalid config: " + validation.getErrors());
        }
        
        // Apply update strategy
        ConfigValue updated = updateStrategy.apply(existing, updatedConfig);
        
        // Update in DB with version bump
        configRepository.save(updated);
        
        // Invalidate cache
        configCache.invalidate(key);
        
        // Publish event for subscribers
        ConfigChangeEvent event = new ConfigChangeEvent(
            updated.getId(),
            key,
            EventType.UPDATED,
            Instant.now()
        );
        eventPublisher.publish(event);
        
        // Commit transaction
        tx.commit();
    } catch (Exception e) {
        // Rollback on failure
        throw new ConfigUpdateFailedException("Failed to update config: " + key, e);
    }
}

Transaction Boundary Design

All configuration operations are wrapped in a single database transaction:

• Read-modify-write operations on the same config key
• Atomic persistence of updated configuration
• Event publishing within the same transaction scope to ensure consistency

Service Interface Contracts

public interface ConfigService {
    ConfigValue getConfig(String key);
    void updateConfig(String key, ConfigValue value);
    void deleteConfig(String key);
    List<ConfigValue> listConfigs();
    void watchConfig(String key, ConfigChangeListener listener);
}

Saga / Orchestration Pattern

For complex configuration operations involving external systems, we would implement a saga pattern:

• Each operation is atomic within local transaction
• Compensation logic for rollback scenarios
• Event-driven coordination across services

Domain Object / Value Object Design

public class ConfigValue {
    private final String id;
    private final String key;
    private final String value;
    private final Long version;
    private final Instant lastModified;
    private final Set<String> tags;
    private final boolean isDeleted;
    
    // Constructor, getters, immutable operations
}

The ConfigValue is designed as an immutable value object with proper equality and hash code implementations to ensure consistency across cache and database layers.

5. Concurrency & Thread Safety

Concurrency Model

The configuration service will use a thread pool model with a fixed-size thread pool for handling concurrent requests. Each request will be processed by a dedicated thread from the pool, ensuring that I/O-bound operations (like database reads/writes or external API calls) don't block other requests. For CPU-intensive operations like configuration validation or transformation, we'll use a separate compute thread pool to avoid blocking the main request processing threads.

Race Condition Prevention Strategy

Race conditions in configuration updates will be prevented through:

1. Database-level atomic operations - Using optimistic locking with version numbers or timestamps for concurrent updates
2. Application-level mutexes - For critical sections where configuration state is read and updated together
3. Transactional boundaries - Ensuring all reads and writes to configuration entities happen within a single transaction
4. Immutable configuration objects - When possible, returning immutable configuration snapshots to prevent modification during read operations

Optimistic vs. Pessimistic Locking Selection

We will use optimistic locking for configuration updates because:

• Configuration changes are infrequent compared to reads
• The system will handle conflicts gracefully with retry logic
• Pessimistic locking would introduce unnecessary contention and reduce throughput
• The configuration service is read-heavy with low write frequency

Deadlock Prevention Strategy

To prevent deadlocks:

1. Consistent ordering of resource acquisition - All operations acquire locks in a consistent order (e.g., by configuration ID, namespace, or name)
2. Timeout-based lock acquisition - Implement timeouts for lock acquisition to prevent indefinite blocking
3. Lock hierarchy - Establish a clear hierarchy (e.g., namespace → config name) to avoid circular dependencies
4. Transaction timeout - Set a maximum transaction duration (e.g., 5 seconds) to prevent long-running transactions

Idempotency Implementation for Concurrent Duplicate Requests

For concurrent duplicate requests, we'll implement idempotency using:

1. Idempotency keys - Clients can provide an idempotency-key header that uniquely identifies the request
2. Request deduplication cache - Store recent requests (with their results) in an in-memory LRU cache with TTL of 5 minutes
3. Database-level deduplication - For write operations, use unique constraints on the combination of client ID + idempotency key to prevent duplicate processing

public class ConfigurationService {
    private final Map<String, CachedResponse> idempotencyCache = new ConcurrentHashMap<>();
    
    public ConfigurationResponse updateConfiguration(
            String idempotencyKey, 
            UpdateConfigRequest request) {
        
        // Check if this is a duplicate request
        CachedResponse cached = idempotencyCache.get(idempotencyKey);
        if (cached != null && !cached.isExpired()) {
            return cached.response;
        }
        
        // Process the request with transactional boundary
        ConfigurationResponse result = processUpdate(request);
        
        // Cache the result for idempotency
        idempotencyCache.put(idempotencyKey, new CachedResponse(result));
        
        return result;
    }
    
    private ConfigurationResponse processUpdate(UpdateConfigRequest request) {
        // Transaction boundary - all DB operations in single transaction
        return configurationRepository.withTransaction(() -> {
            // Read current config with version
            Configuration current = configurationRepository.findById(request.getId());
            
            // Validate and apply changes
            Configuration updated = current.updateWith(request.getChanges());
            
            // Save with optimistic locking (version check)
            configurationRepository.save(updated);
            
            // Publish update event
            eventPublisher.publish(new ConfigurationUpdatedEvent(updated));
            
            return new ConfigurationResponse(updated);
        });
    }
    
    private static class CachedResponse {
        private final ConfigurationResponse response;
        private final long timestamp;
        private static final long TTL_MILLIS = 5 * 60 * 1000; // 5 minutes
        
        public CachedResponse(ConfigurationResponse response) {
            this.response = response;
            this.timestamp = System.currentTimeMillis();
        }
        
        public boolean isExpired() {
            return System.currentTimeMillis() - timestamp > TTL_MILLIS;
        }
    }
}

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant Client
    participant ConfigService
    participant Cache
    participant DB

    Client->>ConfigService: GET /config/{key}
    ConfigService->>Cache: Check if config exists in cache
    Cache-->>ConfigService: Hit or Miss
    alt Cache Hit
        ConfigService-->>Client: Return config from cache
    else Cache Miss
        ConfigService->>DB: Fetch config from database
        DB-->>ConfigService: Return config
        ConfigService->>Cache: Store config in cache
        Cache-->>ConfigService: Cache set success
        ConfigService-->>Client: Return config from DB
    end

sequenceDiagram
    participant Client
    participant ConfigService
    participant Cache
    participant DB
    participant EventPublisher

    Client->>ConfigService: PUT /config/{key} with new value
    ConfigService->>DB: Update config in database
    DB-->>ConfigService: Update success
    ConfigService->>Cache: Invalidate config from cache
    Cache-->>ConfigService: Cache invalidated
    ConfigService->>EventPublisher: Publish config updated event
    EventPublisher-->>ConfigService: Event published
    ConfigService-->>Client: Return success response

sequenceDiagram
    participant ConfigService
    participant Cache
    participant EventConsumer
    participant Subscriber

    ConfigService->>Cache: Invalidate config in cache
    Cache-->>ConfigService: Cache invalidated
    ConfigService->>EventConsumer: Publish config updated event
    EventConsumer->>Subscriber: Notify subscribers of config change
    Subscriber-->>EventConsumer: Acknowledge receipt
    EventConsumer-->>ConfigService: Event processing complete

7. Caching Design

Cache Strategy Selection

For the Dynamic Configuration service, we'll use a cache-aside pattern with write-through behavior for critical configuration updates. This approach provides the best balance between consistency and performance, ensuring that configuration changes are immediately available to all service instances while maintaining cache freshness.

Cache Key Design

Cache keys will follow a structured namespace pattern:

config:{environment}:{configKey}:{version}
config:metadata:{configKey}

Example:

config:prod:feature-flag-1:v2
config:metadata:feature-flag-1

TTL Strategy

Data Type	TTL	Reason
Configuration Values	5 minutes	Frequent updates, short-lived cache
Metadata	1 hour	Stable metadata, less frequent changes
Configuration History	24 hours	Audit trail for rollback scenarios

Cache Eviction Policy

We'll use LRU (Least Recently Used) eviction policy with a secondary TTL-based cleanup. This ensures that:

1. Least accessed configurations are evicted first
2. Stale data is automatically cleaned up based on TTL
3. Memory pressure is managed proactively

Cache Stampede Prevention

To prevent cache stampedes:

1. Probabilistic early expiry: Cache entries expire 20% before TTL (e.g., 4 minutes instead of 5)
2. Distributed locks: Use Redis distributed locks when cache misses occur for the same key
3. Cache warming: Pre-warm hot configuration keys during service startup

Distributed Cache Architecture

We'll use Redis Cluster with:

• 3 master nodes for high availability
• 3 replica nodes per shard for read scaling
• Redis Sentinel for automatic failover
• Cache sharding based on environment and config key prefix

Cache Invalidation Strategy

We'll implement event-driven invalidation:

1. When configuration is updated, publish a config.updated event
2. All service instances subscribe to this event and invalidate local cache entries
3. Use Redis Pub/Sub for low-latency, reliable invalidation
4. For critical updates, implement immediate cache warming to ensure availability

// Example cache invalidation logic in Java
public class ConfigCacheInvalidator {
    private final RedisTemplate<String, String> redisTemplate;
    
    public void invalidateConfig(String environment, String configKey) {
        String cacheKey = "config:" + environment + ":" + configKey + ":*";
        Set<String> keys = redisTemplate.keys(cacheKey);
        
        if (keys != null && !keys.isEmpty()) {
            redisTemplate.delete(keys);
        }
        
        // Also invalidate metadata
        String metadataKey = "config:metadata:" + configKey;
        redisTemplate.delete(metadataKey);
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and persistence concerns. Each entity has its own repository interface with standardized CRUD operations.

public interface ConfigurationRepository {
    Optional<Configuration> findById(String id);
    List<Configuration> findAll();
    List<Configuration> findByEnvironmentAndNamespace(String environment, String namespace);
    Configuration save(Configuration config);
    void delete(String id);
    boolean existsById(String id);
}

public interface ConfigurationHistoryRepository {
    List<ConfigurationHistory> findByConfigId(String configId);
    void save(ConfigurationHistory history);
}

ORM vs Raw Query Decisions

For the configuration service, we use JPA/Hibernate for primary entities due to its maturity, type safety, and ability to handle complex relationships. Raw queries are used for performance-critical operations like bulk updates or analytics.

// JPA Entity - Type safe, supports lazy loading and transactions
@Entity
@Table(name = "configurations")
public class Configuration {
    @Id
    @GeneratedValue(strategy = GenerationType.UUID)
    private UUID id;
    
    @Column(name = "key", nullable = false)
    private String key;
    
    @Lob
    @Column(name = "value", nullable = false)
    private String value;
    
    @Column(name = "environment", nullable = false)
    private String environment;
    
    @Column(name = "namespace", nullable = false)
    private String namespace;
    
    @Column(name = "version", nullable = false)
    private Long version;
    
    // Getters and setters
}

// Raw Query for performance-critical operations
@Repository
public class ConfigurationRawRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    public List<Configuration> findLatestByNamespace(String namespace) {
        String jpql = """
            SELECT c FROM Configuration c 
            WHERE c.namespace = :namespace 
            AND c.version = (
                SELECT MAX(c2.version) FROM Configuration c2 
                WHERE c2.namespace = :namespace AND c2.key = c.key
            )
        """;
        
        return entityManager.createQuery(jpql, Configuration.class)
                           .setParameter("namespace", namespace)
                           .getResultList();
    }
}

Read vs Write Separation

We implement a CQRS pattern for the configuration service to separate read and write operations:

// Write side - handles mutations
@Repository
public class ConfigurationWriteRepository {
    @PersistenceContext
    private EntityManager entityManager;
    
    public Configuration save(Configuration config) {
        entityManager.persist(config);
        return config;
    }
    
    public void delete(String id) {
        Configuration config = entityManager.find(Configuration.class, id);
        if (config != null) {
            entityManager.remove(config);
        }
    }
}

// Read side - optimized for queries
@Repository
public class ConfigurationReadRepository {
    @PersistenceContext
    private EntityManager entityManager;
    
    public List<Configuration> findByNamespace(String namespace) {
        String jpql = "SELECT c FROM Configuration c WHERE c.namespace = :namespace";
        return entityManager.createQuery(jpql, Configuration.class)
                           .setParameter("namespace", namespace)
                           .getResultList();
    }
    
    public Configuration findByKeyAndNamespace(String key, String namespace) {
        String jpql = "SELECT c FROM Configuration c WHERE c.key = :key AND c.namespace = :namespace";
        return entityManager.createQuery(jpql, Configuration.class)
                           .setParameter("key", key)
                           .setParameter("namespace", namespace)
                           .getSingleResult();
    }
}

Connection Pool Configuration

Connection pool is configured with HikariCP for optimal performance and resource management:

# application.yml
spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      validation-timeout: 5000
      leak-detection-threshold: 60000

N+1 Query Prevention Strategy

N+1 query problem is mitigated using eager loading and batch fetching:

// Using @BatchSize for collections to reduce queries
@Entity
@Table(name = "configurations")
public class Configuration {
    @Id
    private UUID id;
    
    @OneToMany(mappedBy = "configuration", fetch = FetchType.LAZY)
    @BatchSize(size = 20)
    private List<ConfigurationHistory> history;
    
    // Other fields...
}

// Using JOIN FETCH in JPQL to avoid N+1
@Repository
public class ConfigurationRepositoryImpl {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    public List<Configuration> findAllWithHistory() {
        String jpql = """
            SELECT c FROM Configuration c 
            JOIN FETCH c.history h 
            WHERE c.environment = :environment
        """;
        
        return entityManager.createQuery(jpql, Configuration.class)
                           .setParameter("environment", "production")
                           .getResultList();
    }
}

Batch Operation Design

Batch operations are implemented for bulk configuration updates:

@Repository
public class ConfigurationBatchRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    public void batchUpdate(List<Configuration> configs) {
        String jpql = "UPDATE Configuration c SET c.value = :value, c.version = c.version + 1 WHERE c.id = :id";
        TypedQuery<Configuration> query = entityManager.createQuery(jpql, Configuration.class);
        
        for (Configuration config : configs) {
            query.setParameter("id", config.getId());
            query.setParameter("value", config.getValue());
            query.executeUpdate();
        }
    }
    
    public void batchSave(List<Configuration> configs) {
        for (int i = 0; i < configs.size(); i++) {
            entityManager.persist(configs.get(i));
            
            // Flush every 1000 entities to prevent memory issues
            if (i % 1000 == 0) {
                entityManager.flush();
                entityManager.clear();
            }
        }
    }
}

Transaction Management

All repository operations are wrapped in proper transaction boundaries:

@Service
@Transactional
public class ConfigurationService {
    
    @Autowired
    private ConfigurationWriteRepository writeRepo;
    
    @Autowired
    private ConfigurationReadRepository readRepo;
    
    public Configuration updateConfiguration(String id, String newValue) {
        Configuration config = writeRepo.findById(id).orElseThrow(
            () -> new ConfigurationNotFoundException("Configuration not found: " + id)
        );
        
        // Update logic
        config.setValue(newValue);
        config.setVersion(config.getVersion() + 1);
        
        return writeRepo.save(config);
    }
}

Data Access Layer Metrics

Metrics are collected for monitoring database performance:

@Component
public class DatabaseMetrics {
    private final MeterRegistry meterRegistry;
    
    public DatabaseMetrics(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
    }
    
    public Timer.Sample startQueryTimer(String operation) {
        return Timer.start(meterRegistry);
    }
    
    public void recordQueryDuration(Timer.Sample sample, String operation) {
        sample.stop(Timer.builder("db.query.duration")
                         .tag("operation", operation)
                         .register(meterRegistry));
    }
}

The repository design balances type safety, performance, and maintainability while supporting the service layer's requirements for high availability and low-latency configuration access.

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The configuration service will use a hybrid synchronous/asynchronous approach. The primary synchronous operations include:

• Configuration creation/update/deletion (immediate consistency requirement)
• Configuration retrieval (low latency requirement)

Asynchronous processing is used for:

• Configuration change propagation to downstream services
• Audit logging and analytics aggregation
• Configuration validation and policy enforcement (background jobs)
• Rollback/recovery operations for failed configuration changes

The decision to use async processing for propagation is based on the fact that downstream services can tolerate eventual consistency and the configuration service should not block client requests for these operations.

Message schema design

{
  "version": "1.0",
  "event_type": "configuration.changed",
  "timestamp": "2023-10-15T14:30:00Z",
  "source": "config-service",
  "data": {
    "configuration_id": "cfg-12345",
    "key": "feature-flag-1",
    "value": "true",
    "environment": "production",
    "version": 5,
    "change_type": "UPDATE"
  },
  "metadata": {
    "trace_id": "abc-123-def-456",
    "correlation_id": "corr-789"
  }
}

Dead letter queue strategy and poison message handling

Poison messages will be routed to a dead letter queue (DLQ) after 3 failed delivery attempts. The DLQ will store messages with:

• Original message content
• Failure reason and stack trace
• Retry count and timestamp of last failure
• Service context (source, target)

A separate monitoring system will alert on DLQ queue size and individual poison messages. Manual intervention will be required to process these messages.

At-least-once vs. exactly-once delivery

We will implement at-least-once delivery with idempotency handling. This is chosen because:

• Configuration changes are generally idempotent (re-applying same config has same effect)
• The downstream services can handle duplicate events gracefully
• Simpler implementation than exactly-once semantics

Retry policy for consumers

Retry strategy:

• Max attempts: 3
• Backoff formula: exponential with jitter (1s, 2s, 4s ± 10% random)
• Non-retryable errors: Invalid configuration format, permission denied
• Retryable errors: Network timeouts, service unavailable

Outbox pattern for reliable event publishing

We will implement the outbox pattern to ensure message delivery reliability:

@Entity
@Table(name = "event_outbox")
public class EventOutbox {
    @Id
    private String id;
    
    @Column(name = "event_type")
    private String eventType;
    
    @Column(name = "payload")
    private String payload;
    
    @Column(name = "status")
    private EventStatus status;
    
    @Column(name = "retry_count")
    private int retryCount;
    
    @Column(name = "created_at")
    private LocalDateTime createdAt;
    
    @Column(name = "processed_at")
    private LocalDateTime processedAt;
}

public enum EventStatus {
    PENDING, PROCESSING, PROCESSED, FAILED
}

Sequence diagram for async flows

sequenceDiagram
    participant ConfigService
    participant EventPublisher
    participant MessageBroker
    participant ConsumerService

    ConfigService->>EventPublisher: Publish configuration.changed event
    EventPublisher->>MessageBroker: Send to topic
    MessageBroker-->>ConsumerService: Deliver event
    ConsumerService->>MessageBroker: Acknowledge receipt
    MessageBroker->>EventPublisher: Confirm delivery
    EventPublisher->>ConfigService: Update event status to PROCESSED

10. Error Handling Design

Error Classification Hierarchy

The error handling strategy follows a hierarchical classification to enable proper routing and response generation:

Error
├── DomainError (business logic violations)
│   ├── ValidationException (input validation failures)
│   ├── BusinessLogicException (rule violations, state transitions)
│   └── ResourceNotFoundException (entities not found)
├── InfrastructureError (system-level issues)
│   ├── DatabaseException (DB connection, query failures)
│   ├── ExternalServiceException (third-party API issues)
│   └── CacheException (cache access failures)
└── SystemError (internal system issues)
    ├── InternalServerException (unhandled exceptions)
    └── ConfigurationException (service misconfiguration)

Error Propagation Strategy

Errors are handled at their point of occurrence but follow a consistent propagation pattern:

1. Domain-level errors are caught and wrapped in appropriate domain exceptions
2. Infrastructure errors are logged with context and re-thrown as system errors or converted to domain errors when appropriate
3. System errors are logged and wrapped in internal server exceptions for consistent error responses

Retry Strategy with Exponential Backoff and Jitter

For external service calls and transient infrastructure issues, a retry mechanism is implemented:

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;

    public <T> T executeWithRetry(Supplier<T> operation) {
        int attempts = 0;
        Exception lastException = null;

        while (attempts < MAX_RETRIES) {
            try {
                return operation.get();
            } catch (ExternalServiceException e) {
                lastException = e;
                attempts++;
                
                if (attempts >= MAX_RETRIES) {
                    throw new RetryFailedException("Operation failed after " + MAX_RETRIES + " attempts", e);
                }

                long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempts) * 
                                   (1 + JITTER_FACTOR * (Math.random() - 0.5)));
                try {
                    Thread.sleep(delay);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new RetryFailedException("Retry interrupted", ie);
                }
            }
        }
        throw new RetryFailedException("Operation failed after " % MAX_RETRIES + " attempts", lastException);
    }
}

Circuit Breaker Design

For external dependencies, a circuit breaker pattern prevents cascading failures:

public class CircuitBreaker {
    private final String name;
    private final int failureThreshold;
    private final long timeoutMs;
    private final int resetTimeoutMs;

    private volatile CircuitState state = CircuitState.CLOSED;
    private volatile int failureCount = 0;
    private volatile long lastFailureTime = 0;

    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }

    public <T> T execute(Supplier<T> operation) {
        switch (state) {
            case CLOSED:
                return executeClosed(operation);
            case OPEN:
                return executeOpen(operation);
            case HALF_OPEN:
                return executeHalfOpen(operation);
            default:
                throw new IllegalStateException("Unknown circuit state: " + state);
        }
    }

    private <T> T executeClosed(Supplier<T> operation) {
        try {
            T result = operation.get();
            failureCount = 0; // Reset on success
            return result;
        } catch (Exception e) {
            failureCount++;
            lastFailureTime = System.currentTimeMillis();
            
            if (failureCount >= failureThreshold) {
                state = CircuitState.OPEN;
                throw new CircuitBreakerOpenException("Circuit breaker opened for " + name, e);
            }
            throw e;
        }
    }

    private <T> T executeOpen(Supplier<T> operation) {
        if (System.currentTimeMillis() - lastFailureTime > resetTimeoutMs) {
            state = CircuitState.HALF_OPEN;
            return executeHalfOpen(operation);
        }
        throw new CircuitBreakerOpenException("Circuit breaker is open for " + name);
    }

    private <T> T executeHalfOpen(Supplier<T> operation) {
        try {
            T result = operation.get();
            state = CircuitState.CLOSED;
            return result;
        } catch (Exception e) {
            state = CircuitState.OPEN;
            throw new CircuitBreakerOpenException("Circuit breaker re-opened for " + name, e);
        }
    }
}

Fallback Behavior Design

Fallbacks are implemented for critical external dependencies:

public class FallbackService {
    private final ExternalConfigClient client;
    
    public ConfigResponse getConfigWithFallback(String key) {
        try {
            return client.getConfig(key);
        } catch (ExternalServiceException e) {
            // Log the failure for monitoring
            logger.warn("External config service failed, returning default config for key: {}", key, e);
            
            // Return default configuration or cached value
            return getDefaultConfig(key);
        }
    }
    
    private ConfigResponse getDefaultConfig(String key) {
        // Return default config based on key or fallback to static defaults
        return new ConfigResponse(key, "default_value", Collections.emptyMap());
    }
}

Error Logging Format and Alerting Thresholds

Structured logging is implemented with consistent fields:

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);
    
    public void logError(String operation, Exception error, Map<String, Object> context) {
        Map<String, Object> logFields = new HashMap<>();
        logFields.put("timestamp", Instant.now().toString());
        logFields.put("operation", operation);
        logFields.put("errorType", error.getClass().getSimpleName());
        logFields.put("errorMessage", error.getMessage());
        logFields.put("stackTrace", error.getStackTrace());
        logFields.put("context", context);
        
        logger.error("Configuration service error: {}", logFields);
    }
}

Alerting thresholds:

• Critical (P0): Database connection failures, cache unavailability → Alert immediately
• High (P1): External service timeouts, retry failures → Alert after 3 consecutive failures
• Medium (P2): Validation errors, configuration parsing issues → Alert after 5 occurrences in 10 minutes

Error Response Format Standardization

All API errors follow a consistent JSON format:

{
  "error": {
    "code": "CONFIG_NOT_FOUND",
    "message": "Configuration with key 'feature.flag' not found",
    "details": {
      "key": "feature.flag",
      "timestamp": "2023-10-27T10:30:00Z"
    },
    "httpStatus": 404
  }
}

Exception Handling in Service Layer

@Service
public class ConfigService {
    
    @Transactional
    public ConfigResponse getConfig(String key) {
        try {
            // Business logic
            return configRepository.findByKey(key)
                .orElseThrow(() -> new ResourceNotFoundException("Configuration not found for key: " + key));
        } catch (ResourceNotFoundException e) {
            // Re-throw domain errors as-is
            throw e;
        } catch (Exception e) {
            // Wrap infrastructure/system errors
            throw new InternalServerException("Failed to retrieve configuration", e);
        }
    }
}

Retry Configuration Parameters

Parameter	Value	Reason
Max retries	3	Prevents infinite retry loops
Base delay	1000ms	Initial backoff time
Backoff multiplier	2x	Exponential increase to avoid overwhelming systems
Jitter factor	±20%	Prevents thundering herd
Retryable exceptions	ExternalServiceException, DatabaseException	Transient failures that may succeed on retry

Circuit Breaker Configuration

Parameter	Value	Reason
Failure threshold	5	Prevents cascading failures
Timeout	30 seconds	Time to attempt reset
Reset timeout	60 seconds	Time before half-open state
Circuit breaker name	"external-config-service"	Clear identification

Error Handling in Asynchronous Flows

For async operations, error handling includes:

1. Retry logic within async consumers
2. Dead letter queue (DLQ) for permanently failed messages
3. Error logging with full context for debugging
4. Metrics tracking of error rates and retry counts

Error Recovery Patterns

1. Idempotent operations: Ensure retries don't cause side effects
2. State validation: Verify state consistency before retrying
3. Graceful degradation: Return defaults when primary systems fail
4. Monitoring and alerting: Immediate notification of system failures

This error handling design ensures robustness, observability, and graceful degradation while maintaining service availability under various failure conditions.

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using a combination of:

• Regex patterns for format validation (e.g., ^[a-zA-Z0-9_-]+$ for configuration keys)
• Length constraints (max 255 characters for keys, 10k for values)
• Type validation (JSON schema validation for configuration values)
• Sanitization libraries: OWASP Java HTML Sanitizer for any user-provided content that might be rendered

SQL Injection Prevention

• Prepared statements for all database queries using parameterized queries (JDBC/MyBatis)
• Whitelisting of table/column names - no dynamic SQL construction
• Input escaping for any non-parameterized components (e.g., in custom queries)

Authentication Middleware Design

• JWT-based authentication with 2-hour expiration and 15-minute refresh window
• OAuth2 integration for service-to-service authentication (using client credentials flow)
• Authentication middleware that intercepts all requests and validates JWT tokens
• Token revocation endpoint to invalidate tokens during logout or security events

Authorization Enforcement (RBAC)

• Role-based access control with roles: ADMIN, READ_ONLY, CONFIG_EDITOR
• Resource-based permissions:
  • ADMIN: Full CRUD on all configurations
  • CONFIG_EDITOR: Create/Update/Delete on specific namespaces
  • READ_ONLY: Read access only
• Permission checking at service layer using @PreAuthorize annotations (Spring Security)
• Namespace-based access control - users can only access configurations within their assigned namespaces

Sensitive Data Encryption at Rest and in Transit

• At rest: AES-256 encryption using AWS KMS for configuration values stored in DB
• In transit: TLS 1.3 with HSTS enabled for all API endpoints
• Encryption keys managed by AWS KMS with automatic rotation every 90 days

PII Handling and Masking Strategy

• PII data (user IDs, email addresses) in configuration values are encrypted before storage
• Masking in logs: Any PII fields in logs are redacted (e.g., user@example.com → user@***.com)
• Audit logs do not contain raw configuration values - only metadata (key, namespace, user, timestamp)

Secrets Management

• Secrets stored in AWS Secrets Manager with versioning and rotation policies
• Application secrets: JWT signing keys, database credentials, external service tokens
• Rotation policy: 90-day rotation for all secrets with automatic failover
• Access control using IAM policies to restrict access to secrets

Audit Logging Design

• Logged events:
  • Configuration created/updated/deleted
  • User login/logout
  • Access denied attempts
• Log format:

{
  "timestamp": "2023-10-15T14:30:00Z",
  "user_id": "user-123",
  "action": "CONFIG_UPDATE",
  "resource_type": "configuration",
  "resource_id": "config-key-456",
  "namespace": "prod-app",
  "ip_address": "10.0.0.1",
  "user_agent": "Mozilla/5.0...",
  "status": "SUCCESS"
}

• Retention policy: 90 days for audit logs with archiving to S3 for compliance
• Alerting: Security events (failed auth, unauthorized access) trigger immediate alerts to security team

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

• Service Layer: 85% code coverage, focusing on business logic and edge cases
• Repository Layer: 90% coverage with mock database interactions
• Domain Objects: 100% coverage for value objects and immutable entities
• Error Handling Paths: 100% coverage for all exception scenarios
• Validation Logic: 95% coverage for input validation rules

Integration Testing Strategy

• Database Integration Tests:

  • Use testcontainers for PostgreSQL/MySQL with schema migration
  • Test transaction boundaries and concurrent access patterns
  • Validate read-after-write consistency in distributed setup

• External Service Mocks:

  • WireMock for external configuration source integrations (e.g., Consul, etcd)
  • Test failure scenarios and retry logic with mocked network issues

• Event Integration Tests:

  • Use embedded Redis or Kafka for event-driven flows
  • Validate message consumption and processing consistency

Contract Testing

• Pact Broker Integration:
  • Define consumer-driven contracts for configuration service clients
  • Test API boundary changes and backward compatibility
  • Validate JSON schema compliance for configuration payloads

Performance Testing Tool and Key Scenarios

• Tool: k6 (JavaScript-based load testing)
• Key Scenarios:
  • Concurrent configuration fetch requests: 1000 concurrent users with 50% read, 50% write
  • High-frequency configuration updates: 1000 updates/sec with 50% cache hits
  • Large configuration payload handling: 1MB+ JSON configurations under load
  • Cache warming and invalidation under burst traffic

Test Data Management Strategy

• Fixtures:

  • YAML-based configuration templates for different environments (dev, staging, prod)
  • Predefined test configurations with realistic feature flags and values

• Factories:

  • Factory pattern for generating valid configuration objects with randomization
  • Support for generating edge case configurations (empty, malformed, large payloads)

• Seeding:

  • Database seed scripts for initial test setup
  • Configuration versioning with rollback capability during testing
  • Environment-specific test data sets with proper isolation between test runs

13. Observability Design

Structured Logging Schema

The logging system will use structured JSON format to ensure consistency and enable efficient querying. Required fields include:

{
  "timestamp": "2023-10-27T14:30:00.000Z",
  "level": "INFO",
  "service": "config-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user_12345",
  "entity_id": "config_abcde",
  "request_id": "req_98765",
  "event_type": "config_updated",
  "message": "Configuration updated successfully",
  "error_code": "CONFIG_UPDATE_FAILED",
  "error_details": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid configuration format"
  },
  "duration_ms": 45,
  "method": "PUT",
  "path": "/config/v1/feature-flag",
  "status_code": 200,
  "ip_address": "192.168.1.1",
  "user_agent": "Mozilla/5.0...",
  "tags": ["feature_flag", "dynamic_config"]
}

PII masking rules:

• User IDs are masked in logs when not needed for debugging
• IP addresses are truncated to network level (e.g., 192.168.1.0/24)
• API keys and tokens are completely redacted

Distributed Tracing Instrumentation

Using OpenTelemetry for distributed tracing with custom spans:

Key Operations to Trace:

1. config_fetch - Fetch configuration for client
2. config_update - Update configuration via API
3. config_validation - Validate configuration schema
4. config_broadcast - Broadcast config changes to clients

Span Attributes:

// Example custom span for config update operation
Span span = tracer.spanBuilder("config_update")
    .setAttribute("config_id", configId)
    .setAttribute("config_type", "feature_flag")
    .setAttribute("user_id", userId)
    .setAttribute("client_version", clientVersion)
    .setAttribute("environment", environment)
    .startSpan();

Metrics Design

RED Method Metrics:

Metric Name	Type	Labels	Description
config_requests_total	Counter	method, path, status_code, environment	Total config requests
config_request_duration_seconds	Histogram	method, path, status_code, environment	Request processing time
config_errors_total	Counter	error_type, method, path, environment	Config operation errors
config_cache_hits_total	Counter	cache_type, environment	Cache hit statistics
config_updates_total	Counter	config_type, environment	Configuration updates
config_clients_connected	Gauge	environment	Active client connections

Additional Metrics:

• config_validation_failures - Counter for validation errors
• config_broadcast_duration_seconds - Histogram for broadcast times
• config_cache_misses_total - Counter for cache misses

Health Check Endpoint Design

GET /health
GET /ready
GET /live
GET /startup

Health Check Responses:

{
  "status": "healthy",
  "timestamp": "2023-10-27T14:30:00.000Z",
  "service": "config-service",
  "version": "1.2.3",
  "dependencies": {
    "database": "healthy",
    "redis": "healthy",
    "message_broker": "healthy"
  },
  "uptime_seconds": 3600,
  "metrics_collector": "healthy"
}

Alerting Thresholds

Critical Alerts (require on-call escalation):

• config_errors_total > 100 per minute (5 min window)
• config_request_duration_seconds p99 > 500ms (1 min window)
• config_cache_misses_total > 500 per minute (5 min window)
• Database connection pool exhaustion > 10% of capacity

Warning Alerts (monitoring):

• config_request_duration_seconds p95 > 200ms (1 min window)
• config_errors_total > 10 per minute (5 min window)
• Cache hit ratio < 80% for 15 minutes
• Redis memory usage > 85% of limit

On-call escalation path:

1. Slack alert to config team
2. Email notification to engineering lead
3. Page to SRE team if no response within 15 minutes

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• Configuration refresh strategy: Should clients pull configuration changes or receive push notifications? Push requires a pub/sub system that may not be available in all environments, while pull introduces polling overhead and potential staleness.
• Rollback policy: How should we handle configuration rollbacks when a bad version is deployed? Should this be automatic or require manual intervention?
• Feature flag vs. configuration: When to use feature flags instead of simple configuration values? This decision affects the complexity and cost of the service.
• Multi-environment support: Should we support configuration per environment (dev, staging, prod) or treat all environments as a single namespace with overrides?

Known trade-offs and justifications

• Consistency vs. availability: The service must balance between strong consistency (all clients get the same config immediately) and high availability (clients can continue working even if config service is down). We've chosen eventual consistency with cache warming to favor availability.
• Granularity vs. complexity: Fine-grained configuration per client or group vs. simpler global configuration. We chose a hierarchical approach (global → environment → client) to balance complexity with flexibility.
• Real-time vs. batch updates: Real-time configuration changes vs. periodic batch refreshes. We've opted for real-time with caching to reduce latency while maintaining performance.
• Caching strategy vs. freshness: Using aggressive caching for performance vs. ensuring configuration freshness. We've implemented a hybrid approach with TTLs and cache warming to maintain balance.

Risks and mitigation strategies

• Cache stampede: High concurrent client requests could overwhelm the service. Mitigation includes probabilistic early expiry and distributed locks for cache warming.
• Configuration drift: Different environments or clients may end up with different configurations. Mitigation involves strict versioning, audit trails, and environment-specific validation.
• Service availability impact: If the configuration service goes down, all clients may fail. Mitigation includes client-side fallbacks and graceful degradation strategies.
• Security misconfigurations: Improper access control or data exposure. Mitigation includes strict RBAC, encryption at rest and in transit, and comprehensive audit logging.

Technical debt acknowledged

• Configuration schema evolution: As the service grows, managing backward compatibility in configuration schemas becomes increasingly complex. We've implemented versioned configuration schemas with clear migration paths but acknowledge this will require ongoing maintenance.
• Eventual consistency complexity: The eventual consistency model introduces complexity in client-side state management and error handling. This is acceptable for the use case but requires careful client implementation.
• Multi-tenant configuration isolation: While we've implemented basic tenant isolation, more sophisticated multi-tenancy features (like cross-tenant configuration inheritance) are not yet supported and would require significant refactoring.
• Audit logging overhead: Comprehensive audit trails increase storage and processing costs. We've implemented a configurable audit log level but acknowledge this could become a performance bottleneck at scale.