State Machine Design Patterns and Implementation

Comprehensive low-level design guide for implementing robust state machines with event-driven transitions and persistence capabilities.

PublishedJuly 12, 2026
Length7,719 words
Reading time39 min read

Backend LLD

State machines are fundamental constructs for modeling system behavior through well-defined states and transitions. This document presents a production-ready low-level design approach for implementing flexible, scalable state machines with proper error handling and persistence support.

1. Problem Statement & Requirements

Clarifying questions

What specific system or domain does this state machine apply to? Is it for order lifecycle, ticket booking, user account status, or another domain?
What is the expected scale of state transitions? Will we be handling millions of concurrent state changes or is this a smaller-scale system?
What consistency requirements are there? Should state transitions be strongly consistent, eventually consistent, or can eventual consistency work for some operations?
What is the read/write ratio for state data? Is this primarily a write-heavy system or does it require frequent reads of current state?
What are the latency sensitivity requirements? Are there strict SLAs for state transitions (e.g., <100ms for order status change)?
What authentication and authorization model is in place? Will state transitions be restricted to specific roles or users?
Are there external systems that need to be notified of state changes? Do we need to integrate with third-party services or external APIs?

Functional requirements

Support state transitions for a defined set of entities (e.g., orders, tickets, user accounts)
Maintain audit trail of all state transitions with timestamps and actors
Allow querying current state of entities
Support conditional state transitions (e.g., only allow transition from "pending" to "confirmed" if payment is received)
Enforce state transition rules (e.g., cannot go from "completed" to "pending")
Support rollback or revert capability for specific state transitions (if needed)
Handle concurrent state changes without data corruption
Support bulk state update operations where appropriate

Non-functional requirements

Latency targets: p50: 20ms, p95: 50ms, p99: 100ms for state transition operations
Throughput: 50,000 state transitions per second across all entities
Availability: 99.9% uptime with graceful degradation capability
Scalability: Support horizontal scaling to handle peak loads (e.g., 1M concurrent state transitions)
Durability: All state transitions must be persisted to disk and not lost during system failures
Consistency: Strong consistency for state reads within 100ms of write

Constraints and assumptions

State machine will be implemented in a distributed system with eventual consistency for cross-service operations
All state transitions must be idempotent to handle retries and network issues
State data will be stored in a relational database with ACID guarantees for each transition operation
System assumes that state transitions are infrequent compared to reads (read-heavy workload)
No external dependencies for core state machine functionality
System will not support real-time state change notifications (e.g., WebSocket push) in this design

Out of scope

Real-time state change notifications or push-based updates to clients
Complex business logic validation beyond simple state transition rules (e.g., payment validation, inventory checks)
Multi-step workflows or complex conditional logic beyond simple state transitions
Integration with external systems for state change notifications (e.g., sending emails or calling webhooks)
Support for state machine versioning or backward compatibility of state definitions
UI or dashboard components for managing state machines
Support for state machine configuration via external APIs (e.g., runtime modification of state rules)
Integration with message queues or event streaming platforms for async state change processing

2. API Design

API Style Selection

For this state machine system, I'll select gRPC as the primary API style due to:

Strong typing: gRPC's Protocol Buffers provide compile-time type safety, reducing runtime errors
Efficient serialization: Binary protocol is more efficient than JSON for high-throughput systems
Built-in streaming support: Enables real-time state change notifications and event streams
Service contracts: Clear, versioned interfaces that support evolution over time
Cross-language support: Enables multi-language microservices ecosystem

Endpoint Definitions

Method	Path	Description	Auth	Rate Limit
GetState	`/v1/state/{id}`	Retrieve current state of a state machine instance	JWT	100 RPS
Transition	`/v1/state/{id}/transition`	Trigger a state transition	JWT	50 RPS
ListStates	`/v1/states`	List all state machine instances	JWT	20 RPS
GetHistory	`/v1/state/{id}/history`	Retrieve state transition history	JWT	50 RPS
ResetState	`/v1/state/{id}/reset`	Reset state machine to initial state	JWT	10 RPS

Request and Response Schemas

// StateMachine.proto
syntax = "proto3";

package state_machine;

message StateRequest {
  string id = 1;
  string state_type = 2;
}

message StateResponse {
  string id = 1;
  string current_state = 2;
  string state_type = 3;
  int64 created_at = 4;
  int64 updated_at = 5;
  repeated StateTransition history = 6;
}

message StateTransition {
  string from_state = 1;
  string to_state = 2;
  string event_type = 3;
  int64 timestamp = 4;
  string actor_id = 5;
  optional string reason = 6;
}

message TransitionRequest {
  string id = 1;
  string event_type = 2;
  optional string actor_id = 3;
  optional string reason = 4;
  optional string metadata = 5;
}

message TransitionResponse {
  string id = 1;
  string current_state = 2;
  string previous_state = 3;
  repeated StateTransition history = 4;
  bool success = 5;
}

message ListStatesRequest {
  string state_type = 1;
  int32 page_size = 2;
  string page_token = 3;
}

message ListStatesResponse {
  repeated StateResponse states = 1;
  string next_page_token = 2;
}

message ResetRequest {
  string id = 1;
  optional string actor_id = 2;
  optional string reason = 3;
}

message ResetResponse {
  string id = 1;
  string new_state = 2;
  bool success = 3;
}

HTTP Status Codes and Error Response Format

Standard HTTP status codes:

200 OK - Successful GET, PUT, POST
204 No Content - Successful DELETE or state reset
400 Bad Request - Invalid request parameters
401 Unauthorized - Missing or invalid authentication
403 Forbidden - Insufficient permissions
404 Not Found - State machine instance not found
429 Too Many Requests - Rate limit exceeded
500 Internal Server Error - Unexpected server error

Standardized error response format:

{
  "error": {
    "code": "STATE_TRANSITION_FAILED",
    "message": "Cannot transition from 'ACTIVE' to 'CANCELLED'",
    "details": {
      "state_id": "sm-123",
      "from_state": "ACTIVE",
      "to_state": "CANCELLED"
    },
    "timestamp": "2023-10-15T14:30:00Z"
  }
}

API Versioning Strategy

Versioned using path prefix: /v1/ with future versions as /v2/, etc.

Pagination Design

Cursor-based pagination using page_token field in request/response. This approach:

Provides consistent results even when data changes
Avoids issues with offset-based pagination when items are inserted/deleted
Is more efficient for large datasets

Authentication and Authorization

All endpoints require JWT authentication with scopes:

state.read - Read access to state machines
state.write - Write access to state machines
state.reset - Reset capability for state machines

Rate Limiting Strategy

Rate limiting implemented per endpoint with:

Token bucket algorithm for burst handling
Per-user rate limits (100 RPS GET, 50 RPS POST)
IP-based fallback for anonymous requests
Redis-based rate limiting with sliding window

Idempotency Design

For mutation endpoints, idempotency key strategy:

Clients provide X-Idempotency-Key header
Server stores operation result by key
If same key is used, return cached result
Key format: state-transition-{state_id}-{event_type}-{timestamp}

// IdempotencyKeyGenerator.java
public class IdempotencyKeyGenerator {
    public static String generateKey(String stateId, String eventType, String actorId) {
        return String.format("state-transition-%s-%s-%s", 
                           stateId, eventType, actorId);
    }
}

3. Data Model Design

Entity-Relationship diagram (Mermaid)

erDiagram
    STATE_MACHINE {
        string id PK
        string name
        string initial_state
        string current_state
        jsonb states
        jsonb transitions
        timestamp created_at
        timestamp updated_at
        string created_by
        string updated_by
    }

    STATE_MACHINE_EVENT {
        string id PK
        string state_machine_id FK
        string event_name
        string source_state
        string target_state
        timestamp occurred_at
        jsonb metadata
    }

    STATE_MACHINE_STATE {
        string id PK
        string state_machine_id FK
        string state_name
        boolean is_final
        jsonb properties
        timestamp created_at
    }

    STATE_MACHINE_TRANSITION {
        string id PK
        string state_machine_id FK
        string from_state
        string to_state
        string event_name
        jsonb conditions
        jsonb actions
        timestamp created_at
    }

    STATE_MACHINE {
        string id PK
    }

    STATE_MACHINE_EVENT {
        string state_machine_id FK
    }

    STATE_MACHINE_STATE {
        string state_machine_id FK
    }

    STATE_MACHINE_TRANSITION {
        string state_machine_id FK
    }

Table / Collection schema definitions with field types, constraints, indices

state_machines

CREATE TABLE state_machines (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL,
    initial_state VARCHAR(100) NOT NULL,
    current_state VARCHAR(100),
    states JSONB NOT NULL,
    transitions JSONB NOT NULL,
    created_by VARCHAR(255) NOT NULL,
    updated_by VARCHAR(255),
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    deleted_at TIMESTAMP WITH TIME ZONE
);

CREATE INDEX idx_state_machines_name ON state_machines(name);
CREATE INDEX idx_state_machines_created_at ON state_machines(created_at);
CREATE INDEX idx_state_machines_current_state ON state_machines(current_state);
CREATE INDEX idx_state_machines_deleted_at ON state_machines(deleted_at) WHERE deleted_at IS NULL;

state_machine_events

CREATE TABLE state_machine_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    state_machine_id UUID NOT NULL REFERENCES state_machines(id) ON DELETE CASCADE,
    event_name VARCHAR(255) NOT NULL,
    source_state VARCHAR(100) NOT NULL,
    target_state VARCHAR(100),
    occurred_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    metadata JSONB,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_state_machine_events_state_machine_id ON state_machine_events(state_machine_id);
CREATE INDEX idx_state_machine_events_event_name ON state_machine_events(event_name);
CREATE INDEX idx_state_machine_events_occurred_at ON state_machine_events(occurred_at);

state_machine_states

CREATE TABLE state_machine_states (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    state_machine_id UUID NOT NULL REFERENCES state_machines(id) ON DELETE CASCADE,
    state_name VARCHAR(100) NOT NULL,
    is_final BOOLEAN DEFAULT FALSE,
    properties JSONB,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_state_machine_states_state_machine_id ON state_machine_states(state_machine_id);
CREATE INDEX idx_state_machine_states_state_name ON state_machine_states(state_name);

state_machine_transitions

CREATE TABLE state_machine_transitions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    state_machine_id UUID NOT NULL REFERENCES state_machines(id) ON DELETE CASCADE,
    from_state VARCHAR(100) NOT NULL,
    to_state VARCHAR(100) NOT NULL,
    event_name VARCHAR(255) NOT NULL,
    conditions JSONB,
    actions JSONB,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_state_machine_transitions_state_machine_id ON state_machine_transitions(state_machine_id);
CREATE INDEX idx_state_machine_transitions_from_to_states ON state_machine_transitions(from_state, to_state);
CREATE INDEX idx_state_machine_transitions_event_name ON state_machine_transitions(event_name);

Primary key strategy (UUID vs. auto-increment vs. ULID — with justification)

Using UUID as primary key for all tables to support:

Distributed system compatibility (no coordination needed for ID generation)
Multi-region deployment capability
Audit trail integrity (can be joined across systems)
No hotspots in database sharding
Better security (no sequential predictability)

Index design (composite, partial, covering indices — with justification per index)

idx_state_machines_name: For efficient querying by state machine name
idx_state_machines_created_at: For time-based queries and audit trails
idx_state_machines_current_state: For filtering state machines by current state (common query pattern)
idx_state_machines_deleted_at: Partial index to efficiently filter active records only
idx_state_machine_events_state_machine_id: For efficient event lookups by state machine
idx_state_machine_events_event_name: For filtering events by event type
idx_state_machine_events_occurred_at: For time-series analysis and chronological queries
idx_state_machine_states_state_machine_id: For efficient state lookups by state machine
idx_state_machine_states_state_name: For direct state name lookups
idx_state_machine_transitions_state_machine_id: For efficient transition lookups by state machine
idx_state_machine_transitions_from_to_states: Composite index for fast transition lookups by state pair
idx_state_machine_transitions_event_name: For event-driven transition filtering

Data partitioning / sharding strategy

Partition by created_at timestamp monthly to support:

Time-based data lifecycle management
Improved query performance for recent state machine operations
Easier archival of older data
Better distribution of read/write loads across time periods

Soft delete vs. hard delete decision with justification

Soft delete using deleted_at timestamp field because:

Audit trail preservation for compliance and debugging
Ability to restore accidentally deleted state machines
Historical analysis of system evolution over time
Data integrity for dependent event tracking and reporting

Audit trail design (who, what, when)

All state machine operations capture:

created_by and updated_by fields for user tracking
created_at and updated_at timestamps for temporal tracking
state_machine_events table captures all state transitions with metadata
Full audit trail maintained in state_machine_events for compliance and debugging

Data archival strategy and retention policy

Daily Archival: State machine events older than 30 days are moved to archive storage (S3 or similar)
Monthly Archival: State machine definitions older than 1 year are archived
Quarterly Cleanup: Completely remove archived state machines after 3 years of inactivity
Retention Policy:
- Active state machines: Keep forever (with soft delete)
- Event history: Keep 30 days for operational analysis
- Historical state machine definitions: Keep 1 year for compliance
Archival Format: JSON format with metadata preserved for full audit capability

4. Class Design & Service Layer

classDiagram
    class StateMachine {
        -currentState: State
        -stateTransitions: Map~State, Set~State~~
        +handleEvent(event: Event): Result
        +getCurrentState(): State
    }
    
    class State {
        -name: String
        -actions: List~Action~
        +execute(context: Context): Result
    }
    
    class Event {
        -name: String
        -payload: Map~String, Object~
    }
    
    class Context {
        -data: Map~String, Object~
        -userId: String
        -entityId: String
    }
    
    class Action {
        -name: String
        -handler: EventHandler
        +execute(context: Context): Result
    }
    
    class EventHandler {
        +handle(context: Context): Result
    }
    
    StateMachine --> State : manages
    State --> Action : contains
    StateMachine --> Event : processes
    Context --> State : provides context for
    Action --> EventHandler : delegates to

stateDiagram-v2
    [*] --> Created
    Created --> Pending : create()
    Pending --> Active : approve()
    Active --> Completed : complete()
    Active --> Cancelled : cancel()
    Pending --> Cancelled : reject()
    Cancelled --> [*]
    Completed --> [*]

Service Class Structure and Responsibilities

The state machine service layer will include:

StateMachineService - Main orchestrator for state transitions
StateTransitionValidator - Validates state transitions before execution
EventPublisher - Publishes state change events to message queue
StateRepository - Persists current state and audit logs

Interface / Abstract Class Design

public interface StateMachine {
    Result handleEvent(Event event);
    State getCurrentState();
}

public interface EventHandler {
    Result handle(Context context);
}

public abstract class AbstractStateMachine implements StateMachine {
    protected State currentState;
    protected Map<State, Set<State>> validTransitions;
    
    @Override
    public Result handleEvent(Event event) {
        if (!isValidTransition(event)) {
            throw new InvalidStateException("Invalid state transition");
        }
        
        return executeTransition(event);
    }
    
    protected abstract boolean isValidTransition(Event event);
    protected abstract Result executeTransition(Event event);
}

Business Logic Encapsulation Patterns

We'll use the State Pattern to encapsulate state-specific behavior:

public class OrderStateMachine extends AbstractStateMachine {
    private final EventPublisher eventPublisher;
    private final StateRepository stateRepository;
    
    public OrderStateMachine(EventPublisher eventPublisher, 
                           StateRepository stateRepository) {
        this.eventPublisher = eventPublisher;
        this.stateRepository = stateRepository;
        initializeTransitions();
    }
    
    @Override
    protected boolean isValidTransition(Event event) {
        return validTransitions.getOrDefault(currentState, Collections.emptySet())
                              .contains(event.getTargetState());
    }
    
    @Override
    protected Result executeTransition(Event event) {
        State previousState = currentState;
        currentState = event.getTargetState();
        
        // Update database
        stateRepository.updateState(event.getEntityId(), currentState, previousState);
        
        // Publish event
        eventPublisher.publish(new StateChangeEvent(
            event.getEntityId(), 
            previousState, 
            currentState,
            event.getMetadata()
        ));
        
        return new Result("SUCCESS", "State transition completed");
    }
    
    private void initializeTransitions() {
        validTransitions = new HashMap<>();
        validTransitions.put(OrderState.CREATED, Set.of(OrderState.PENDING));
        validTransitions.put(OrderState.PENDING, Set.of(OrderState.ACTIVE, OrderState.CANCELLED));
        validTransitions.put(OrderState.ACTIVE, Set.of(OrderState.COMPLETED, OrderState.CANCELLED));
    }
}

Critical-Path Method Implementation

public class OrderService {
    private final OrderRepository orderRepository;
    private final StateMachineService stateMachineService;
    private final EventPublisher eventPublisher;
    private final TransactionManager transactionManager;
    
    public Result processOrderEvent(String orderId, OrderEvent event) {
        return transactionManager.executeInTransaction(() -> {
            // Load order with pessimistic locking
            Order order = orderRepository.findByIdWithLock(orderId);
            
            // Validate order state and event
            if (!stateMachineService.canHandleEvent(order.getState(), event)) {
                throw new InvalidOrderStateException(
                    "Cannot process event " + event.getType() + 
                    " in current state " + order.getState()
                );
            }
            
            // Execute state transition
            StateChangeResult stateChange = stateMachineService.handleEvent(
                new EventBuilder()
                    .withEntityId(orderId)
                    .withEventType(event.getType())
                    .withMetadata(event.getMetadata())
                    .build()
            );
            
            // Update order with new state
            order.setState(stateChange.getNewState());
            order.setLastUpdated(Instant.now());
            
            // Save updated order
            orderRepository.save(order);
            
            // Publish domain events
            eventPublisher.publish(
                new OrderStateChangedEvent(
                    order.getId(),
                    order.getState(),
                    stateChange.getPreviousState(),
                    event.getMetadata()
                )
            );
            
            return new Result("SUCCESS", "Order processed successfully");
        });
    }
}

Transaction Boundary Design

A single transaction includes:

Loading the entity with pessimistic locking
Validating state transition
Updating the entity state in database
Publishing domain events (in same transaction for reliability)

Service Interface Contracts

public interface OrderService {
    Result processOrderEvent(String orderId, OrderEvent event);
    OrderState getCurrentState(String orderId);
    List<OrderState> getValidTransitions(String orderId);
}

Saga / Orchestration Pattern

For distributed transactions involving external systems:

public class OrderSaga {
    private final SagaManager sagaManager;
    
    public void handleOrderCreation(Order order) {
        sagaManager.startSaga(
            new OrderCreationSaga(order)
                .withCompensationHandler(this::handleOrderCancellation)
        );
    }
}

Domain Object / Value Object Design

public class Order {
    private String id;
    private OrderState state;
    private Map<String, Object> metadata;
    private Instant createdAt;
    private Instant lastUpdated;
    
    // Getters and setters
}

public enum OrderState {
    CREATED, PENDING, ACTIVE, COMPLETED, CANCELLED
}

public class OrderEvent {
    private String orderId;
    private String eventType;
    private Map<String, Object> metadata;
}

5. Concurrency & Thread Safety

Concurrency Model

The system uses a thread-per-request model with a fixed-size thread pool for handling concurrent operations. For state machine transitions, we employ a combination of:

Java's java.util.concurrent for thread-safe operations
ReentrantLock for fine-grained locking where needed
AtomicReference and AtomicBoolean for lock-free state updates
ThreadLocal for per-request context (e.g., trace IDs, user contexts)

Race Condition Prevention Strategy

Race conditions in state machine transitions are mitigated through:

Optimistic locking using version numbers on state entities
Atomic operations for state value updates
Synchronized blocks for critical sections where necessary
Immutable state objects to prevent concurrent modification

Optimistic vs. Pessimistic Locking Selection

We choose optimistic locking for state machine transitions because:

Low contention: State transitions are infrequent and typically involve single entities
Better throughput: Avoids blocking threads during lock acquisition
Scalability: Reduces lock contention in high-concurrency scenarios
Simpler recovery: Failed updates can be retried with minimal overhead

Implementation uses a version field in the state entity:

public class StateMachineEntity {
    private Long version;
    private String currentState;
    
    // Getters and setters
}

Deadlock Prevention Strategy

Deadlock prevention is implemented through:

Lock ordering: All state transitions acquire locks in a consistent order (e.g., by entity ID)
Timeout-based acquisition: Use tryLock() with timeout to prevent indefinite blocking
Hierarchical lock management: Lock entities in a consistent hierarchy (e.g., by domain, then by ID)
Deadlock detection: Periodic monitoring of lock wait chains

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is handled through:

Idempotency keys in request headers (e.g., X-Idempotency-Key)
Request deduplication cache using Redis or in-memory store with TTL
State transition validation: Check if the requested state change is already applied
Version-based idempotency: If version matches, no-op the operation

public class IdempotencyService {
    private final RedisTemplate<String, String> redisTemplate;
    
    public boolean isIdempotent(String idempotencyKey) {
        String key = "idempotency:" + idempotencyKey;
        return redisTemplate.hasKey(key);
    }
    
    public void markIdempotent(String idempotencyKey) {
        String key = "idempotency:" + idempotencyKey;
        redisTemplate.opsForValue().set(key, "processed", 30, TimeUnit.MINUTES);
    }
}

Thread Safety in State Machine Operations

State machine operations are made thread-safe through:

Immutable state objects - once created, state values don't change
Thread-local state tracking for operation context
Synchronized access to shared mutable state
Atomic reference updates for state transitions

public class ThreadSafeStateMachine {
    private final AtomicReference<State> currentState = new AtomicReference<>();
    
    public boolean transitionTo(State newState) {
        return currentState.compareAndSet(currentState.get(), newState);
    }
}

6. Sequence Diagrams for Key Flows

Happy Path Sequence Diagram

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Service as StateMachineService
    participant Repo as StateMachineRepository
    participant DB as Database

    U->>API: POST /state-machines/{id}/transition
    API->>Service: validateRequest()
    Service->>Repo: findById(id)
    Repo->>DB: SELECT * FROM state_machines WHERE id = ?
    DB-->>Repo: return StateMachineEntity
    Repo-->>Service: return StateMachine
    Service->>Service: validateTransition(fromState, toState)
    Service->>Repo: updateState(id, newState)
    Repo->>DB: UPDATE state_machines SET current_state = ? WHERE id = ?
    DB-->>Repo: return updated StateMachine
    Repo-->>Service: return updated StateMachine
    Service->>Service: publishStateChangeEvent()
    Service-->>API: return success response
    API-->>U: 200 OK

Error / Edge Case Sequence Diagram

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Service as StateMachineService
    participant Repo as StateMachineRepository

    U->>API: POST /state-machines/{id}/transition
    API->>Service: validateRequest()
    Service->>Repo: findById(id)
    Repo-->>Service: return null (not found)
    Service->>Service: throw StateMachineNotFoundException
    Service-->>API: return 404 Not Found
    API-->>U: 404 Not Found

Async Flow Sequence Diagram

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Service as StateMachineService
    participant EventPublisher as EventPublisher
    participant EventBus as MessageBus

    U->>API: POST /state-machines/{id}/transition
    API->>Service: validateRequest()
    Service->>Service: validateTransition(fromState, toState)
    Service->>EventPublisher: publishStateChangeEvent()
    EventPublisher->>EventBus: send event to queue
    EventBus-->>EventPublisher: event acknowledged
    EventPublisher-->>Service: return success
    Service-->>API: return 200 OK
    Note over EventBus,EventPublisher: Async processing of state change events

Error Handling Flow

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Service as StateMachineService
    participant Repo as StateMachineRepository
    participant DB as Database

    U->>API: POST /state-machines/{id}/transition
    API->>Service: validateRequest()
    Service->>Repo: findById(id)
    Repo->>DB: SELECT * FROM state_machines WHERE id = ?
    DB-->>Repo: return StateMachineEntity
    Repo-->>Service: return StateMachine
    Service->>Service: validateTransition(fromState, toState)
    Service->>Service: throw InvalidTransitionException
    Service-->>API: return 400 Bad Request
    API-->>U: 400 Bad Request with error details

7. Caching Design

Cache Strategy Selection

For this system, we'll use a cache-aside pattern with LRU eviction policy. This approach provides the best balance between simplicity, performance, and control over cache behavior. Cache-aside allows us to explicitly manage when data is loaded into cache, enabling better control over cache warming and invalidation strategies.

Cache Key Design

Cache keys will follow a structured namespace pattern:

{namespace}:{entityType}:{entityId}:{version}

Example key structure for user data:

user:profile:12345:v1

TTL Configuration

Data Type	TTL	Reason
User Profile	5 minutes	Frequent updates, short-lived
User Preferences	1 hour	Less frequent changes
Product Catalog	24 hours	Stable data, expensive to rebuild
Session Tokens	30 minutes	Security-sensitive, short-lived
Order Status	15 minutes	Critical for UI responsiveness

Cache Eviction Policy

We'll implement LRU (Least Recently Used) eviction policy with a secondary TTL-based cleanup. This ensures that:

Frequently accessed items remain in cache
Memory pressure is managed through TTL expiration
Cache warming can be controlled via explicit key refresh

Cache Stampede Prevention

To prevent cache stampedes:

Probabilistic early expiry: 20% of TTL is used for early refresh
Distributed locks: For cache population operations using Redis distributed locks
Cache warming: Pre-warm hot keys during low-traffic periods

Distributed Cache Architecture

We'll use Redis Cluster with:

3 master nodes for data distribution
3 replica nodes per shard for high availability
Redis Sentinel for automatic failover
Memory eviction policy set to allkeys-lru

Cache Invalidation Strategy

We'll implement event-driven invalidation:

Write-through pattern: Cache is updated on write operations
Event-driven invalidation: Cache entries invalidated via publish-subscribe when data changes
Explicit purge: Manual cache clearing for administrative operations

Cache Implementation Example

@Component
public class CacheService {
    private final RedisTemplate<String, Object> redisTemplate;
    private final ObjectMapper objectMapper;
    
    public <T> T getFromCache(String key, Class<T> type) {
        String cachedValue = redisTemplate.opsForValue().get(key);
        if (cachedValue != null) {
            try {
                return objectMapper.readValue(cachedValue, type);
            } catch (IOException e) {
                log.error("Failed to deserialize cached value for key: {}", key, e);
                return null;
            }
        }
        return null;
    }
    
    public <T> void putInCache(String key, T value, Duration ttl) {
        try {
            String jsonValue = objectMapper.writeValueAsString(value);
            redisTemplate.opsForValue().set(key, jsonValue, ttl);
        } catch (IOException e) {
            log.error("Failed to serialize object for caching: {}", value, e);
        }
    }
    
    public void invalidateCache(String key) {
        redisTemplate.delete(key);
    }
    
    public boolean isCacheAvailable() {
        try {
            return redisTemplate.getConnectionFactory().getConnection().ping().equals("PONG");
        } catch (Exception e) {
            return false;
        }
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain objects and persistence concerns. Each entity will have its own repository interface and concrete implementation.

public interface OrderRepository {
    Optional<Order> findById(UUID orderId);
    List<Order> findByUserId(UUID userId);
    Order save(Order order);
    void delete(UUID orderId);
    Page<Order> findAll(Pageable pageable);
}

@Repository
public class OrderRepositoryImpl implements OrderRepository {
    private final JdbcTemplate jdbcTemplate;
    private final OrderRowMapper rowMapper;
    
    @Override
    public Optional<Order> findById(UUID orderId) {
        String sql = "SELECT * FROM orders WHERE id = ?";
        try {
            return Optional.of(jdbcTemplate.queryForObject(sql, rowMapper, orderId));
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }
    
    @Override
    public Order save(Order order) {
        if (order.getId() == null) {
            return createOrder(order);
        } else {
            return updateOrder(order);
        }
    }
    
    private Order createOrder(Order order) {
        String sql = """
            INSERT INTO orders (id, user_id, status, created_at, updated_at) 
            VALUES (?, ?, ?, ?, ?)
            """;
        KeyHolder keyHolder = new GeneratedKeyHolder();
        jdbcTemplate.update(connection -> {
            PreparedStatement ps = connection.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS);
            ps.setObject(1, order.getId());
            ps.setObject(2, order.getUserId());
            ps.setString(3, order.getStatus().name());
            ps.setTimestamp(4, Timestamp.valueOf(order.getCreatedAt()));
            ps.setTimestamp(5, Timestamp.valueOf(order.getUpdatedAt()));
            return ps;
        }, keyHolder);
        
        order.setId((UUID) keyHolder.getKey());
        return order;
    }
    
    private Order updateOrder(Order order) {
        String sql = """
            UPDATE orders 
            SET status = ?, updated_at = ? 
            WHERE id = ?
            """;
        jdbcTemplate.update(sql, order.getStatus().name(), 
                           Timestamp.valueOf(order.getUpdatedAt()), order.getId());
        return order;
    }
}

ORM vs. Raw Query Decisions

For this system, we use a hybrid approach:

ORM for complex domain objects: Hibernate is used for entities with complex relationships and business logic
Raw queries for performance-critical operations: Direct JDBC is used for read-heavy operations like analytics or batch processing
JPA for standard CRUD: Standard JPA repositories are used for common operations

Read vs. Write Separation

The CQRS (Command Query Responsibility Segregation) pattern is applied to separate read and write operations:

// Write model - used for mutations
@Repository
public class OrderWriteRepository {
    // Implementation for write operations
}

// Read model - optimized for queries
@Repository
public class OrderReadRepository {
    // Implementation optimized for read operations with denormalized data
    
    public List<OrderSummary> findRecentOrdersByUserId(UUID userId) {
        String sql = """
            SELECT o.id, o.status, o.created_at, o.total_amount
            FROM orders o 
            WHERE o.user_id = ? 
            ORDER BY o.created_at DESC
            LIMIT 50
            """;
        return jdbcTemplate.query(sql, new OrderSummaryRowMapper(), userId);
    }
}

Connection Pool Configuration

Connection pool is configured with:

Pool size: 20-50 connections based on concurrent load
Timeout: 30 seconds for connection acquisition
Validation query: SELECT 1 to verify connection health

@Configuration
public class DatabaseConfig {
    @Bean
    public HikariDataSource dataSource() {
        HikariConfig config = new HikariConfig();
        config.setJdbcUrl("jdbc:postgresql://localhost:5432/myapp");
        config.setUsername("user");
        config.setPassword("password");
        config.setMaximumPoolSize(30);
        config.setConnectionTimeout(30000);
        config.setIdleTimeout(600000);
        config.setMaxLifetime(1800000);
        config.setValidationTimeout(5000);
        config.setLeakDetectionThreshold(60000);
        return new HikariDataSource(config);
    }
}

N+1 Query Prevention Strategy

To prevent N+1 query problems:

Eager loading: Use JOINs in queries to fetch related data
Batch loading: Implement batch operations for collections

// Instead of N+1 queries, use batch loading
@Query("SELECT o FROM Order o JOIN FETCH o.items WHERE o.userId = :userId")
List<Order> findOrdersWithItemsByUserId(@Param("userId") UUID userId);

// Batch operation for order items
@Modifying
@Query("UPDATE OrderItem oi SET oi.status = :status WHERE oi.orderId IN :orderIds")
void updateOrderItemsStatus(@Param("orderIds") List<UUID> orderIds, 
                           @Param("status") OrderItemStatus status);

Batch Operation Design

Batch operations are implemented for high-throughput scenarios:

public class OrderBatchRepository {
    
    public void batchCreateOrders(List<Order> orders) {
        String sql = """
            INSERT INTO orders (id, user_id, status, created_at, updated_at) 
            VALUES (?, ?, ?, ?, ?)
            """;
            
        jdbcTemplate.batchUpdate(orders, (ps, order) -> {
            ps.setObject(1, order.getId());
            ps.setObject(2, order.getUserId());
            ps.setString(3, order.getStatus().name());
            ps.setTimestamp(4, Timestamp.valueOf(order.getCreatedAt()));
            ps.setTimestamp(5, Timestamp.valueOf(order.getUpdatedAt()));
        });
    }
    
    public void batchUpdateOrders(List<Order> orders) {
        String sql = """
            UPDATE orders 
            SET status = ?, updated_at = ? 
            WHERE id = ?
            """;
            
        jdbcTemplate.batchUpdate(orders, (ps, order) -> {
            ps.setString(1, order.getStatus().name());
            ps.setTimestamp(2, Timestamp.valueOf(order.getUpdatedAt()));
            ps.setObject(3, order.getId());
        });
    }
}

Transaction Management

Transaction boundaries are carefully defined to ensure data consistency:

@Service
@Transactional
public class OrderService {
    
    @Autowired
    private OrderRepository orderRepository;
    
    @Autowired
    private OrderEventPublisher eventPublisher;
    
    public Order createOrder(Order order) {
        // Transaction boundary starts here
        Order saved = orderRepository.save(order);
        
        // Publish event within same transaction
        eventPublisher.publishOrderCreated(saved);
        
        return saved;
        // Transaction commits here
    }
}

Data Access Layer Testing

Unit tests for repository layer use embedded database:

@ExtendWith(SpringExtension.class)
@SpringBootTest
@TestPropertySource(properties = {
    "spring.datasource.url=jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1",
    "spring.jpa.hibernate.ddl-auto=create-drop"
})
class OrderRepositoryTest {
    
    @Autowired
    private OrderRepository orderRepository;
    
    @Test
    void shouldSaveAndRetrieveOrder() {
        // Given
        Order order = new Order(UUID.randomUUID(), 
                               UUID.randomUUID(), 
                               OrderStatus.PENDING);
        
        // When
        Order saved = orderRepository.save(order);
        Optional<Order> found = orderRepository.findById(saved.getId());
        
        // Then
        assertThat(found).isPresent();
        assertThat(found.get().getId()).isEqualTo(saved.getId());
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The state machine implementation will use a hybrid synchronous/asynchronous approach. Critical state transitions that must be immediately consistent (e.g., order placement, payment confirmation) will be handled synchronously to ensure immediate consistency and error handling. Non-critical transitions that can tolerate eventual consistency (e.g., sending confirmation emails, updating analytics) will be handled asynchronously through message queues.

The primary state transitions that trigger async processing:

ORDER_CREATED → PAYMENT_CONFIRMED (send email notification)
PAYMENT_CONFIRMED → ORDER_SHIPPED (update inventory, send shipping notification)
ORDER_SHIPPED → ORDER_DELIVERED (send delivery confirmation, update analytics)
ORDER_CANCELLED → REFUND_INITIATED (process refund, send cancellation email)

Message schema design with sample payload

{
  "version": "1.0",
  "event_type": "order.shipped",
  "timestamp": "2023-10-15T14:30:00Z",
  "correlation_id": "order-uuid-12345",
  "payload": {
    "order_id": "uuid-12345",
    "status": "SHIPPED",
    "shipment_tracking_number": "TRK123456789",
    "shipped_at": "2023-10-15T14:30:00Z"
  }
}

Dead letter queue strategy and poison message handling

Poison messages will be routed to a dead-letter queue (DLQ) after 3 failed delivery attempts. Messages in the DLQ will be automatically moved to a manual review queue with a 24-hour TTL before permanent deletion. The DLQ will be monitored by an alerting system that triggers on message count thresholds exceeding 100 messages in a 5-minute window.

At-least-once vs. exactly-once delivery

At-least-once delivery is selected for this feature because:

State transitions are idempotent and can safely be processed multiple times
The system uses a state machine with explicit state transitions that prevent duplicate processing
The cost of implementing exactly-once delivery (distributed transactions, idempotency keys) outweighs the benefits for this use case

Retry policy for consumers

Retry policy:

Max attempts: 3
Backoff formula: Exponential with jitter (1s, 2s, 4s ± 1s)
Retryable errors: Network timeouts, service unavailable (503), rate limiting (429)
Non-retryable errors: Invalid payload (400), authentication failed (401)

Outbox pattern for reliable event publishing

The outbox pattern is implemented to ensure eventual consistency between state changes and message publishing:

@Entity
@Table(name = "order_events")
public class OrderEvent {
    @Id
    private UUID id;
    
    private UUID orderId;
    private String eventType;
    private String payload;
    private EventStatus status;
    private LocalDateTime createdAt;
    private LocalDateTime processedAt;
    
    public enum EventStatus {
        PENDING, PROCESSING, PROCESSED, FAILED
    }
}

Sequence diagram for async flows

sequenceDiagram
    participant OrderService
    participant EventPublisher
    participant MessageQueue
    participant EmailConsumer
    participant InventoryConsumer
    
    OrderService->>EventPublisher: Publish ORDER_SHIPPED event
    EventPublisher->>MessageQueue: Queue event message
    MessageQueue->>EmailConsumer: Deliver email notification
    MessageQueue->>InventoryConsumer: Deliver inventory update
    
    EmailConsumer-->>EventPublisher: Acknowledge receipt
    InventoryConsumer-->>EventPublisher: Acknowledge receipt
    
    EventPublisher->>OrderService: Update event status to PROCESSED

10. Error Handling Design

Error Classification Hierarchy

The error handling strategy follows a hierarchical classification system to enable proper routing and response generation:

Domain Errors - Business logic violations (e.g., insufficient funds, invalid state transitions)
Validation Errors - Input data format or constraints violations (e.g., missing required fields, invalid email)
Infrastructure Errors - System-level issues (e.g., database connection failure, timeout)
Authorization Errors - Access control violations (e.g., insufficient permissions)
Internal System Errors - Unexpected system failures (e.g., unhandled exceptions, missing dependencies)

Error Propagation Strategy

Errors are propagated up the call stack with appropriate wrapping to preserve context:

// Example error propagation in service layer
public class OrderService {
    public void processOrder(OrderRequest request) throws OrderProcessingException {
        try {
            validateOrder(request);
            // ... business logic
            saveOrder(order);
        } catch (ValidationException e) {
            throw new OrderProcessingException("Failed to process order due to validation error", e);
        } catch (DatabaseException e) {
            throw new OrderProcessingException("Failed to save order due to database error", e);
        }
    }
}

Retry Strategy with Exponential Backoff and Jitter

For transient infrastructure errors, a retry mechanism is implemented using exponential backoff with jitter:

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;

    public <T> T executeWithRetry(Supplier<T> operation) throws Exception {
        int attempts = 0;
        while (attempts < MAX_RETRIES) {
            try {
                return operation.get();
            } catch (TransientException e) {
                if (attempts >= MAX_RETRIES - 1) throw e;
                
                long delay = calculateBackoffDelay(attempts);
                Thread.sleep(delay);
                attempts++;
            }
        }
        throw new RuntimeException("Max retries exceeded");
    }

    private long calculateBackoffDelay(int attempt) {
        long baseDelay = (long) (BASE_DELAY_MS * Math.pow(2, attempt));
        long jitter = (long) (baseDelay * JITTER_FACTOR * (Math.random() - 0.5) * 2);
        return baseDelay + jitter;
    }
}

Circuit Breaker Design

For external dependencies, a circuit breaker pattern is implemented to prevent cascading failures:

public class CircuitBreaker {
    private final int failureThreshold;
    private final long timeout;
    private final int successThreshold;
    
    private volatile CircuitState state = CircuitState.CLOSED;
    private volatile int failureCount = 0;
    private volatile long lastFailureTime = 0;

    public <T> T execute(Supplier<T> operation) throws Exception {
        switch (state) {
            case CLOSED:
                return executeClosed(operation);
            case OPEN:
                return executeOpen(operation);
            case HALF_OPEN:
                return executeHalfOpen(operation);
        }
        throw new IllegalStateException("Invalid circuit state");
    }

    private <T> T executeClosed(Supplier<T> operation) throws Exception {
        try {
            T result = operation.get();
            onSuccess();
            return result;
        } catch (Exception e) {
            onFailure(e);
            throw e;
        }
    }

    private <T> T executeOpen(Supplier<T> operation) throws Exception {
        if (System.currentTimeMillis() - lastFailureTime > timeout) {
            state = CircuitState.HALF_OPEN;
            throw new CircuitBreakerOpenException("Circuit is half-open");
        }
        throw new CircuitBreakerOpenException("Circuit is open");
    }

    private <T> T executeHalfOpen(Supplier<T> operation) throws Exception {
        try {
            T result = operation.get();
            state = CircuitState.CLOSED;
            failureCount = 0;
            return result;
        } catch (Exception e) {
            state = CircuitState.OPEN;
            throw e;
        }
    }

    private void onSuccess() {
        if (state == CircuitState.HALF_OPEN) {
            failureCount = 0;
        }
    }

    private void onFailure(Exception e) {
        if (state == CircuitState.CLOSED) {
            failureCount++;
            lastFailureTime = System.currentTimeMillis();
            if (failureCount >= failureThreshold) {
                state = CircuitState.OPEN;
            }
        }
    }

    enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }
}

Fallback Behavior Design

Each external dependency has a fallback implementation to ensure graceful degradation:

public class OrderService {
    private final PaymentGateway paymentGateway;
    private final InventoryService inventoryService;

    public OrderResult processOrder(OrderRequest request) {
        try {
            return paymentGateway.processPayment(request.getPaymentDetails());
        } catch (PaymentException e) {
            // Fallback to manual processing or logging
            logger.warn("Payment gateway failed, falling back to manual processing", e);
            return new OrderResult(OrderStatus.MANUAL_REVIEW, "Payment failed, requires manual review");
        }
    }

    public InventoryCheckResult checkInventory(List<OrderItem> items) {
        try {
            return inventoryService.checkAvailability(items);
        } catch (InventoryException e) {
            // Fallback to default inventory check or return error
            logger.warn("Inventory service unavailable, using fallback logic", e);
            return new InventoryCheckResult(true, "Fallback: Assuming inventory available");
        }
    }
}

Error Logging Format and Alerting Thresholds

Structured logging is implemented with consistent error fields:

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);

    public void logError(String operation, Exception error, Map<String, Object> context) {
        logger.error(
            "Operation failed: {} | Error code: {} | User ID: {} | Entity ID: {} | Stack trace: {}",
            operation,
            getErrorCode(error),
            context.get("userId"),
            context.get("entityId"),
            error.getMessage(),
            error
        );
    }

    private String getErrorCode(Exception error) {
        if (error instanceof ValidationException) return "VALIDATION_ERROR";
        if (error instanceof AuthorizationException) return "AUTHORIZATION_ERROR";
        if (error instanceof DatabaseException) return "DATABASE_ERROR";
        return "INTERNAL_ERROR";
    }
}

Alerting thresholds are set for critical errors:

Error Type	Alert Threshold	Severity
Database connection failures	>5 errors in 5 minutes	Critical
Payment gateway timeouts	>10 errors in 10 minutes	Warning
Authorization failures	>50 errors in 1 hour	Critical
Validation errors	>100 errors in 1 hour	Warning

Error Response Format

Standardized JSON error responses are returned to clients:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid email format provided",
    "details": {
      "field": "email",
      "value": "invalid-email"
    },
    "timestamp": "2023-10-15T14:30:00Z",
    "requestId": "req-12345"
  }
}

Error Handling in Async Flows

For asynchronous operations, error handling includes dead-letter queue processing:

public class AsyncErrorHandler {
    private final DeadLetterQueue deadLetterQueue;
    
    public void handleAsyncError(String messageId, Exception error) {
        // Log the error
        logger.error("Async operation failed: {}", messageId, error);
        
        // Move to dead letter queue for manual inspection
        deadLetterQueue.enqueue(messageId, error);
        
        // Send alert to monitoring system
        alertService.sendAlert("Async operation failed", error);
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using a combination of:

Schema validation with JSON Schema for structured data (e.g., request bodies, query parameters)
Custom validation rules using Apache BVal or Bean Validation annotations in Java
Sanitization with OWASP Java HTML Sanitizer for untrusted text inputs to prevent XSS
Length and format constraints (e.g., email regex, UUID format) to prevent malformed data

SQL Injection Prevention

Prepared statements are used exclusively for all database queries in Java using JDBC or JPA
Parameterized queries are enforced via ORM (Hibernate) with strict type mapping
No dynamic SQL string concatenation - all queries are built using parameterized approaches
Input escaping for any non-parameterized fallback paths (e.g., raw SQL in exception cases)

Authentication Middleware Design

JWT-based authentication for API access with 2-hour expiration and refresh token support
OAuth2 integration via Okta or Auth0 for third-party identity providers (if applicable)
Authentication middleware intercepts all requests and validates JWT tokens using HS256 signing
Token revocation via Redis-based blacklisting for session management (refresh tokens can be invalidated)

Authorization Enforcement

Role-Based Access Control (RBAC) with hierarchical permissions:
- ADMIN → full access to all resources
- USER → read/write access to own resources only
- MANAGER → read access to all, write access to team resources

Authorization checks performed at service layer before any domain operation:

if (!hasPermission(user, resource, action)) {
    throw new AccessDeniedException("Insufficient permissions");
}

Resource ownership verification using database joins or precomputed user-resource mappings

Sensitive Data Encryption

At-rest encryption using AES-256 with keys managed by AWS KMS or HashiCorp Vault
In-transit encryption enforced via TLS 1.3 for all API endpoints (HTTPS only)
PII encryption using field-level encryption (FLE) for sensitive fields like email, phone, SSN
Key rotation policy every 90 days with automatic backup and failover

PII Handling and Masking Strategy

PII fields (email, phone, SSN) are encrypted at rest and masked in logs

Log masking using regex patterns to redact PII from structured logs:

String masked = logMessage.replaceAll("\\b\\d{3}-?\\d{2}-?\\d{4}\\b", "XXX-XX-XXXX");

PII in URLs and query parameters are stripped or encoded before logging
Audit trail logs only non-sensitive metadata (user ID, timestamp, action type)

Secrets Management

Secrets stored in AWS Secrets Manager or HashiCorp Vault with automatic rotation
Environment-specific secrets (DB credentials, API keys) loaded via secure config providers
No hardcoded secrets in source code or configuration files
Secrets rotation policy every 90 days with automated failover in case of key rotation

Audit Logging Design

Audit events captured for all sensitive operations:
- User login/logout
- Resource creation/modification/deletion
- Permission changes
- Admin actions (e.g., user ban, role change)

Event format includes:

{
  "timestamp": "2023-10-15T14:30:00Z",
  "userId": "uuid-123",
  "action": "USER_LOGIN",
  "resourceType": "USER",
  "resourceId": "uuid-456",
  "ipAddress": "192.168.1.100",
  "userAgent": "Mozilla/5.0...",
  "details": {}
}

Retention policy logs retained for 90 days with archiving to S3 for compliance
Compliance monitoring alerts triggered on unauthorized access patterns or admin abuse

12. Testing Strategy

Unit Testing Scope and Coverage Targets

Unit testing will target 85% code coverage for business logic, with a minimum of 90% coverage for critical paths and error handling branches. Each service class will have dedicated unit tests covering:

Happy path scenarios with valid inputs
All error conditions and boundary cases
State transition validation
Concurrent access handling
Integration with external dependencies (mocked)

Key areas to test include:

State validation logic for all valid state transitions
Business rule enforcement during state changes
Idempotency handling in concurrent scenarios
Event publishing and error rollback behavior

Integration Testing Strategy

Integration testing will use testcontainers for database and external service simulation, focusing on:

Database persistence correctness across state transitions
Event publishing to message brokers (e.g., Kafka/Redis)
External dependency interactions (payment gateway, notification service)
Transaction boundary validation

Test data will be managed using factory patterns and fixtures to ensure consistent test environments. Embedded databases (H2 for in-memory testing) will be used for faster unit tests, while real database integration tests will run against containerized instances.

Contract Testing

Contract testing will be implemented using Pact for API contracts between services. This ensures that:

External service consumers can validate their expectations
Changes to state machine behavior don't break client integrations
Event schemas and payloads maintain backward compatibility

Performance Testing Tool

Performance testing will be conducted using k6 with the following key scenarios:

Concurrent state transition requests (100-1000 concurrent users)
High-volume state change operations (10k+ requests per minute)
Latency targets: p50 < 20ms, p95 < 100ms, p99 < 500ms
Throughput targets: 1000+ state transitions/sec under load

Test Data Management Strategy

Test data will be managed using:

Factory patterns for generating valid test entities
Fixture files for consistent baseline data
Database seeding scripts for integration test setup
Data cleanup procedures to ensure test isolation

Test data will include various state combinations, edge cases, and concurrent access scenarios. Each test case will have a clean database state to prevent cross-contamination between tests.

// Example unit test for state transition validation
@Test
public void testValidStateTransition() {
    // Given: Valid initial state and transition request
    StateMachine stateMachine = new StateMachine();
    stateMachine.setCurrentState(OrderState.CREATED);
    
    // When: Valid transition is requested
    OrderState newState = stateMachine.transition(OrderState.CONFIRMED);
    
    // Then: State should be updated correctly
    assertEquals(OrderState.CONFIRMED, newState);
    verify(eventPublisher).publish(any(StateChangeEvent.class));
}

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective search, filtering, and correlation across services. The logging structure includes:

{
  "timestamp": "2023-10-27T14:30:00.000Z",
  "level": "INFO|WARN|ERROR",
  "service": "booking-service",
  "trace_id": "1234567890abcdef",
  "span_id": "abcdef1234567890",
  "user_id": "user-123",
  "entity_id": "booking-456",
  "operation": "createBooking",
  "message": "Booking created successfully",
  "error_code": "BOOKING_CREATION_FAILED",
  "error_details": {
    "stack_trace": "...",
    "cause": "Invalid seat selection"
  },
  "duration_ms": 125,
  "request_id": "req-789"
}

PII masking rules:

User IDs and email addresses are masked in logs using hash-based obfuscation
Credit card numbers and other sensitive fields are completely redacted

Distributed Tracing Instrumentation

Using OpenTelemetry, we'll instrument key operations with custom spans:

Span types and attributes:

booking.create - Attributes: booking_id, user_id, seat_id
payment.process - Attributes: payment_method, amount, currency
inventory.check - Attributes: seat_id, show_time, venue_id

Custom span example in Java:

Span span = tracer.spanBuilder("booking.create")
    .setAttribute("booking_id", booking.getId())
    .setAttribute("user_id", userId)
    .startSpan();

try (Scope scope = span.makeCurrent()) {
    // Business logic
} finally {
    span.end();
}

Metrics Design

RED Method metrics per service endpoint:

Metric Name	Type	Labels	Description
`booking_requests_total`	Counter	operation, status, error_code	Total booking operations
`booking_processing_duration_seconds`	Histogram	operation	Request processing time
`booking_errors_total`	Counter	operation, error_code	Failed booking operations
`active_bookings_count`	Gauge	status	Current active bookings

Example histogram bucket configuration:

Histogram.Timer timer = Histogram.startTimer();
// ... operation logic ...
timer.observeDuration();

Health Check Endpoint Design

Endpoints and their checks:

GET /health
GET /ready
GET /live
GET /startup

Response format:

{
  "status": "healthy|unhealthy",
  "checks": {
    "database": {
      "status": "healthy",
      "details": "Connected to primary DB"
    },
    "cache": {
      "status": "healthy",
      "details": "Redis connected"
    },
    "dependencies": {
      "status": "healthy",
      "details": "All external services reachable"
    }
  },
  "timestamp": "2023-10-27T14:30:00.000Z"
}

Alerting Thresholds

Critical alerts (escalation to on-call):

booking_errors_total > 50 errors/minute for 5 minutes → Critical
booking_processing_duration_seconds p99 > 500ms for 10 minutes → Warning
database_connection_errors_total > 10 errors/minute for 2 minutes → Critical

Warning alerts (monitoring team):

booking_requests_total drops below 50% of baseline for 15 minutes → Warning
active_bookings_count > 1M active bookings → Warning

On-call escalation path:

First level: Slack channel with alert details
Second level: SMS to primary on-call engineer
Third level: Escalation to engineering manager if no response in 15 minutes

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

State machine scope definition: Should the state machine be generic enough to support multiple domain entities (orders, tickets, bookings) or specific to one entity type? This decision impacts code reuse vs. domain-specific optimization.
Event-driven vs. synchronous state transitions: How much of the state transition logic should be asynchronous? This affects system complexity and eventual consistency guarantees.
State persistence strategy: Should state machine states be stored in the same database as the main entity or in a dedicated state store? This impacts data consistency and query performance.
Versioning strategy for state machines: How should we handle state machine evolution over time? Should we support multiple versions or enforce strict backward compatibility?

Known trade-offs and justifications

Trade-off	Option 1	Option 2	Justification
State persistence	Store in main entity table	Dedicated state store	Option 1 simplifies queries but increases coupling. Option 2 provides better separation of concerns and scalability but adds complexity with distributed state management
State transition logic	In-process business logic	Event-driven asynchronous	Option 1 provides immediate consistency but can block operations. Option 2 enables loose coupling and better scalability but introduces eventual consistency
State machine reusability	Monolithic, single-use state machines	Generic, configurable state machines	Option 1 offers better performance and type safety but reduces code reuse. Option 2 increases flexibility but adds complexity in configuration and potential runtime errors
State persistence granularity	Full state serialization	Delta-based state changes	Option 1 ensures complete state recovery but increases storage and network overhead. Option 2 reduces storage costs but requires more complex state reconstruction logic

Risks and mitigation strategies

Race conditions in concurrent state transitions: Risk of multiple concurrent requests modifying the same state machine instance simultaneously. Mitigation: Implement optimistic locking with version numbers and retry logic for state transitions.
Event loss in asynchronous processing: Risk of losing state transition events when using message queues. Mitigation: Implement dead letter queue handling and event replay capabilities with idempotency keys.
State machine evolution complexity: Risk of breaking existing state machines when evolving the state model. Mitigation: Implement versioned state machine definitions with backward compatibility support and comprehensive testing.
Performance bottlenecks in state retrieval: Risk of high latency when loading complex state machine instances. Mitigation: Implement caching strategies and lazy loading for non-critical state information.
Inconsistent state across distributed systems: Risk of different services having inconsistent views of the same state machine. Mitigation: Use distributed consensus protocols or event sourcing for maintaining consistent state across microservices.

Technical debt acknowledged

State machine configuration complexity: The generic state machine design introduces significant configuration overhead that requires extensive documentation and tooling support to maintain.
Event-driven complexity: The asynchronous state transition approach introduces additional complexity in debugging and monitoring, particularly around event ordering and delivery guarantees.
Version management overhead: Supporting multiple state machine versions requires ongoing maintenance of migration paths and version compatibility matrices.
Testing complexity: Comprehensive testing of state machines requires extensive test coverage for all possible state transitions and edge cases, which can become unwieldy as the state machine grows in complexity.
Performance monitoring gaps: The distributed nature of state machines makes it harder to trace and monitor performance characteristics across the entire state transition lifecycle.