Order State Machine Design Patterns and Implementation

An order state machine is a critical component in e-commerce and transactional systems that manages the lifecycle of orders through various states while ensuring data consistency and system reliability across concurrent operations.

1. Problem Statement & Requirements

Clarifying questions

1. What is the expected scale of orders?
   • Is this a high-volume system (e.g., millions of orders per day) or low-medium volume?
2. What are the consistency requirements for order state transitions?
   • Should all state changes be atomic, or can eventual consistency be acceptable?
3. What are the key entities and their relationships?
   • Is an order tied to a user, product(s), payment, shipping, etc.?
4. What is the read/write ratio for orders?
   • Is it more common to query order state than update it?
5. What are the latency sensitivity requirements?
   • Should order state transitions happen in <100ms for real-time use cases?
6. What authentication model is expected?
   • Is it a public-facing API or internal service with JWT/OAuth2?
7. Are there external systems involved in order processing?
   • E.g., payment gateway, inventory system, shipping provider?

Functional requirements

• Order creation: Create a new order with associated items, user, and payment details.
• State transitions: Support state changes from CREATED → PAID → SHIPPED → DELIVERED.
• State query: Retrieve current state of an order.
• State rollback: Optionally revert to a previous state (e.g., from SHIPPED back to PAID).
• Order cancellation: Cancel an order if it's in a cancellable state.
• Audit logging: Track all state transitions with timestamps and user/actor information.

Non-functional requirements

• Latency targets:
  • p50: <50ms for state query
  • p95: <100ms for state query
  • p99: <200ms for state query
• Throughput: Support 10,000 concurrent order state transitions per second.
• Availability: 99.9% uptime for order state management service.
• Idempotency: State change operations must be idempotent to handle retries gracefully.

Constraints and assumptions

• Assumption: Order state transitions are initiated by internal services or authenticated users via a secure API.
• Assumption: The system supports only one order lifecycle per order ID.
• Constraint: Order state must be consistent across all reads and writes, with no race conditions.
• Constraint: State transitions must be auditable for compliance and debugging.

Out of scope

• Payment processing: Separate payment system integration.
• Inventory management: Inventory updates are handled by a separate service.
• Email/SMS notifications: Event-driven notification system is out of scope.
• Refund handling: Refunds are handled by a separate financial service.
• Multi-currency support: All orders are processed in a single currency.

2. API Design

API Style Selection

For order management system, I'll select RESTful API with JSON payloads for the following reasons:

• Familiarity and widespread adoption across teams
• Clear resource-based URLs that align with domain entities
• Strong ecosystem support for tooling and documentation (Swagger/OpenAPI)
• Good balance between simplicity and capability for this feature set

Endpoint Definitions

Method	Path	Description	Auth
GET	/v1/orders/{orderId}	Retrieve order details	JWT
POST	/v1/orders	Create new order	JWT
PUT	/v1/orders/{orderId}/status	Update order status	JWT
DELETE	/v1/orders/{orderId}	Cancel order (if allowed)	JWT
GET	/v1/orders/{orderId}/items	List order items	JWT
POST	/v1/orders/{orderId}/items	Add item to order	JWT

Request and Response Schemas

Create Order Request

{
  "customerId": "uuid-string",
  "items": [
    {
      "productId": "uuid-string",
      "quantity": 2,
      "price": 19.99
    }
  ],
  "shippingAddress": {
    "street": "123 Main St",
    "city": "San Francisco",
    "state": "CA",
    "zipCode": "94105"
  }
}

Create Order Response

{
  "orderId": "uuid-string",
  "status": "PENDING",
  "customerId": "uuid-string",
  "items": [
    {
      "itemId": "uuid-string",
      "productId": "uuid-string",
      "quantity": 2,
      "price": 19.99
    }
  ],
  "totalAmount": 39.98,
  "createdAt": "2023-10-15T14:30:00Z",
  "updatedAt": "2023-10-15T14:30:00Z"
}

Update Order Status Request

{
  "status": "CONFIRMED"
}

Update Order Status Response

{
  "orderId": "uuid-string",
  "status": "CONFIRMED",
  "updatedAt": "2023-10-15T14:35:00Z"
}

HTTP Status Codes and Error Response Format

Standard HTTP status codes will be used:

• 200 OK - Successful GET, PUT, DELETE
• 201 Created - Successful POST (create)
• 400 Bad Request - Invalid request body or parameters
• 401 Unauthorized - Missing or invalid auth token
• 403 Forbidden - Insufficient permissions
• 404 Not Found - Resource not found
• 422 Unprocessable Entity - Business logic validation failed
• 429 Too Many Requests - Rate limiting exceeded
• 500 Internal Server Error - Unexpected server error

Standardized error response format:

{
  "error": {
    "code": "ORDER_NOT_FOUND",
    "message": "Order with ID abc-123 not found",
    "details": {
      "orderId": "abc-123"
    },
    "timestamp": "2023-10-15T14:30:00Z"
  }
}

API Versioning Strategy

Versioning will be done via path prefix:

• /v1/orders - Current version
• Future: /v2/orders for breaking changes

Pagination Design

Cursor-based pagination will be used:

• ?cursor=abc123&limit=20
• Provides better consistency and handles data changes during pagination
• Avoids issues with offset-based pagination when items are inserted/deleted

Authentication and Authorization

All endpoints require JWT Bearer token authentication. Authorization will be implemented via:

• JWT claims validation (customer ID in token matches order owner)
• Role-based access control (customer, admin, service roles)

Rate Limiting

Rate limiting will be implemented per endpoint:

• POST /v1/orders: 100 requests/minute
• PUT /v1/orders/{orderId}/status: 50 requests/minute
• GET /v1/orders/{orderId}: 200 requests/minute

Rate limiting will be enforced using Redis-based token bucket algorithm.

Idempotency Design

For mutation endpoints, idempotency keys will be supported:

• Header: Idempotency-Key: uuid-string
• For duplicate requests, system will return the same response as first request
• Idempotency keys are stored in Redis with TTL of 5 minutes

// Example usage in controller
@PostMapping("/v1/orders")
public ResponseEntity<Order> createOrder(
    @RequestHeader("Idempotency-Key") String idempotencyKey,
    @RequestBody CreateOrderRequest request) {
    
    // Check if this is a duplicate request
    Order existing = orderService.findOrCreateByIdempotencyKey(idempotencyKey, request);
    return ResponseEntity.status(HttpStatus.CREATED).body(existing);
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    ORDER {
        string id PK
        string user_id FK
        string status
        timestamp created_at
        timestamp updated_at
        timestamp shipped_at
        timestamp delivered_at
        string shipping_address
        string billing_address
        decimal total_amount
        string currency
    }

    ORDER_ITEM {
        string id PK
        string order_id FK
        string product_id
        integer quantity
        decimal unit_price
        decimal total_price
    }

    PAYMENT {
        string id PK
        string order_id FK
        string status
        string payment_method
        timestamp created_at
        timestamp updated_at
        decimal amount
        string currency
    }

    INVENTORY {
        string product_id PK
        integer available_quantity
        integer reserved_quantity
        timestamp last_updated
    }

    USER {
        string id PK
        string email
        string name
        timestamp created_at
    }

    ORDER ||--o{ ORDER_ITEM : contains
    ORDER ||--o{ PAYMENT : has
    ORDER ||--o{ INVENTORY : affects
    USER ||--o{ ORDER : places

Table Schema Definitions

orders table

CREATE TABLE orders (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL REFERENCES users(id),
    status VARCHAR(50) NOT NULL DEFAULT 'CREATED',
    shipping_address TEXT NOT NULL,
    billing_address TEXT NOT NULL,
    total_amount DECIMAL(10,2) NOT NULL,
    currency VARCHAR(3) NOT NULL DEFAULT 'USD',
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    shipped_at TIMESTAMP,
    delivered_at TIMESTAMP,
    
    INDEX idx_orders_user_id (user_id),
    INDEX idx_orders_status (status),
    INDEX idx_orders_created_at (created_at)
);

order_items table

CREATE TABLE order_items (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    order_id UUID NOT NULL REFERENCES orders(id) ON DELETE CASCADE,
    product_id VARCHAR(100) NOT NULL,
    quantity INTEGER NOT NULL,
    unit_price DECIMAL(10,2) NOT NULL,
    total_price DECIMAL(10,2) NOT NULL,
    
    INDEX idx_order_items_order_id (order_id),
    INDEX idx_order_items_product_id (product_id)
);

payments table

CREATE TABLE payments (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    order_id UUID NOT NULL REFERENCES orders(id) ON DELETE CASCADE,
    status VARCHAR(50) NOT NULL DEFAULT 'PENDING',
    payment_method VARCHAR(50) NOT NULL,
    amount DECIMAL(10,2) NOT NULL,
    currency VARCHAR(3) NOT NULL DEFAULT 'USD',
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_payments_order_id (order_id),
    INDEX idx_payments_status (status)
);

inventory table

CREATE TABLE inventory (
    product_id VARCHAR(100) PRIMARY KEY,
    available_quantity INTEGER NOT NULL DEFAULT 0,
    reserved_quantity INTEGER NOT NULL DEFAULT 0,
    last_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_inventory_available (available_quantity),
    INDEX idx_inventory_reserved (reserved_quantity)
);

users table

CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email VARCHAR(255) UNIQUE NOT NULL,
    name VARCHAR(255) NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_users_email (email)
);

Primary Key Strategy

UUIDs for all entities - chosen over auto-increment for:

• Global uniqueness across distributed systems
• Prevention of sequential guessing attacks
• Better sharding capabilities with hash-based partitioning
• Support for cross-service references without ID conflicts

Index Design

Index Name	Table	Columns	Type	Purpose
idx_orders_user_id	orders	user_id	Standard	User order lookup
idx_orders_status	orders	status	Standard	Status-based filtering
idx_orders_created_at	orders	created_at	Standard	Time-range queries
idx_order_items_order_id	order_items	order_id	Standard	Order item retrieval
idx_order_items_product_id	order_items	product_id	Standard	Product-based queries
idx_payments_order_id	payments	order_id	Standard	Payment lookup by order
idx_payments_status	payments	status	Standard	Payment status queries
idx_inventory_available	inventory	available_quantity	Standard	Stock availability checks

Data Partitioning / Sharding Strategy

Shard by user_id hash modulo 8 - ensures:

• Even distribution of orders across shards
• User data locality for performance
• Scalable horizontal partitioning strategy

Soft Delete vs. Hard Delete

Soft delete with deleted_at timestamp - chosen for:

• Audit trail preservation
• Data recovery capability
• Compliance with financial audit requirements
• Preventing data loss from accidental deletion

Audit Trail Design

Audit log table with structured fields:

CREATE TABLE audit_logs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    entity_type VARCHAR(50) NOT NULL,
    entity_id UUID NOT NULL,
    operation VARCHAR(20) NOT NULL,
    user_id UUID,
    old_values JSONB,
    new_values JSONB,
    timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_audit_logs_entity (entity_type, entity_id),
    INDEX idx_audit_logs_timestamp (timestamp)
);

Data Archival Strategy

Data Type	Retention Period	Archive Location	Reason
Completed orders (status: DELIVERED)	5 years	Cold storage (S3 Glacier)	Legal compliance
Cancelled orders	2 years	Cold storage (S3 Glacier)	Audit and dispute resolution
Pending orders	30 days	Hot storage (PostgreSQL)	Active order management
Payment records	7 years	Cold storage (S3 Glacier)	Financial audit requirements

// Example of soft-delete pattern in Java
public class Order {
    private String id;
    private String userId;
    private OrderStatus status;
    private boolean deleted = false;
    private LocalDateTime deletedAt;
    
    // Soft delete method
    public void softDelete() {
        this.deleted = true;
        this.deletedAt = LocalDateTime.now();
        // Optionally update status to ARCHIVED or similar
    }
    
    public boolean isDeleted() {
        return deleted;
    }
}

4. Class Design & Service Layer

UML Class Diagram

classDiagram
    class Order {
        -String id
        -OrderStatus status
        -List~OrderItem~ items
        -BigDecimal totalAmount
        -String customerId
        -ZonedDateTime createdAt
        -ZonedDateTime updatedAt
        +placeOrder()
        +cancelOrder()
        +processPayment()
        +shipOrder()
        +completeOrder()
    }

    class OrderItem {
        -String productId
        -Integer quantity
        -BigDecimal price
    }

    class OrderStatus {
        <<enumeration>>
        CREATED
        PAID
        SHIPPED
        DELIVERED
        CANCELLED
    }

    class OrderService {
        +placeOrder(OrderRequest)
        +cancelOrder(String orderId)
        +processPayment(String orderId)
        +shipOrder(String orderId)
        +completeOrder(String orderId)
        +getOrderStatus(String orderId)
    }

    class OrderEventPublisher {
        +publishOrderEvent(OrderEvent)
    }

    class OrderEvent {
        -String orderId
        -OrderStatus fromStatus
        -OrderStatus toStatus
        -ZonedDateTime timestamp
        -String eventType
    }

    class OrderStateTransitionValidator {
        +validateTransition(Order, OrderStatus)
    }

    class OrderRepository {
        +save(Order)
        +findById(String)
        +findByCustomerId(String)
    }

    Order --> OrderStatus : has
    Order --> OrderItem : contains
    OrderService --> Order : operates on
    OrderService --> OrderEventPublisher : publishes events
    OrderService --> OrderStateTransitionValidator : validates
    OrderService --> OrderRepository : persists data
    OrderEventPublisher --> OrderEvent : emits

State Machine Diagram

stateDiagram-v2
    [*] --> CREATED
    CREATED --> PAID : processPayment()
    CREATED --> CANCELLED : cancelOrder()
    PAID --> SHIPPED : shipOrder()
    PAID --> CANCELLED : cancelOrder()
    SHIPPED --> DELIVERED : completeOrder()
    SHIPPED --> CANCELLED : cancelOrder()
    DELIVERED --> [*]
    CANCELLED --> [*]

Service Class Structure and Responsibilities

The OrderService class encapsulates all business logic for order lifecycle management. It handles:

• Order creation and validation
• State transitions with proper validation
• Event publishing for audit and async processing
• Integration with external systems (payment, shipping)
• Error handling and domain-specific exception management

Interface / Abstract Class Design

public interface OrderRepository {
    Order save(Order order);
    Optional<Order> findById(String id);
    List<Order> findByCustomerId(String customerId);
}

public interface OrderEventPublisher {
    void publishOrderEvent(OrderEvent event);
}

public interface OrderStateTransitionValidator {
    boolean validateTransition(Order order, OrderStatus targetStatus);
}

Business Logic Encapsulation Patterns

Strategy Pattern Implementation for State Validation

public interface OrderStateValidator {
    boolean validate(Order order, OrderStatus targetStatus);
}

@Component
public class OrderStateValidatorStrategy {
    private final Map<OrderStatus, OrderStateValidator> validators;

    public OrderStateValidatorStrategy() {
        this.validators = new HashMap<>();
        this.validators.put(OrderStatus.CREATED, new CreatedStateValidator());
        this.validators.put(OrderStatus.PAID, new PaidStateValidator());
        this.validators.put(OrderStatus.SHIPPED, new ShippedStateValidator());
        this.validators.put(OrderStatus.DELIVERED, new DeliveredStateValidator());
        this.validators.put(OrderStatus.CANCELLED, new CancelledStateValidator());
    }

    public boolean validate(Order order, OrderStatus targetStatus) {
        OrderStateValidator validator = validators.get(targetStatus);
        if (validator == null) {
            throw new IllegalArgumentException("No validator found for status: " + targetStatus);
        }
        return validator.validate(order, targetStatus);
    }

    private static class CreatedStateValidator implements OrderStateValidator {
        @Override
        public boolean validate(Order order, OrderStatus targetStatus) {
            return targetStatus == OrderStatus.PAID || targetStatus == OrderStatus.CANCELLED;
        }
    }

    private static class PaidStateValidator implements OrderStateValidator {
        @Override
        public boolean validate(Order order, OrderStatus targetStatus) {
            return targetStatus == OrderStatus.SHIPPED || targetStatus == OrderStatus.CANCELLED;
        }
    }

    private static class ShippedStateValidator implements OrderStateValidator {
        @Override
        public boolean validate(Order order, OrderStatus targetStatus) {
            return targetStatus == OrderStatus.DELIVERED || targetStatus == OrderStatus.CANCELLED;
        }
    }

    private static class DeliveredStateValidator implements OrderStateValidator {
        @Override
        public boolean validate(Order order, OrderStatus targetStatus) {
            return targetStatus == OrderStatus.DELIVERED;
        }
    }

    private static class CancelledStateValidator implements OrderStateValidator {
        @Override
        public boolean validate(Order order, OrderStatus targetStatus) {
            return targetStatus == OrderStatus.CANCELLED;
        }
    }
}

Critical-Path Method Implementation

@Transactional
public Order completeOrder(String orderId) {
    // 1. Load order with pessimistic lock to prevent race conditions
    Order order = orderRepository.findById(orderId)
        .orElseThrow(() -> new OrderNotFoundException("Order not found: " + orderId));
    
    // 2. Validate state transition
    if (!orderStateValidator.validateTransition(order, OrderStatus.DELIVERED)) {
        throw new InvalidOrderStateException(
            "Cannot complete order from status: " + order.getStatus());
    }
    
    // 3. Update order state
    OrderStatus oldStatus = order.getStatus();
    order.setStatus(OrderStatus.DELIVERED);
    order.setUpdatedAt(Instant.now());
    
    // 4. Save updated order
    Order updatedOrder = orderRepository.save(order);
    
    // 5. Publish event for async processing and audit
    OrderEvent event = new OrderEvent(
        orderId,
        oldStatus,
        OrderStatus.DELIVERED,
        Instant.now(),
        "ORDER_COMPLETED"
    );
    
    orderEventPublisher.publishOrderEvent(event);
    
    // 6. Trigger any async workflows (e.g., inventory update, notifications)
    // This would typically be done via message queue or async task
    
    return updatedOrder;
}

Transaction Boundary Design

Each order operation is wrapped in a single database transaction:

• Place Order: Creates new order, validates items, reserves inventory
• Process Payment: Updates order status to PAID, calls payment gateway
• Ship Order: Updates status to SHIPPED, calls shipping service
• Complete Order: Updates status to DELIVERED, triggers post-delivery actions
• Cancel Order: Updates status to CANCELLED, refunds payment if needed

Service Interface Contracts

public interface OrderService {
    Order placeOrder(OrderRequest request);
    Order cancelOrder(String orderId);
    Order processPayment(String orderId);
    Order shipOrder(String orderId);
    Order completeOrder(String orderId);
    OrderStatus getOrderStatus(String orderId);
    List<Order> getOrdersByCustomer(String customerId);
}

Saga / Orchestration Pattern

For distributed transactions involving external systems (payment, shipping), we implement a saga pattern:

public class OrderSaga {
    private final OrderRepository orderRepository;
    private final PaymentService paymentService;
    private final ShippingService shippingService;

    public void handleOrderPlacement(Order order) {
        // Start saga
        try {
            // Step 1: Process payment
            String paymentId = paymentService.processPayment(order);
            
            // Step 2: Ship order
            String shippingId = shippingService.scheduleShipping(order);
            
            // Step 3: Complete order
            order.setStatus(OrderStatus.SHIPPED);
            orderRepository.save(order);
            
        } catch (Exception e) {
            // Compensate: reverse operations
            compensateFailedOrder(order);
            throw new OrderProcessingException("Order processing failed", e);
        }
    }

    private void compensateFailedOrder(Order order) {
        // Cancel payment
        if (order.getPaymentId() != null) {
            paymentService.refundPayment(order.getPaymentId());
        }
        
        // Cancel shipping
        if (order.getShippingId() != null) {
            shippingService.cancelShipping(order.getShippingId());
        }
        
        // Update order status to cancelled
        order.setStatus(OrderStatus.CANCELLED);
        orderRepository.save(order);
    }
}

Domain Object / Value Object Design

public class Order {
    private String id;
    private OrderStatus status;
    private List<OrderItem> items;
    private BigDecimal totalAmount;
    private String customerId;
    private ZonedDateTime createdAt;
    private ZonedDateTime updatedAt;
    
    // Getters and setters
}

public class OrderItem {
    private String productId;
    private Integer quantity;
    private BigDecimal price;
    
    // Getters and setters
}

public enum OrderStatus {
    CREATED,
    PAID,
    SHIPPED,
    DELIVERED,
    CANCELLED
}

public class OrderEvent {
    private String orderId;
    private OrderStatus fromStatus;
    private OrderStatus toStatus;
    private Instant timestamp;
    private String eventType;
    
    // Getters and setters
}

5. Concurrency & Thread Safety

Concurrency Model

The system uses a thread-per-request model with a thread pool managed by the application server (e.g., Tomcat or Netty). Each incoming request is handled by a dedicated thread from the pool, ensuring that concurrent order processing operations don't interfere with each other at the application level.

Race Condition Prevention Strategy

Race conditions are mitigated through:

1. Optimistic locking using version numbers on order entities to detect concurrent modifications
2. Database-level atomic operations for critical state transitions (e.g., UPDATE orders SET status = 'SHIPPED' WHERE id = ? AND version = ?)
3. Application-level synchronization for critical sections where necessary (e.g., when updating order totals and inventory in a single transaction)
4. Idempotency keys for external API calls to prevent duplicate processing

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected because:

• Order state transitions are infrequent and typically involve minimal concurrent access
• The system prioritizes throughput over strict consistency in the face of contention
• It avoids blocking threads and reduces latency for concurrent operations
• The version field in the database serves as a concurrency token, allowing conflict detection without blocking

Deadlock Prevention Strategy

To prevent deadlocks:

1. Consistent ordering of database locks - All operations acquire locks in a consistent order (e.g., by ID or status)
2. Timeout configuration - Database queries timeout after 5 seconds to prevent indefinite blocking
3. Transaction scope minimization - Transactions are kept as small as possible to reduce lock duration
4. Retry logic with exponential backoff - When a deadlock is detected, the system retries the operation after a random delay

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is implemented using a dedicated idempotency key store:

public class IdempotencyService {
    private final RedisTemplate<String, String> redisTemplate;
    
    public boolean isIdempotent(String key) {
        return redisTemplate.hasKey("idempotency:" + key);
    }
    
    public void markIdempotent(String key) {
        redisTemplate.opsForValue().set(
            "idempotency:" + key, 
            "processed", 
            Duration.ofHours(24)
        );
    }
}

For order creation operations, the client provides an idempotencyKey in the request header. The system checks if this key has been processed before, and if so, returns the existing order instead of creating a duplicate.

The idempotency store uses Redis with TTL to prevent memory leaks and ensure cleanup of old keys. A background job periodically cleans up expired entries.

Thread Safety in Business Logic

All service methods are designed to be stateless and thread-safe:

• No shared mutable state across threads
• Immutable value objects are used for order items and totals
• Thread-local storage is only used for request-scoped metadata (e.g., user context)
• Synchronization is only applied to critical sections where necessary (e.g., when updating order totals with concurrent inventory updates)

Concurrency Control in Order State Transitions

State transitions are protected by database-level atomic operations:

@Transactional
public Order updateOrderStatus(Long orderId, String newStatus, Integer expectedVersion) {
    int updatedRows = orderRepository.updateStatus(
        orderId, 
        newStatus, 
        expectedVersion
    );
    
    if (updatedRows == 0) {
        throw new ConcurrencyException("Order was modified by another process");
    }
    
    return orderRepository.findById(orderId);
}

This pattern ensures that only one concurrent operation can successfully update an order's status, preventing race conditions during state transitions.

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant U as User
    participant O as OrderService
    participant DB as Database
    participant E as EventPublisher

    U->>O: CreateOrderRequest(orderItems, customer)
    O->>DB: Save new Order (status: CREATED)
    O->>E: Publish OrderCreatedEvent
    O->>U: CreateOrderResponse(orderId, status: CREATED)

    Note over O,DB: Background processing
    O->>DB: Update Order (status: CONFIRMED)
    O->>E: Publish OrderConfirmedEvent

sequenceDiagram
    participant U as User
    participant O as OrderService
    participant DB as Database
    participant E as EventPublisher
    participant P as PaymentService

    U->>O: CreateOrderRequest(orderItems, customer)
    O->>DB: Save new Order (status: CREATED)
    O->>E: Publish OrderCreatedEvent
    O->>U: CreateOrderResponse(orderId, status: CREATED)

    O->>P: ProcessPayment(paymentDetails)
    P-->>O: PaymentProcessedResponse
    O->>DB: Update Order (status: CONFIRMED)
    O->>E: Publish OrderConfirmedEvent

sequenceDiagram
    participant U as User
    participant O as OrderService
    participant DB as Database
    participant E as EventPublisher
    participant P as PaymentService
    participant S as ShippingService

    U->>O: CreateOrderRequest(orderItems, customer)
    O->>DB: Save new Order (status: CREATED)
    O->>E: Publish OrderCreatedEvent
    O->>U: CreateOrderResponse(orderId, status: CREATED)

    O->>P: ProcessPayment(paymentDetails)
    P-->>O: PaymentProcessedResponse
    O->>DB: Update Order (status: CONFIRMED)
    O->>E: Publish OrderConfirmedEvent

    O->>S: ScheduleShipping(orderId)
    S-->>O: ShippingScheduledResponse
    O->>DB: Update Order (status: SHIPPED)
    O->>E: Publish OrderShippedEvent

sequenceDiagram
    participant U as User
    participant O as OrderService
    participant DB as Database
    participant E as EventPublisher
    participant P as PaymentService
    participant S as ShippingService

    U->>O: CreateOrderRequest(orderItems, customer)
    O->>DB: Save new Order (status: CREATED)
    O->>E: Publish OrderCreatedEvent
    O->>U: CreateOrderResponse(orderId, status: CREATED)

    O->>P: ProcessPayment(paymentDetails)
    P-->>O: PaymentFailedResponse
    O->>DB: Update Order (status: FAILED)
    O->>E: Publish OrderFailedEvent

sequenceDiagram
    participant U as User
    participant O as OrderService
    participant DB as Database
    participant E as EventPublisher
    participant P as PaymentService

    U->>O: CreateOrderRequest(orderItems, customer)
    O->>DB: Save new Order (status: CREATED)
    O->>E: Publish OrderCreatedEvent
    O->>U: CreateOrderResponse(orderId, status: CREATED)

    O->>P: ProcessPayment(paymentDetails)
    P-->>O: PaymentProcessedResponse
    O->>DB: Update Order (status: CONFIRMED)
    O->>E: Publish OrderConfirmedEvent

    Note over O,DB: Later, user cancels order
    O->>DB: Update Order (status: CANCELLED)
    O->>E: Publish OrderCancelledEvent

7. Caching Design

Cache Strategy Selection

For order state machine operations, we'll use cache-aside pattern with LRU eviction policy. This approach provides the best balance between performance and consistency for read-heavy order state queries while allowing fine-grained control over cache invalidation.

Cache Key Design

Cache keys will follow the namespace pattern: order:{orderId}:{state} with TTL-based expiration.

public class OrderCacheKey {
    private static final String ORDER_STATE_NAMESPACE = "order";
    
    public static String buildOrderStateKey(String orderId, String state) {
        return String.format("%s:%s:%s", ORDER_STATE_NAMESPACE, orderId, state);
    }
    
    public static String buildOrderKey(String orderId) {
        return String.format("%s:%s", ORDER_STATE_NAMESPACE, orderId);
    }
}

TTL Configuration

Data Type	TTL	Reason
Order State	5 minutes	Frequent reads, state changes are infrequent
Order Summary	10 minutes	Less frequent updates, longer TTL acceptable
Order History	24 hours	Historical data rarely accessed, longer retention

Cache Eviction Policy

LRU (Least Recently Used) with probabilistic early expiry to prevent cache stampede:

public class CacheManager {
    private final RedisTemplate<String, Object> redisTemplate;
    private static final int CACHE_TTL_SECONDS = 300; // 5 minutes
    
    public Object getCachedValue(String key) {
        String cacheKey = OrderCacheKey.buildOrderStateKey(key, "state");
        Object cached = redisTemplate.opsForValue().get(cacheKey);
        
        // Implement probabilistic early expiry
        if (cached != null && shouldRefreshCache(cacheKey)) {
            // Refresh in background or return stale
            return cached;
        }
        
        return cached;
    }
    
    private boolean shouldRefreshCache(String cacheKey) {
        // Check if cache is approaching expiration
        Long ttl = redisTemplate.getExpire(cacheKey);
        return ttl != null && ttl < 60; // Refresh if less than 1 minute left
    }
}

Cache Stampede Prevention

Using distributed locks to prevent thundering herd:

public class CacheStampedePrevention {
    private static final String LOCK_PREFIX = "cache_lock:";
    
    public Object getCachedWithLock(String key, Supplier<Object> cacheLoader) {
        String lockKey = LOCK_PREFIX + key;
        String lockValue = UUID.randomUUID().toString();
        
        try {
            // Try to acquire lock
            if (redisTemplate.opsForValue().setIfAbsent(lockKey, lockValue, 
                Duration.ofSeconds(10), TimeUnit.SECONDS)) {
                
                // Load from cache or database
                Object result = cacheLoader.get();
                
                // Update cache with fresh data
                String cacheKey = OrderCacheKey.buildOrderStateKey(key, "state");
                redisTemplate.opsForValue().set(cacheKey, result, 
                    Duration.ofSeconds(CACHE_TTL_SECONDS));
                
                return result;
            } else {
                // Wait and retry
                Thread.sleep(100);
                return redisTemplate.opsForValue().get(cacheKey);
            }
        } finally {
            // Release lock if owned
            releaseLock(lockKey, lockValue);
        }
    }
    
    private void releaseLock(String lockKey, String lockValue) {
        String script = "if redis.call('get', KEYS[1]) == ARGV[1] then " +
                       "return redis.call('del', KEYS[1]) else return 0 end";
        redisTemplate.execute(new DefaultRedisScript<>(script, Long.class), 
                             Collections.singletonList(lockKey), lockValue);
    }
}

Distributed Cache Architecture

Redis cluster with:

• 3 master nodes for high availability
• 3 replica nodes per master for read scaling
• Automatic failover using Redis Sentinel
• Cache sharding by order ID hash

Cache Invalidation Strategy

Event-driven invalidation using message broker:

@Component
public class OrderStateCacheInvalidator {
    private final RedisTemplate<String, Object> redisTemplate;
    private final MessagePublisher messagePublisher;
    
    @EventListener
    public void handleOrderStateChanged(OrderStateChangedEvent event) {
        // Invalidate all relevant cache entries
        String orderId = event.getOrderId();
        
        // Remove order state cache
        String stateKey = OrderCacheKey.buildOrderStateKey(orderId, "state");
        redisTemplate.delete(stateKey);
        
        // Remove order summary cache
        String summaryKey = OrderCacheKey.buildOrderKey(orderId);
        redisTemplate.delete(summaryKey);
        
        // Publish cache invalidation event for other services
        messagePublisher.publish(new CacheInvalidatedEvent(orderId));
    }
}

Cache Read-Through Pattern

public class OrderStateCacheService {
    private final RedisTemplate<String, Object> redisTemplate;
    private final OrderRepository orderRepository;
    
    public OrderState getOrCreateOrderState(String orderId) {
        String cacheKey = OrderCacheKey.buildOrderStateKey(orderId, "state");
        
        // Try cache first
        OrderState cached = (OrderState) redisTemplate.opsForValue().get(cacheKey);
        if (cached != null) {
            return cached;
        }
        
        // Cache miss - load from database
        OrderState orderState = orderRepository.findOrderState(orderId);
        if (orderState != null) {
            // Cache the result
            redisTemplate.opsForValue().set(cacheKey, orderState, 
                Duration.ofSeconds(CACHE_TTL_SECONDS));
        }
        
        return orderState;
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain objects and persistence concerns. Each entity has its own repository interface with standardized CRUD operations.

public interface OrderRepository {
    Optional<Order> findById(UUID orderId);
    List<Order> findByUserId(UUID userId);
    Order save(Order order);
    void delete(UUID orderId);
    Page<Order> findAll(Pageable pageable);
}

public interface OrderItemRepository {
    List<OrderItem> findByOrderId(UUID orderId);
    void saveAll(List<OrderItem> orderItems);
}

ORM vs Raw Query Decisions

For the core Order entity, we use JPA/Hibernate as the ORM for its maturity and ease of development. For high-volume read operations on order items, raw SQL queries are used to optimize performance and reduce object mapping overhead.

@Repository
public class OrderRepositoryImpl implements OrderRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    @Override
    public Optional<Order> findById(UUID orderId) {
        return Optional.ofNullable(
            entityManager.find(Order.class, orderId)
        );
    }
    
    @Override
    public Order save(Order order) {
        if (order.getId() == null) {
            entityManager.persist(order);
        } else {
            entityManager.merge(order);
        }
        return order;
    }
    
    // Raw query for performance-critical operations
    @Query("SELECT o FROM Order o WHERE o.userId = :userId AND o.status IN :statuses")
    public List<Order> findByUserIdAndStatuses(
        @Param("userId") UUID userId,
        @Param("statuses") List<OrderStatus> statuses
    );
}

Read vs Write Separation

We implement CQRS pattern for read-heavy operations to optimize performance:

// Read model - optimized for querying
public class OrderReadModel {
    private UUID id;
    private String userId;
    private OrderStatus status;
    private BigDecimal totalAmount;
    private LocalDateTime createdAt;
    
    // Getters and setters
}

// Write model - optimized for mutations
public class Order {
    private UUID id;
    private String userId;
    private OrderStatus status;
    private List<OrderItem> items;
    private BigDecimal totalAmount;
    private LocalDateTime createdAt;
    private LocalDateTime updatedAt;
    
    // Business logic and validation methods
}

Connection Pool Configuration

Connection pool is configured with HikariCP for optimal performance:

spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      leak-detection-threshold: 60000

N+1 Query Prevention Strategy

Eager loading is used for order items to prevent N+1 queries:

@Query("SELECT o FROM Order o JOIN FETCH o.items WHERE o.id = :orderId")
public Optional<Order> findByIdWithItems(@Param("orderId") UUID orderId);

// Using @EntityGraph for explicit fetch plan
@EntityGraph(attributePaths = {"items", "items.product"})
@Query("SELECT o FROM Order o WHERE o.id = :orderId")
public Optional<Order> findByIdWithItemsAndProduct(@Param("orderId") UUID orderId);

Batch Operation Design

Batch operations are implemented for bulk order creation:

@Repository
public class OrderBatchRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    public void saveAll(List<Order> orders) {
        for (int i = 0; i < orders.size(); i++) {
            entityManager.persist(orders.get(i));
            if (i % 50 == 0) { // Flush every 50 entities
                entityManager.flush();
                entityManager.clear();
            }
        }
    }
    
    public void saveOrderItemsInBatch(List<OrderItem> items) {
        String sql = "INSERT INTO order_items (id, order_id, product_id, quantity, price) VALUES (?, ?, ?, ?, ?)";
        try (PreparedStatement stmt = entityManager.unwrap(Session.class).connection().prepareStatement(sql)) {
            for (OrderItem item : items) {
                stmt.setObject(1, item.getId());
                stmt.setObject(2, item.getOrderId());
                stmt.setObject(3, item.getProductId());
                stmt.setInt(4, item.getQuantity());
                stmt.setBigDecimal(5, item.getPrice());
                stmt.addBatch();
            }
            stmt.executeBatch();
        } catch (SQLException e) {
            throw new DataAccessException("Failed to batch save order items", e);
        }
    }
}

Transaction Management

Repository methods are transactionally isolated:

@Service
@Transactional
public class OrderService {
    
    @Autowired
    private OrderRepository orderRepository;
    
    public Order createOrder(Order order) {
        // Transaction boundary starts here
        Order saved = orderRepository.save(order);
        // Event publishing happens within same transaction
        eventPublisher.publish(new OrderCreatedEvent(saved));
        return saved;
    }
}

Read-Through Cache Integration

Repository methods integrate with caching layer:

@Repository
public class CachedOrderRepository implements OrderRepository {
    
    private final OrderRepository delegate;
    private final CacheManager cacheManager;
    
    public CachedOrderRepository(OrderRepository delegate, CacheManager cacheManager) {
        this.delegate = delegate;
        this.cacheManager = cacheManager;
    }
    
    @Override
    @Cacheable(value = "orders", key = "#orderId")
    public Optional<Order> findById(UUID orderId) {
        return delegate.findById(orderId);
    }
    
    @Override
    @CacheEvict(value = "orders", key = "#order.id")
    public Order save(Order order) {
        return delegate.save(order);
    }
}

Data Access Layer Testing

@MockBean
private OrderRepository orderRepository;

@Test
public void testFindByIdWithItems() {
    // Given
    Order order = createSampleOrder();
    when(orderRepository.findByIdWithItems(any(UUID.class))).thenReturn(Optional.of(order));
    
    // When
    Optional<Order> result = orderRepository.findByIdWithItems(UUID.randomUUID());
    
    // Then
    assertTrue(result.isPresent());
    assertEquals(2, result.get().getItems().size());
}

The repository design provides clean abstraction, handles performance optimizations like batch operations and eager loading, and maintains proper transaction boundaries while supporting caching and CQRS patterns for scalable data access.

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The order state machine operates with a hybrid synchronous/asynchronous architecture. The core order creation and state transitions are handled synchronously to ensure immediate consistency for critical operations like payment processing and inventory validation. However, certain downstream operations that don't block the user experience are handled asynchronously:

1. Synchronous flows:

   • Order creation with payment processing
   • Inventory reservation and validation
   • Order state transitions (e.g., from CREATED to CONFIRMED)

2. Asynchronous flows:

   • Order confirmation email notifications
   • Inventory release on timeout or cancellation
   • Payment refund processing (if order is canceled)
   • Order analytics and reporting
   • Inventory adjustment notifications to external systems

This approach ensures that users receive immediate feedback for critical operations while allowing non-blocking operations to be handled in the background.

Message schema design with sample payload

{
  "version": "1.0",
  "eventId": "550e8400-e29b-41d4-a716-446655440000",
  "eventType": "order.created",
  "timestamp": "2023-10-27T14:30:00Z",
  "payload": {
    "orderId": "ORD-20231027-001",
    "customerId": "CUST-12345",
    "status": "CREATED",
    "items": [
      {
        "productId": "PROD-001",
        "quantity": 2,
        "price": 29.99
      }
    ],
    "totalAmount": 59.98,
    "paymentStatus": "PENDING"
  },
  "source": "order-service",
  "correlationId": "corr-123456789"
}

Dead letter queue strategy and poison message handling

Poison messages are handled through a dead-letter queue (DLQ) strategy:

1. Retry mechanism: Messages are retried up to 3 times with exponential backoff (1s, 2s, 4s)
2. DLQ routing: Messages that fail after max retries are routed to a dedicated DLQ topic
3. Manual intervention: Messages in DLQ are flagged for manual inspection and can be replayed after fixes
4. Monitoring: DLQ queue size and error rates are monitored with alerts

At-least-once vs. exactly-once delivery

At-least-once delivery is selected for this feature because:

1. Idempotency at consumer level: Order processing consumers implement idempotent operations
2. Event sourcing approach: Events are stored with unique IDs to prevent duplicate processing
3. Business tolerance: Re-sending order confirmation emails or analytics events is acceptable

The system uses message deduplication through unique event IDs and implements idempotency keys for critical operations.

Retry policy for consumers

public class OrderEventConsumer {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double BACKOFF_MULTIPLIER = 2.0;
    
    public void processOrderEvent(OrderEvent event) {
        int attempts = 0;
        while (attempts < MAX_RETRIES) {
            try {
                handleEvent(event);
                return;
            } catch (RetryableException e) {
                attempts++;
                if (attempts >= MAX_RETRIES) {
                    moveMessageToDLQ(event);
                    throw new RuntimeException("Failed after " + MAX_RETRIES + " attempts", e);
                }
                long delay = (long) (BASE_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempts - 1));
                Thread.sleep(delay);
            } catch (NonRetryableException e) {
                moveMessageToDLQ(event);
                throw e;
            }
        }
    }
}

Outbox pattern for reliable event publishing

The outbox pattern is implemented to ensure eventual consistency between database writes and message publishing:

@Service
public class OrderService {
    @Autowired
    private OrderRepository orderRepository;
    
    @Autowired
    private EventPublisher eventPublisher;
    
    @Transactional
    public Order createOrder(OrderRequest request) {
        Order order = new Order();
        order.setStatus(OrderStatus.CREATED);
        order.setCustomerId(request.getCustomerId());
        
        // Save order to database
        Order savedOrder = orderRepository.save(order);
        
        // Publish event via outbox pattern
        OutboxEvent outboxEvent = new OutboxEvent();
        outboxEvent.setAggregateId(savedOrder.getId());
        outboxEvent.setEventType("order.created");
        outboxEvent.setPayload(mapper.writeValueAsString(order));
        outboxEvent.setStatus(OutboxEventStatus.PENDING);
        
        // Save event to outbox table
        outboxRepository.save(outboxEvent);
        
        return savedOrder;
    }
}

Sequence diagram for async flows

sequenceDiagram
    participant OrderService
    participant DB
    participant OutboxRepo
    participant MessageBroker
    participant EmailConsumer
    participant InventoryConsumer
    
    OrderService->>DB: Save Order (CREATED)
    OrderService->>OutboxRepo: Insert OutboxEvent
    OrderService->>MessageBroker: Publish order.created event
    MessageBroker-->>EmailConsumer: Deliver event
    MessageBroker-->>InventoryConsumer: Deliver event
    
    loop Process Events
        EmailConsumer->>DB: Check if email sent
        EmailConsumer->>EmailService: Send confirmation email
        InventoryConsumer->>DB: Check if inventory reserved
        InventoryConsumer->>InventoryService: Reserve items
    end

10. Error Handling Design

Error Classification Hierarchy

The system implements a hierarchical error classification to enable proper handling and routing:

Error
├── ValidationException (400)
│   ├── InvalidInputException
│   └── ConstraintViolationException
├── BusinessLogicException (400)
│   ├── InsufficientStockException
│   ├── OrderAlreadySubmittedException
│   └── InvalidOrderStateException
├── InfrastructureException (500)
│   ├── DatabaseException
│   ├── ExternalServiceException
│   └── TimeoutException
└── AuthorizationException (403)
    └── AccessDeniedException

Error Propagation Strategy

• Domain-level errors: Caught and converted to domain-specific exceptions before being propagated up the call stack
• Infrastructure errors: Bubble up as-is to allow for circuit breaker and retry logic at higher levels
• Validation errors: Handled immediately at service boundaries with structured error responses
• Business logic errors: Translated to appropriate HTTP status codes and logged for audit trail

Retry Strategy with Exponential Backoff and Jitter

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long INITIAL_DELAY_MS = 1000;
    private static final double BACKOFF_MULTIPLIER = 2.0;
    private static final double JITTER_RANGE = 0.1;

    public <T> T executeWithRetry(Supplier<T> operation) {
        int attempts = 0;
        long delay = INITIAL_DELAY_MS;

        while (true) {
            try {
                return operation.get();
            } catch (ExternalServiceException e) {
                attempts++;
                if (attempts >= MAX_RETRIES) {
                    throw new RetryExhaustedException("Operation failed after " + MAX_RETRIES + " attempts", e);
                }

                // Apply jitter to prevent thundering herd
                long jitter = (long) (delay * JITTER_RANGE * (Math.random() - 0.5) * 2);
                long actualDelay = delay + jitter;

                try {
                    Thread.sleep(actualDelay);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new RetryExhaustedException("Retry interrupted", ie);
                }

                delay = (long) (delay * BACKOFF_MULTIPLIER);
            }
        }
    }
}

Circuit Breaker Design

public class CircuitBreaker {
    private final int failureThreshold = 5;
    private final long timeout = 30000; // 30 seconds
    private final int successThreshold = 5;
    
    private volatile CircuitState state = CircuitState.CLOSED;
    private volatile int failureCount = 0;
    private volatile long lastFailureTime = 0;
    
    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }

    public <T> T execute(Supplier<T> operation) {
        switch (state) {
            case CLOSED:
                return executeClosed(operation);
            case OPEN:
                return executeOpen(operation);
            case HALF_OPEN:
                return executeHalfOpen(operation);
            default:
                throw new IllegalStateException("Unknown circuit state");
        }
    }

    private <T> T executeClosed(Supplier<T> operation) {
        try {
            T result = operation.get();
            onSuccess();
            return result;
        } catch (Exception e) {
            onFailure(e);
            throw e;
        }
    }

    private <T> T executeOpen(Supplier<T> operation) {
        if (System.currentTimeMillis() - lastFailureTime > timeout) {
            state = CircuitState.HALF_OPEN;
            return executeHalfOpen(operation);
        }
        throw new CircuitBreakerOpenException("Circuit is open");
    }

    private <T> T executeHalfOpen(Supplier<T> operation) {
        try {
            T result = operation.get();
            onSuccess();
            return result;
        } catch (Exception e) {
            onFailure(e);
            throw e;
        }
    }

    private void onSuccess() {
        failureCount = 0;
        if (state == CircuitState.HALF_OPEN) {
            state = CircuitState.CLOSED;
        }
    }

    private void onFailure(Exception e) {
        failureCount++;
        lastFailureTime = System.currentTimeMillis();
        
        if (failureCount >= failureThreshold) {
            state = CircuitState.OPEN;
        }
    }
}

Fallback Behavior Design

public class OrderService {
    private final CircuitBreaker circuitBreaker = new CircuitBreaker();
    
    public Order getFallbackOrder(String orderId) {
        // Return cached or default order state
        return new Order()
            .setId(orderId)
            .setStatus(OrderStatus.FAILED)
            .setErrorMessage("Order processing failed, fallback applied");
    }
    
    public Order processOrder(String orderId) {
        return circuitBreaker.execute(() -> {
            try {
                return orderProcessor.process(orderId);
            } catch (ExternalServiceException e) {
                // Fallback to cached state or default
                return getFallbackOrder(orderId);
            }
        });
    }
}

Error Logging Format and Alerting Thresholds

Structured logging follows this schema:

{
  "timestamp": "2023-10-15T14:30:00Z",
  "trace_id": "abc123def456",
  "span_id": "xyz789uvw012",
  "user_id": "user-12345",
  "entity_id": "order-98765",
  "error_code": "INSUFFICIENT_STOCK",
  "error_message": "Not enough stock available for item SKU-12345",
  "error_type": "BusinessLogicException",
  "stack_trace": "...",
  "http_status": 400,
  "service_name": "order-service"
}

Alerting thresholds:

Metric	Warning Level	Critical Level	Alerting Channel
Order processing failures	> 10 per minute	> 50 per minute	Slack + PagerDuty
Circuit breaker open	> 10 consecutive failures	> 50 consecutive failures	Slack + PagerDuty
Database connection timeouts	> 5 per minute	> 20 per minute	Slack + PagerDuty
External service timeouts	> 15 per minute	> 50 per minute	Slack + PagerDuty

Error Response Format

Standardized JSON error response:

{
  "error": {
    "code": "INSUFFICIENT_STOCK",
    "message": "Not enough stock available for item SKU-12345",
    "details": {
      "itemId": "SKU-12345",
      "requestedQuantity": 10,
      "availableQuantity": 5
    },
    "timestamp": "2023-10-15T14:30:00Z",
    "traceId": "abc123def456"
  }
}

Retry Exhaustion Handling

public class RetryExhaustedException extends RuntimeException {
    public RetryExhaustedException(String message, Throwable cause) {
        super(message, cause);
    }
    
    public RetryExhaustedException(String message) {
        super(message);
    }
}

When retry attempts are exhausted, the system logs a critical alert and sends an alert to on-call engineers via PagerDuty or Slack with full context including:

• The operation that failed
• The number of retries attempted
• The final error encountered
• The timestamp and trace ID for debugging
• The user context (if available)

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using a combination of:

• Bean Validation (JSR-303) for structured validation rules
• Custom validators for business-specific constraints (e.g., order item quantity limits)
• OWASP Java HTML Sanitizer for any user-provided text fields to prevent XSS
• Regex patterns for strict format validation (e.g., email, phone number)

public class OrderRequest {
    @NotNull
    @Min(1)
    private Long customerId;
    
    @ValidItems
    private List<OrderItem> items;
    
    @Email
    @NotBlank
    private String customerEmail;
}

SQL Injection Prevention

• All database queries use parameterized prepared statements
• No dynamic query building from user input
• ORM framework (Hibernate) is used exclusively for data access
• Input sanitization for any non-query string interpolation

Authentication Middleware Design

• JWT-based authentication with 2-hour expiration for session tokens
• OAuth2 integration for third-party identity providers (e.g., Google, Facebook)
• Authentication filter intercepts all requests and validates JWT tokens
• Token refresh mechanism with 15-minute sliding window for access tokens

Authorization Enforcement

• Role-Based Access Control (RBAC) implemented at service layer:
  • CUSTOMER role can only view/update their own orders
  • ADMIN role has full CRUD access to all orders
  • OPERATIONS role can cancel orders and update status

@PreAuthorize("hasRole('ADMIN') or (hasRole('CUSTOMER') and @orderAuthorizationService.isOwner(#orderId, #userId))")
public void cancelOrder(@PathVariable String orderId, @AuthenticationPrincipal UserPrincipal user) {
    // Implementation
}

Sensitive Data Encryption

• At Rest: AES-256 encryption using AWS KMS for order data and customer PII
• In Transit: TLS 1.3 with HSTS header enforcement for all API endpoints
• Customer PII: Credit card data is never stored; only tokenized references to payment processors

PII Handling and Masking Strategy

• Customer email: Fully masked in logs (e.g., j***@gmail.com)
• Phone numbers: Partially masked (e.g., +1 ***-***-1234)
• Full name: Only stored in encrypted format, never logged in plain text
• Audit logs: PII fields are redacted or replaced with hash values

Secrets Management

• AWS Secrets Manager for storing:
  • Database credentials (rotated every 90 days)
  • JWT signing keys (rotated every 30 days)
  • Payment processor API keys
• Secrets rotated automatically using AWS Lambda functions with CloudWatch events

Audit Logging Design

• Events logged: Order creation, status changes, cancellation, refund processing
• Log format:

{
  "timestamp": "2023-10-27T14:30:00Z",
  "userId": "user-123",
  "action": "ORDER_CANCELLED",
  "entityId": "order-456",
  "ipAddress": "192.168.1.100",
  "userAgent": "Mozilla/5.0...",
  "metadata": {
    "oldStatus": "PENDING",
    "newStatus": "CANCELLED"
  }
}

• Retention policy: 3 years for compliance, 90 days for operational logs
• Log aggregation: Centralized logging via AWS CloudWatch with alerting rules

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

• 90%+ code coverage for business logic in service layer classes
• 100% test coverage for all state transition logic and validation rules
• 95%+ test coverage for data access layer methods (repositories)
• Target: 85%+ for utility classes and helper functions
• Test pyramid ratio: 70% unit tests, 20% integration tests, 10% end-to-end tests

Integration Testing Strategy

• Database integration tests using Testcontainers with PostgreSQL 13+ for all CRUD operations and transaction boundaries
• External service mocks using WireMock or Mockito for payment gateway, inventory system, and notification services
• Event-driven integration tests using embedded Kafka or RabbitMQ to validate async workflows
• Cross-service boundary tests for order creation and state change propagation between microservices (if applicable)
• Test data isolation using database transaction rollback per test method to ensure clean state

Contract Testing

• Pact Broker integration for API contract testing between Order Service and dependent services (Payment, Inventory, Notification)
• Consumer-driven contracts for all external API integrations
• Versioned contract schemas with semantic versioning to prevent breaking changes in downstream services

Performance Testing Tool and Key Scenarios

• k6 for load testing and performance validation:
  • Concurrent order creation: 1000 concurrent users creating orders with varying payloads
  • State transition stress test: Simulate high-frequency state changes (e.g., cancel, confirm, ship)
  • Throughput test: 10,000 orders per minute with 95th percentile < 200ms
  • Latency targets: p50 < 100ms, p95 < 300ms, p99 < 500ms for order creation and state change operations
• JMeter for long-running stress tests (1 hour) to detect memory leaks and resource exhaustion

Test Data Management Strategy

• Factory pattern for generating test data with configurable attributes (status, items, timestamps)
• Database fixtures stored in YAML or JSON format for consistent test data across environments
• Test data seeding scripts that populate database with realistic order history and edge cases (cancelled orders, failed payments)
• Data anonymization for test data containing PII using masking techniques (e.g., email obfuscation, name truncation)
• Test data lifecycle management with automatic cleanup after each test suite run to prevent data pollution

// Example unit test for order state transition logic
@Test
public void testCancelOrderFromPendingState() {
    Order order = orderFactory.createOrderWithStatus(OrderStatus.PENDING);
    
    // Mock repository to return the order
    when(orderRepository.findById(any())).thenReturn(Optional.of(order));
    
    // Execute
    Order cancelledOrder = orderService.cancelOrder("order-123");
    
    // Verify
    assertEquals(OrderStatus.CANCELLED, cancelledOrder.getStatus());
    verify(orderRepository).save(any(Order.class));
    verify(eventPublisher).publish(any(OrderCancelledEvent.class));
}

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective search, filtering, and analysis:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO|WARN|ERROR",
  "service": "order-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user_12345",
  "entity_id": "order_98765",
  "operation": "placeOrder",
  "message": "Order placed successfully",
  "error_code": "ORDER_CREATION_FAILED",
  "error_details": {
    "stack_trace": "...",
    "cause": "Insufficient inventory"
  },
  "request_id": "req_abc123",
  "duration_ms": 450,
  "metadata": {
    "order_type": "standard",
    "payment_method": "credit_card"
  }
}

PII masking rules:

• User IDs are logged as user_XXXXX (masked)
• Credit card numbers are masked to ****-****-****-1234
• Email addresses are truncated to user@***.com

Distributed Tracing Instrumentation

Using OpenTelemetry with custom spans:

// Example span for order placement flow
Span span = tracer.spanBuilder("placeOrder")
    .setAttribute("order.id", orderId)
    .setAttribute("user.id", userId)
    .startSpan();

// Custom attributes for key operations
span.setAttribute("payment.amount", order.getAmount());
span.setAttribute("inventory.items", order.getItems().size());

Key spans to instrument:

1. placeOrder - Root span for order creation
2. validateOrder - Input validation step
3. reserveInventory - Inventory reservation logic
4. processPayment - Payment processing flow
5. sendNotifications - Async notification publishing

Metrics Design

RED metrics pattern applied at service level:

// Rate: Number of orders processed per second
Counter ordersProcessed = meter.counterBuilder("orders.processed")
    .setDescription("Number of orders successfully processed")
    .build();

// Errors: Error rate for order processing
Counter ordersFailed = meter.counterBuilder("orders.failed")
    .setDescription("Number of orders that failed processing")
    .build();

// Duration: Latency for order processing
Histogram orderProcessingDuration = meter.histogramBuilder("orders.processing.duration")
    .setDescription("Duration of order processing in milliseconds")
    .setUnit("ms")
    .build();

Metrics labels:

• operation: placeOrder, validateOrder, reserveInventory
• status: success, failed, timeout
• payment_method: credit_card, paypal, bank_transfer
• order_type: standard, express, bulk

Health Check Endpoint Design

@RestController
@RequestMapping("/health")
public class HealthController {
    
    @GetMapping("/live")
    public ResponseEntity<Map<String, Object>> live() {
        Map<String, Object> response = new HashMap<>();
        response.put("status", "healthy");
        response.put("timestamp", Instant.now());
        return ResponseEntity.ok(response);
    }
    
    @GetMapping("/ready")
    public ResponseEntity<Map<String, Object>> ready() {
        Map<String, Object> response = new HashMap<>();
        response.put("status", "ready");
        response.put("database", databaseHealth());
        response.put("cache", cacheHealth());
        response.put("messageQueue", messageQueueHealth());
        return ResponseEntity.ok(response);
    }
    
    @GetMapping("/startup")
    public ResponseEntity<Map<String, Object>> startup() {
        Map<String, Object> response = new HashMap<>();
        response.put("status", "startup_complete");
        response.put("dependencies", dependenciesHealth());
        return ResponseEntity.ok(response);
    }
}

Alerting Thresholds

Metric	Warning Level	Critical Level	Escalation Path
orders.failed.rate	> 0.1%	> 1%	On-call engineer
orders.processing.duration.p95	> 500ms	> 2s	SRE team
database.connection.pool.utilization	> 80%	> 95%	DBA
cache.hit.rate	< 70%	< 50%	SRE team
message.queue.delay	> 30s	> 2min	Messaging team

Alerts are configured with:

• 5-minute sliding window
• 2-minute delay before escalation
• Slack notifications for warnings, PagerDuty for critical alerts

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• State transition granularity: Should we have fine-grained states (e.g., "payment_processing", "payment_failed") or coarse-grained states (e.g., "payment_complete")? This impacts the complexity of state management and debugging capabilities.
• Order cancellation policy: Should orders be cancellable in all states or only in specific ones? What happens to the associated inventory reservations and payment holds?
• Refund processing strategy: How should refunds be handled for orders that have been partially processed or failed? Should we support partial refunds?

Known trade-offs and justifications

Trade-off	Option 1	Option 2	Justification
State persistence	In-memory state machine	Database-backed state machine	In-memory provides better performance but loses state on restart. Database persistence ensures durability but adds latency and complexity. We chose database persistence for production reliability.
Event-driven vs. synchronous state transitions	Synchronous state changes	Asynchronous event processing	Synchronous ensures immediate consistency but can block operations. Asynchronous allows better scalability and decouples components, but introduces eventual consistency concerns.
State validation vs. optimistic updates	Validate all state transitions before applying	Apply state changes optimistically with rollback capability	Validation provides strong consistency but can be slower. Optimistic updates improve performance but require more complex rollback logic.
State machine vs. explicit state flags	Single source of truth for order state	Multiple boolean flags or enum fields	State machine provides clear, auditable state transitions and prevents invalid state combinations. Flags can lead to inconsistent states and harder-to-maintain code.

Risks and mitigation strategies

• Race conditions in state transitions: Concurrent updates to the same order can cause inconsistent states. Mitigation: Use optimistic locking with version numbers and implement proper retry logic.
• Data inconsistency in distributed systems: If order state is updated across multiple services, there's risk of inconsistency. Mitigation: Implement Saga pattern for distributed transactions and use outbox pattern for reliable event publishing.
• State machine complexity explosion: Adding too many states and transitions can make the system hard to maintain. Mitigation: Keep state machine simple and well-documented, with clear separation of concerns.
• Audit trail gaps: Missing or incomplete audit logs can make debugging difficult. Mitigation: Implement comprehensive audit logging with all state transitions and associated metadata.
• Performance degradation under high load: Complex state transitions can cause bottlenecks. Mitigation: Use caching for frequently accessed order states and implement proper indexing.

Technical debt acknowledged

• Hardcoded state transition logic: Current implementation has hardcoded state transitions that may become difficult to maintain as the system grows. Future improvement: Move to a configuration-driven state machine or rule engine.
• Limited extensibility: The current design assumes a fixed set of order states and transitions. Future enhancement: Design for extensibility to support custom order types or workflows.
• No built-in state history tracking: While we capture state changes, there's no built-in mechanism to track the full history of an order's lifecycle. This could be useful for analytics and debugging.
• Single state machine instance: The current design assumes a single instance of the state machine. In a distributed environment, this could lead to race conditions or inconsistent state across nodes.
• No support for state machine versioning: As business requirements evolve, we may need to handle different versions of the same state machine. Currently, there's no mechanism to support this evolution gracefully.