Distributed Transaction Coordinator Design

A distributed transaction coordinator ensures atomicity and consistency across multiple distributed systems, requiring sophisticated coordination mechanisms to handle failures and maintain data integrity in large-scale distributed environments.

1. Problem Statement & Requirements

Clarifying questions

1. What scale and throughput requirements are expected?
   • Is this for a system handling millions of distributed transactions per second, or more modest scale?
2. What consistency guarantees are required?
   • Is it strictly ACID, or can eventual consistency be acceptable?
3. What are the key entities and their relationships?
   • What types of resources or services participate in distributed transactions?
4. What is the read/write ratio and latency sensitivity?
   • Are most operations read-heavy, or do we expect high write volume with strict latency SLAs?
5. What authentication and authorization model is in place?
   • Will the system be accessed via OAuth2, JWTs, or API keys?
6. Are there any external system integrations?
   • Will it coordinate with third-party systems like payment gateways, inventory services, etc.?
7. What are the key failure scenarios to handle?
   • Should it support transaction rollback, timeout handling, or idempotency?

Functional requirements

• Transaction lifecycle management
  • Create, commit, and rollback distributed transactions across multiple services.
• Transaction state tracking
  • Maintain transaction status (pending, committed, rolled back) and metadata.
• Participant coordination
  • Communicate with transaction participants to prepare, commit, or rollback.
• Transaction timeout handling
  • Automatically abort transactions that exceed a configured timeout.
• Idempotency support
  • Ensure idempotent operations for retries and duplicate requests.
• Event-driven transaction lifecycle
  • Publish events when transaction states change (e.g., committed, rolled back).

Non-functional requirements

• Latency targets
  • p50: <10ms, p95: <50ms, p99: <200ms for transaction creation and status queries.
• Throughput
  • Support 10,000+ concurrent distributed transactions per second.
• Availability
  • 99.9% uptime with graceful degradation capability.
• Scalability
  • Horizontally scalable to support millions of concurrent transactions.

Constraints and assumptions

• Assumption: Transaction participants are implemented using a consistent state management model (e.g., database or in-memory).
• Constraint: Transaction coordinator must not become a single point of failure.
• Assumption: Participants can be reached via HTTP or gRPC.
• Constraint: Transaction state must be durable and recoverable after system failures.

Out of scope

• Transaction participant implementation
  • The design does not include how individual services implement their transactional state.
• Cross-datacenter replication
  • Multi-region support or cross-datacenter transaction coordination is out of scope.
• Complex business logic
  • Business rules or conditional branching within the transaction are not part of this design.
• UI or dashboard
  • No user-facing interface is included in the scope.

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for the distributed transaction coordinator due to:

• Strong typing and schema evolution support via Protobuf
• Efficient binary serialization for cross-service communication
• Built-in support for streaming and bidirectional RPCs
• Native support for versioning and backward compatibility
• Better performance characteristics over HTTP/REST for high-throughput systems

API Endpoints

Method	Path	Description	Auth	Rate Limit
CreateTransaction	/transaction.v1.TransactionService/CreateTransaction	Initiates a new distributed transaction with participants	JWT + Service Auth	100 req/sec
GetTransactionStatus	/transaction.v1.TransactionService/GetTransactionStatus	Queries current transaction state	JWT + Service Auth	500 req/sec
CommitTransaction	/transaction.v1.TransactionService/CommitTransaction	Commits a transaction across all participants	JWT + Service Auth	50 req/sec
RollbackTransaction	/transaction.v1.TransactionService/RollbackTransaction	Rolls back a transaction across all participants	JWT + Service Auth	50 req/sec
GetTransactionParticipants	/transaction.v1.TransactionService/GetTransactionParticipants	Lists all participants in a transaction	JWT + Service Auth	200 req/sec

Request and Response Schemas

CreateTransactionRequest

message CreateTransactionRequest {
  string transaction_id = 1;
  int64 created_at = 2;
  repeated TransactionParticipant participants = 3;
  string description = 4;
  int32 timeout_seconds = 5;
}

message TransactionParticipant {
  string service_name = 1;
  string endpoint_url = 2;
  string resource_id = 3;
  TransactionPhase phase = 4;
}

enum TransactionPhase {
  PHASE_UNKNOWN = 0;
  PHASE_PREPARE = 1;
  PHASE_COMMIT = 2;
  PHASE_ROLLBACK = 3;
}

CreateTransactionResponse

message CreateTransactionResponse {
  string transaction_id = 1;
  TransactionStatus status = 2;
  int64 created_at = 3;
}

message TransactionStatus {
  string transaction_id = 1;
  TransactionState state = 2;
  repeated ParticipantStatus participants = 3;
  int64 created_at = 4;
  int64 updated_at = 5;
}

enum TransactionState {
  STATE_UNKNOWN = 0;
  STATE_ACTIVE = 1;
  STATE_PREPARED = 2;
  STATE_COMMITTED = 3;
  STATE_ROLLED_BACK = 4;
  STATE_TIMEOUT = 5;
}

message ParticipantStatus {
  string service_name = 1;
  string resource_id = 2;
  ParticipantState state = 3;
  int64 updated_at = 4;
}

enum ParticipantState {
  PARTICIPANT_UNKNOWN = 0;
  PARTICIPANT_PREPARE_PENDING = 1;
  PARTICIPANT_PREPARE_SUCCESS = 2;
  PARTICIPANT_PREPARE_FAILURE = 3;
  PARTICIPANT_COMMIT_PENDING = 4;
  PARTICIPANT_COMMIT_SUCCESS = 5;
  PARTICIPANT_COMMIT_FAILURE = 6;
  PARTICIPANT_ROLLBACK_PENDING = 7;
  PARTICIPANT_ROLLBACK_SUCCESS = 8;
  PARTICIPANT_ROLLBACK_FAILURE = 9;
}

HTTP Status Codes and Error Response Format

Status Code	Description
200 OK	Success
400 Bad Request	Invalid request parameters
401 Unauthorized	Missing or invalid authentication
403 Forbidden	Insufficient permissions
429 Too Many Requests	Rate limit exceeded
500 Internal Server Error	Unexpected server error

Standardized error response:

{
  "error": {
    "code": "TRANSACTION_NOT_FOUND",
    "message": "Transaction with ID 'tx-123' not found",
    "details": {
      "transaction_id": "tx-123"
    }
  }
}

API Versioning Strategy

Versioned using semantic versioning: /transaction.v1.TransactionService/...

• Major version (v1) changes are breaking
• Minor versions support backward compatibility
• Patch versions for bug fixes and non-breaking changes

Pagination Design

Cursor-based pagination is used because:

• More efficient for large datasets
• Consistent results even when data changes
• Better performance for deep pagination

Authentication and Authorization

All endpoints require:

• JWT Bearer token with service-level permissions
• Service name validation in JWT claims
• Authorization checks using RBAC model

Rate Limiting Strategy

• CreateTransaction: 100 req/sec (burst: 200)
• GetTransactionStatus: 500 req/sec (burst: 1000)
• CommitTransaction/RollbackTransaction: 50 req/sec (burst: 100)
• GetTransactionParticipants: 200 req/sec (burst: 400)

Rate limiting enforced using Redis with sliding window algorithm.

Idempotency Design

For mutation endpoints:

• Use idempotency_key in request headers
• Store operation metadata with key as hash of request + idempotency_key
• Return same result for duplicate requests within TTL (5 minutes)
• Implement distributed lock to prevent concurrent processing of same key

// Example idempotency check in gRPC service
public CreateTransactionResponse createTransaction(
    CreateTransactionRequest request, 
    ServerCallStreamObserver<CreateTransactionResponse> observer) {
    
    String idempotencyKey = extractIdempotencyKey();
    String operationHash = hashRequest(request, idempotencyKey);
    
    // Check if operation already processed
    if (idempotencyCache.exists(operationHash)) {
        return idempotencyCache.get(operationHash);
    }
    
    // Process transaction...
    CreateTransactionResponse response = processTransaction(request);
    
    // Store operation result
    idempotencyCache.put(operationHash, response, Duration.ofMinutes(5));
    
    return response;
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    TRANSACTION {
        string id PK
        string status
        int participant_count
        int completed_participants
        timestamp created_at
        timestamp updated_at
        timestamp expires_at
        string description
    }

    PARTICIPANT {
        string id PK
        string transaction_id FK
        string endpoint_url
        string status
        timestamp created_at
        timestamp updated_at
        string payload
        string response
    }

    TRANSACTION_EVENT {
        string id PK
        string transaction_id FK
        string event_type
        string payload
        timestamp created_at
    }

    TRANSACTION -> PARTICIPANT : contains
    TRANSACTION -> TRANSACTION_EVENT : has_events

Table Schema Definitions

transactions table

CREATE TABLE transactions (
    id UUID PRIMARY KEY,
    status VARCHAR(20) NOT NULL DEFAULT 'INITIATED',
    participant_count INT NOT NULL,
    completed_participants INT NOT NULL DEFAULT 0,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    expires_at TIMESTAMP NOT NULL,
    description TEXT,
    
    INDEX idx_status (status),
    INDEX idx_expires_at (expires_at),
    INDEX idx_created_at (created_at)
);

participants table

CREATE TABLE participants (
    id UUID PRIMARY KEY,
    transaction_id UUID NOT NULL REFERENCES transactions(id) ON DELETE CASCADE,
    endpoint_url VARCHAR(512) NOT NULL,
    status VARCHAR(20) NOT NULL DEFAULT 'PENDING',
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    payload TEXT,
    response TEXT,
    
    INDEX idx_transaction_id (transaction_id),
    INDEX idx_status (status),
    INDEX idx_created_at (created_at)
);

transaction_events table

CREATE TABLE transaction_events (
    id UUID PRIMARY KEY,
    transaction_id UUID NOT NULL REFERENCES transactions(id) ON DELETE CASCADE,
    event_type VARCHAR(50) NOT NULL,
    payload TEXT,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_transaction_id (transaction_id),
    INDEX idx_event_type (event_type),
    INDEX idx_created_at (created_at)
);

Primary Key Strategy

Using UUIDs as primary keys for all tables to support distributed system requirements:

• UUID v1: Time-based, globally unique, but reveals creation time
• UUID v4: Randomly generated, better for privacy and distribution
• Decision: Using UUID v4 for all primary keys to ensure global uniqueness without leaking temporal information

Index Design

Index Name	Fields	Type	Purpose
idx_status	status	Standard	Fast status-based queries
idx_expires_at	expires_at	Standard	Efficient timeout handling
idx_created_at	created_at	Standard	Audit and time-based filtering
idx_transaction_id	transaction_id	Standard	Fast participant lookups
idx_event_type	event_type	Standard	Event type filtering

Data Partitioning / Sharding Strategy

Partition by created_at timestamp with monthly buckets:

-- Example partitioning strategy for transactions table
CREATE TABLE transactions_2024_01 PARTITION OF transactions
FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');

CREATE TABLE transactions_2024_02 PARTITION OF transactions
FOR VALUES FROM ('2024-02-01') TO ('2024-03-01');

Soft Delete vs. Hard Delete

Decision: Soft delete with deleted_at timestamp field.

ALTER TABLE transactions ADD COLUMN deleted_at TIMESTAMP NULL;
ALTER TABLE participants ADD COLUMN deleted_at TIMESTAMP NULL;

Justification:

• Required for audit trails and compliance
• Allows recovery of deleted transactions in case of system errors
• Enables historical analysis of transaction lifecycle

Audit Trail Design

All data changes are captured in transaction_events table:

public class TransactionEvent {
    private String id;
    private String transactionId;
    private String eventType; // "STATUS_CHANGED", "PARTICIPANT_ADDED", etc.
    private String payload;
    private Instant createdAt;
}

Data Archival Strategy and Retention Policy

Data Type	Retention Period	Archive Location
Completed transactions	30 days	Cold storage (S3 Glacier)
Failed transactions	90 days	Cold storage (S3 Glacier)
In-progress transactions	7 days	Hot storage (RDS)

-- Archive completed transactions older than 30 days
CREATE PROCEDURE archive_completed_transactions()
BEGIN
    INSERT INTO archived_transactions SELECT * FROM transactions 
    WHERE status = 'COMPLETED' AND created_at < DATE_SUB(NOW(), INTERVAL 30 DAY);
    
    DELETE FROM transactions WHERE status = 'COMPLETED' AND created_at < DATE_SUB(NOW(), INTERVAL 30 DAY);
END;

4. Class Design & Service Layer

classDiagram
    class TransactionCoordinator {
        -TransactionStore transactionStore
        -EventPublisher eventPublisher
        -LockManager lockManager
        +initiateTransaction(TransactionRequest request) TransactionId
        +commitTransaction(TransactionId id) boolean
        +rollbackTransaction(TransactionId id) boolean
        +getTransactionStatus(TransactionId id) TransactionStatus
    }

    class Transaction {
        -TransactionId id
        -TransactionStatus status
        -List~TransactionParticipant~ participants
        -long createdAt
        -long updatedAt
        +addParticipant(TransactionParticipant participant)
        +updateStatus(TransactionStatus status)
        +getParticipants() List~TransactionParticipant~
    }

    class TransactionParticipant {
        -String serviceId
        -String endpointUrl
        -TransactionPhase currentPhase
        -long lastUpdated
    }

    class TransactionRequest {
        -String idempotencyKey
        -List~ParticipantRequest~ participants
        -long timeoutMillis
    }

    class ParticipantRequest {
        -String serviceId
        -String endpointUrl
        -String operationType
        -Map~String, Object~ payload
    }

    class TransactionStore {
        +save(Transaction transaction)
        +findById(TransactionId id) Transaction
        +update(Transaction transaction)
        +findAllByStatus(TransactionStatus status) List~Transaction~
    }

    class EventPublisher {
        +publish(Event event)
    }

    class LockManager {
        +acquireLock(String lockKey) boolean
        +releaseLock(String lockKey)
        +isLocked(String lockKey) boolean
    }

    class TransactionStateMachine {
        -TransactionState currentState
        +handleEvent(TransactionEvent event) TransactionState
    }

    class TransactionEvent {
        -TransactionId transactionId
        -TransactionEventType eventType
        -String payload
    }

    TransactionCoordinator --> TransactionStore : persists
    TransactionCoordinator --> EventPublisher : publishes events
    TransactionCoordinator --> LockManager : manages locks
    TransactionCoordinator --> TransactionStateMachine : state transitions
    Transaction --> TransactionParticipant : has many
    TransactionStore --> Transaction : CRUD operations
    EventPublisher --> Event : publishes
    TransactionStateMachine --> TransactionEvent : processes events

    class TransactionPhase {
        PREPARE
        COMMIT
        ROLLBACK
    }

    class TransactionStatus {
        PENDING
        PREPARED
        COMMITTED
        ROLLBACK
        FAILED
    }

    class TransactionEventType {
        TRANSACTION_STARTED
        PARTICIPANT_PREPARED
        PARTICIPANT_COMMITTED
        TRANSACTION_COMMITTED
        TRANSACTION_ROLLED_BACK
    }

stateDiagram-v2
    [*] --> PENDING
    PENDING --> PREPARED : All participants prepared
    PENDING --> ROLLBACK : Timeout or failure
    PREPARED --> COMMITTED : All participants committed
    PREPARED --> ROLLBACK : Any participant failed
    COMMITTED --> [*]
    ROLLBACK --> [*]

Service Layer Design

The Transaction Coordinator service implements a robust distributed transaction management system using the Saga pattern with eventual consistency. The service layer is organized around core responsibilities:

1. Transaction lifecycle management (initiate, commit, rollback)
2. Participant coordination and status tracking
3. State machine orchestration for distributed transactions
4. Event-driven communication with participants

Core Service Classes

public interface TransactionCoordinator {
    TransactionId initiateTransaction(TransactionRequest request);
    boolean commitTransaction(TransactionId id);
    boolean rollbackTransaction(TransactionId id);
    TransactionStatus getTransactionStatus(TransactionId id);
}

public class DistributedTransactionCoordinator implements TransactionCoordinator {
    private final TransactionStore transactionStore;
    private final EventPublisher eventPublisher;
    private final LockManager lockManager;
    private final TransactionStateMachine stateMachine;
    private final ParticipantClient participantClient;
    private final ScheduledExecutorService scheduler;

    public DistributedTransactionCoordinator(
            TransactionStore transactionStore,
            EventPublisher eventPublisher,
            LockManager lockManager,
            TransactionStateMachine stateMachine,
            ParticipantClient participantClient) {
        this.transactionStore = transactionStore;
        this.eventPublisher = eventPublisher;
        this.lockManager = lockManager;
        this.stateMachine = stateMachine;
        this.participantClient = participantClient;
        this.scheduler = Executors.newScheduledThreadPool(2);
    }

    @Override
    public TransactionId initiateTransaction(TransactionRequest request) {
        // Validate request and check idempotency
        TransactionId transactionId = new TransactionId(UUID.randomUUID().toString());
        
        // Create and persist initial transaction state
        Transaction transaction = new Transaction(
            transactionId,
            TransactionStatus.PENDING,
            new ArrayList<>(),
            System.currentTimeMillis()
        );
        
        // Validate all participants can be reached
        for (ParticipantRequest participant : request.getParticipants()) {
            if (!participantClient.canReach(participant.getEndpointUrl())) {
                throw new TransactionException("Participant unreachable: " + participant.getServiceId());
            }
        }
        
        transactionStore.save(transaction);
        
        // Publish transaction started event
        eventPublisher.publish(new TransactionEvent(
            transactionId, 
            TransactionEventType.TRANSACTION_STARTED,
            new ObjectMapper().writeValueAsString(request)
        ));
        
        // Start async preparation phase
        scheduler.schedule(() -> prepareParticipants(transactionId, request), 0, TimeUnit.MILLISECONDS);
        
        return transactionId;
    }

    private void prepareParticipants(TransactionId transactionId, TransactionRequest request) {
        Transaction transaction = transactionStore.findById(transactionId);
        
        // Prepare all participants in parallel
        List<CompletableFuture<Void>> futures = new ArrayList<>();
        
        for (ParticipantRequest participant : request.getParticipants()) {
            CompletableFuture<Void> future = CompletableFuture.runAsync(() -> {
                try {
                    // Acquire lock for participant
                    String lockKey = "participant:" + participant.getServiceId() + ":" + transactionId;
                    if (!lockManager.acquireLock(lockKey)) {
                        throw new TransactionException("Unable to acquire lock for participant: " + participant.getServiceId());
                    }
                    
                    // Send prepare request
                    PrepareResponse response = participantClient.prepare(
                        participant.getEndpointUrl(),
                        new PrepareRequest(transactionId, participant.getOperationType(), participant.getPayload())
                    );
                    
                    if (response.isSuccessful()) {
                        // Update participant status
                        TransactionParticipant participantState = new TransactionParticipant(
                            participant.getServiceId(),
                            participant.getEndpointUrl(),
                            TransactionPhase.PREPARE,
                            System.currentTimeMillis()
                        );
                        
                        transaction.addParticipant(participantState);
                        transactionStore.update(transaction);
                        
                        // Publish participant prepared event
                        eventPublisher.publish(new TransactionEvent(
                            transactionId,
                            TransactionEventType.PARTICIPANT_PREPARED,
                            participant.getServiceId()
                        ));
                    } else {
                        // Handle prepare failure
                        rollbackTransaction(transactionId);
                        throw new TransactionException("Participant failed to prepare: ":
                            participant.getServiceId());
                    }
                } catch (Exception e) {
                    rollbackTransaction(transactionId);
                    throw new TransactionException("Prepare failed for participant: " + 
                        participant.getServiceId(), e);
                }
            });
            
            futures.add(future);
        }
        
        // Wait for all participants to prepare or timeout
        try {
            CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).get(30, TimeUnit.SECONDS);
            
            // Update transaction status to prepared
            transaction.updateStatus(TransactionStatus.PREPARED);
            transactionStore.update(transaction);
            
        } catch (Exception e) {
            rollbackTransaction(transactionId);
            throw new TransactionException("Timeout or failure during prepare phase", e);
        }
    }

    @Override
    public boolean commitTransaction(TransactionId id) {
        Transaction transaction = transactionStore.findById(id);
        
        if (transaction.getStatus() != TransactionStatus.PREPARED) {
            throw new TransactionException("Transaction not in prepared state: " + id);
        }
        
        // Commit all participants
        List<CompletableFuture<Void>> futures = new ArrayList<>();
        
        for (TransactionParticipant participant : transaction.getParticipants()) {
            CompletableFuture<Void> future = CompletableFuture.runAsync(() -> {
                try {
                    CommitResponse response = participantClient.commit(
                        participant.getEndpointUrl(),
                        new CommitRequest(id)
                    );
                    
                    if (!response.isSuccessful()) {
                        throw new TransactionException("Participant failed to commit: " + 
                            participant.getServiceId());
                    }
                    
                    // Update participant status
                    participant.setCurrentPhase(TransactionPhase.COMMIT);
                    transactionStore.update(transaction);
                    
                } catch (Exception e) {
                    rollbackTransaction(id);
                    throw new TransactionException("Commit failed for participant: " + 
                        participant.getServiceId(), e);
                }
            });
            
            futures.add(future);
        }
        
        try {
            CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).get(30, TimeUnit.SECONDS);
            
            // Update transaction status to committed
            transaction.updateStatus(TransactionStatus.COMMITTED);
            transactionStore.update(transaction);
            
            // Publish committed event
            eventPublisher.publish(new TransactionEvent(
                id,
                TransactionEventType.TRANSACTION_COMMITTED,
                "Transaction committed successfully"
            ));
            
            return true;
        } catch (Exception e) {
            rollbackTransaction(id);
            throw new TransactionException("Commit timeout or failure", e);
        }
    }

    @Override
    public boolean rollbackTransaction(TransactionId id) {
        Transaction transaction = transactionStore.findById(id);
        
        // Send rollback to all participants
        List<CompletableFuture<Void>> futures = new ArrayList<>();
        
        for (TransactionParticipant participant : transaction.getParticipants()) {
            CompletableFuture<Void> future = CompletableFuture.runAsync(() -> {
                try {
                    RollbackResponse response = participantClient.rollback(
                        participant.getEndpointUrl(),
                        new RollbackRequest(id)
                    );
                    
                    if (!response.isSuccessful()) {
                        // Log but don't fail the rollback process
                        logger.warn("Participant failed to rollback: {}", participant.getServiceId());
                    }
                    
                    // Update participant status
                    participant.setCurrentPhase(TransactionPhase.ROLLBACK);
                    transactionStore.update(transaction);
                    
                } catch (Exception e) {
                    logger.error("Failed to send rollback to participant: {}", 
                        participant.getServiceId(), e);
                }
            });
            
            futures.add(future);
        }
        
        try {
            CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).get(10, TimeUnit.SECONDS);
            
            // Update transaction status to rolled back
            transaction.updateStatus(TransactionStatus.ROLLBACK);
            transactionStore.update(transaction);
            
            // Publish rolled back event
            eventPublisher.publish(new TransactionEvent(
                id,
                TransactionEventType.TRANSACTION_ROLLED_BACK,
                "Transaction rolled back successfully"
            ));
            
            return true;
        } catch (Exception e) {
            logger.error("Rollback timeout or failure for transaction: {}", id, e);
            return false;
        }
    }

    @Override
    public TransactionStatus getTransactionStatus(TransactionId id) {
        return transactionStore.findById(id).getStatus();
    }
}

Strategy Pattern Implementation

The service layer uses the Strategy pattern for handling different participant operation types:

public interface ParticipantOperationStrategy {
    PrepareResponse prepare(ParticipantRequest request);
    CommitResponse commit(ParticipantRequest request);
    RollbackResponse rollback(ParticipantRequest request);
}

public class ParticipantOperationStrategyFactory {
    private final Map<String, ParticipantOperationStrategy> strategies = new HashMap<>();
    
    public ParticipantOperationStrategyFactory() {
        strategies.put("CREATE_USER", new UserOperationStrategy());
        strategies.put("UPDATE_ACCOUNT", new AccountOperationStrategy());
        strategies.put("TRANSFER_MONEY", new MoneyTransferStrategy());
    }
    
    public ParticipantOperationStrategy getStrategy(String operationType) {
        return strategies.getOrDefault(operationType, new DefaultOperationStrategy());
    }
}

public class UserOperationStrategy implements ParticipantOperationStrategy {
    @Override
    public PrepareResponse prepare(ParticipantRequest request) {
        // Specific logic for user creation preparation
        return new PrepareResponse(true, "User prepare successful");
    }
    
    @Override
    public CommitResponse commit(ParticipantRequest request) {
        // Specific logic for user creation commit
        return new CommitResponse(true, "User commit successful");
    }
    
    @Override
    public RollbackResponse rollback(ParticipantRequest request) {
        // Specific logic for user creation rollback
        return new RollbackResponse(true, "User rollback successful");
    }
}

Critical Path Method Implementation

public boolean commitTransaction(TransactionId id) {
    // Transaction boundary - all operations within a single transaction context
    try (TransactionContext tx = transactionManager.begin()) {
        // Lock the transaction to prevent concurrent modifications
        String lockKey = "transaction:" + id;
        if (!lockManager.acquireLock(lockKey, 30, TimeUnit.SECONDS)) {
            throw new TransactionException("Unable to acquire transaction lock: " + id);
        }
        
        // Load current transaction state
        Transaction transaction = transactionStore.findById(id);
        if (transaction == null) {
            throw new TransactionException("Transaction not found: " + id);
        }
        
        // Validate transaction state
        if (transaction.getStatus() != TransactionStatus.PREPARED) {
            throw new TransactionException("Transaction not in prepared state: " + id);
        }
        
        // Prepare commit operation with optimistic locking
        long expectedVersion = transaction.getVersion();
        transaction.setVersion(transaction.getVersion() + 1);
        
        // Execute commit on all participants in parallel with timeout
        List<CompletableFuture<Void>> futures = new ArrayList<>();
        
        for (TransactionParticipant participant : transaction.getParticipants()) {
            CompletableFuture<Void> future = CompletableFuture.runAsync(() -> {
                try {
                    // Send commit request with retry logic
                    CommitRequest commitRequest = new CommitRequest(id, transaction.getVersion());
                    
                    // Retry up to 3 times with exponential backoff
                    int maxRetries = 3;
                    int retryCount = 0;
                    Exception lastException = null;
                    
                    while (retryCount < maxRetries) {
                        try {
                            CommitResponse response = participantClient.commit(
                                participant.getEndpointUrl(), commitRequest);
                            
                            if (response.isSuccessful()) {
                                break; // Success, exit retry loop
                            } else {
                                throw new TransactionException("Participant commit failed: " + 
                                    participant.getServiceId());
                            }
                        } catch (Exception e) {
                            lastException = e;
                            retryCount++;
                            
                            if (retryCount < maxRetries) {
                                // Exponential backoff with jitter
                                long delay = (long) Math.pow(2, retryCount) * 1000;
                                Thread.sleep(delay + new Random().nextInt(500));
                            }
                        }
                    }
                    
                    if (retryCount == maxRetries && lastException != null) {
                        throw new TransactionException("Commit failed after retries", lastException);
                    }
                    
                } catch (Exception e) {
                    throw new TransactionException("Failed to commit participant: " + 
                        participant.getServiceId(), e);
                }
            });
            
            futures.add(future);
        }
        
        // Wait for all participants to commit or timeout
        try {
            CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]))
                .get(30, TimeUnit.SECONDS);
        } catch (TimeoutException e) {
            // Handle timeout - initiate rollback
            logger.warn("Commit timeout for transaction: {}", id);
            rollbackTransaction(id);
            throw new TransactionException("Commit timeout - rolled back transaction", e);
        }
        
        // Update transaction status atomically
        transaction.updateStatus(TransactionStatus.COMMITTED);
        transactionStore.update(transaction);
        
        // Publish commit event
        TransactionEvent commitEvent = new TransactionEvent(
            id, 
            TransactionEventType.TRANSACTION_COMMITTED,
            "Transaction committed successfully"
        );
        eventPublisher.publish(commitEvent);
        
        // Release transaction lock
        lockManager.releaseLock(lockKey);
        
        // Commit the database transaction
        tx.commit();
        
        return true;
    } catch (Exception e) {
        // Handle any transaction errors
        logger.error("Failed to commit transaction: {}", id, e);
        throw new TransactionException("Transaction commit failed", e);
    }
}

Transaction Boundary Design

The transaction boundary encompasses:

1. Database operations (transaction persistence)
2. Participant communication (network calls)
3. State management (status updates)
4. Event publishing (audit trail)

Each transaction operation is wrapped in a database transaction to ensure atomicity and consistency. The system uses optimistic locking to prevent concurrent modification conflicts.

Service Interface Contracts

public interface TransactionCoordinator {
    /**
     * Initiates a new distributed transaction with specified participants.
     * @param request Transaction initiation request
     * @return TransactionId for tracking the transaction
     * @throws TransactionException if initiation fails
     */
    TransactionId initiateTransaction(TransactionRequest request);
    
    /**
     * Commits a prepared transaction.
     * @param id Transaction identifier
     * @return true if successful, false otherwise
     * @throws TransactionException if commit fails
     */
    boolean commitTransaction(TransactionId id);
    
    /**
     * Rolls back a transaction in any state.
     * @param id Transaction identifier
     * @return true if successful, false otherwise
     */
    boolean rollbackTransaction(TransactionId id);
    
    /**
     * Gets the current status of a transaction.
     * @param id Transaction identifier
     * @return Current transaction status
     */
    TransactionStatus getTransactionStatus(TransactionId id);
}

Saga Orchestration Pattern

The system implements a saga pattern where the coordinator manages the global transaction state and orchestrates participant operations. Each participant is responsible for local consistency, while the coordinator ensures global consistency through coordination.

Domain Object Design

public class Transaction {
    private final TransactionId id;
    private TransactionStatus status;
    private final List<TransactionParticipant> participants;
    private final long createdAt;
    private volatile long updatedAt;
    private volatile long version = 0;

    public Transaction(TransactionId id, TransactionStatus status, 
                      List<TransactionParticipant> participants, long createdAt) {
        this.id = id;
        this.status = status;
        this.participants = new ArrayList<>(participants);
        this.createdAt = createdAt;
        this.updatedAt = createdAt;
    }

    public void addParticipant(TransactionParticipant participant) {
        this.participants.add(participant);
        this.updatedAt = System.currentTimeMillis();
    }

    public void updateStatus(TransactionStatus status) {
        this.status = status;
        this.updatedAt = System.currentTimeMillis();
    }

    // Getters and setters...
}

public class TransactionParticipant {
    private final String serviceId;
    private final String endpointUrl;
    private TransactionPhase currentPhase;
    private final long createdAt;
    private volatile long lastUpdated;

    public TransactionParticipant(String serviceId, String endpointUrl, 
                                TransactionPhase currentPhase, long createdAt) {
        this.serviceId = serviceId;
        this.endpointUrl = endpointUrl;
        this.currentPhase = currentPhase;
        this.createdAt = createdAt;
        this.lastUpdated = createdAt;
    }

    // Getters and setters...
}

5. Concurrency & Thread Safety

Concurrency Model

The distributed transaction coordinator operates in a multi-threaded environment where multiple concurrent transactions can be processed simultaneously. The system uses a thread pool with fixed-size workers to handle transaction operations, ensuring predictable resource usage and avoiding thread starvation. Each transaction coordinator instance runs on a dedicated set of threads, with separate pools for:

• Transaction state management (read/write operations)
• Event processing and async handling
• Retry and timeout monitoring

Race Condition Prevention Strategy

Race conditions are mitigated through:

1. Immutable transaction state objects - Once a transaction enters a committed or rolled back state, it cannot be modified
2. Atomic operations - All state transitions use atomic operations or database-level atomic updates
3. Locking mechanisms - Per-transaction locks to prevent concurrent modifications during critical operations
4. Versioned state updates - Optimistic concurrency control with version numbers to detect and handle conflicts

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected for the following reasons:

• High throughput scenarios with low contention
• Transactions that are typically read-heavy or have short durations
• Better scalability characteristics compared to pessimistic locking
• Reduced lock wait times and improved system responsiveness

Implementation approach:

public class TransactionState {
    private final String transactionId;
    private volatile TransactionStatus status;
    private final int version; // For optimistic locking
    
    public boolean updateStatusIfVersionMatches(TransactionStatus newStatus, int expectedVersion) {
        // Atomic compare-and-swap operation
        return CAS.update(this, newStatus, expectedVersion);
    }
}

Deadlock Prevention Strategy

Deadlock prevention is implemented through:

1. Transaction ordering - All transactions acquire locks in a consistent global order (e.g., by transaction ID)
2. Timeout-based detection - Transaction manager enforces maximum wait times for lock acquisition
3. Lock hierarchy - Defined lock acquisition order to prevent circular waits:
   • First: Transaction coordinator locks
   • Then: Participant resource locks (in ID-sorted order)
4. Deadlock detection algorithm - Periodic cycle detection in the wait-for graph

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is enforced via:

1. Idempotency keys - Clients provide unique identifiers for each request
2. Request deduplication cache - In-memory cache with TTL to store processed requests
3. Database-level uniqueness constraints - Unique indexes on idempotency key + operation type

@Service
public class TransactionCoordinator {
    private final Map<String, CompletableFuture<TransactionResult>> idempotencyCache = 
        new ConcurrentHashMap<>();
    
    public CompletableFuture<TransactionResult> processTransaction(
            String idempotencyKey, TransactionRequest request) {
        
        return idempotencyCache.computeIfAbsent(idempotencyKey, key -> {
            // Check if already processed
            TransactionResult existing = transactionRepository.findResultByIdempotencyKey(key);
            if (existing != null) {
                return CompletableFuture.completedFuture(existing);
            }
            
            // Process new transaction
            return CompletableFuture.supplyAsync(() -> {
                try {
                    return executeTransaction(request);
                } catch (Exception e) {
                    // Handle and cache error results
                    return new TransactionResult(key, TransactionStatus.FAILED, e.getMessage());
                }
            });
        });
    }
}

Thread Safety Patterns

1. Immutable objects - Transaction state and metadata are immutable once created
2. Thread-local storage - For per-thread transaction context (e.g., trace IDs, user info)
3. Synchronized collections - Use of ConcurrentHashMap and CopyOnWriteArrayList for shared state
4. ReentrantReadWriteLocks - For scenarios where read-heavy operations need to be concurrent but writes must be exclusive

Transaction Isolation Level

The coordinator maintains Read Committed isolation level to:

• Prevent dirty reads (uncommitted changes from other transactions)
• Allow for better concurrency than Serializable
• Support high throughput workloads with acceptable consistency guarantees

Concurrency Control for Distributed Participants

For distributed transactions involving multiple participants:

1. Two-phase commit (2PC) coordination - Ensures atomicity across all participants
2. Participant state tracking - Thread-safe state machine for each participant with concurrent access protection
3. Timeout management - Per-participant timeout handling to prevent indefinite blocking

public class DistributedTransaction {
    private final Map<String, ParticipantState> participants = new ConcurrentHashMap<>();
    
    public synchronized void updateParticipantStatus(String participantId, 
                                                   ParticipantStatus status) {
        participants.computeIfAbsent(participantId, k -> new ParticipantState())
                   .setStatus(status);
    }
}

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant Client
    participant Coordinator
    participant ParticipantA
    participant ParticipantB
    participant ParticipantC

    Client->>Coordinator: BeginTransaction()
    Coordinator->>ParticipantA: Prepare(A)
    Coordinator->>ParticipantB: Prepare(B)
    Coordinator->>ParticipantC: Prepare(C)
    ParticipantA-->>Coordinator: PrepareResponse(A)
    ParticipantB-->>Coordinator: PrepareResponse(B)
    ParticipantC-->>Coordinator: PrepareResponse(C)
    Coordinator->>ParticipantA: Commit(A)
    Coordinator->>ParticipantB: Commit(B)
    Coordinator->>ParticipantC: Commit(C)
    ParticipantA-->>Coordinator: CommitResponse(A)
    ParticipantB-->>Coordinator: CommitResponse(B)
    ParticipantC-->>Coordinator: CommitResponse(C)
    Coordinator-->>Client: TransactionComplete()

sequenceDiagram
    participant Client
    participant Coordinator
    participant ParticipantA
    participant ParticipantB

    Client->>Coordinator: BeginTransaction()
    Coordinator->>ParticipantA: Prepare(A)
    Coordinator->>ParticipantB: Prepare(B)
    ParticipantA-->>Coordinator: PrepareResponse(A)
    ParticipantB-->>Coordinator: PrepareResponse(B)
    Coordinator->>ParticipantA: Rollback(A)
    Coordinator->>ParticipantB: Rollback(B)
    ParticipantA-->>Coordinator: RollbackResponse(A)
    ParticipantB-->>Coordinator: RollbackResponse(B)
    Coordinator-->>Client: TransactionAborted()

sequenceDiagram
    participant Client
    participant Coordinator
    participant ParticipantA
    participant ParticipantB

    Client->>Coordinator: BeginTransaction()
    Coordinator->>ParticipantA: Prepare(A)
    Coordinator->>ParticipantB: Prepare(B)
    ParticipantA-->>Coordinator: PrepareResponse(A)
    ParticipantB-->>Coordinator: PrepareResponse(B)
    Coordinator->>ParticipantA: Commit(A)
    ParticipantA-->>Coordinator: CommitResponse(A)
    Coordinator->>ParticipantB: Commit(B)
    ParticipantB-->>Coordinator: CommitResponse(B)
    Coordinator->>ParticipantA: Commit(A)
    ParticipantA-->>Coordinator: CommitResponse(A)
    Coordinator->>ParticipantB: Commit(B)
    ParticipantB-->>Coordinator: CommitResponse(B)
    Coordinator-->>Client: TransactionComplete()

7. Caching Design

Cache Strategy Selection

For a distributed transaction coordinator, we'll use cache-aside pattern with write-through behavior for critical transaction state data. This approach provides the best balance between consistency and performance for a system where transaction state must be reliably available.

Cache Key Design

Cache keys will follow this namespace strategy:

transaction:{txId}:status
transaction:{txId}:participants
transaction:{txId}:metadata
transaction:{txId}:lock:{resourceId}

TTL Strategy

Data Type	TTL	Reason
Transaction Status	15 minutes	Allows for transaction cleanup while preventing stale data
Participant List	30 minutes	Long enough to handle retries but not indefinitely
Transaction Metadata	5 minutes	Frequent updates, short-lived
Locks	2 minutes	Short-lived to prevent deadlocks

Cache Eviction Policy

We'll use LRU (Least Recently Used) eviction policy with a secondary TTL-based cleanup to handle edge cases where items are accessed infrequently but still need to be cleaned up.

Cache Stampede Prevention

To prevent cache stampedes:

1. Probabilistic Early Expiry: Cache entries expire 20% early to prevent simultaneous refreshes
2. Distributed Locks: Use Redis distributed locks when refreshing cache entries to prevent multiple concurrent refreshes
3. Cache Warm-up: Pre-warm hot transaction data during system startup

Distributed Cache Architecture

We'll use Redis Cluster with:

• 3 master nodes for high availability
• 3 replica nodes per master for read scaling
• Redis Sentinel for automatic failover
• Cache sharding based on transaction ID hash

Cache Invalidation Strategy

We'll implement event-driven invalidation:

1. Transaction state changes trigger cache invalidation events
2. Cache entries are invalidated with a delay to allow for concurrent reads
3. Critical transaction metadata is explicitly purged on completion or rollback

// Cache invalidation example for transaction status change
public class TransactionCacheInvalidator {
    private final RedisTemplate<String, Object> redisTemplate;
    
    public void invalidateTransactionCache(String transactionId) {
        String statusKey = "transaction:" + transactionId + ":status";
        String participantsKey = "transaction:" + transactionId + ":participants";
        String metadataKey = "transaction:" + transactionId + ":metadata";
        
        // Use pipeline for atomic invalidation
        redisTemplate.executePipelined(
            (RedisCallback<Object>) connection -> {
                connection.del(statusKey.getBytes());
                connection.del(participantsKey.getBytes());
                connection.del(metadataKey.getBytes());
                return null;
            }
        );
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and database operations. Each domain entity has its own repository interface with standardized CRUD operations.

public interface TransactionRepository {
    Transaction save(Transaction transaction);
    Optional<Transaction> findById(UUID id);
    List<Transaction> findByStatus(TransactionStatus status);
    void update(Transaction transaction);
    void delete(UUID id);
    List<Transaction> findAllPaginated(int page, int size);
}

public interface TransactionParticipantRepository {
    void save(TransactionParticipant participant);
    List<TransactionParticipant> findByTransactionId(UUID transactionId);
    void update(TransactionParticipant participant);
}

ORM vs. Raw Query Decisions

For the distributed transaction coordinator, we use a hybrid approach:

• ORM for complex domain entities: Hibernate/JPA is used for Transaction and TransactionParticipant entities due to their complex relationships and the need for object-relational mapping.
• Raw queries for performance-critical operations: For high-frequency status updates and transaction state lookups, raw SQL is used to minimize overhead.

@Repository
public class TransactionRepositoryImpl implements TransactionRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    @Override
    public Transaction save(Transaction transaction) {
        entityManager.persist(transaction);
        return transaction;
    }
    
    @Override
    public Optional<Transaction> findById(UUID id) {
        return Optional.ofNullable(entityManager.find(Transaction.class, id));
    }
    
    @Override
    public List<Transaction> findByStatus(TransactionStatus status) {
        return entityManager.createQuery(
            "SELECT t FROM Transaction t WHERE t.status = :status", 
            Transaction.class)
            .setParameter("status", status)
            .getResultList();
    }
    
    @Override
    public void update(Transaction transaction) {
        entityManager.merge(transaction);
    }
    
    @Override
    public void delete(UUID id) {
        Transaction transaction = entityManager.find(Transaction.class, id);
        if (transaction != null) {
            entityManager.remove(transaction);
        }
    }
    
    @Override
    public List<Transaction> findAllPaginated(int page, int size) {
        return entityManager.createQuery(
            "SELECT t FROM Transaction t ORDER BY t.createdAt DESC", 
            Transaction.class)
            .setFirstResult(page * size)
            .setMaxResults(size)
            .getResultList();
    }
    
    // Raw query for performance-critical operations
    @Query(value = "SELECT t.id, t.status FROM transaction t WHERE t.status = :status AND t.created_at < :cutoff", 
           nativeQuery = true)
    public List<Object[]> findStaleTransactions(@Param("status") TransactionStatus status, 
                                                @Param("cutoff") LocalDateTime cutoff);
}

Read vs. Write Separation

The CQRS pattern is applied to separate read and write operations:

// Write model - used for transaction creation and modification
@Repository
public class TransactionWriteRepository {
    @PersistenceContext
    private EntityManager entityManager;
    
    public void save(Transaction transaction) {
        entityManager.persist(transaction);
    }
    
    public void update(Transaction transaction) {
        entityManager.merge(transaction);
    }
}

// Read model - optimized for querying transaction status
@Repository
public class TransactionReadRepository {
    @PersistenceContext
    private EntityManager entityManager;
    
    public TransactionStatusView getStatus(UUID transactionId) {
        return entityManager.createQuery(
            "SELECT new com.example.transaction.view.TransactionStatusView(t.id, t.status, t.createdAt) " +
            "FROM Transaction t WHERE t.id = :id", 
            TransactionStatusView.class)
            .setParameter("id", transactionId)
            .getSingleResult();
    }
}

Connection Pool Configuration

Connection pooling is configured with HikariCP for optimal performance:

spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      validation-timeout: 5000
      leak-detection-threshold: 60000

N+1 Query Prevention Strategy

To prevent N+1 query problems, eager loading is used for related entities:

@Entity
@Table(name = "transactions")
@NamedEntityGraph(
    name = "Transaction.withParticipants",
    attributeNodes = @NamedAttributeNode(value = "participants", fetchMode = FetchType.EAGER)
)
public class Transaction {
    @Id
    private UUID id;
    
    @OneToMany(mappedBy = "transaction", fetch = FetchType.EAGER)
    private List<TransactionParticipant> participants;
    
    // Other fields...
}

Batch Operation Design

Batch operations are implemented for bulk transaction status updates:

@Repository
public class BatchTransactionRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    public void batchUpdateStatus(List<UUID> transactionIds, TransactionStatus newStatus) {
        String jpql = "UPDATE Transaction t SET t.status = :newStatus WHERE t.id IN :ids";
        entityManager.createQuery(jpql)
            .setParameter("newStatus", newStatus)
            .setParameter("ids", transactionIds)
            .executeUpdate();
    }
    
    public void batchSaveParticipants(List<TransactionParticipant> participants) {
        // Use JDBC batch operations for better performance
        String sql = "INSERT INTO transaction_participants (id, transaction_id, participant_id, status) VALUES (?, ?, ?, ?)";
        
        try (Connection conn = dataSource.getConnection();
             PreparedStatement stmt = conn.prepareStatement(sql)) {
            
            for (TransactionParticipant participant : participants) {
                stmt.setObject(1, participant.getId());
                stmt.setObject(2, participant.getTransactionId());
                stmt.setString(3, participant.getParticipantId());
                stmt.setString(4, participant.getStatus().name());
                stmt.addBatch();
            }
            
            stmt.executeBatch();
        } catch (SQLException e) {
            throw new DataAccessException("Failed to batch save participants", e);
        }
    }
}

Transaction Management

The repository layer uses proper transaction management:

@Service
@Transactional
public class TransactionService {
    
    @Autowired
    private TransactionRepository transactionRepository;
    
    public Transaction createTransaction(Transaction transaction) {
        // This will be executed within a single database transaction
        return transactionRepository.save(transaction);
    }
    
    @Transactional(readOnly = true)
    public List<Transaction> getActiveTransactions() {
        return transactionRepository.findByStatus(TransactionStatus.ACTIVE);
    }
}

Data Access Layer Design Summary

The repository layer provides:

1. Type Safety: Strong typing through JPA entities
2. Separation of Concerns: Clear distinction between read and write operations
3. Performance Optimization: Hybrid ORM/raw query approach with batch operations
4. Scalability: Connection pooling and efficient query execution
5. Maintainability: Clean abstraction that allows for easy database migration or replacement

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The distributed transaction coordinator operates primarily in synchronous mode for coordination, but leverages asynchronous messaging for:

• Transaction state change notifications (e.g., COMMITTED, ROLLED_BACK)
• Compensation operation triggers
• Audit logging and monitoring events

The coordinator itself is designed to be stateless and can handle multiple concurrent distributed transactions. It uses a message broker for decoupling transaction participants from the coordinator, enabling independent scaling of components.

Message schema design

{
  "version": "1.0",
  "event_type": "transaction_state_change",
  "timestamp": "2023-10-27T14:30:00Z",
  "transaction_id": "uuid-1234567890",
  "state": "COMMITTED|ROLLED_BACK",
  "participant_id": "participant-123",
  "details": {
    "compensation_required": true,
    "compensation_topic": "compensation-topic"
  }
}

Dead letter queue strategy and poison message handling

Messages that fail to be processed after maximum retries are moved to a dead letter queue (DLQ) with a dedicated topic:

• DLQ Topic: dlq.transaction-coordinator
• Poison Message Handling:
  • Messages that fail more than 3 times are routed to DLQ
  • DLQ messages are flagged with metadata is_poison=true
  • Manual intervention required for DLQ message processing
  • Alerting triggered when DLQ queue size exceeds threshold (50 messages)

At-least-once vs. exactly-once delivery

Decision: At-least-once delivery with idempotency guarantees.

Justification:

• Distributed transactions must ensure all participants reach the same state
• Idempotency keys prevent duplicate processing of transaction steps
• Message brokers (e.g., Kafka) support exactly-once semantics when configured with idempotent producers and consumer groups

Retry policy for consumers

public class TransactionEventConsumer {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double BACKOFF_MULTIPLIER = 2.0;
    
    public void processTransactionEvent(TransactionEvent event) {
        int attempts = 0;
        while (attempts < MAX_RETRIES) {
            try {
                handleEvent(event);
                return;
            } catch (RetryableException e) {
                attempts++;
                if (attempts >= MAX_RETRIES) {
                    moveMessageToDLQ(event);
                    throw new RuntimeException("Max retries exceeded", e);
                }
                long delay = (long) (BASE_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempts));
                Thread.sleep(delay);
            } catch (NonRetryableException e) {
                moveMessageToDLQ(event);
                throw e;
            }
        }
    }
}

Outbox pattern for reliable event publishing

Decision: Implement outbox pattern to ensure transactional consistency between business logic and event publishing.

Implementation:

• Events are written to an event_outbox table within the same database transaction as business logic
• A background worker periodically polls and publishes events to message broker
• Events are marked as published once successfully delivered

@Entity
@Table(name = "event_outbox")
public class EventOutbox {
    @Id
    private UUID id;
    
    @Column(name = "event_payload")
    private String payload;
    
    @Column(name = "event_type")
    private String eventType;
    
    @Column(name = "status")
    private EventStatus status = EventStatus.PENDING;
    
    @Column(name = "created_at")
    private LocalDateTime createdAt;
}

Sequence diagram for async flows

sequenceDiagram
    participant TC as TransactionCoordinator
    participant ParticipantA as ParticipantA
    participant ParticipantB as ParticipantB
    participant Broker as MessageBroker
    participant DLQ as DeadLetterQueue

    TC->>ParticipantA: Prepare request
    ParticipantA-->>TC: Prepare response (ACK)
    TC->>ParticipantB: Prepare request
    ParticipantB-->>TC: Prepare response (ACK)
    TC->>TC: Validate all prepare responses
    TC->>Broker: Publish COMMITTED event
    Broker-->>ParticipantA: Commit event
    ParticipantA-->>Broker: Acknowledge
    Broker-->>ParticipantB: Commit event
    ParticipantB-->>Broker: Acknowledge
    Broker->>TC: Event delivered confirmation
    TC->>TC: Mark transaction as COMMITTED
    Broker->>DLQ: If delivery fails, move to DLQ

10. Error Handling Design

Error Classification Hierarchy

The error handling strategy follows a hierarchical classification to enable proper routing and recovery:

1. Domain Errors - Business logic violations (e.g., insufficient funds, invalid state transitions)
2. Validation Errors - Input data issues (e.g., missing fields, format violations)
3. Infrastructure Errors - System-level failures (e.g., database timeouts, network issues)
4. External Service Errors - Third-party integration problems (e.g., downstream API failures)

public class DistributedTransactionError extends RuntimeException {
    private final ErrorType errorType;
    private final String errorCode;
    
    public enum ErrorType {
        DOMAIN, VALIDATION, INFRASTRUCTURE, EXTERNAL_SERVICE
    }
    
    public DistributedTransactionError(ErrorType errorType, String errorCode, String message) {
        super(message);
        this.errorType = errorType;
        this.errorCode = errorCode;
    }
}

Error Propagation Strategy

• Domain and Validation errors: Bubbled up to the API layer for proper HTTP response formatting
• Infrastructure errors: Handled at service layer with retry logic or circuit breaker patterns
• External service errors: Logged and potentially retried based on error classification

Retry Strategy with Exponential Backoff and Jitter

For infrastructure and external service errors, a retry mechanism is implemented with exponential backoff and jitter to prevent thundering herd:

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_RANGE = 0.2; // ±20% jitter
    
    public <T> T executeWithRetry(Supplier<T> operation) {
        int attempts = 0;
        while (attempts < MAX_RETRIES) {
            try {
                return operation.get();
            } catch (InfrastructureException e) {
                attempts++;
                if (attempts >= MAX_RETRIES) throw e;
                
                long delay = calculateDelay(attempts);
                Thread.sleep(delay);
            }
        }
        throw new RuntimeException("Max retries exceeded");
    }
    
    private long calculateDelay(int attempt) {
        long delay = (long) Math.pow(2, attempt) * BASE_DELAY_MS;
        long jitter = (long) (delay * JITTER_RANGE * (Math.random() - 0.5) * 2);
        return delay + jitter;
    }
}

Circuit Breaker Design

For external service dependencies, a circuit breaker pattern is implemented to prevent cascading failures:

public class CircuitBreaker {
    private static final int FAILURE_THRESHOLD = 5;
    private static final long TIMEOUT_MS = 60000;
    private static final int RESET_TIMEOUT_MS = 30000;
    
    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private long lastFailureTime = 0;
    
    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }
    
    public <T> T execute(Supplier<T> operation) {
        switch (state) {
            case CLOSED:
                return executeClosed(operation);
            case OPEN:
                return executeOpen(operation);
            case HALF_OPEN:
                return executeHalfOpen(operation);
        }
        throw new IllegalStateException("Invalid circuit state");
    }
    
    private <T> T executeClosed(Supplier<T> operation) {
        try {
            T result = operation.get();
            failureCount = 0; // Reset on success
            return result;
        } catch (Exception e) {
            failureCount++;
            lastFailureTime = System.currentTimeMillis();
            
            if (failureCount >= FAILURE_THRESHOLD) {
                state = CircuitState.OPEN;
            }
            throw e;
        }
    }
    
    private <T> T executeOpen(Supplier<T> operation) {
        if (System.currentTimeMillis() - lastFailureTime > RESET_TIMEOUT_MS) {
            state = CircuitState.HALF_OPEN;
            return executeHalfOpen(operation);
        }
        throw new CircuitBreakerOpenException("Circuit is currently open");
    }
    
    private <T> T executeHalfOpen(Supplier<T> operation) {
        try {
            T result = operation.get();
            state = CircuitState.CLOSED;
            return result;
        } catch (Exception e) {
            state = CircuitState.OPEN;
            throw e;
        }
    }
}

Fallback Behavior Design

Fallbacks are implemented for critical external dependencies with graceful degradation:

public class FallbackService {
    public TransactionResult getTransactionStatusFallback(String transactionId) {
        // Return cached or default state
        return new TransactionResult(
            transactionId, 
            TransactionStatus.UNKNOWN,
            "Fallback status due to external service failure"
        );
    }
}

Error Logging Format and Alerting Thresholds

Structured logging is implemented with consistent fields:

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);
    
    public void logError(String operation, String errorCode, Exception e, Map<String, Object> context) {
        logger.error(
            "Operation: {}, Error Code: {}, Message: {}, Context: {}",
            operation, errorCode, e.getMessage(), context,
            e
        );
        
        // Alerting logic based on error severity
        if (isCriticalError(errorCode)) {
            sendAlertToMonitoringSystem(operation, errorCode, e);
        }
    }
    
    private boolean isCriticalError(String errorCode) {
        return errorCode.startsWith("INFRA") || errorCode.startsWith("EXTERNAL");
    }
}

Alerting Thresholds

Error Type	Warning Threshold	Critical Threshold	Alerting Channel
Infrastructure errors	5 per minute	20 per minute	Slack/Email
External service errors	3 per minute	15 per minute	PagerDuty
Domain validation errors	10 per hour	50 per hour	Email

Error Response Format

Standardized JSON response for API errors:

{
  "error": {
    "code": "TRANSACTION_NOT_FOUND",
    "message": "Transaction with ID abc123 not found",
    "details": {
      "transactionId": "abc123",
      "timestamp": "2023-05-15T10:30:00Z"
    }
  }
}

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using Jakarta Bean Validation (JSR-380) with custom constraints for business logic validation. For sensitive fields like transaction IDs and participant identifiers, regex patterns ensure only alphanumeric and hyphen characters are accepted. HTML sanitization is performed using OWASP Java HTML Sanitizer for any user-facing content fields to prevent XSS attacks.

SQL Injection Prevention

All database queries are constructed using parameterized prepared statements with Hibernate's Criteria API to prevent SQL injection. Raw SQL is never used for user inputs, and any dynamic query construction uses proper escaping mechanisms.

Authentication Middleware Design

Authentication is handled via JWT (JSON Web Tokens) with RS256 signature algorithm. The authentication middleware validates token signature, expiration, and audience claims before allowing access to protected endpoints. Token refresh mechanism is implemented with a separate endpoint that accepts refresh tokens stored in secure HTTP-only cookies.

Authorization Enforcement

Role-Based Access Control (RBAC) is implemented at the service layer using Spring Security annotations. Permissions are mapped to roles in a centralized policy store, with fine-grained checks for transaction coordinator operations like initiating, committing, or aborting distributed transactions. Authorization is enforced at the method level using @PreAuthorize annotations.

Sensitive Data Encryption

All sensitive transaction data is encrypted at rest using AES-256-GCM encryption with keys rotated quarterly. Transaction payloads and participant state information are encrypted using a master key derived from AWS KMS or equivalent cloud HSM service. In-transit data uses TLS 1.3 with HSTS header enforcement.

PII Handling and Masking Strategy

PII data is never stored in plain text. Transaction participant identifiers are masked using a deterministic hashing algorithm (SHA-256) with salt for audit purposes. Any logs or monitoring systems that might contain PII data are sanitized before being written to log aggregation services.

Secrets Management

All secrets including database credentials, JWT signing keys, and external service tokens are managed through AWS Secrets Manager or HashiCorp Vault. The application fetches secrets at startup and refreshes them periodically via a background task to prevent credential exposure.

Audit Logging Design

All security-relevant events are logged with structured JSON format including:

• Event type (transaction_initiated, transaction_committed, authorization_failed)
• Timestamp in ISO8601 format
• User/Service ID (masked for PII)
• IP address (redacted in logs)
• Transaction ID
• Outcome (success/failure)
• Session ID

Audit logs are retained for 90 days and stored in a separate encrypted log storage system with read access restricted to security team members only. All audit events are signed with a digital signature for tamper-proof logging.

Security Headers Implementation

All HTTP responses include security headers:

• Content-Security-Policy: "default-src 'self'; script-src 'self' 'unsafe-inline'; object-src 'none'"
• X-Content-Type-Options: "nosniff"
• X-Frame-Options: "DENY"
• X-XSS-Protection: "1; mode=block"
• Strict-Transport-Security: "max-age=31536000; includeSubDomains"
• Referrer-Policy: "strict-origin-when-cross-origin"

Vulnerability Scanning and Dependency Management

Regular dependency scanning is performed using OWASP Dependency-Check or similar tools to identify known vulnerabilities in third-party libraries. Security patches are applied within 48 hours of release, and all dependencies are pinned to specific versions in the build configuration.

12. Testing Strategy

Unit Testing Scope and Coverage Targets

Unit testing will cover 85%+ code coverage for business logic, with a focus on service layer methods and domain object operations. Each service class will have dedicated test classes, with individual methods tested for:

• Happy path scenarios
• Error conditions and boundary cases
• Null input validation
• Concurrency edge cases

For the distributed transaction coordinator, unit tests will validate:

• Transaction state transitions
• Participant registration and heartbeat handling
• Retry logic for failed participants
• Idempotency handling for duplicate requests

Integration Testing Strategy

Integration testing will use testcontainers for database and messaging system simulation, ensuring realistic behavior of:

• Transaction coordinator with database persistence
• Message broker interactions (e.g., Kafka/Redis)
• External service call mocks (e.g., participant service endpoints)

Test containers will be used for:

• PostgreSQL instances for transaction state persistence
• Redis instances for caching and coordination
• Kafka clusters for event-driven communication

Contract Testing

Contract testing will be implemented using Pact for consumer-provider contract verification. Key areas:

• Transaction coordinator API contracts with participant services
• Event schema contracts between transaction coordinator and event system
• External service integration contracts (e.g., payment gateway, notification service)

Performance Testing Tool and Key Scenarios

Performance testing will use k6 for load testing distributed transaction coordinator with:

• Concurrent transaction creation (1000+ simultaneous transactions)
• Transaction participant failure simulation under load
• High throughput transaction state persistence (10,000+ transactions/sec)
• Message processing rate testing (Kafka/Redis message throughput)

Key performance scenarios:

• 100 concurrent distributed transactions with 5 participants each
• 50% participant failure rate under load
• Transaction timeout handling under high contention

Test Data Management Strategy

Test data will be managed using:

• Factory patterns for generating valid transaction objects with various states
• Fixtures for common transaction scenarios (success, partial failure, timeout)
• Seeding scripts for initial database state (participant registry, transaction history)

Test data will include:

• Transaction states (created, in_progress, committed, rolled_back)
• Participant statuses (registered, healthy, failed, timeout)
• Transaction metadata (start_time, timeout_duration, participant_count)

// Example test data factory for distributed transaction
public class TransactionFactory {
    public static DistributedTransaction createTransactionWithParticipants(
            int participantCount, 
            TransactionState initialState) {
        DistributedTransaction transaction = new DistributedTransaction();
        transaction.setId(UUID.randomUUID());
        transaction.setState(initialState);
        transaction.setCreatedAt(Instant.now());
        
        List<Participant> participants = new ArrayList<>();
        for (int i = 0; i < participantCount; i++) {
            Participant participant = new Participant();
            participant.setId(UUID.randomUUID());
            participant.setEndpoint("http://participant-" + i + ".service");
            participant.setState(ParticipantState.REGISTERED);
            participants.add(participant);
        }
        transaction.setParticipants(participants);
        
        return transaction;
    }
}

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective search, filtering, and analysis:

{
  "timestamp": "2023-10-27T14:30:00.000Z",
  "level": "INFO|WARN|ERROR",
  "service": "distributed-transaction-coordinator",
  "trace_id": "1234567890abcdef1234567890abcdef",
  "span_id": "abcdef1234567890abcdef1234567890",
  "user_id": "user-123",
  "transaction_id": "txn-abc123def456",
  "operation": "coordinator.handleCommit",
  "message": "Committing transaction",
  "error_code": "TXN_COMMIT_FAILED",
  "error_details": {
    "cause": "Participant timeout",
    "participant_id": "participant-xyz"
  },
  "duration_ms": 150,
  "host": "coordinator-01.example.com"
}

PII masking rules:

• User IDs are masked in logs (e.g., user-123 → user-***)
• Sensitive fields like passwords or tokens are completely omitted
• Full PII fields (email, phone) are truncated to first 2 characters

Distributed Tracing Instrumentation

We use OpenTelemetry for distributed tracing with custom spans:

Key Spans to Instrument:

1. distributed_transaction.start - Transaction initiation
2. participant_registration - Participant registration with status
3. coordinator.vote_request - Vote request to participants
4. participant_response - Participant response handling
5. coordinator.commit - Commit operation execution
6. participant_commit - Participant commit execution

Span Attributes:

• Transaction ID (required)
• Participant ID (when relevant)
• Vote status (VOTE_COMMIT/VOTE_ABORT)
• Operation type (START, VOTE, COMMIT, ABORT)
• Retry count (for retryable operations)

Metrics Design

RED Method Metrics:

Metric Name	Type	Description	Labels
transaction_count	Counter	Total transaction operations	operation_type, status
transaction_duration_ms	Histogram	Transaction processing time	operation_type, status
participant_errors	Counter	Participant error counts	participant_id, error_type
transaction_retries	Counter	Retry attempts per operation	operation_type, retry_count
active_transactions	Gauge	Current active transactions	none

Labels for Granularity:

• operation_type: START, VOTE, COMMIT, ABORT
• status: SUCCESS, FAILED, TIMEOUT, RETRY
• participant_id: Unique identifier for each participant
• error_type: TIMEOUT, NETWORK_ERROR, INVALID_RESPONSE

Health Check Endpoint Design

GET /health
GET /ready
GET /live
GET /startup

Response Format:

{
  "status": "healthy|unhealthy",
  "checks": {
    "database": {
      "status": "healthy",
      "timestamp": "2023-10-27T14:30:00.000Z"
    },
    "message_queue": {
      "status": "healthy",
      "timestamp": "2023-10-27T14:30:00.000Z"
    },
    "cache": {
      "status": "healthy",
      "timestamp": "2023-10-27T14:30:00.000Z"
    }
  },
  "timestamp": "2023-10-27T14:30:00.000Z"
}

Health Check Logic:

• /live - Basic liveness check (process running)
• /ready - Readiness check (dependencies ready)
• /health - Comprehensive health status (all components)
• /startup - Startup validation (initial configuration)

Alerting Thresholds

Critical Alerts (escalation to on-call):

Metric	Threshold	Alert Level
transaction_duration_ms (p99)	> 5000ms	CRITICAL
transaction_count (rate)	< 100 ops/sec	CRITICAL
participant_errors (rate)	> 50 errors/min	CRITICAL
active_transactions	> 1000 concurrent	WARNING

Warning Alerts:

Metric	Threshold	Alert Level
transaction_duration_ms (p95)	> 2000ms	WARNING
transaction_retries (rate)	> 10 retries/min	WARNING
participant_errors (rate)	> 5 errors/min	WARNING

Alerting Rules:

• Multiple consecutive failures trigger escalation
• Alert threshold violations must be logged with full context
• On-call rotation documented in system documentation

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

1. Transaction scope granularity - Should we support micro-transactions (per-service) or macro-transactions (cross-service) that span multiple services? This decision impacts the complexity of coordination and error recovery strategies.

2. Consistency model - Do we prioritize strong consistency (two-phase commit) or eventual consistency with idempotent operations? This affects the performance characteristics and system behavior under network partitions.

3. Service boundary definition - How granular should the distributed transaction boundaries be? Should each service own its own transaction or should we support cross-service atomic operations?

4. Retry policy configuration - Should retry policies be configurable per service or standardized across the system? This impacts operational flexibility and complexity.

Known trade-offs and justifications

1. Latency vs. Reliability Trade-off - Using synchronous coordination protocols (like 2PC) provides strong consistency but introduces significant latency and single points of failure. Asynchronous approaches like Saga or eventual consistency improve performance but introduce complexity in handling partial failures and data consistency.

2. Complexity vs. Simplicity Trade-off - Implementing a full-featured distributed transaction coordinator increases system complexity significantly. Simpler approaches like idempotent operations and event-driven architectures reduce complexity but may not provide the same level of transactional guarantees.

3. Resource utilization vs. Performance Trade-off - Maintaining extensive state in the coordinator (transaction metadata, participant status) consumes significant memory and disk space. This can impact performance under high load but is necessary for proper transaction management.

4. Network resilience vs. Atomicity Trade-off - Supporting network partitions and partial failures requires complex state management and recovery mechanisms, which can compromise the atomicity guarantees of distributed transactions.

Risks and mitigation strategies

1. Deadlock Risk - Distributed deadlocks can occur when multiple services are waiting for each other's resources. Mitigation includes implementing timeout mechanisms, deadlock detection algorithms, and using consistent ordering of resource acquisition.

2. System-wide Availability Risk - The coordinator itself becomes a single point of failure that can bring down the entire distributed transaction system. Mitigation includes implementing high availability with replication and failover mechanisms.

3. Data Inconsistency Risk - Network partitions or system failures can lead to inconsistent states across services. Mitigation includes implementing compensating actions, idempotent operations, and robust error recovery mechanisms.

4. Performance Degradation Risk - Coordination overhead can significantly impact system performance under high load. Mitigation includes optimizing state management, implementing efficient caching strategies, and using asynchronous processing where appropriate.

5. Complexity Accumulation Risk - Distributed transaction systems can become increasingly complex as more services and use cases are added. Mitigation includes maintaining clear architectural boundaries, providing comprehensive documentation, and implementing proper abstraction layers.

Technical debt acknowledged

1. State Management Complexity - The coordinator maintains extensive state about transaction participants and their status, creating technical debt in terms of storage requirements, memory usage, and state synchronization complexity.

2. Error Recovery Complexity - Handling various failure modes (network partitions, service unavailability, participant failures) introduces significant complexity in error recovery logic and state management.

3. Version Compatibility Debt - Supporting multiple versions of transaction protocols and service implementations creates ongoing maintenance burden and potential compatibility issues.

4. Monitoring and Debugging Debt - Distributed transaction systems are notoriously difficult to debug due to their distributed nature, requiring extensive logging and monitoring capabilities that add to system complexity.

5. Service Coupling Debt - The coordinator creates tight coupling between services and the coordination layer, making it harder to evolve services independently and potentially leading to performance bottlenecks.