Transaction Manager System Design

A transaction manager is a critical backend component that ensures data consistency and integrity across distributed systems, handling complex scenarios like distributed transactions, rollback recovery, and concurrent access control in high-throughput environments.

1. Problem Statement & Requirements

Clarifying questions

1. What scale and throughput requirements are expected for the transaction manager?
   • Is it designed to handle millions of concurrent transactions or is it more for moderate-scale operations?
2. What consistency guarantees are required (e.g., ACID, eventual consistency)?
   • Should the system support strong consistency or is eventual consistency acceptable?
3. What are the key entities and their relationships in the transaction system?
   • Are there specific entities like Transaction, Participant, CommitLog, etc.?
4. What is the expected read/write ratio for transaction data?
   • Is it predominantly reads or writes, and how does that impact caching and storage strategies?
5. Are there specific latency sensitivity requirements (e.g., p99 under 100ms)?
   • Is sub-100ms response time critical for transaction processing?
6. How is authentication and authorization handled in the system?
   • Is it based on OAuth2, JWTs, or API keys?
7. Are there any external systems that need to be integrated with the transaction manager?
   • For example, integration with external payment gateways or third-party services.

Functional requirements

• Support for distributed transaction management (e.g., 2PC or Saga patterns)
• Transaction lifecycle management (start, commit, rollback)
• Participant registration and coordination
• Transaction state tracking and persistence
• Support for transaction timeout and cleanup
• Transaction visibility and audit trail capabilities

Non-functional requirements

• Latency targets: p50 < 20ms, p95 < 50ms, p99 < 100ms for transaction operations
• Throughput: 5,000+ concurrent transactions per second (with burst up to 10,000)
• Availability: 99.9% uptime (99.99% for critical transaction paths)
• Durability: All committed transactions must persist and be recoverable
• Scalability: Horizontal scaling capability to support multi-region deployment

Constraints and assumptions

• The system assumes a single transaction manager instance or a highly available cluster of instances (e.g., Raft-based consensus)
• Participants in distributed transactions are assumed to be stateless or capable of handling compensating actions
• Transaction data is expected to be stored in a durable storage backend (e.g., relational database or distributed key-value store)
• The system assumes that all participants support the same transaction protocol (e.g., 2PC or Saga)
• Transaction state is expected to be highly consistent and not stale

Out of scope

• Integration with external systems (e.g., payment gateways or third-party services)
• Support for XA transactions or other legacy transaction protocols
• Complex business logic or domain-specific validation within the transaction manager itself (this is handled by participants)
• UI or dashboard for transaction monitoring and management
• Transaction recovery from system failures (e.g., crash recovery) — handled by external systems or persistence layer

2. API Design

API Style Selection

REST is selected for this Transaction Manager design due to its widespread adoption, clear semantics, and ease of integration with existing systems. REST's stateless nature aligns well with transaction management where each request should contain all necessary information to process the transaction independently.

Endpoint Definitions

Method	Path	Description	Auth
POST	/v1/transactions	Create a new transaction	JWT
GET	/v1/transactions/{id}	Retrieve transaction details	JWT
PUT	/v1/transactions/{id}/status	Update transaction status	JWT
GET	/v1/transactions/{id}/revisions	Get transaction revision history	JWT
POST	/v1/transactions/{id}/revert	Revert transaction to a previous revision	JWT
GET	/v1/transactions/{id}/events	Get transaction event stream	JWT

Request and Response Schemas

Create Transaction Request

{
  "amount": 100.00,
  "currency": "USD",
  "description": "Payment for order #12345",
  "metadata": {
    "orderId": "12345",
    "customerId": "cust-001"
  }
}

Create Transaction Response

{
  "id": "txn-1234567890",
  "amount": 100.00,
  "currency": "USD",
  "status": "PENDING",
  "description": "Payment for order #12345",
  "createdAt": "2023-10-15T10:30:00Z",
  "updatedAt": "2023-10-15T10:30:00Z",
  "revisions": []
}

Transaction Status Update Request

{
  "status": "COMPLETED"
}

Transaction Revision Response

[
  {
    "id": "rev-1234567890",
    "transactionId": "txn-1234567890",
    "status": "PENDING",
    "createdAt": "2023-10-15T10:30:00Z",
    "updatedAt": "2023-10-15T10:30:00Z"
  },
  {
    "id": "rev-1234567891",
    "transactionId": "txn-1234567890",
    "status": "COMPLETED",
    "createdAt": "2023-10-15T10:35:00Z",
    "updatedAt": "2023-10-15T10:35:00Z"
  }
]

HTTP Status Codes and Error Response Format

Standard HTTP status codes will be used:

• 200 OK - Successful GET, PUT, DELETE
• 201 Created - Successful POST
• 400 Bad Request - Invalid request body or parameters
• 401 Unauthorized - Missing or invalid auth token
• 403 Forbidden - Insufficient permissions
• 404 Not Found - Transaction not found
• 422 Unprocessable Entity - Business logic validation errors
• 500 Internal Server Error - Unexpected server error

Standardized error response format:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Amount must be greater than zero",
    "details": [
      {
        "field": "amount",
        "message": "Amount must be greater than zero"
      }
    ]
  }
}

API Versioning Strategy

Versioned using path prefix (/v1/). This approach provides clear separation between different API versions and allows for independent evolution of each version. Versioning is done at the service level rather than individual endpoints to maintain consistency.

Pagination Design

Cursor-based pagination is selected over offset-based pagination for better performance and consistency at scale. Cursor-based pagination avoids issues with data changes between page requests and provides more predictable performance characteristics.

Authentication and Authorization

Authentication will use JWT tokens with the following scopes:

• transactions:read - Read access to transaction data
• transactions:write - Create and update transactions
• transactions:revert - Revert transaction to previous state

Authorization will be implemented at the service layer using RBAC principles with role-based access control.

Rate Limiting

Rate limiting will be implemented per endpoint:

• Create transaction: 100 requests/minute
• Get transaction details: 500 requests/minute
• Update transaction status: 200 requests/minute
• Revert transaction: 50 requests/minute

Rate limiting will be enforced using a token bucket algorithm with distributed implementation using Redis to maintain consistency across service instances.

Idempotency Design

Idempotency keys will be supported for mutation endpoints:

• POST /v1/transactions - Idempotency key ensures duplicate creation requests are handled correctly
• PUT /v1/transactions/{id}/status - Idempotency key ensures duplicate status updates are handled correctly

Idempotency keys will be validated using a distributed cache (Redis) to ensure uniqueness across concurrent requests.

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    TRANSACTION {
        string id PK
        string status
        string initiator_user_id
        string transaction_type
        timestamp created_at
        timestamp updated_at
        timestamp committed_at
        timestamp rolled_back_at
        jsonb metadata
        string error_message
    }

    TRANSACTION_PARTICIPANT {
        string transaction_id FK
        string participant_id PK
        string participant_type
        string status
        timestamp created_at
        timestamp updated_at
        jsonb data
    }

    TRANSACTION_EVENT {
        string id PK
        string transaction_id FK
        string event_type
        string status
        timestamp created_at
        jsonb payload
        string error_message
    }

    TRANSACTION_LOCK {
        string lock_id PK
        string transaction_id FK
        string resource_type
        string resource_id
        timestamp acquired_at
        timestamp released_at
        string lock_mode
    }

    TRANSACTION {
        "string id" ||--o{ TRANSACTION_PARTICIPANT : "has"
        "string id" ||--o{ TRANSACTION_EVENT : "has"
        "string id" ||--o{ TRANSACTION_LOCK : "owns"
    }

Table Schema Definitions

transactions table

CREATE TABLE transactions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    status VARCHAR(50) NOT NULL DEFAULT 'INITIATED',
    initiator_user_id UUID NOT NULL,
    transaction_type VARCHAR(100) NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    committed_at TIMESTAMP WITH TIME ZONE,
    rolled_back_at TIMESTAMP WITH TIME ZONE,
    metadata JSONB DEFAULT '{}',
    error_message TEXT,
    
    CONSTRAINT chk_status CHECK (status IN ('INITIATED', 'COMMITTED', 'ROLLED_BACK', 'FAILED')),
    CONSTRAINT chk_transaction_type CHECK (transaction_type IS NOT NULL AND transaction_type != '')
);

CREATE INDEX idx_transactions_initiator_user_id ON transactions (initiator_user_id);
CREATE INDEX idx_transactions_created_at ON transactions (created_at);
CREATE INDEX idx_transactions_status ON transactions (status);
CREATE INDEX idx_transactions_type ON transactions (transaction_type);

transaction_participants table

CREATE TABLE transaction_participants (
    transaction_id UUID NOT NULL REFERENCES transactions(id) ON DELETE CASCADE,
    participant_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    participant_type VARCHAR(100) NOT NULL,
    status VARCHAR(50) NOT NULL DEFAULT 'PENDING',
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    data JSONB DEFAULT '{}',
    
    CONSTRAINT chk_status CHECK (status IN ('PENDING', 'COMMITTED', 'ROLLED_BACK', 'FAILED'))
);

CREATE INDEX idx_transaction_participants_transaction_id ON transaction_participants (transaction_id);
CREATE INDEX idx_transaction_participants_type ON transaction_participants (participant_type);

transaction_events table

CREATE TABLE transaction_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    transaction_id UUID NOT NULL REFERENCES transactions(id) ON DELETE CASCADE,
    event_type VARCHAR(100) NOT NULL,
    status VARCHAR(50) NOT NULL DEFAULT 'PENDING',
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    payload JSONB DEFAULT '{}',
    error_message TEXT,
    
    CONSTRAINT chk_status CHECK (status IN ('PENDING', 'PROCESSED', 'FAILED'))
);

CREATE INDEX idx_transaction_events_transaction_id ON transaction_events (transaction_id);
CREATE INDEX idx_transaction_events_event_type ON transaction_events (event_type);

transaction_locks table

CREATE TABLE transaction_locks (
    lock_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    transaction_id UUID NOT NULL REFERENCES transactions(id) ON DELETE CASCADE,
    resource_type VARCHAR(100) NOT NULL,
    resource_id UUID NOT NULL,
    acquired_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    released_at TIMESTAMP WITH TIME ZONE,
    lock_mode VARCHAR(20) NOT NULL DEFAULT 'EXCLUSIVE',
    
    CONSTRAINT chk_lock_mode CHECK (lock_mode IN ('SHARED', 'EXCLUSIVE'))
);

CREATE INDEX idx_transaction_locks_transaction_id ON transaction_locks (transaction_id);
CREATE INDEX idx_transaction_locks_resource ON transaction_locks (resource_type, resource_id);
CREATE INDEX idx_transaction_locks_acquired_at ON transaction_locks (acquired_at);

Primary Key Strategy

UUID is selected as the primary key strategy for all tables because:

1. Distributed system compatibility: UUIDs are globally unique and don't require coordination between services
2. Scalability: No need for centralized ID generation, avoiding bottlenecks in high-throughput systems
3. No sequential patterns: Prevents potential security concerns and makes it harder to predict IDs
4. Cross-service consistency: Enables seamless integration with microservices that may generate their own IDs

Index Design

Index Name	Columns	Type	Purpose
idx_transactions_initiator_user_id	initiator_user_id	Standard	Fast lookup by user
idx_transactions_created_at	created_at	Standard	Time-based queries and cleanup
idx_transactions_status	status	Standard	Filter by transaction state
idx_transactions_type	transaction_type	Standard	Query by type
idx_transaction_participants_transaction_id	transaction_id	Standard	Fast participant lookups
idx_transaction_participants_type	participant_type	Standard	Filter by participant type
idx_transaction_events_transaction_id	transaction_id	Standard	Fast event lookups
idx_transaction_events_event_type	event_type	Standard	Filter by event type
idx_transaction_locks_transaction_id	transaction_id	Standard	Fast lock lookups
idx_transaction_locks_resource	resource_type, resource_id	Composite	Fast resource lock lookups
idx_transaction_locks_acquired_at	acquired_at	Standard	Cleanup of stale locks

Data Partitioning / Sharding Strategy

Shard by transaction type and creation date:

1. Partitioning key: transaction_type + created_at
2. Shard strategy:
   • Transaction type (e.g., payment, transfer) as the first level
   • Monthly date range for the second level (e.g., 2024-01, 2024-02)
3. Implementation:
   • Use PostgreSQL partitioning with range partitions by created_at
   • Each partition can be stored on different storage tiers (SSD vs. HDD)
   • Maintain a global view through materialized views or application-level sharding

Soft Delete vs. Hard Delete Decision

Soft delete is used because:

1. Audit requirements: Need to maintain historical transaction state for compliance and analytics
2. Recovery capability: Ability to restore transactions in case of accidental deletion
3. Referential integrity: Prevents breaking foreign key relationships in reporting systems
4. Data lineage tracking: Maintains complete audit trail for debugging and compliance

Audit Trail Design

Fields to capture in audit logs:

• transaction_id: Unique identifier for the transaction
• user_id: Initiator of the transaction
• operation_type: CREATE, UPDATE, DELETE, COMMIT, ROLLBACK
• timestamp: When operation occurred
• old_values: Previous state (if applicable)
• new_values: New state (if applicable)
• status: Transaction status before/after operation
• ip_address: Originating IP (if available)
• user_agent: Client information

Data Archival Strategy and Retention Policy

Archival strategy:

1. Hot storage: Latest 30 days of data in primary database (SSD)
2. Warm storage: 30-365 days in secondary database (HDD)
3. Cold storage: > 365 days archived to S3 with metadata stored in database

Retention policy:

• Active transactions: 30 days (soft delete)
• Committed transactions: 7 years (compliance requirement)
• Failed/rolled back transactions: 30 days (for debugging and analytics)
• Archived data: Per compliance requirements (5-7 years minimum)

public class Transaction {
    private String id;
    private String status;
    private String initiatorUserId;
    private String transactionType;
    private LocalDateTime createdAt;
    private LocalDateTime updatedAt;
    private LocalDateTime committedAt;
    private LocalDateTime rolledBackAt;
    private Map<String, Object> metadata;
    private String errorMessage;

    // Getters and setters
}

4. Class Design & Service Layer

classDiagram
    class TransactionManager {
        +createTransaction(TransactionRequest) TransactionResponse
        +commitTransaction(String) boolean
        +rollbackTransaction(String) boolean
        +getTransactionStatus(String) TransactionStatus
        +lockResource(String, String) boolean
        +unlockResource(String, String) boolean
    }

    class TransactionService {
        -transactionRepository TransactionRepository
        -eventPublisher EventPublisher
        -lockManager LockManager
        -transactionValidator TransactionValidator
        +createTransaction(TransactionRequest) TransactionResponse
        +commitTransaction(String) boolean
        +rollbackTransaction(String) boolean
        +getTransactionStatus(String) TransactionStatus
    }

    class Transaction {
        -id String
        -status TransactionStatus
        -createdAt Instant
        -updatedAt Instant
        -resources List~ResourceLock~
        -version int
        +getId() String
        +getStatus() TransactionStatus
        +setStatus(TransactionStatus) void
        +getResources() List~ResourceLock~
        +addResource(ResourceLock) void
    }

    class ResourceLock {
        -resourceId String
        -lockType LockType
        -lockedBy String
        -acquiredAt Instant
    }

    class TransactionRequest {
        -id String
        -resources List~ResourceLock~
        -timeoutSeconds int
        -metadata Map~String, String~
    }

    class TransactionResponse {
        -transactionId String
        -status TransactionStatus
        -message String
    }

    class LockManager {
        -lockStore Map~String, ResourceLock~
        +acquireLock(String, String) boolean
        +releaseLock(String, String) boolean
        +isLocked(String) boolean
    }

    class TransactionValidator {
        +validateRequest(TransactionRequest) ValidationResult
        +validateResourceLocks(List~ResourceLock~) ValidationResult
    }

    class EventPublisher {
        +publishTransactionEvent(TransactionEvent) void
    }

    class TransactionEvent {
        -type EventType
        -transactionId String
        -status TransactionStatus
        -timestamp Instant
    }

    TransactionManager --> TransactionService : uses
    TransactionService --> TransactionRepository : persists
    TransactionService --> LockManager : manages locks
    TransactionService --> EventPublisher : publishes events
    TransactionService --> TransactionValidator : validates
    Transaction --> TransactionRepository : persisted as
    ResourceLock --> Transaction : owns
    LockManager --> ResourceLock : manages
    EventPublisher --> TransactionEvent : publishes

stateDiagram-v2
    [*] --> Created
    Created --> Prepared : createTransaction()
    Prepared --> Committed : commitTransaction()
    Prepared --> RolledBack : rollbackTransaction()
    Committed --> [*]
    RolledBack --> [*]

Service Class Structure and Responsibilities

The Transaction Manager system is composed of several key services:

1. TransactionService - Core business logic for transaction lifecycle management
2. LockManager - Resource locking and concurrency control
3. TransactionValidator - Input validation and business rule enforcement
4. EventPublisher - Asynchronous event publishing for transaction state changes

Interface / Abstract Class Design

public interface TransactionRepository {
    Transaction save(Transaction transaction);
    Optional<Transaction> findById(String id);
    void delete(String id);
    List<Transaction> findByStatus(TransactionStatus status);
}

public interface LockManager {
    boolean acquireLock(String resourceId, String transactionId);
    boolean releaseLock(String resourceId, String transactionId);
    boolean isLocked(String resourceId);
}

public interface EventPublisher {
    void publishTransactionEvent(TransactionEvent event);
}

public enum TransactionStatus {
    CREATED, PREPARED, COMMITTED, ROLLED_BACK
}

public enum LockType {
    READ, WRITE
}

Business Logic Encapsulation Patterns

Strategy Pattern for Transaction Validation:

public interface TransactionValidationStrategy {
    ValidationResult validate(TransactionRequest request);
}

public class ResourceLockValidationStrategy implements TransactionValidationStrategy {
    private final LockManager lockManager;
    
    public ResourceLockValidationStrategy(LockManager lockManager) {
        this.lockManager = lockManager;
    }
    
    @Override
    public ValidationResult validate(TransactionRequest request) {
        // Validate that resources can be locked
        for (ResourceLock resource : request.getResources()) {
            if (lockManager.isLocked(resource.getResourceId())) {
                return ValidationResult.failure("Resource " + resource.getResourceId() + " is already locked");
            }
        }
        return ValidationResult.success();
    }
}

Critical-Path Method Implementation

public TransactionResponse createTransaction(TransactionRequest request) {
    // Begin transaction boundary - all operations within this method are atomic
    Transaction transaction = new Transaction();
    transaction.setId(UUID.randomUUID().toString());
    transaction.setStatus(TransactionStatus.CREATED);
    transaction.setCreatedAt(Instant.now());
    
    // Validate request
    ValidationResult validation = transactionValidator.validateRequest(request);
    if (!validation.isSuccess()) {
        throw new TransactionValidationException(validation.getMessage());
    }
    
    // Acquire locks for all resources
    List<ResourceLock> resourceLocks = new ArrayList<>();
    for (ResourceLock resource : request.getResources()) {
        if (!lockManager.acquireLock(resource.getResourceId(), transaction.getId())) {
            // Rollback any acquired locks
            rollbackAcquiredLocks(resourceLocks, transaction.getId());
            throw new ResourceLockedException("Failed to acquire lock for resource: " + resource.getResourceId());
        }
        resource.setLockedBy(transaction.getId());
        resource.setAcquiredAt(Instant.now());
        resourceLocks.add(resource);
    }
    
    // Update transaction with locks
    transaction.setResources(resourceLocks);
    transaction.setStatus(TransactionStatus.PREPARED);
    transaction.setUpdatedAt(Instant.now());
    
    // Save transaction to database
    Transaction savedTransaction = transactionRepository.save(transaction);
    
    // Publish event for monitoring and audit
    TransactionEvent event = new TransactionEvent(
        EventType.TRANSACTION_CREATED,
        savedTransaction.getId(),
        savedTransaction.getStatus(),
        Instant.now()
    );
    eventPublisher.publishTransactionEvent(event);
    
    return new TransactionResponse(savedTransaction.getId(), TransactionStatus.PREPARED, "Transaction created successfully");
}

Transaction Boundary Design

Each transaction operation is bounded by:

1. Database transaction scope (JDBC/ORM transaction)
2. Lock acquisition/releasing scope
3. Event publishing scope

The critical path ensures that all steps either complete successfully or are rolled back atomically.

Service Interface Contracts

public interface TransactionManager {
    TransactionResponse createTransaction(TransactionRequest request);
    boolean commitTransaction(String transactionId);
    boolean rollbackTransaction(String transactionId);
    TransactionStatus getTransactionStatus(String transactionId);
    boolean lockResource(String resourceId, String transactionId);
    boolean unlockResource(String resourceId, String transactionId);
}

Saga / Orchestration Pattern

For distributed transactions, we implement a saga pattern:

public class DistributedTransactionSaga {
    private final List<SagaStep> steps;
    private final String transactionId;
    
    public void execute() {
        List<SagaStep> executedSteps = new ArrayList<>();
        
        try {
            for (SagaStep step : steps) {
                step.execute();
                executedSteps.add(step);
            }
            
            // Commit all steps
            for (SagaStep step : executedSteps) {
                step.commit();
            }
        } catch (Exception e) {
            // Rollback all executed steps
            for (int i = executedSteps.size() - 1; i >= 0; i--) {
                executedSteps.get(i).rollback();
            }
            throw new TransactionFailedException("Saga failed, rolled back", e);
        }
    }
}

Domain Object / Value Object Design

public class Transaction {
    private String id;
    private TransactionStatus status;
    private Instant createdAt;
    private Instant updatedAt;
    private List<ResourceLock> resources;
    private int version;
    
    // Getters and setters with proper validation
}

public class ResourceLock {
    private String resourceId;
    private LockType lockType;
    private String lockedBy;
    private Instant acquiredAt;
    
    // Constructor, getters, and setters
}

public class TransactionRequest {
    private String id;
    private List<ResourceLock> resources;
    private int timeoutSeconds;
    private Map<String, String> metadata;
    
    // Constructor, getters, and setters
}

5. Concurrency & Thread Safety

Concurrency Model

The Transaction Manager will operate in a multi-threaded environment with the following concurrency model:

• Thread-per-request pattern for handling concurrent transaction requests
• Async/Await pattern for non-blocking I/O operations (e.g., database operations, external service calls)
• Thread-safe shared state using atomic operations and synchronized blocks where necessary
• Immutable domain objects to prevent race conditions in transaction state changes

Race Condition Prevention Strategy

Race conditions will be mitigated through:

1. Fine-grained locking on transaction-specific resources (e.g., transaction ID, participant locks)
2. Optimistic concurrency control using version numbers for transaction state updates
3. Immutable object patterns for transaction metadata to prevent concurrent modification
4. Thread-local storage for per-request transaction context to avoid cross-contamination

Optimistic vs. Pessimistic Locking Selection

Optimistic locking selected with the following rationale:

• High read-to-write ratio: Transaction manager primarily handles reads for status checks and transaction state queries
• Low contention: Transactions are typically short-lived with minimal overlap in resource access
• Better scalability: Avoids blocking threads and reduces lock contention in high-concurrency scenarios
• Improved throughput: Allows concurrent reads while ensuring consistency through version validation

Implementation approach:

// Transaction entity with optimistic locking
@Entity
@Table(name = "transactions")
public class Transaction {
    @Id
    private UUID id;
    
    @Version
    private Long version; // Used for optimistic locking
    
    private TransactionStatus status;
    private String participantId;
    private LocalDateTime createdAt;
    private LocalDateTime updatedAt;
}

Deadlock Prevention Strategy

Deadlock prevention mechanisms:

1. Lock ordering protocol: All transactions acquire locks in a consistent order (e.g., by transaction ID)
2. Timeout-based lock acquisition: 5-second timeout for lock acquisition with retry logic
3. Deadlock detection algorithm: Periodic check for circular wait conditions in transaction graph
4. Transaction timeout enforcement: Maximum 30-second transaction lifetime to prevent indefinite lock holding

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is implemented through:

1. Idempotency key strategy: Unique identifiers provided by clients for duplicate request detection
2. Database-level uniqueness constraints on idempotency keys with transaction IDs
3. In-memory cache of recent idempotency keys with TTL (5 minutes) to handle concurrent requests
4. Idempotent operation state tracking to return same result for repeated calls

// Idempotency key handling in transaction manager
public class TransactionIdempotencyService {
    private final Map<String, TransactionResult> recentResults = new ConcurrentHashMap<>();
    private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
    
    public TransactionResult executeWithIdempotency(
            String idempotencyKey, 
            Supplier<TransactionResult> operation) {
        
        // Check if this idempotency key was recently processed
        TransactionResult cached = recentResults.get(idempotencyKey);
        if (cached != null) {
            return cached;
        }
        
        // Execute operation and cache result
        TransactionResult result = operation.get();
        recentResults.put(idempotencyKey, result);
        
        // Schedule cache cleanup
        scheduler.schedule(() -> recentResults.remove(idempotencyKey), 5, TimeUnit.MINUTES);
        
        return result;
    }
}

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant Client
    participant TransactionManager
    participant TransactionRepository
    participant Database
    participant EventPublisher

    Client->>TransactionManager: initiateTransaction(request)
    TransactionManager->>TransactionRepository: save(transaction)
    TransactionRepository->>Database: persist transaction
    Database-->>TransactionRepository: transactionId
    TransactionRepository-->>TransactionManager: transactionId
    TransactionManager->>EventPublisher: publish transaction.created event
    EventPublisher-->>TransactionManager: event published
    TransactionManager-->>Client: transactionId

sequenceDiagram
    participant Client
    participant TransactionManager
    participant TransactionRepository
    participant Database
    participant EventPublisher

    Client->>TransactionManager: rollbackTransaction(transactionId)
    TransactionManager->>TransactionRepository: findById(transactionId)
    TransactionRepository-->>TransactionManager: transaction
    alt transaction is not committed
        TransactionManager->>Database: rollback transaction
        Database-->>TransactionManager: rollback success
        TransactionManager->>EventPublisher: publish transaction.rolledBack event
        EventPublisher-->>TransactionManager: event published
        TransactionManager-->>Client: success
    else transaction is committed
        TransactionManager-->>Client: error - cannot rollback committed transaction
    end

sequenceDiagram
    participant Client
    participant TransactionManager
    participant TransactionRepository
    participant Database
    participant EventPublisher
    participant ExternalService

    Client->>TransactionManager: processTransaction(request)
    TransactionManager->>TransactionRepository: save(transaction)
    TransactionRepository->>Database: persist transaction
    Database-->>TransactionRepository: transactionId
    TransactionRepository-->>TransactionManager: transactionId
    TransactionManager->>ExternalService: call external service
    ExternalService-->>TransactionManager: response
    TransactionManager->>Database: update transaction status
    Database-->>TransactionManager: update success
    TransactionManager->>EventPublisher: publish transaction.completed event
    EventPublisher-->>TransactionManager: event published
    TransactionManager-->>Client: success

7. Caching Design

Cache Strategy Selection

For a Transaction Manager, we'll use cache-aside pattern with write-through semantics for critical transaction data and write-behind for audit trail data. This provides the best balance of consistency, performance, and operational simplicity.

Cache Key Design

Cache keys will follow a namespace-based pattern:

transaction:{transactionId}
transaction:status:{transactionId}
transaction:metadata:{transactionId}
audit:{transactionId}:{timestamp}

TTL Strategy

Data Type	TTL	Reason
Transaction metadata	5 minutes	Frequent reads, short-lived
Transaction status	1 hour	Long-running transactions may span hours
Audit logs	24 hours	Compliance and debugging requirements
User session cache	30 minutes	Session timeout alignment

Cache Eviction Policy

We'll implement LRU (Least Recently Used) eviction policy with TTL-based soft expiration to handle cache stampede prevention and memory pressure.

Cache Stampede Prevention

We'll implement probabilistic early expiry with 20% early expiration and distributed locks for cache warming operations:

public class CacheManager {
    private final RedisTemplate<String, Object> redisTemplate;
    private final DistributedLockService lockService;
    
    public Object getWithStampedeProtection(String key, Supplier<Object> loader) {
        String cacheKey = "cache:" + key;
        String lockKey = "lock:" + key;
        
        // Try to get from cache first
        Object cachedValue = redisTemplate.opsForValue().get(cacheKey);
        if (cachedValue != null) {
            return cachedValue;
        }
        
        // Acquire distributed lock for cache warming
        if (lockService.acquireLock(lockKey, 5000)) {
            try {
                // Double-check after lock acquisition
                cachedValue = redisTemplate.opsForValue().get(cacheKey);
                if (cachedValue != null) {
                    return cachedValue;
                }
                
                // Load fresh data
                Object newValue = loader.get();
                
                // Set with early expiration (80% of TTL)
                redisTemplate.opsForValue().set(
                    cacheKey, 
                    newValue, 
                    Duration.ofMinutes(5).multipliedBy(4).dividedBy(5)
                );
                
                return newValue;
            } finally {
                lockService.releaseLock(lockKey);
            }
        }
        
        // Fallback to direct load if lock acquisition fails
        return loader.get();
    }
}

Distributed Cache Architecture

We'll use Redis Cluster with:

• 3 master nodes for high availability
• 3 replica nodes per master for read scaling
• Redis Sentinel for automatic failover
• Cache sharding with consistent hashing

Cache Invalidation Strategy

We'll implement event-driven invalidation using a publish-subscribe model:

@Component
public class TransactionCacheInvalidator {
    private final RedisTemplate<String, Object> redisTemplate;
    private final ApplicationEventPublisher eventPublisher;
    
    @EventListener
    public void handleTransactionCommitted(TransactionCommittedEvent event) {
        String transactionId = event.getTransactionId();
        
        // Invalidate all related cache entries
        Set<String> keysToInvalidate = new HashSet<>();
        keysToInvalidate.add("transaction:" + transactionId);
        keysToInvalidate.add("transaction:status:" + transactionId);
        
        // Use Redis Pipeline for batch invalidation
        redisTemplate.delete(keysToInvalidate);
    }
    
    @EventListener
    public void handleTransactionRolledBack(TransactionRolledBackEvent event) {
        String transactionId = event.getTransactionId();
        
        // Invalidate status cache and metadata
        redisTemplate.delete(Arrays.asList(
            "transaction:" + transactionId,
            "transaction:status:" + transactionId,
            "transaction:metadata:" + transactionId
        ));
    }
}

Cache Read-Through Strategy

For high-frequency transaction status queries, we'll implement read-through with automatic cache population:

@Service
public class TransactionStatusService {
    private final CacheManager cacheManager;
    private final TransactionRepository transactionRepository;
    
    public TransactionStatus getTransactionStatus(String transactionId) {
        String cacheKey = "transaction:status:" + transactionId;
        
        return (TransactionStatus) cacheManager.getWithStampedeProtection(
            cacheKey,
            () -> {
                Transaction transaction = transactionRepository.findById(transactionId);
                return transaction != null ? transaction.getStatus() : null;
            }
        );
    }
}

Cache Warm-up Strategy

We'll implement background cache warming for frequently accessed transaction IDs:

@Component
public class TransactionCacheWarmer {
    private final ScheduledExecutorService scheduler = 
        Executors.newScheduledThreadPool(2);
    
    @PostConstruct
    public void scheduleCacheWarmup() {
        scheduler.scheduleAtFixedRate(
            this::warmUpFrequentlyAccessedTransactions,
            0, 30, TimeUnit.SECONDS
        );
    }
    
    private void warmUpFrequentlyAccessedTransactions() {
        // Get recent transactions from database
        List<String> recentTransactionIds = transactionRepository
            .findRecentlyAccessedTransactionIds(1000);
            
        // Pre-populate cache with metadata
        recentTransactionIds.forEach(id -> {
            String key = "transaction:metadata:" + id;
            cacheManager.put(key, fetchTransactionMetadata(id));
        });
    }
}

This caching design provides optimal performance for the transaction manager while maintaining consistency and handling concurrent access patterns. The combination of cache-aside with proper invalidation ensures that stale data doesn't persist, and the stampede prevention mechanisms maintain system stability under high load.

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain objects and persistence concerns. Each entity will have its own repository interface and implementation.

public interface TransactionRepository {
    Transaction save(Transaction transaction);
    Optional<Transaction> findById(UUID id);
    List<Transaction> findByStatus(TransactionStatus status);
    List<Transaction> findByUserIdAndStatus(UUID userId, TransactionStatus status);
    void updateStatus(UUID id, TransactionStatus newStatus);
    void delete(UUID id);
}

@Repository
public class JdbcTransactionRepository implements TransactionRepository {
    private final JdbcTemplate jdbcTemplate;
    private final ObjectMapper objectMapper;

    public JdbcTransactionRepository(JdbcTemplate jdbcTemplate, ObjectMapper objectMapper) {
        this.jdbcTemplate = jdbcTemplate;
        this.objectMapper = objectMapper;
    }

    @Override
    public Transaction save(Transaction transaction) {
        String sql = """
            INSERT INTO transactions (
                id, user_id, amount, currency, status, metadata, created_at, updated_at
            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
            """;
        
        jdbcTemplate.update(sql, 
            transaction.getId(),
            transaction.getUserId(),
            transaction.getAmount(),
            transaction.getCurrency(),
            transaction.getStatus().name(),
            objectMapper.writeValueAsString(transaction.getMetadata()),
            transaction.getCreatedAt(),
            transaction.getUpdatedAt()
        );
        
        return transaction;
    }

    @Override
    public Optional<Transaction> findById(UUID id) {
        String sql = "SELECT * FROM transactions WHERE id = ?";
        try {
            return Optional.of(jdbcTemplate.queryForObject(sql, new TransactionRowMapper(), id));
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }

    @Override
    public List<Transaction> findByStatus(TransactionStatus status) {
        String sql = "SELECT * FROM transactions WHERE status = ? ORDER BY created_at DESC";
        return jdbcTemplate.query(sql, new TransactionRowMapper(), status.name());
    }

    @Override
    public List<Transaction> findByUserIdAndStatus(UUID userId, TransactionStatus status) {
        String sql = "SELECT * FROM transactions WHERE user_id = ? AND status = ? ORDER BY created_at DESC";
        return jdbcTemplate.query(sql, new TransactionRowMapper(), userId, status.name());
    }

    @Override
    public void updateStatus(UUID id, TransactionStatus newStatus) {
        String sql = "UPDATE transactions SET status = ?, updated_at = ? WHERE id = ?";
        jdbcTemplate.update(sql, newStatus.name(), Instant.now(), id);
    }

    @Override
    public void delete(UUID id) {
        String sql = "DELETE FROM transactions WHERE id = ?";
        jdbcTemplate.update(sql, id);
    }
}

ORM vs. Raw Query Decisions

For the Transaction Manager, we use raw SQL queries with JdbcTemplate for better performance and control over database operations. This approach is preferred because:

1. Performance: Raw queries are more efficient than ORM-generated queries for simple CRUD operations
2. Control: Direct control over query execution and optimization
3. Debugging: Easier to debug and profile database operations
4. Complexity: The transaction model is relatively simple, so ORM overhead isn't justified

Read vs. Write Separation

We implement read-write separation using a CQRS pattern where:

• Read operations (e.g., findByUserIdAndStatus) go to read replicas or read-only database instances
• Write operations (e.g., save, updateStatus) go to the primary database

@Repository
public class ReadTransactionRepository {
    private final JdbcTemplate readJdbcTemplate;

    public ReadTransactionRepository(JdbcTemplate readJdbcTemplate) {
        this.readJdbcTemplate = readJdbcTemplate;
    }

    public List<Transaction> findByUserIdAndStatus(UUID userId, TransactionStatus status) {
        String sql = "SELECT * FROM transactions WHERE user_id = ? AND status = ? ORDER BY created_at DESC";
        return readJdbcTemplate.query(sql, new TransactionRowMapper(), userId, status.name());
    }
}

Connection Pool Configuration

Connection pool is configured with:

• Max pool size: 20 connections
• Min idle connections: 5
• Connection timeout: 30 seconds
• Validation query: SELECT 1

@Configuration
public class DatabaseConfig {
    @Bean
    public DataSource dataSource() {
        HikariConfig config = new HikariConfig();
        config.setJdbcUrl("jdbc:postgresql://localhost:5432/transaction_db");
        config.setUsername("user");
        config.setPassword("password");
        config.setMaximumPoolSize(20);
        config.setMinimumIdle(5);
        config.setConnectionTimeout(30000);
        config.setValidationTimeout(5000);
        config.setLeakDetectionThreshold(60000);
        return new HikariDataSource(config);
    }
}

N+1 Query Prevention Strategy

To prevent N+1 query problems, we use eager loading and batch operations:

// Batch fetch for transactions by user ID
public List<Transaction> findTransactionsByUserIds(List<UUID> userIds) {
    String sql = "SELECT * FROM transactions WHERE user_id IN (" + 
        userIds.stream().map(id -> "?").collect(Collectors.joining(",")) + 
        ") ORDER BY created_at DESC";
    
    // Use batch parameters to avoid N+1
    return jdbcTemplate.query(sql, userIds.toArray(), new TransactionRowMapper());
}

Batch Operation Design

Batch operations are implemented for bulk transaction processing:

@Repository
public class BatchTransactionRepository {
    private final JdbcTemplate jdbcTemplate;

    public BatchTransactionRepository(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }

    public void batchSave(List<Transaction> transactions) {
        String sql = """
            INSERT INTO transactions (
                id, user_id, amount, currency, status, metadata, created_at, updated_at
            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
            """;
        
        List<Object[]> batchArgs = transactions.stream()
            .map(transaction -> new Object[] {
                transaction.getId(),
                transaction.getUserId(),
                transaction.getAmount(),
                transaction.getCurrency(),
                transaction.getStatus().name(),
                objectMapper.writeValueAsString(transaction.getMetadata()),
                transaction.getCreatedAt(),
                transaction.getUpdatedAt()
            })
            .collect(Collectors.toList());
        
        jdbcTemplate.batchUpdate(sql, batchArgs);
    }
}

Transaction Management

All repository operations are wrapped in proper transaction boundaries:

@Service
@Transactional
public class TransactionService {
    private final TransactionRepository transactionRepository;
    
    public Transaction createTransaction(Transaction transaction) {
        // This will be executed in a database transaction
        return transactionRepository.save(transaction);
    }
}

Data Access Layer Design Summary

The repository layer follows these principles:

1. Single Responsibility: Each repository handles one domain entity
2. Abstraction: Database-specific implementation details are hidden
3. Testability: Easy to mock and test with in-memory databases
4. Performance: Raw SQL queries for optimal performance
5. Scalability: Support for read-write separation and batch operations

The design supports high-throughput transaction processing while maintaining clean separation of concerns between domain logic and data access.

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The Transaction Manager will use a hybrid synchronous/asynchronous approach. The core transaction processing (validation, execution, and state updates) will happen synchronously to ensure immediate consistency. However, the following operations will be handled asynchronously:

1. Notification Events: Sending transaction completion notifications to external systems (e.g., accounting, audit services)
2. Audit Trail Events: Publishing detailed transaction logs for compliance and monitoring
3. Compensation Operations: When a transaction fails, compensating actions (like rolling back partial changes) will be queued for asynchronous execution
4. Reporting Events: Aggregating transaction statistics for dashboards and analytics

The decision to make these asynchronous is based on the principle of loose coupling and improved system resilience. These operations don't block the primary transaction flow, allowing for better performance and scalability under high load.

Message schema design with sample payload

We'll use Avro for message schema definition to support schema evolution and compatibility checking:

{
  "type": "record",
  "name": "TransactionEvent",
  "fields": [
    {
      "name": "id",
      "type": "string"
    },
    {
      "name": "transactionId",
      "type": "string"
    },
    {
      "name": "eventType",
      "type": {
        "type": "enum",
        "name": "TransactionEventType",
        "symbols": ["CREATED", "COMPLETED", "FAILED", "COMPENSATED"]
      }
    },
    {
      "name": "timestamp",
      "type": "long"
    },
    {
      "name": "details",
      "type": {
        "type": "record",
        "name": "TransactionDetails",
        "fields": [
          {
            "name": "amount",
            "type": "double"
          },
          {
            "name": "currency",
            "type": "string"
          },
          {
            "name": "sourceAccount",
            "type": "string"
          },
          {
            "name": "destinationAccount",
            "type": "string"
          }
        ]
      }
    },
    {
      "name": "correlationId",
      "type": "string"
    }
  ]
}

Dead letter queue strategy and poison message handling

For messages that fail to process after retries, we'll implement a dead letter queue (DLQ) strategy:

1. Retry Configuration: Messages will be retried up to 3 times with exponential backoff (1s, 2s, 4s)
2. DLQ Handling: Messages that fail after max retries will be moved to a dedicated DLQ topic
3. Manual Re-processing: Failed messages in DLQ will be flagged with metadata indicating the failure reason and timestamp
4. Monitoring: DLQ messages will trigger alerts for manual investigation

At-least-once vs. exactly-once delivery

We will implement at-least-once delivery for transaction events because:

1. Transaction systems typically prioritize consistency over strict ordering
2. The system can handle duplicate events gracefully through idempotency checks
3. The complexity of exactly-once semantics (idempotent producers, transactional consumers) is not justified for this use case
4. Most downstream services can handle duplicate events without data corruption

Retry policy for consumers

Retry policy configuration:

public class TransactionEventRetryPolicy {
    private static final int MAX_RETRIES = 3;
    private static final long INITIAL_DELAY_MS = 1000;
    private static final double BACKOFF_MULTIPLIER = 2.0;
    private static final long MAX_DELAY_MS = 30000; // 30 seconds
    
    public static long calculateDelay(int attempt) {
        if (attempt <= 0) return 0;
        
        long delay = (long) (INITIAL_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempt - 1));
        return Math.min(delay, MAX_DELAY_MS);
    }
    
    public static boolean shouldRetry(int attempt, Exception error) {
        // Don't retry on certain fatal errors
        if (error instanceof IllegalArgumentException || 
            error instanceof IllegalStateException) {
            return false;
        }
        
        return attempt < MAX_RETRIES;
    }
}

Outbox pattern for reliable event publishing

We will implement the outbox pattern to ensure that events are published reliably:

1. Event Storage: Transaction events are first stored in an event_outbox table within the same transaction as the business operation
2. Event Publishing: A separate background process (or scheduled job) polls this outbox table and publishes events to the message broker
3. Idempotency: Events are marked as published once successfully delivered to prevent duplicates
4. Transaction Boundaries: The outbox pattern ensures that event publishing is part of the same transaction as the business operation, preventing partial writes

Sequence diagram for async flows

sequenceDiagram
    participant TM as TransactionManager
    participant DB as Database
    participant EO as EventOutbox
    participant MQ as MessageQueue
    participant Consumer as EventConsumer
    
    TM->>DB: Begin transaction
    TM->>DB: Create transaction record
    TM->>DB: Update account balances
    TM->>EO: Insert event to outbox table
    TM->>DB: Commit transaction
    
    loop Polling for events
        EO->>MQ: Publish event to queue
        MQ->>Consumer: Deliver event
        Consumer->>DB: Process event (e.g., send notification)
    end

10. Error Handling Design

Error Classification Hierarchy

Errors are categorized into three main types to guide handling and propagation:

1. Domain Errors - Business logic violations (e.g., insufficient funds, invalid state transitions)
2. Infrastructure Errors - System-level issues (e.g., database connection failures, network timeouts)
3. Validation Errors - Input data issues (e.g., missing fields, malformed data)

Each error type has specific handling strategies and should be mapped to appropriate HTTP status codes.

Error Propagation Strategy

Errors follow a "bubble up" pattern with specific handling at each layer:

• Repository Layer: Wrap infrastructure errors in custom exceptions (e.g., DatabaseException)
• Service Layer: Catch and transform domain errors into business-specific exceptions (e.g., InsufficientFundsException)
• Controller Layer: Map service exceptions to appropriate HTTP responses with standardized error format

Retry Strategy with Exponential Backoff and Jitter

Retry logic is implemented using exponential backoff with jitter to prevent thundering herd:

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.5;

    public <T> T executeWithRetry(Supplier<T> operation) throws Exception {
        int attempts = 0;
        while (attempts < MAX_RETRIES) {
            try {
                return operation.get();
            } catch (Exception e) {
                if (attempts == MAX_RETRIES - 1 || !isRetryable(e)) {
                    throw e;
                }
                
                long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempts) 
                    * (1 + JITTER_FACTOR * (Math.random() - 0.5)));
                Thread.sleep(delay);
                attempts++;
            }
        }
        throw new RuntimeException("Max retries exceeded");
    }

    private boolean isRetryable(Exception e) {
        return e instanceof TimeoutException || 
               e instanceof ConnectionException ||
               e instanceof RetryableServiceException;
    }
}

Circuit Breaker Design

Circuit breaker pattern is implemented using the Hystrix-style approach:

public class CircuitBreaker {
    private final int failureThreshold = 5;
    private final long timeoutMs = 30000;
    private final long resetTimeoutMs = 60000;
    
    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private long lastFailureTime = 0;

    public <T> T execute(Supplier<T> operation) throws Exception {
        switch (state) {
            case CLOSED:
                return executeWithCircuitClosed(operation);
            case OPEN:
                if (System.currentTimeMillis() - lastFailureTime > resetTimeoutMs) {
                    state = CircuitState.HALF_OPEN;
                    return executeWithCircuitHalfOpen(operation);
                } else {
                    throw new CircuitBreakerOpenException("Circuit is open");
                }
            case HALF_OPEN:
                return executeWithCircuitHalfOpen(operation);
        }
        throw new IllegalStateException("Invalid circuit state");
    }

    private enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }
}

Fallback Behavior Design

Fallbacks are implemented for critical dependencies:

public class TransactionService {
    private final PaymentGateway paymentGateway;
    private final FallbackHandler fallbackHandler;

    public TransactionResult processTransaction(TransactionRequest request) {
        try {
            return paymentGateway.process(request);
        } catch (PaymentGatewayException e) {
            // Fallback to manual processing or alternative payment method
            return fallbackHandler.handlePaymentFailure(request, e);
        }
    }
}

Error Logging Format and Alerting Thresholds

Structured logging includes trace context and error metadata:

{
  "timestamp": "2023-10-15T14:30:45.123Z",
  "trace_id": "abc123def456",
  "span_id": "xyz789uvw012",
  "user_id": "user-12345",
  "error_code": "INSUFFICIENT_FUNDS",
  "error_message": "Insufficient funds in account for transaction",
  "service": "transaction-manager",
  "method": "processTransaction",
  "request_id": "req-56789"
}

Alerting thresholds:

Error Type	Alert Threshold	Severity
Infrastructure errors	> 10 occurrences in 5 minutes	Critical
Domain validation errors	> 100 occurrences in 1 hour	Warning
Business logic errors	> 50 occurrences in 1 hour	Info

Error Response Format

Standardized JSON error response:

{
  "error": {
    "code": "INSUFFICIENT_FUNDS",
    "message": "Insufficient funds in account for transaction",
    "details": {
      "accountId": "acc-12345",
      "requestedAmount": 100.00,
      "availableBalance": 50.00
    },
    "timestamp": "2023-10-15T14:30:45.123Z",
    "requestId": "req-56789"
  }
}

Error Handling in Transaction Context

Transaction-specific error handling includes:

1. Rollback Strategy: All transactional operations must be rolled back on any failure
2. Compensation Logic: For distributed transactions, implement compensation operations
3. Idempotency Handling: Retryable errors must not cause duplicate side effects

public class TransactionErrorHandling {
    public void handleTransactionFailure(Transaction transaction, Exception error) {
        // Log the failure with full context
        logger.error("Transaction failed", 
            "transactionId", transaction.getId(),
            "userId", transaction.getUserId(),
            "error", error.getMessage());

        // Attempt rollback if transaction is in progress
        if (transaction.isInProgress()) {
            transaction.rollback();
        }

        // Send alert for critical errors
        if (isCriticalError(error)) {
            alertService.sendAlert("Transaction failure", 
                "Transaction " + transaction.getId() + " failed with critical error");
        }
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using a combination of:

• Bean Validation (JSR-303) for structured validation rules
• Custom validators for business logic constraints (e.g., transaction amount ranges)
• OWASP Java HTML Sanitizer for any text fields that might contain user-generated content
• Regular expression validation for format constraints (e.g., email, phone number patterns)

public class TransactionRequest {
    @NotNull(message = "Amount is required")
    @DecimalMin(value = "0.01", message = "Amount must be greater than zero")
    private BigDecimal amount;
    
    @NotBlank(message = "Currency is required")
    @Pattern(regexp = "^[A-Z]{3}$", message = "Currency must be 3-letter ISO code")
    private String currency;
    
    @ValidTransactionReference
    private String reference;
}

SQL Injection Prevention

• Prepared statements exclusively used for all database queries
• No dynamic query building from user input
• Parameterized queries with strict type binding
• Database schema validation to prevent schema-injection attacks

Authentication Middleware Design

• JWT-based authentication with RS256 signing algorithm
• OAuth2 Authorization Code flow for external identity providers (Google, GitHub)
• Authentication filter intercepting all requests to validate JWT tokens
• Token refresh mechanism with sliding window expiration (30 min access, 7 day refresh)

@Component
public class JwtAuthenticationFilter extends OncePerRequestFilter {
    @Override
    protected void doFilterInternal(HttpServletRequest request, 
                                   HttpServletResponse response,
                                   FilterChain filterChain) throws ServletException, IOException {
        String token = extractToken(request);
        if (token != null && jwtService.validateToken(token)) {
            Authentication auth = jwtService.getAuthentication(token);
            SecurityContextHolder.getContext().setAuthentication(auth);
        }
        filterChain.doFilter(request, response);
    }
}

Authorization Enforcement

• Role-Based Access Control (RBAC) with hierarchical permissions
• Fine-grained authorization using Spring Security annotations (@PreAuthorize)
• Resource-level permissions checking user ownership for transaction operations
• Permission caching to reduce database lookups during authorization checks

@PreAuthorize("hasRole('ADMIN') or @transactionAuthorizationService.canAccessTransaction(#transactionId)")
public Transaction getTransaction(String transactionId) {
    return transactionRepository.findById(transactionId);
}

Sensitive Data Encryption

• At-rest encryption: AES-256-GCM using AWS KMS managed keys for transaction data
• In-transit encryption: TLS 1.3 with HSTS header enforcement
• PII masking: Credit card numbers masked to XXXX-XXXX-XXXX-1234 format in logs
• Key rotation: 90-day automatic key rotation policy

PII Handling and Masking Strategy

• Credit card numbers: Never stored in plaintext; only last 4 digits retained for audit
• User names and addresses: Stored encrypted with customer-specific keys
• Log masking: All PII fields in logs are masked using configurable patterns
• Audit trail: Only minimal PII retained for compliance (user ID, transaction type)

Secrets Management

• AWS Secrets Manager for storing database credentials, API keys, and encryption keys
• Secret rotation: 30-day automatic rotation for all secrets with proper key distribution
• Least privilege access: IAM roles with minimal required permissions to access secrets
• Environment-specific secret paths: /prod/transaction-manager/ vs /staging/transaction-manager/

Audit Logging Design

• Comprehensive audit events: Transaction creation, modification, cancellation, and approval
• Structured log format including trace_id, user_id, transaction_id, operation_type, timestamp
• PII masking in logs: All sensitive fields are masked before logging
• Retention policy: 3 years for compliance, 90 days for operational logs

{
  "timestamp": "2023-10-15T14:30:45.123Z",
  "trace_id": "abc123def456",
  "user_id": "user_12345",
  "transaction_id": "txn_abc123",
  "operation_type": "CREATE",
  "amount": "100.00",
  "currency": "USD",
  "status": "PENDING"
}

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

• Service Layer: 85% code coverage, focusing on business logic and edge cases
• Repository Layer: 90% coverage with all CRUD operations tested
• Domain Objects: 100% coverage for value objects and immutable entities
• Exception Handling: 100% coverage for all error paths and validation failures
• Integration Points: 95% coverage for external service mocks and database interactions

Integration Testing Strategy

• Test Containers: Use Docker-based test containers for PostgreSQL, Redis, and Kafka to ensure real infrastructure behavior
• Embedded Databases: H2 for fast unit tests with schema compatibility validation
• WireMock: For external service stubbing (payment gateways, notification services)
• Database Transaction Isolation: Integration tests run within transactional boundaries to ensure clean state isolation

Contract Testing

• Pact Framework: For consumer-driven contract testing between Transaction Manager and downstream services (Payment Service, Notification Service)
• Versioned Contracts: Maintain contract versions with semantic versioning to support backward compatibility
• Contract Validation: Run contract tests as part of CI pipeline for every service change

Performance Testing Tool and Key Scenarios

• k6: Use k6 for load testing with realistic user scenarios
• Key Scenarios:
  • Concurrent transaction creation (1000 concurrent requests)
  • Transaction status update under high load
  • Batch transaction processing (100 transactions in batch)
  • Transaction rollback under stress conditions
• Metrics Tracked: p50/p95/p99 response times, throughput, error rates

Test Data Management Strategy

• Fixtures: Use JSON/YAML fixtures for static test data (user profiles, transaction types)
• Factories: Implement Factory classes in Java for generating test data with realistic constraints
• Seeding: Database seed scripts for initial state setup, including sample transactions and user data
• Test Data Isolation: Each test runs in isolated database schema or transaction to prevent cross-contamination

// Example of factory pattern for test data generation
public class TransactionFactory {
    public static Transaction createTransaction(String status, String type) {
        return Transaction.builder()
                .id(UUID.randomUUID())
                .userId("user-123")
                .amount(BigDecimal.valueOf(100.00))
                .currency("USD")
                .status(status)
                .type(type)
                .createdAt(Instant.now())
                .build();
    }
}

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective search, filtering, and analysis:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO|WARN|ERROR",
  "service": "transaction-manager",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user-123",
  "transaction_id": "txn-456",
  "operation": "create_transaction",
  "message": "Transaction created successfully",
  "duration_ms": 150,
  "error_code": "TXN_001",
  "error_stack": "com.example.TransactionException: Invalid transaction data",
  "request_id": "req-789"
}

PII masking rules:

• User IDs and transaction IDs are logged as hashed values (SHA-256 with salt)
• Sensitive fields like account numbers are masked using **** pattern
• All sensitive fields in request/response payloads are redacted from logs

Distributed Tracing Instrumentation

OpenTelemetry is used for distributed tracing with custom spans:

Key Spans to Instrument:

1. transaction_manager.handle_create_transaction - Root span for transaction creation
2. transaction_manager.validate_transaction - Validation logic
3. transaction_manager.persist_transaction - Database persistence
4. transaction_manager.publish_transaction_event - Event publishing

Attributes to Capture:

Span span = tracer.spanBuilder("transaction_manager.handle_create_transaction")
    .setAttribute("transaction_id", transactionId)
    .setAttribute("user_id", userId)
    .setAttribute("amount", amount)
    .setAttribute("currency", currency)
    .setAttribute("source_account_id", sourceAccountId)
    .setAttribute("destination_account_id", destinationAccountId)
    .startSpan();

Metrics Design

RED Method Metrics:

Metric Name	Type	Labels	Description
transaction_manager_transaction_count	Counter	status, operation_type	Transaction count by operation type and status
transaction_manager_transaction_duration	Histogram	operation_type, status	Transaction processing duration in milliseconds
transaction_manager_transaction_errors	Counter	operation_type, error_code	Error count by operation and error code
transaction_manager_active_transactions	Gauge	operation_type	Current active transaction count

Sample Histogram Configuration:

Histogram transactionDuration = meter.histogramBuilder("transaction_manager_transaction_duration")
    .setDescription("Transaction processing duration in milliseconds")
    .setUnit("milliseconds")
    .build();

Sample Counter Configuration:

Counter transactionCount = meter.counterBuilder("transaction_manager_transaction_count")
    .setDescription("Number of transactions processed")
    .build();

Health Check Endpoint Design

Health Endpoints:

1. /health - Overall service health
2. /ready - Service ready to serve requests
3. /live - Process liveness check
4. /startup - Startup health validation

Response Format:

{
  "status": "healthy|unhealthy",
  "checks": [
    {
      "name": "database_connection",
      "status": "healthy|unhealthy",
      "details": "Database connected successfully"
    },
    {
      "name": "cache_connection",
      "status": "healthy|unhealthy",
      "details": "Cache connected successfully"
    }
  ],
  "timestamp": "2023-10-27T14:30:45.123Z"
}

Alerting Thresholds

Critical Alerts (Escalation to On-Call):

Metric	Threshold	Alert Condition
transaction_manager_transaction_errors	> 10 per minute	Error rate exceeds threshold
transaction_manager_transaction_duration	p99 > 5000ms	Transaction processing time exceeds threshold
transaction_manager_active_transactions	> 10000	Too many concurrent transactions
transaction_manager_database_connection_errors	> 5 per minute	Database connection failures

Warning Alerts (Monitoring):

Metric	Threshold	Alert Condition
transaction_manager_transaction_duration	p95 > 2000ms	Transaction processing time exceeds threshold
transaction_manager_transaction_errors	> 5 per minute	Error rate increases
transaction_manager_active_transactions	> 5000	High concurrent transaction count

On-Call Escalation Path:

1. Alert triggers → Slack notification to on-call engineer
2. If no response within 5 minutes → PagerDuty escalation to primary team
3. If no response within 10 minutes → Secondary team notification

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• Transaction scope granularity: Should transactions span multiple services (distributed) or be confined to single service boundaries? This decision directly impacts whether we need distributed transaction protocols like Saga or two-phase commit, which significantly increase system complexity.
• Eventual consistency vs. strong consistency: For cross-service operations, how much eventual consistency is acceptable? This affects whether we can use optimistic locking or need pessimistic locking strategies, and impacts the user experience for concurrent operations.
• Idempotency key strategy: Should idempotency keys be generated client-side or server-side? Client-side generation allows for better retry handling but introduces risk of key collisions; server-side generation is safer but requires coordination.
• Error recovery policy: Should failed transactions be automatically retried or require manual intervention? This affects system resilience and the need for robust error handling and alerting systems.

Known trade-offs and justifications

• Performance vs. Safety: Using distributed transactions (e.g., Saga pattern) provides strong consistency but introduces latency and complexity. We could opt for eventual consistency with compensating transactions, trading safety for performance.
• Flexibility vs. Simplicity: Supporting multiple transaction types (e.g., local, distributed, batch) increases flexibility but adds complexity to the system design and maintenance. A simpler approach with a single transaction model may be more maintainable.
• Cache Invalidation vs. Cache Performance: Using cache-aside pattern improves read performance but requires careful invalidation to prevent stale data. Write-through or write-behind patterns provide better consistency but may impact performance.
• Eventual Consistency vs. High Availability: Using eventual consistency for cross-service operations can improve availability but introduces potential data inconsistencies that must be handled gracefully in the application layer.

Risks and mitigation strategies

• Deadlock Risk: Concurrent transaction processing can lead to deadlocks when multiple transactions attempt to acquire locks in different orders. Mitigation includes implementing a consistent lock ordering strategy and using timeout mechanisms.
• Transaction Timeout Risk: Long-running transactions can cause resource contention and timeouts. Mitigation includes implementing transaction timeout policies, monitoring long-running transactions, and providing early warning systems.
• Data Inconsistency Risk: Distributed transactions can lead to partial failures and data inconsistency. Mitigation includes implementing compensating transactions, maintaining audit logs, and providing rollback capabilities.
• Scalability Risk: High transaction volume can overwhelm the system. Mitigation includes implementing proper sharding strategies, using asynchronous processing for non-critical operations, and monitoring transaction throughput.
• Security Risk: Transaction data may contain sensitive information. Mitigation includes implementing proper encryption, access controls, and audit logging to track all transaction activities.

Technical debt acknowledged

• Complexity of Distributed Transaction Handling: The implementation of distributed transaction protocols introduces significant complexity that may be difficult to maintain and debug in the long term. This debt is acceptable for now but requires ongoing attention.
• Eventual Consistency Handling: The system's reliance on eventual consistency introduces potential race conditions and data inconsistency that must be carefully managed through compensating logic and monitoring.
• Idempotency Implementation Complexity: The need to handle idempotent operations across multiple services introduces complexity in tracking and managing duplicate requests, which may lead to increased maintenance overhead.
• Caching Invalidation Complexity: The cache invalidation strategy requires careful handling to prevent stale reads, and the implementation may become complex as more entities are added to the system.
• Monitoring and Alerting Debt: The system's observability layer requires ongoing development to ensure proper monitoring of transaction health, performance, and error rates. Without proper investment in these areas, operational issues may go unnoticed.