File Versioning System Design

Comprehensive low-level design for a scalable file versioning system with snapshot, rollback, and storage optimization capabilities.

PublishedJuly 20, 2025
Length7,813 words
Reading time40 min read

Backend LLD

A robust file versioning system requires careful consideration of storage efficiency, consistency guarantees, and performance characteristics to support concurrent access and reliable recovery operations across distributed systems.

1. Problem Statement & Requirements

Clarifying questions

What scale and throughput requirements are expected?
- Expected concurrent users: 10K–100K
- Peak file upload rate: 1000 uploads/sec?
- Total storage capacity needed: 1PB?
What consistency and durability guarantees are required?
- Strong consistency for version retrieval?
- Is eventual consistency acceptable for file metadata?
What are the key entities and their relationships?
- Are files stored in a flat structure or hierarchical?
- Do versions have parent-child relationships?
What's the read/write ratio and latency sensitivity?
- Is file retrieval more frequent than uploads?
- What are the SLA targets for file access (e.g., p99 < 200ms)?
What authentication and authorization model is in place?
- Are users authenticated via JWT or OAuth2?
- Is RBAC or ABAC used for file access control?
Are there any external integrations or dependencies?
- Does the system integrate with cloud storage (e.g., S3, GCS)?
- Is there a need to support file sharing or collaboration?
What’s out of scope for this design?
- Are file content deduplication, compression, or encryption handled by other systems?

Functional requirements

File Upload: Users can upload new versions of a file with the same name.
Version Retrieval: Retrieve any specific version of a file by ID or timestamp.
File Metadata Management: Store and retrieve metadata (size, type, owner, last modified).
Version History: Maintain a complete history of file versions with timestamps and user info.
Rollback Capability: Revert to a previous version of a file.
File Deletion: Soft delete files with version history preserved.

Non-functional requirements

Requirement	Target
p50 latency (file retrieval)	< 100ms
p95 latency (file retrieval)	< 200ms
p99 latency (file retrieval)	< 500ms
Throughput (uploads/sec)	> 1000
Availability	99.9% (5 minutes of downtime/year)
Data durability	99.99% (data loss risk < 0.01%)
Concurrent users	Support up to 100K simultaneous users

Constraints and assumptions

Storage backend: Assumed to be a cloud storage provider (e.g., AWS S3, GCP Cloud Storage).
File size limits: Max 5GB per file.
Version limit: Maximum of 100 versions per file (configurable).
No real-time collaboration: File versioning is not tied to concurrent editing.
Single region: All data and processing are assumed to be in one region (multi-region support is out of scope).

Out of scope

File content deduplication: Not handled here.
Real-time collaboration features: e.g., live editing, comments.
Cross-region replication or multi-datacenter support.
File content encryption: Assumed to be handled by the storage backend.
Webhooks or real-time notifications for file changes.
File search or indexing capabilities.

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for the file versioning system due to:

Strong typing and automatic code generation for all supported languages
Efficient binary serialization with smaller payload sizes compared to JSON
Native support for streaming and bidirectional communication (useful for large file transfers)
Built-in support for versioning through proto files and semantic versioning
Better performance characteristics for high-throughput scenarios

Endpoint Definitions

Method	Path	Description	Auth
CreateVersion	`POST /v1/files/{fileId}/versions`	Creates a new version of a file	JWT
GetVersion	`GET /v1/files/{fileId}/versions/{versionId}`	Retrieves a specific file version	JWT
ListVersions	`GET /v1/files/{fileId}/versions`	Lists all versions of a file	JWT
RestoreVersion	`POST /v1/files/{fileId}/versions/{versionId}/restore`	Restores a file to a specific version	JWT
DeleteVersion	`DELETE /v1/files/{fileId}/versions/{versionId}`	Deletes a specific file version	JWT
GetVersionContent	`GET /v1/files/{fileId}/versions/{versionId}/content`	Retrieves the content of a specific version	JWT

Request and Response Schemas

CreateVersion Request

{
  "fileId": "string",
  "description": "string (optional)",
  "metadata": {
    "key1": "value1",
    "key2": "value2"
  }
}

CreateVersion Response

{
  "versionId": "string",
  "fileId": "string",
  "versionNumber": "integer",
  "description": "string (optional)",
  "metadata": {
    "key1": "value1",
    "key2": "value2"
  },
  "createdAt": "timestamp",
  "size": "integer",
  "checksum": "string"
}

GetVersion Response

{
  "versionId": "string",
  "fileId": "string",
  "versionNumber": "integer",
  "description": "string (optional)",
  "metadata": {
    "key1": "value1",
    "key2": "value2"
  },
  "createdAt": "timestamp",
  "size": "integer",
  "checksum": "string"
}

ListVersions Response

{
  "versions": [
    {
      "versionId": "string",
      "fileId": "string",
      "versionNumber": "integer",
      "description": "string (optional)",
      "metadata": {
        "key1": "value1",
        "key2": "value2"
      },
      "createdAt": "timestamp",
      "size": "integer",
      "checksum": "string"
    }
  ],
  "nextCursor": "string (optional)"
}

RestoreVersion Response

{
  "message": "string",
  "restoredVersionId": "string"
}

HTTP Status Codes and Error Response Format

Status Code	Description
200	Success
201	Created
400	Bad Request
401	Unauthorized
403	Forbidden
404	Not Found
429	Rate Limited
500	Internal Server Error

Standardized error response format:

{
  "error": {
    "code": "string",
    "message": "string",
    "details": "object (optional)"
  }
}

API Versioning Strategy

Versioned using path prefix: /v1/ with semantic versioning. Future versions will use /v2/ and so on.

Pagination Design

Cursor-based pagination is used because:

Provides consistent results even when data changes
More efficient than offset-based pagination for large datasets
Better performance characteristics with large datasets

Authentication and Authorization

All endpoints require JWT authentication. Authorization is handled through RBAC:

Users must have read permission to list/get versions
Users must have write permission to create/restore/delete versions

Rate Limiting

Rate limiting per endpoint:

CreateVersion: 100 requests/minute
GetVersion/ListVersions: 500 requests/minute
RestoreVersion/DeleteVersion: 50 requests/minute

Rate limiting enforced using Redis with token bucket algorithm.

Idempotency Design

Idempotency keys are supported for mutation endpoints:

X-Idempotency-Key header in requests
Server stores operation state for key duration (30 seconds)
Duplicate requests with same key return same result

// Example idempotency handling in gRPC service
public CreateVersionResponse createVersion(CreateVersionRequest request) {
    String idempotencyKey = request.getIdempotencyKey();
    
    if (idempotencyKey != null) {
        CreateVersionResponse cached = idempotencyCache.get(idempotencyKey);
        if (cached != null) {
            return cached;
        }
    }
    
    // Process request...
    CreateVersionResponse response = processCreateVersion(request);
    
    if (idempotencyKey != null) {
        idempotencyCache.put(idempotencyKey, response, Duration.ofSeconds(30));
    }
    
    return response;
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    FILE {
        string id PK
        string name
        string ownerId
        string storagePath
        string mimeType
        int64 sizeBytes
        datetime createdAt
        datetime updatedAt
        bool isDeleted
    }

    FILE_VERSION {
        string id PK
        string fileId FK
        string versionNumber
        string storagePath
        int64 sizeBytes
        datetime createdAt
        string createdBy
        bool isCurrent
    }

    FILE_HISTORY {
        string id PK
        string fileId FK
        string versionId FK
        string action
        datetime timestamp
        string userId
        string metadata
    }

    FILE {
        string id PK
    }

    FILE_VERSION {
        string fileId FK
    }

    FILE_HISTORY {
        string fileId FK
        string versionId FK
    }

    FILE ||--o{ FILE_VERSION : "has"
    FILE ||--o{ FILE_HISTORY : "has"

Table Schema Definitions

-- Files table
CREATE TABLE files (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL,
    owner_id UUID NOT NULL,
    storage_path TEXT NOT NULL,
    mime_type VARCHAR(100),
    size_bytes BIGINT NOT NULL DEFAULT 0,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    is_deleted BOOLEAN DEFAULT FALSE,
    
    INDEX idx_files_owner_id (owner_id),
    INDEX idx_files_created_at (created_at),
    INDEX idx_files_is_deleted (is_deleted)
);

-- File versions table
CREATE TABLE file_versions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    file_id UUID NOT NULL REFERENCES files(id) ON DELETE CASCADE,
    version_number VARCHAR(50) NOT NULL,
    storage_path TEXT NOT NULL,
    size_bytes BIGINT NOT NULL DEFAULT 0,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    created_by UUID NOT NULL,
    is_current BOOLEAN DEFAULT FALSE,
    
    INDEX idx_file_versions_file_id (file_id),
    INDEX idx_file_versions_version_number (version_number),
    INDEX idx_file_versions_created_at (created_at),
    UNIQUE(file_id, version_number)
);

-- File history table
CREATE TABLE file_history (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    file_id UUID NOT NULL REFERENCES files(id) ON DELETE CASCADE,
    version_id UUID REFERENCES file_versions(id) ON DELETE SET NULL,
    action VARCHAR(50) NOT NULL, -- 'created', 'updated', 'deleted', 'restored'
    timestamp TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    user_id UUID NOT NULL,
    metadata JSONB,
    
    INDEX idx_file_history_file_id (file_id),
    INDEX idx_file_history_timestamp (timestamp),
    INDEX idx_file_history_action (action)
);

Primary Key Strategy

UUIDs for all primary keys - chosen over auto-increment for:

Global uniqueness across distributed systems
Prevention of enumeration attacks
Support for horizontal sharding without key conflicts
Better cacheability and distribution

Index Design Justification

Files table:
- idx_files_owner_id: For owner-based queries (e.g., list user's files)
- idx_files_created_at: For time-based queries and pagination
- idx_files_is_deleted: For soft-delete filtering
File versions table:
- idx_file_versions_file_id: For file-based version queries
- idx_file_versions_version_number: For version lookups
- idx_file_versions_created_at: For chronological queries
File history table:
- idx_file_history_file_id: For file audit trails
- idx_file_history_timestamp: For time-based history queries
- idx_file_history_action: For action-based filtering

Data Partitioning Strategy

Sharding by owner_id (user ID):

Files are partitioned by user ID to ensure data locality
Each user's files and their versions remain in the same shard
This enables efficient querying by owner without cross-shard joins

Soft Delete vs. Hard Delete

Soft delete with is_deleted flag - chosen because:

Enables recovery of accidentally deleted files
Maintains audit trail integrity
Supports compliance requirements for data retention
Allows for version restoration without complex recovery logic

Audit Trail Design

File history table captures all changes:

Records action (create, update, delete, restore)
Timestamps for chronological tracking
User ID for accountability
Metadata field for additional context (e.g., previous version, reason for change)

Data Archival Strategy

30-day retention policy:

Files marked as deleted after 30 days are archived to cold storage
Version history older than 6 months is purged
Archive retention: 1 year for compliance

-- Example archiving procedure (simplified)
CREATE PROCEDURE archive_old_files()
BEGIN
    -- Move files older than 30 days to archive table
    INSERT INTO file_archive SELECT * FROM files WHERE is_deleted = TRUE AND updated_at < NOW() - INTERVAL '30 days';
    
    -- Purge history older than 6 months
    DELETE FROM file_history WHERE timestamp < NOW() - INTERVAL '6 months';
END;

4. Class Design & Service Layer

classDiagram
    class FileVersionService {
        +createFileVersion(FileVersionRequest) FileVersion
        +getFileVersions(String) List~FileVersion~
        +getFileContent(String, int) byte[]
        +revertToVersion(String, int) FileVersion
        +deleteFileVersion(String, int) void
    }
    
    class FileVersionRepository {
        +save(FileVersion) FileVersion
        +findById(String, int) Optional~FileVersion~
        +findAllByFileId(String) List~FileVersion~
        +delete(FileVersion) void
    }
    
    class FileContentStorage {
        +storeContent(String, byte[]) String
        +getContent(String) byte[]
        +deleteContent(String) void
    }
    
    class EventPublisher {
        +publish(FileVersionEvent) void
    }
    
    class FileVersion {
        -id: String
        -fileId: String
        -versionNumber: int
        -contentHash: String
        -size: long
        -createdBy: String
        -createdAt: Instant
        -contentId: String
        +getContent() byte[]
    }
    
    class FileVersionEvent {
        -eventType: String
        -fileId: String
        -versionNumber: int
        -timestamp: Instant
    }
    
    FileVersionService --> FileVersionRepository : uses
    FileVersionService --> FileContentStorage : stores/retrieves
    FileVersionService --> EventPublisher : publishes
    FileVersionRepository --> FileVersion : manages
    FileContentStorage --> FileVersion : references content

stateDiagram-v2
    [*] --> Active
    Active --> Archived : archive()
    Active --> Deleted : delete()
    Archived --> Active : restore()
    Deleted --> [*]

Service Class Structure and Responsibilities

The system implements a service-oriented architecture with the following key services:

FileVersionService - Main orchestration layer for file versioning operations
FileVersionRepository - Data access abstraction for version records
FileContentStorage - Content storage abstraction (S3, local disk, etc.)
EventPublisher - Event publishing for audit and async processing

Interface Design

public interface FileVersionService {
    FileVersion createFileVersion(FileVersionRequest request);
    List<FileVersion> getFileVersions(String fileId);
    byte[] getFileContent(String fileId, int versionNumber);
    FileVersion revertToVersion(String fileId, int versionNumber);
    void deleteFileVersion(String fileId, int versionNumber);
}

public interface FileVersionRepository {
    FileVersion save(FileVersion fileVersion);
    Optional<FileVersion> findById(String fileId, int versionNumber);
    List<FileVersion> findAllByFileId(String fileId);
    void delete(FileVersion fileVersion);
}

public interface FileContentStorage {
    String storeContent(String contentId, byte[] content);
    byte[] getContent(String contentId);
    void deleteContent(String contentId);
}

Business Logic Patterns

Strategy Pattern for Content Storage

public interface ContentStorageStrategy {
    String storeContent(String contentId, byte[] content);
    byte[] getContent(String contentId);
    void deleteContent(String contentId);
}

public class S3StorageStrategy implements ContentStorageStrategy {
    private final AmazonS3 s3Client;
    
    @Override
    public String storeContent(String contentId, byte[] content) {
        // Store to S3 with proper metadata
        return s3Client.putObject(
            new PutObjectRequest("file-content-bucket", contentId, 
                new ByteArrayInputStream(content), null)
        ).getEtag();
    }
    
    @Override
    public byte[] getContent(String contentId) {
        return s3Client.getObject("file-content-bucket", contentId).getObjectContent().readAllBytes();
    }
    
    @Override
    public void deleteContent(String contentId) {
        s3Client.deleteObject("file-content-bucket", contentId);
    }
}

public class LocalStorageStrategy implements ContentStorageStrategy {
    private final Path storagePath;
    
    @Override
    public String storeContent(String contentId, byte[] content) {
        Path filePath = storagePath.resolve(contentId);
        try {
            Files.write(filePath, content);
            return filePath.toString();
        } catch (IOException e) {
            throw new StorageException("Failed to store content", e);
        }
    }
    
    @Override
    public byte[] getContent(String contentId) {
        Path filePath = storagePath.resolve(contentId);
        try {
            return Files.readAllBytes(filePath);
        } catch (IOException e) {
            throw new StorageException("Failed to read content", e);
        }
    }
    
    @Override
    public void deleteContent(String contentId) {
        try {
            Files.deleteIfExists(storagePath.resolve(contentId));
        } catch (IOException e) {
            throw new StorageException("Failed to delete content", e);
        }
    }
}

Critical-Path Method Implementation

public FileVersion createFileVersion(FileVersionRequest request) {
    String fileId = request.getFileId();
    String contentId = UUID.randomUUID().toString();
    
    // Transaction boundary - all operations in single DB transaction
    return transactionTemplate.execute(status -> {
        try {
            // 1. Validate file exists and user has permission
            validateFileAccess(fileId, request.getCreatedBy());
            
            // 2. Generate content hash for deduplication
            String contentHash = generateContentHash(request.getContent());
            
            // 3. Check if content already exists (deduplication)
            Optional<FileVersion> existing = fileVersionRepository
                .findByContentHash(contentHash);
            
            FileVersion newVersion;
            if (existing.isPresent()) {
                // 4. Use existing content - just create new version reference
                FileVersion existingVersion = existing.get();
                newVersion = FileVersion.builder()
                    .fileId(fileId)
                    .versionNumber(getNextVersionNumber(fileId))
                    .contentHash(contentHash)
                    .size(request.getContent().length)
                    .createdBy(request.getCreatedBy())
                    .contentId(existingVersion.getContentId())
                    .build();
            } else {
                // 5. Store content and create new version
                String storedContentId = contentStorage.storeContent(contentId, request.getContent());
                
                newVersion = FileVersion.builder()
                    .fileId(fileId)
                    .versionNumber(getNextVersionNumber(fileId))
                    .contentHash(contentHash)
                    .size(request.getContent().length)
                    .createdBy(request.getCreatedBy())
                    .contentId(storedContentId)
                    .build();
            }
            
            // 6. Save version record
            FileVersion savedVersion = fileVersionRepository.save(newVersion);
            
            // 7. Publish event for audit and async processing
            eventPublisher.publish(FileVersionEvent.builder()
                .eventType("FILE_VERSION_CREATED")
                .fileId(fileId)
                .versionNumber(savedVersion.getVersionNumber())
                .timestamp(Instant.now())
                .build());
            
            return savedVersion;
            
        } catch (Exception e) {
            // Rollback content storage if needed
            if (contentId != null && !contentId.isEmpty()) {
                try {
                    contentStorage.deleteContent(contentId);
                } catch (Exception cleanupError) {
                    // Log but don't fail the main operation
                    log.warn("Failed to cleanup content after version creation failure", cleanupError);
                }
            }
            throw new FileVersioningException("Failed to create file version", e);
        }
    });
}

Transaction Boundary Design

All file versioning operations are wrapped in database transactions to ensure atomicity:

Create version: content storage + DB record creation
Revert version: DB update + content retrieval
Delete version: DB delete + content cleanup

Service Interface Contracts

public class FileVersionRequest {
    private String fileId;
    private byte[] content;
    private String createdBy;
    private String description;
}

public class FileVersion {
    private String id; // UUID
    private String fileId;
    private int versionNumber;
    private String contentHash; // SHA-256 of content
    private long size;
    private String createdBy;
    private Instant createdAt;
    private String contentId; // Reference to storage
}

Saga Pattern for Distributed Transactions

For systems with distributed storage, a saga pattern is implemented:

public class FileVersionSaga {
    private final List<SagaStep> steps;
    
    public void execute(SagaContext context) {
        for (int i = 0; i < steps.size(); i++) {
            try {
                steps.get(i).execute(context);
            } catch (Exception e) {
                // Compensate previous steps
                for (int j = i - 1; j >= 0; j--) {
                    steps.get(j).compensate(context);
                }
                throw new SagaExecutionException("Saga failed at step " + i, e);
            }
        }
    }
}

Domain Object Design

@Builder
public class FileVersion {
    private String id; // UUID
    private String fileId;
    private int versionNumber;
    private String contentHash; // SHA-256 of content
    private long size;
    private String createdBy;
    private Instant createdAt;
    private String contentId; // Reference to storage
    
    public byte[] getContent() {
        return fileContentStorage.getContent(contentId);
    }
    
    public boolean isLatestVersion() {
        return versionNumber == getLatestVersionNumber(fileId);
    }
}

5. Concurrency & Thread Safety

Concurrency Model

The file versioning system will use a thread-safe, asynchronous processing model with:

Java's CompletableFuture for async operations
Thread pool with fixed-size executor (2x CPU cores) for concurrent file operations
ReadWriteLocks for protecting shared mutable state during version operations
Immutable value objects to prevent concurrent modification issues

Race Condition Prevention Strategy

Race conditions are mitigated through:

Optimistic locking using version numbers on file metadata to detect concurrent modifications
Database-level atomic operations for incrementing version counters and updating metadata
Transactional boundary enforcement to ensure atomicity of file operations
Idempotency keys for handling duplicate requests in concurrent environments

Optimistic vs. Pessimistic Locking

Optimistic locking selected because:

File versioning typically has low write contention (most operations are reads)
Reduces lock wait times and improves throughput
Better scalability for concurrent file access operations
Version numbers provide sufficient conflict detection

Deadlock Prevention Strategy

Deadlock prevention mechanisms:

Lock ordering - Always acquire locks in consistent order (e.g., by file ID, then version number)
Timeout-based acquisition - Set 500ms timeout for lock acquisition to prevent indefinite waits
Hierarchical locking - Use file-level locks before version-level locks to avoid circular dependencies
Thread-safe operation queue - Use blocking queues to serialize operations on the same file

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is implemented using:

Idempotency keys passed in request headers (X-Idempotency-Key)
Cache-based deduplication for recent operations (5-minute TTL)
Database-level uniqueness constraints on operation keys
Request fingerprinting to detect and reject duplicate operations

public class FileVersioningService {
    private final Map<String, CompletableFuture<Void>> operationCache = new ConcurrentHashMap<>();
    private final ReadWriteLock fileLock = new ReentrantReadWriteLock();
    
    public CompletableFuture<FileVersion> createVersion(
            String fileId, 
            CreateVersionRequest request) {
        
        String idempotencyKey = request.getIdempotencyKey();
        if (idempotencyKey != null) {
            return operationCache.computeIfAbsent(idempotencyKey, key -> {
                return CompletableFuture.supplyAsync(() -> {
                    try {
                        // Acquire write lock for file operations
                        fileLock.writeLock().lock();
                        
                        // Check if operation already completed (idempotent)
                        FileVersion existing = fileVersionRepository.findByKey(key);
                        if (existing != null) {
                            return existing;
                        }
                        
                        // Perform version creation logic
                        FileVersion newVersion = createNewFileVersion(fileId, request);
                        
                        // Publish version created event
                        eventPublisher.publish(new VersionCreatedEvent(
                            newVersion.getId(), 
                            newVersion.getFileId()));
                            
                        return newVersion;
                    } finally {
                        fileLock.writeLock().unlock();
                    }
                });
            });
        }
        
        // Non-idempotent path - proceed with normal operation
        return CompletableFuture.supplyAsync(() -> {
            fileLock.writeLock().lock();
            try {
                return createNewFileVersion(fileId, request);
            } finally {
                fileLock.writeLock().unlock();
            }
        });
    }
}

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Auth as Auth Service
    participant FS as File Service
    participant Storage as Object Storage
    participant EventBus as Event Bus

    U->>API: Upload file v1
    API->>Auth: Validate JWT
    Auth-->>API: Authenticated
    API->>FS: CreateFileRequest
    FS->>Storage: Store file v1
    Storage-->>FS: File ID
    FS->>FS: CreateFileVersion
    FS->>EventBus: Publish FileUploadedEvent
    EventBus-->>FS: Ack
    FS-->>API: Success Response
    API-->>U: File uploaded successfully

    Note over U,FS: User uploads new version of existing file
    U->>API: Upload file v2 (same name)
    API->>Auth: Validate JWT
    Auth-->>API: Authenticated
    API->>FS: CreateFileVersionRequest
    FS->>FS: Check if file exists
    FS->>Storage: Store file v2
    Storage-->>FS: File ID
    FS->>FS: CreateFileVersion
    FS->>EventBus: Publish FileVersionCreatedEvent
    EventBus-->>FS: Ack
    FS-->>API: Success Response
    API-->>U: File version updated successfully

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Auth as Auth Service
    participant FS as File Service
    participant Storage as Object Storage
    participant EventBus as Event Bus

    U->>API: Delete file version 1
    API->>Auth: Validate JWT
    Auth-->>API: Authenticated
    API->>FS: DeleteFileVersionRequest
    FS->>FS: Get file version info
    FS->>Storage: Delete file v1 from storage
    Storage-->>FS: Success
    FS->>FS: Soft delete version record
    FS->>EventBus: Publish FileVersionDeletedEvent
    EventBus-->>FS: Ack
    FS-->>API: Success Response
    API-->>U: File version deleted successfully

    Note over U,FS: User tries to delete non-existent file
    U->>API: Delete file version 999
    API->>Auth: Validate JWT
    Auth-->>API: Authenticated
    API->>FS: DeleteFileVersionRequest
    FS->>FS: Get file version info
    FS-->>API: FileVersionNotFound
    API-->>U: Error: File version not found

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Auth as Auth Service
    participant FS as File Service
    participant Storage as Object Storage
    participant EventBus as Event Bus

    U->>API: Download file v2
    API->>Auth: Validate JWT
    Auth-->>API: Authenticated
    API->>FS: DownloadFileVersionRequest
    FS->>FS: Get file version info
    FS->>Storage: Get file v2 from storage
    Storage-->>FS: File content
    FS-->>API: Success Response with file content
    API-->>U: File downloaded successfully

    Note over U,FS: Async processing for file metadata update
    FS->>EventBus: Publish FileMetadataUpdatedEvent
    EventBus-->>FS: Ack

7. Caching Design

Cache Strategy Selection

For file versioning system, we'll use cache-aside pattern with write-through for critical metadata operations. This approach provides:

Explicit cache control - cache operations are explicitly managed by the application
Consistency guarantees - cache and database remain consistent through explicit invalidation
Flexibility for different data access patterns - can optimize per use case

Cache Key Design

Cache keys will follow a structured namespace pattern:

{namespace}:{entity_type}:{entity_id}:{version?}:{metadata_key?}

Examples:

file:metadata:1234567890abcdef
file:version:1234567890abcdef:1
file:permissions:1234567890abcdef:user:john@example.com

TTL Strategy

Data Type	TTL	Reason
File metadata (read)	5 minutes	Frequent reads, infrequent updates
File version list	1 minute	Recent versions frequently accessed
User permissions	30 seconds	High security sensitivity
File content hash	1 hour	Computationally expensive to recalculate
Recent file access	10 minutes	User activity patterns

Cache Eviction Policy

We'll use LRU (Least Recently Used) with TTL fallback:

// Pseudo-code for cache eviction strategy
public class FileVersionCache {
    private final Cache<String, CachedFileData> cache = 
        Caffeine.newBuilder()
            .maximumSize(10000)
            .expireAfterWrite(Duration.ofMinutes(5))
            .evictionListener((key, value, cause) -> {
                // Log cache eviction for monitoring
                logger.info("Evicted key: {}, cause: {}", key, cause);
            })
            .build();
}

Cache Stampede Prevention

To prevent cache stampedes:

Probabilistic early expiry: 80% of TTL for cache entries
Distributed locks for expensive cache population operations

public class CacheStampedePrevention {
    private final RedissonClient redisson;
    
    public CachedFileData getCachedFile(String fileId) {
        String cacheKey = "file:metadata:" + fileId;
        CachedFileData cached = cache.getIfPresent(cacheKey);
        
        if (cached == null) {
            // Use distributed lock to prevent stampede
            String lockKey = "lock:" + cacheKey;
            RLock lock = redisson.getLock(lockKey);
            
            try {
                if (lock.tryLock(100, TimeUnit.MILLISECONDS)) {
                    // Double-check pattern
                    cached = cache.getIfPresent(cacheKey);
                    if (cached == null) {
                        cached = fetchAndCacheFile(fileId, cacheKey);
                    }
                } else {
                    // Fallback to database read
                    return fetchFromDatabase(fileId);
                }
            } finally {
                lock.unlock();
            }
        }
        
        return cached;
    }
}

Distributed Cache Architecture

We'll use Redis Cluster with:

3 master nodes for high availability
3 replica nodes per master for read scaling
Redis Sentinel for automatic failover
Cache sharding based on file ID hash

Cache Invalidation Strategy

We'll implement event-driven invalidation:

Write-through: Cache is updated on file metadata changes
Event-driven invalidation: File modification events trigger cache purges

@Component
public class FileCacheInvalidator {
    
    @EventListener
    public void handleFileUpdated(FileUpdatedEvent event) {
        String fileId = event.getFileId();
        
        // Invalidate all related cache entries
        cache.invalidate("file:metadata:" + fileId);
        cache.invalidate("file:version:" + fileId);
        cache.invalidate("file:permissions:" + fileId);
        
        // Invalidate user-specific entries if needed
        cache.invalidateAll(
            cache.asMap().keySet().stream()
                .filter(key -> key.startsWith("file:permissions:" + fileId + ":user:"))
                .collect(Collectors.toList())
        );
    }
}

Cache Hit/Miss Ratio Monitoring

We'll track:

Cache hit ratio (target: >90%)
Cache miss ratio by operation type
Cache size and memory usage

@Monitor
public class CacheMetrics {
    private final Counter cacheHits = Counter.build()
        .name("file_version_cache_hits_total")
        .help("Total cache hits")
        .register();
        
    private final Counter cacheMisses = Counter.build()
        .name("file_version_cache_misses_total")
        .help("Total cache misses")
        .register();
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain objects and persistence concerns. Each entity has its own repository interface with standardized CRUD operations.

public interface FileRepository {
    Optional<File> findById(UUID fileId);
    List<File> findByUserId(UUID userId, Pageable pageable);
    File save(File file);
    void deleteById(UUID fileId);
    boolean existsById(UUID fileId);
}

public interface FileVersionRepository {
    Optional<FileVersion> findById(UUID versionId);
    List<FileVersion> findByFileId(UUID fileId, Pageable pageable);
    FileVersion save(FileVersion version);
    void deleteById(UUID versionId);
    List<FileVersion> findLatestVersionsByFileIds(List<UUID> fileIds);
}

ORM vs Raw Query Decisions

For the core entities (File and FileVersion), we use JPA/Hibernate for its rich feature set including lazy loading, entity relationships, and query optimization. For high-volume read operations like version listing, we use raw SQL queries to reduce overhead and allow for complex filtering.

@Query("SELECT f FROM File f WHERE f.userId = :userId AND f.deletedAt IS NULL")
List<File> findByUserId(@Param("userId") UUID userId);

@Query(value = "SELECT * FROM file_versions fv WHERE fv.file_id = :fileId ORDER BY fv.version_number DESC LIMIT :limit", nativeQuery = true)
List<FileVersion> findLatestVersionsByFileId(@Param("fileId") UUID fileId, @Param("limit") int limit);

Read vs Write Separation

We implement CQRS pattern for the file versioning system to separate read and write operations:

public interface FileReadRepository {
    FileMetadata findFileMetadataById(UUID fileId);
    List<FileVersionSummary> findLatestVersionsByFileIds(List<UUID> fileIds);
    FileVersionContent findContentById(UUID contentId);
}

public interface FileWriteRepository {
    File save(File file);
    FileVersion save(FileVersion version);
    void deleteFileVersions(UUID fileId);
}

Connection Pool Configuration

Connection pool is configured with:

Max pool size: 20
Min idle connections: 5
Connection timeout: 30 seconds
Validation query: SELECT 1
Idle timeout: 600 seconds

@Configuration
public class DatabaseConfig {
    @Bean
    public DataSource dataSource() {
        HikariDataSource dataSource = new HikariDataSource();
        dataSource.setJdbcUrl("jdbc:postgresql://localhost:5432/fileversioning");
        dataSource.setMaximumPoolSize(20);
        dataSource.setMinimumIdle(5);
        dataSource.setConnectionTimeout(30000);
        dataSource.setIdleTimeout(600000);
        dataSource.setValidationTimeout(5000);
        dataSource.setLeakDetectionThreshold(60000);
        return dataSource;
    }
}

N+1 Query Prevention Strategy

We prevent N+1 queries by:

Using eager loading with JOIN FETCH for related entities
Implementing batch loading for multiple file versions

@Query("SELECT f FROM File f LEFT JOIN FETCH f.versions v WHERE f.id IN :fileIds")
List<File> findByIdsWithVersions(@Param("fileIds") List<UUID> fileIds);

@Query("SELECT fv FROM FileVersion fv WHERE fv.fileId = :fileId ORDER BY fv.versionNumber DESC")
List<FileVersion> findLatestVersionsByFileId(@Param("fileId") UUID fileId);

Batch Operation Design

Batch operations are implemented for bulk file version creation:

public class BatchFileVersionRepository {
    
    public void saveBatch(List<FileVersion> versions) {
        String sql = "INSERT INTO file_versions (id, file_id, version_number, content_id, created_at) VALUES (?, ?, ?, ?, ?)";
        
        try (Connection conn = dataSource.getConnection();
             PreparedStatement stmt = conn.prepareStatement(sql)) {
            
            for (FileVersion version : versions) {
                stmt.setObject(1, version.getId());
                stmt.setObject(2, version.getFileId());
                stmt.setInt(3, version.getVersionNumber());
                stmt.setObject(4, version.getContentId());
                stmt.setTimestamp(5, Timestamp.valueOf(version.getCreatedAt()));
                stmt.addBatch();
            }
            
            stmt.executeBatch();
        } catch (SQLException e) {
            throw new DataAccessException("Failed to batch save file versions", e);
        }
    }
}

Transaction Management

All repository operations are wrapped in proper transaction boundaries:

@Transactional
public class FileRepositoryImpl implements FileRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    @Override
    public File save(File file) {
        if (file.getId() == null) {
            entityManager.persist(file);
        } else {
            entityManager.merge(file);
        }
        return file;
    }
    
    @Override
    @Transactional(readOnly = true)
    public Optional<File> findById(UUID fileId) {
        return Optional.ofNullable(entityManager.find(File.class, fileId));
    }
}

Data Access Layer Abstraction

The repository layer is abstracted through a service interface to allow for easy mocking and testing:

public interface FileVersionService {
    FileVersion createVersion(UUID fileId, String content);
    List<FileVersion> getVersions(UUID fileId, int limit);
    FileVersion getVersion(UUID versionId);
    void deleteVersion(UUID versionId);
}

Query Optimization

We implement query optimization through:

Composite indexes on frequently queried fields
Partial indexes for filtered queries
Covering indexes for read-heavy operations

-- Composite index for file version queries
CREATE INDEX idx_file_version_file_id_version_number ON file_versions(file_id, version_number DESC);

-- Partial index for active files
CREATE INDEX idx_file_active_user ON files(user_id, deleted_at) WHERE deleted_at IS NULL;

-- Covering index for metadata retrieval
CREATE INDEX idx_file_covering ON files(id, user_id, name, size, created_at) 
INCLUDE (version_count, deleted_at);

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The file versioning system will use a hybrid synchronous/asynchronous approach. Core operations like file upload, version creation, and metadata updates will be handled synchronously to ensure immediate consistency for the user. However, heavy operations like file deduplication, content analysis, and archival will be processed asynchronously to avoid blocking user requests.

Key async triggers:

File upload completion → triggers content analysis and deduplication
Version creation → triggers metadata indexing and notification
File deletion → triggers archival and cleanup operations

Message schema design with sample payload

{
  "version": "1.0",
  "eventId": "uuid-12345",
  "eventType": "file.version.created",
  "timestamp": "2023-10-15T14:30:00Z",
  "payload": {
    "fileId": "f-abc123",
    "versionId": "v-def456",
    "userId": "u-xyz789",
    "originalFileId": "f-original123",
    "size": 1024,
    "contentType": "application/pdf",
    "checksum": "sha256:abc123...",
    "metadata": {
      "title": "Document.pdf",
      "tags": ["important", "draft"]
    }
  },
  "source": "file-service",
  "schemaVersion": "1.0"
}

Dead letter queue strategy and poison message handling

All async operations will use a dead letter queue (DLQ) pattern. Messages that fail more than 3 times will be moved to a DLQ with a separate monitoring and manual intervention process. Poison messages will be logged with full context including the original message, error stack trace, and retry count.

At-least-once vs. exactly-once delivery

We will implement at-least-once delivery with idempotency keys to handle duplicate messages gracefully. This is chosen over exactly-once because:

The file versioning system can safely handle duplicate operations
Exactly-once requires complex coordination that adds unnecessary complexity
At-least-once provides sufficient consistency for user-facing operations

Retry policy for consumers

public class RetryPolicy {
    private static final int MAX_RETRIES = 3;
    private static final Duration BASE_DELAY = Duration.ofSeconds(1);
    private static final Duration MAX_DELAY = Duration.ofMinutes(5);
    
    public boolean shouldRetry(int attempt, Exception error) {
        if (attempt >= MAX_RETRIES) return false;
        
        // Retry on transient errors
        return error instanceof TransientException || 
               error instanceof ServiceUnavailableException ||
               error instanceof TimeoutException;
    }
    
    public Duration getDelay(int attempt) {
        long delayMs = (long) Math.min(
            BASE_DELAY.toMillis() * Math.pow(2, attempt),
            MAX_DELAY.toMillis()
        );
        
        // Add jitter (±25%)
        long jitter = (long) (delayMs * 0.25 * (Math.random() - 0.5));
        return Duration.ofMillis(delayMs + jitter);
    }
}

Outbox pattern for reliable event publishing

The outbox pattern will be implemented to ensure message delivery reliability:

public class EventOutbox {
    private final EventRepository eventRepository;
    private final MessagePublisher messagePublisher;
    
    @Transactional
    public void publishEvent(Event event) {
        // Store event in outbox table
        EventOutboxEntry entry = new EventOutboxEntry(event);
        eventRepository.save(entry);
        
        // Publish to message broker
        try {
            messagePublisher.publish(entry.getEvent());
            // Mark as published in outbox
            entry.setPublished(true);
            eventRepository.save(entry);
        } catch (Exception e) {
            // Handle publish failure - will be retried by background job
            entry.setFailed(true);
            eventRepository.save(entry);
        }
    }
}

Sequence diagram for async flows

sequenceDiagram
    participant Client
    participant FileService
    participant EventPublisher
    participant MessageBroker
    participant ContentAnalyzer
    participant Deduplicator
    
    Client->>FileService: Upload file
    FileService->>FileService: Create version record
    FileService->>EventPublisher: Publish file.uploaded event
    EventPublisher->>MessageBroker: Send to queue
    MessageBroker-->>ContentAnalyzer: Process content analysis
    MessageBroker-->>Deduplicator: Process deduplication
    
    ContentAnalyzer->>FileService: Update file metadata
    Deduplicator->>FileService: Update file references
    
    FileService->>Client: Return version info

10. Error Handling Design

Error Classification Hierarchy

The error handling strategy follows a hierarchical classification approach to enable proper routing and response:

Error
├── DomainError (business logic violations)
│   ├── ValidationError (input validation failures)
│   ├── BusinessLogicError (rule violations, state transitions)
│   └── ResourceConflictError (e.g., version already exists)
├── InfrastructureError (system-level issues)
│   ├── DatabaseError (connection, query failures)
│   ├── StorageError (file system, S3 issues)
│   └── ExternalServiceError (third-party API failures)
└── InternalError (system bugs, unexpected states)
    ├── UnexpectedError (unhandled exceptions)
    └── ConfigurationError (misconfigured components)

Error Propagation Strategy

Domain and Infrastructure errors are caught at service boundaries and translated to appropriate HTTP responses
Internal errors bubble up to global error handlers for logging and system-level alerts
Validation errors are returned immediately with 400 status codes
Business logic errors return 409 or 422 depending on context (conflict vs. validation failure)
Infrastructure errors are handled with retry logic where appropriate, then escalated to monitoring

Retry Strategy

For infrastructure and external service errors, a retry mechanism with exponential backoff is implemented:

public class RetryConfig {
    private static final int MAX_RETRIES = 3;
    private static final long INITIAL_DELAY_MS = 1000;
    private static final double BACKOFF_MULTIPLIER = 2.0;
    private static final double JITTER_RANGE = 0.1; // 10% jitter
    
    public static long calculateDelay(int attempt) {
        long delay = (long) (INITIAL_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempt));
        long jitter = (long) (delay * JITTER_RANGE * (Math.random() - 0.5) * 2);
        return delay + jitter;
    }
}

Circuit Breaker Design

For external dependencies like storage services, a circuit breaker pattern is implemented:

public class CircuitBreaker {
    private final int failureThreshold = 5;
    private final long timeoutMs = 60000;
    private final int successThreshold = 5;
    
    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private long lastFailureTime = 0;
    
    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }
    
    public boolean canExecute() {
        switch (state) {
            case CLOSED:
                return true;
            case HALF_OPEN:
                return failureCount < successThreshold;
            case OPEN:
                if (System.currentTimeMillis() - lastFailureTime > timeoutMs) {
                    state = CircuitState.HALF_OPEN;
                    return true;
                }
                return false;
        }
        return false;
    }
    
    public void recordSuccess() {
        if (state == CircuitState.HALF_OPEN) {
            failureCount = 0;
            state = CircuitState.CLOSED;
        }
    }
    
    public void recordFailure() {
        lastFailureTime = System.currentTimeMillis();
        failureCount++;
        
        if (failureCount >= failureThreshold) {
            state = CircuitState.OPEN;
        }
    }
}

Fallback Behavior Design

For critical external dependencies, fallbacks are implemented:

public class StorageFallback {
    private final StorageService primaryStorage;
    private final BackupStorageService backupStorage;
    
    public FileMetadata getFileMetadata(String fileId) {
        try {
            return primaryStorage.getFileMetadata(fileId);
        } catch (StorageException e) {
            // Fallback to backup storage for read operations
            if (e.isReadOperation()) {
                return backupStorage.getFileMetadata(fileId);
            }
            throw e; // Re-throw for write operations
        }
    }
}

Error Logging Format

Structured logging follows this schema:

{
  "timestamp": "2023-10-15T14:30:45.123Z",
  "trace_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "span_id": "f0e9d8c7-b6a5-4321-fedc-ba9876543210",
  "user_id": "user-123",
  "request_id": "req-456",
  "error_code": "FILE_VERSION_CONFLICT",
  "error_message": "Version already exists for file: file-789",
  "error_type": "ResourceConflictError",
  "component": "FileVersioningService",
  "stack_trace": "...",
  "request_context": {
    "method": "POST",
    "path": "/files/file-789/versions",
    "params": {},
    "headers": {}
  }
}

Alerting Thresholds

Key metrics with alerting thresholds:

Metric	Alert Level	Threshold	Action
`file_versioning_errors_total`	Warning	> 10/minute	Slack alert
`file_versioning_errors_total`	Critical	> 100/minute	On-call alert
`file_versioning_retry_count`	Warning	> 5/minute	Slack alert
`circuit_breaker_opened`	Critical	> 1/minute	On-call alert
`storage_error_rate`	Warning	> 5%	Slack alert

Error Response Format

Standardized error response format:

{
  "error": {
    "code": "FILE_VERSION_CONFLICT",
    "message": "Version already exists for file: file-789",
    "details": {
      "file_id": "file-789",
      "version_number": 3
    },
    "timestamp": "2023-10-15T14:30:45.123Z",
    "request_id": "req-456"
  }
}

Error Handling in Service Layer

public class FileVersioningService {
    
    public FileVersion createFileVersion(String fileId, CreateFileVersionRequest request) {
        try {
            // Transaction boundary
            return transactionTemplate.execute(status -> {
                try {
                    validateRequest(request);
                    FileVersion fileVersion = buildFileVersion(fileId, request);
                    
                    // Business logic validation
                    if (isVersionConflict(fileVersion)) {
                        throw new ResourceConflictError(
                            "Version already exists for file: " + fileId,
                            "FILE_VERSION_CONFLICT"
                        );
                    }
                    
                    // Persist to DB
                    FileVersion saved = fileVersionRepository.save(fileVersion);
                    
                    // Publish event
                    eventPublisher.publish(new FileVersionCreatedEvent(saved));
                    
                    return saved;
                } catch (ValidationException e) {
                    // Log and rethrow as domain error
                    logger.warn("Validation failed for file version creation", e);
                    throw new ValidationError(e.getMessage(), "VALIDATION_ERROR");
                } catch (StorageException e) {
                    // Retry logic with circuit breaker
                    if (circuitBreaker.canExecute()) {
                        try {
                            // Retry once with backoff
                            Thread.sleep(RetryConfig.calculateDelay(1));
                            return retryCreateFileVersion(fileId, request);
                        } catch (InterruptedException ie) {
                            Thread.currentThread().interrupt();
                            throw new InfrastructureError("Retry interrupted", "RETRY_INTERRUPTED");
                        }
                    }
                    throw new StorageError("Storage service unavailable", "STORAGE_SERVICE_UNAVAILABLE");
                }
            });
        } catch (Exception e) {
            // Log internal error for monitoring
            logger.error("Unexpected error in file version creation", e);
            throw new InternalError("Internal system error", "INTERNAL_ERROR");
        }
    }
}

11. Security Design

Input validation and sanitization strategy

All file paths, names, and metadata are validated using a combination of regex patterns and length constraints. Path components are sanitized to prevent directory traversal attacks by rejecting any input containing .. sequences or absolute paths. File names are restricted to alphanumeric characters, underscores, hyphens, and periods, with a maximum length of 255 characters. The validation is performed using Apache Commons Validator for regex patterns and custom sanitization logic.

SQL injection prevention

All database queries are constructed using parameterized prepared statements to prevent SQL injection. No dynamic query construction or string concatenation is used for database operations. The repository layer uses MyBatis with parameterized queries, and all user inputs are properly escaped or bound as parameters.

Authentication middleware design

Authentication is handled through JWT tokens passed in the Authorization header. The authentication middleware verifies token signature, expiration, and audience claims before allowing access to protected endpoints. A custom filter in Spring Security intercepts all requests and validates the JWT token before proceeding to the service layer.

Authorization enforcement

Role-based access control (RBAC) is implemented at the service layer using Spring Security annotations. Users must have appropriate roles (e.g., FILE_OWNER, READER, WRITER) to perform actions on files. Permissions are checked before any file operation, and the system enforces that users can only access files they own or have been granted access to.

Sensitive data encryption at rest and in transit

All file content is encrypted using AES-256-GCM encryption before being stored on disk. The encryption keys are managed through AWS KMS (Key Management Service) with automatic key rotation every 90 days. All data in transit is encrypted using TLS 1.3 with HSTS header enforcement to prevent man-in-the-middle attacks.

PII handling and masking strategy

File metadata that may contain PII (such as user names or email addresses) is masked in logs and audit trails. For example, email addresses are displayed as user***@domain.com. The system does not store or log PII in cleartext format, and all sensitive fields are encrypted at rest.

Secrets management

Secrets such as encryption keys, database credentials, and external API tokens are managed using AWS Secrets Manager. The application retrieves secrets at startup and caches them in memory with automatic refresh on expiration. Secrets are rotated every 90 days using AWS Lambda functions.

Audit logging design

All file operations (create, update, delete, access) are logged with structured JSON format including:

Timestamp
User ID (masked)
Operation type (create, update, delete, access)
File ID
IP address (masked)
Session ID (masked)

The audit logs are stored in a separate encrypted database table with retention policy of 3 years. Logs are also published to CloudWatch for real-time monitoring and alerting.

// Example of input validation in file service
public class FileValidationService {
    private static final Pattern FILE_NAME_PATTERN = 
        Pattern.compile("^[a-zA-Z0-9_.\\- ]{1,255}$");
    private static final Pattern SAFE_PATH_PATTERN = 
        Pattern.compile("^[a-zA-Z0-9_\\-\\/\\.]+$");
    
    public void validateFileName(String fileName) {
        if (fileName == null || fileName.isEmpty()) {
            throw new ValidationException("File name cannot be null or empty");
        }
        if (!FILE_NAME_PATTERN.matcher(fileName).matches()) {
            throw new ValidationException("Invalid file name format");
        }
        if (fileName.contains("..")) {
            throw new ValidationException("Directory traversal not allowed");
        }
    }
    
    public void validateFilePath(String filePath) {
        if (filePath == null || filePath.isEmpty()) {
            throw new ValidationException("File path cannot be null or empty");
        }
        if (!SAFE_PATH_PATTERN.matcher(filePath).matches()) {
            throw new ValidationException("Invalid file path format");
        }
        if (filePath.startsWith("/")) {
            throw new ValidationException("Absolute paths not allowed");
        }
    }
}

12. Testing Strategy

Unit Testing Scope and Coverage Targets

Target coverage: 85% line coverage, 90% branch coverage for core business logic
Scope: All service classes, domain objects, value objects, and utility functions
Test patterns:
- Parameterized tests for boundary conditions (file size limits, version limits)
- Mock-based testing for external dependencies (storage, messaging)
- State machine transition tests for version lifecycle management
- Strategy pattern implementation tests

Integration Testing Strategy

Database integration: Test with embedded H2 database for fast execution, with real PostgreSQL in CI/CD pipeline
Storage integration: Use MinIO mock server for S3-compatible storage operations (file upload/download, versioning)
Messaging integration: WireMock for event broker interaction testing (e.g., Redis or Kafka)
Testcontainers: For end-to-end integration with real storage and database instances in CI pipeline
Key test scenarios:
- File upload with version creation and storage
- Version retrieval and restoration
- Concurrent access to same file versions
- Storage cleanup and archival

Contract Testing

Framework: Pact for consumer-driven contract testing between file versioning service and storage service
Contract scope:
- File upload API contract (request/response format)
- Version metadata contract (version ID, timestamp, size)
- Storage interaction contract (S3-compatible operations)
Testing strategy:
- Consumer tests for versioning service
- Provider tests for storage service
- Versioned contract schema validation

Performance Testing Tool and Key Scenarios

Tool: k6 for load testing, JMeter for stress testing
Key performance scenarios:
- Concurrent file uploads (1000 simultaneous requests)
- Version retrieval under load (500 concurrent version fetches)
- Large file versioning (1GB files with multiple versions)
- Storage cleanup batch operations (10,000 version deletions)
Metrics tracked:
- p50/p95/p99 response times (target <200ms for 95%)
- Throughput (1000 ops/sec target)
- Memory usage and GC pressure during high load

Test Data Management Strategy

Fixtures: Pre-defined test files with known content and metadata for consistency
Factories:
- FileVersionFactory for generating valid version objects
- StorageObjectFactory for simulating storage layer behavior
Seeding:
- Seed test database with realistic version history (100+ files, 5-20 versions each)
- Use test data generators for boundary condition testing (empty files, max size files)
Data isolation:
- Use separate test databases per parallel test execution
- Clear test data between test runs using transaction rollback or truncate
Data lifecycle:
- Test data retention: 1 week for CI, 30 days for local dev
- Automated cleanup of test data after each test run or suite

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective search, filtering, and analysis:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO|WARN|ERROR",
  "service": "file-versioning-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user-123",
  "entity_id": "file-456",
  "operation": "uploadFile",
  "request_id": "req-789",
  "message": "File uploaded successfully",
  "duration_ms": 150,
  "error_code": "FILE_UPLOAD_FAILED",
  "error_details": {
    "message": "Permission denied",
    "stack_trace": "...",
    "cause": "INSUFFICIENT_PERMISSIONS"
  },
  "metadata": {
    "file_size_bytes": 1024000,
    "version_number": 3,
    "storage_location": "/storage/files/file-456-v3"
  }
}

PII masking rules:

User IDs should be masked in logs when not required for debugging
File paths and content should never appear in logs
Sensitive metadata fields (e.g., file content hash) should be redacted or truncated

Distributed Tracing Instrumentation

OpenTelemetry integration with custom spans:

Key operations to trace:

file.upload - Upload file and create new version
file.retrieve - Retrieve specific version or latest
file.delete - Delete a specific version or all versions
file.diff - Compare two file versions

Span attributes to capture:

// Example for upload operation
Span currentSpan = Span.current();
currentSpan.setAttribute("file.id", fileId);
currentSpan.setAttribute("file.version", versionNumber);
currentSpan.setAttribute("file.size.bytes", fileSizeBytes);
currentSpan.setAttribute("user.id", userId);
currentSpan.setAttribute("storage.provider", "s3");

Metrics Design

RED Method metrics per service endpoint:

Metric Name	Type	Labels	Description
`file_versioning_requests_total`	Counter	operation, status, user_id	Total count of all file operations
`file_versioning_request_duration_seconds`	Histogram	operation, status	Response time distribution
`file_versioning_errors_total`	Counter	operation, error_type	Error counts by operation and type
`file_versioning_storage_bytes`	Gauge	storage_type, file_type	Current storage usage
`file_versioning_active_versions`	Gauge	file_id	Active version count per file

Label design:

operation: upload, download, delete, list, diff
status: success, client_error, server_error, timeout
error_type: validation_failed, storage_error, permission_denied, etc.
storage_type: s3, local, gcs
file_type: document, image, video, etc.

Health Check Endpoint Design

GET /health
GET /ready
GET /live
GET /startup

Response format:

{
  "status": "healthy|unhealthy",
  "timestamp": "2023-10-27T14:30:45.123Z",
  "service": "file-versioning-service",
  "checks": {
    "database": {
      "status": "healthy",
      "latency_ms": 5
    },
    "storage": {
      "status": "healthy",
      "latency_ms": 12
    },
    "cache": {
      "status": "healthy",
      "latency_ms": 2
    }
  },
  "version": "1.2.3"
}

Health check criteria:

/live: Basic service liveness, no external dependencies
/ready: Service ready to serve requests (DB and cache connected)
/health: Full health including all dependencies
/startup: Service is in a stable state after initialization

Alerting Thresholds

Metric	Warning Level	Critical Level	Escalation Path
`file_versioning_requests_total`	10k req/min	50k req/min	On-call engineer
`file_versioning_request_duration_seconds`	500ms p95	2s p95	SRE team
`file_versioning_errors_total`	10 errors/min	50 errors/min	SRE team
`file_versioning_storage_bytes`	80% used	95% used	Platform team
`file_versioning_active_versions`	100k versions	500k versions	Platform team

Alerts should be configured with:

Alerting window: 5-minute sliding window
Alerting delay: 30 seconds to avoid flapping
Multi-level escalation (warning → critical → on-call)
Slack/Email notifications with context and severity levels

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

Version retention policy: Should we implement a soft-deletion model where old versions are marked as deleted but not immediately purged, or should we enforce strict version limits (e.g., keep last 10 versions)? This affects storage costs and recovery complexity.
Storage backend strategy: Should we store file content in a dedicated object storage (S3, GCS) or embed it within the database? This decision impacts cost, scalability, and data consistency.
Granularity of versioning: Should we version at file level or support granular versioning within a file (e.g., for collaborative editing)? This affects the complexity of the data model and API design.
User-facing version control: Should we expose version history to users or keep it internal for system-level recovery? This impacts UI/UX design and feature scope.

Known trade-offs and justifications

Trade-off	Option A	Option B	Justification
Storage vs. Performance	Store full file content in DB	Store only metadata, keep content in external storage	Full content in DB provides better consistency but increases DB size and query times. External storage reduces DB load but adds complexity in managing content consistency.
Consistency vs. Availability	Strong consistency for version operations	Eventual consistency with optimistic concurrency control	Strong consistency ensures data integrity but can impact availability under high load. Eventual consistency allows better availability but requires handling conflicts and race conditions.
Real-time vs. Batch versioning	Real-time version creation on every file update	Batch versioning with periodic snapshots	Real-time versions provide better auditability but increase write load. Batch versioning reduces overhead but may miss intermediate states.
User experience vs. system complexity	Provide detailed version diff UI	Simplified version list with basic metadata	Rich diff UI provides better user experience but increases complexity in diff computation and rendering. Simple version list is easier to implement and maintain but offers less insight into changes.

Risks and mitigation strategies

Storage cost explosion: Large files with frequent updates can lead to exponential storage growth. Mitigation: Implement version retention policies, content deduplication, and compression strategies.
Performance degradation under high concurrent writes: Multiple simultaneous file updates can cause lock contention and slow query execution. Mitigation: Use optimistic locking, implement proper indexing, and design for horizontal scaling.
Data consistency issues: Race conditions in version creation can lead to lost updates or inconsistent state. Mitigation: Implement proper transaction boundaries, use atomic operations, and design idempotent operations.
Content integrity failure: Corrupted or lost file content in external storage can break version history. Mitigation: Implement checksum validation, content integrity checks, and redundant storage strategies.
Audit trail gaps: Missing version events or incomplete metadata can compromise compliance. Mitigation: Implement comprehensive audit logging, use distributed tracing, and enforce mandatory metadata fields.

Technical debt acknowledged

Limited diff computation capability: Current design assumes simple file content storage without sophisticated diff algorithms. This creates technical debt for future features requiring detailed change tracking.
No content compression: Storing raw file content without compression increases storage costs and bandwidth usage, creating technical debt for scalability.
Hardcoded version limits: Implementation currently uses hardcoded version limits instead of configurable policies, creating technical debt for flexible versioning strategies.
No cross-file version dependencies: The system doesn't support versioning of related files (e.g., a document and its associated images), creating technical debt for complex collaborative workflows.
Synchronous operation bottleneck: Version creation and file operations are currently synchronous, which can become a bottleneck at scale. This creates technical debt for async processing improvements.