File Upload Service Design: Scalable Storage and Retrieval Architecture

This document presents a production-ready low-level design for a file upload service that handles millions of concurrent uploads while maintaining high availability, efficient storage management, and robust content delivery capabilities across distributed infrastructure.

1. Problem Statement & Requirements

Clarifying questions

1. What scale expectations are there for this file upload service?

   • How many concurrent uploads per second?
   • What's the expected file size range (e.g., 1KB to 10GB)?
   • What's the peak concurrent upload capacity?

2. What consistency requirements exist for file metadata and content?

   • Is eventual consistency acceptable for file status updates?
   • Do we need strong consistency for file retrieval after upload?

3. What are the key entities and their relationships?

   • Will files belong to users, teams, or projects?
   • Do we need support for file versioning or lifecycle management?

4. What's the read/write ratio and latency sensitivity?

   • Are uploads more frequent than downloads?
   • What are the acceptable latency targets (e.g., p50/p95/p99) for upload vs. download?

5. What authentication and authorization model is expected?

   • Will it support OAuth2, JWT tokens, or API keys?
   • Are there role-based access controls (RBAC) for file visibility?

6. Are there external integrations or dependencies?

   • Will files be stored in cloud storage (e.g., S3, GCS)?
   • Do we need to integrate with other systems (e.g., virus scanning, metadata extraction)?

7. What’s out of scope for this design?

   • Are features like file sharing, collaboration, or content moderation in scope?
   • Is file deletion and archival handled by this service?

Functional requirements

• Users can upload files via HTTP POST requests.
• Files are stored in a durable storage system (e.g., cloud object store).
• Metadata about uploaded files is persisted in the database.
• Users can retrieve file metadata and content via GET requests.
• File upload status is tracked (e.g., queued, processing, completed).
• Support for multipart file uploads.
• File content is securely stored and accessible only to authorized users.

Non-functional requirements

• Latency Targets:

  • p50: <200ms for upload initiation
  • p95: <500ms for upload initiation
  • p99: <1s for upload initiation

• Throughput:

  • Support at least 10,000 concurrent uploads per second.

• Availability:

  • 99.9% uptime (43.2 minutes of downtime/month max).

• Security:

  • All file transfers must be encrypted in transit (TLS 1.3+).
  • File content must be encrypted at rest (AES-256).

Constraints and assumptions

• Assumption: Files are uploaded to a cloud object store (e.g., AWS S3) with metadata stored in a relational database.
• Assumption: The service is stateless and horizontally scalable.
• Constraint: File size limits (e.g., max 10GB per file).
• Constraint: Uploads must be idempotent to handle retries.
• Assumption: Users are authenticated via JWT tokens.

Out of scope

• File sharing or collaboration features.
• Content moderation or virus scanning.
• File lifecycle management (e.g., automatic deletion).
• Multi-region replication or cross-cloud storage support.
• File search or indexing capabilities.

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for the file upload service due to:

• Efficient binary serialization reducing payload size
• Strong typing and auto-generated client/server code
• Built-in support for streaming uploads (for large files)
• Native support for bidirectional streaming for progress tracking
• Better performance characteristics over HTTP/REST for binary data

Endpoint Definitions

Method	Path	Description	Auth	Rate Limit
POST	/v1/files	Upload a new file	JWT + RBAC	100 req/min
GET	/v1/files/{fileId}	Retrieve file metadata	JWT + RBAC	500 req/min
DELETE	/v1/files/{fileId}	Delete file permanently	JWT + RBAC	50 req/min
GET	/v1/files/{fileId}/download	Download file content	JWT + RBAC	200 req/min
POST	/v1/files/{fileId}/metadata	Update file metadata	JWT + RBAC	100 req/min

Request and Response Schemas

Upload File Request (gRPC):

message UploadFileRequest {
  string filename = 1;
  int64 size_bytes = 2;
  string content_type = 3;
  bytes chunk_data = 4; // For streaming
  string upload_id = 5; // For resumable uploads
}

message UploadFileResponse {
  string file_id = 1;
  string filename = 2;
  int64 size_bytes = 3;
  string content_type = 4;
  string status = 5; // "UPLOADING", "COMPLETED", "FAILED"
  string upload_url = 6; // For chunked uploads
}

File Metadata Response:

{
  "file_id": "uuid-12345",
  "filename": "document.pdf",
  "size_bytes": 1024000,
  "content_type": "application/pdf",
  "created_at": "2023-10-15T14:30:00Z",
  "updated_at": "2023-10-15T14:30:00Z",
  "status": "COMPLETED",
  "owner_id": "user-abc123",
  "tags": ["document", "important"]
}

Error Response Format:

{
  "error": {
    "code": "INVALID_ARGUMENT",
    "message": "File size exceeds maximum limit of 10MB",
    "details": [
      {
        "field": "size_bytes",
        "message": "Must be less than 10485760 bytes"
      }
    ]
  }
}

HTTP Status Codes

Status Code	Usage
200 OK	Successful GET, PUT, PATCH
201 Created	Successful POST (file created)
400 Bad Request	Invalid input, missing fields
401 Unauthorized	Missing or invalid auth token
403 Forbidden	Insufficient permissions
404 Not Found	File or resource not found
409 Conflict	Duplicate file name (if configured)
429 Too Many Requests	Rate limit exceeded
500 Internal Server Error	Unexpected server error

API Versioning Strategy

Versioned using path prefix: /v1/ with future versions as /v2/, etc. This approach:

• Allows for clean separation between major versions
• Enables independent evolution of endpoints
• Maintains backward compatibility for existing clients

Pagination Design

Cursor-based pagination is used because:

• More stable than offset-based when data changes
• Better performance for large datasets
• Easier to implement with composite indexes

Authentication and Authorization

All endpoints require JWT token with RBAC:

• Upload: file:upload permission
• Download: file:download permission
• Metadata: file:read permission
• Delete: file:delete permission

Rate Limiting Strategy

Rate limiting enforced at API gateway level:

• Upload endpoint: 100 requests/minute per user
• Download endpoint: 200 requests/minute per user
• Metadata endpoints: 500 requests/minute per user
• Delete endpoint: 50 requests/minute per user

Idempotency Design

For mutation endpoints, idempotency key strategy:

• Client includes X-Idempotency-Key header with UUID
• Service stores operation state by key for 1 hour
• If same key received, return cached result with same status code

gRPC Streaming Support

For large file uploads, gRPC supports:

• UploadFile: Bidirectional streaming for chunked uploads
• ProgressStream: Server-side progress updates during upload
• UploadStatus: Real-time status updates for ongoing uploads

File Upload Flow

1. Client initiates upload with metadata
2. Service validates file and creates upload session
3. Client streams chunks using bidirectional gRPC stream
4. Service validates each chunk and stores to temporary location
5. Finalize upload when all chunks received
6. Move file from temp storage to permanent location
7. Publish file created event for downstream processing

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    USER ||--o{ FILE : "uploads"
    FILE ||--o{ FILE_VERSION : "has"
    FILE_VERSION ||--o{ FILE_CHUNK : "contains"
    FILE ||--o{ FILE_ACCESS_LOG : "accessed"
    
    USER {
        string id PK
        string email
        string name
        timestamp created_at
        timestamp updated_at
    }
    
    FILE {
        string id PK
        string user_id FK
        string name
        string content_type
        int64 size_bytes
        string storage_path
        string status
        timestamp created_at
        timestamp updated_at
        timestamp deleted_at
    }
    
    FILE_VERSION {
        string id PK
        string file_id FK
        int version_number
        string storage_path
        timestamp created_at
    }
    
    FILE_CHUNK {
        string id PK
        string file_version_id FK
        int chunk_index
        string storage_path
        timestamp created_at
    }
    
    FILE_ACCESS_LOG {
        string id PK
        string file_id FK
        string user_id FK
        string action
        timestamp accessed_at
    }

Table Schema Definitions

Users Table

CREATE TABLE users (
    id VARCHAR(36) PRIMARY KEY,
    email VARCHAR(255) UNIQUE NOT NULL,
    name VARCHAR(255) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    
    INDEX idx_email (email)
);

Files Table

CREATE TABLE files (
    id VARCHAR(36) PRIMARY KEY,
    user_id VARCHAR(36) NOT NULL,
    name VARCHAR(512) NOT NULL,
    content_type VARCHAR(128),
    size_bytes BIGINT UNSIGNED NOT NULL,
    storage_path VARCHAR(1024) NOT NULL,
    status ENUM('UPLOADING', 'UPLOADED', 'PROCESSING', 'AVAILABLE', 'FAILED') NOT NULL DEFAULT 'UPLOADING',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    deleted_at TIMESTAMP NULL,
    
    INDEX idx_user_id (user_id),
    INDEX idx_status (status),
    INDEX idx_created_at (created_at),
    INDEX idx_deleted_at (deleted_at),
    
    FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);

File Versions Table

CREATE TABLE file_versions (
    id VARCHAR(36) PRIMARY KEY,
    file_id VARCHAR(36) NOT NULL,
    version_number INT NOT NULL,
    storage_path VARCHAR(1024) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_file_id (file_id),
    INDEX idx_version_number (version_number),
    
    FOREIGN KEY (file_id) REFERENCES files(id) ON DELETE CASCADE,
    UNIQUE KEY uk_file_version (file_id, version_number)
);

File Chunks Table

CREATE TABLE file_chunks (
    id VARCHAR(36) PRIMARY KEY,
    file_version_id VARCHAR(36) NOT NULL,
    chunk_index INT NOT NULL,
    storage_path VARCHAR(1024) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_file_version_id (file_version_id),
    INDEX idx_chunk_index (chunk_index),
    
    FOREIGN KEY (file_version_id) REFERENCES file_versions(id) ON DELETE CASCADE,
    UNIQUE KEY uk_file_version_chunk (file_version_id, chunk_index)
);

File Access Logs Table

CREATE TABLE file_access_logs (
    id VARCHAR(36) PRIMARY KEY,
    file_id VARCHAR(36) NOT NULL,
    user_id VARCHAR(36) NOT NULL,
    action ENUM('VIEW', 'DOWNLOAD', 'DELETE') NOT NULL,
    accessed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_file_id (file_id),
    INDEX idx_user_id (user_id),
    INDEX idx_accessed_at (accessed_at),
    
    FOREIGN KEY (file_id) REFERENCES files(id) ON DELETE CASCADE,
    FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);

Primary Key Strategy

UUID v4 is used for all primary keys because:

• Globally unique identifiers prevent collisions across distributed systems
• No sequential patterns that could expose system structure or enable enumeration attacks
• Support for distributed database sharding without coordination
• Better performance in high-concurrency environments where auto-increment can become a bottleneck

Index Design

1. Users table:

   • idx_email: For efficient user lookup by email address

2. Files table:

   • idx_user_id: For efficient file listing by user
   • idx_status: For filtering files by upload status (e.g., "failed" files)
   • idx_created_at: For time-based file queries (e.g., recent uploads)
   • idx_deleted_at: For soft-deleted file filtering

3. File Versions table:

   • idx_file_id: For efficient version lookups by file
   • idx_version_number: For version-based queries

4. File Chunks table:

   • idx_file_version_id: For efficient chunk lookups by version
   • idx_chunk_index: For ordered chunk retrieval

5. File Access Logs table:

   • idx_file_id: For audit trail queries by file
   • idx_user_id: For user access history
   • idx_accessed_at: For time-based access logs

Data Partitioning / Sharding Strategy

Shard by User ID (Hash-based):

• Files are partitioned by user_id hash modulo N shards
• Each shard contains files for a subset of users
• This ensures even distribution and avoids hotspots on user-heavy accounts

Soft Delete vs. Hard Delete Decision

Soft Delete is implemented for files with deleted_at timestamp:

• Prevents accidental data loss during operations or system errors
• Supports audit trails and compliance requirements
• Allows for recovery in case of accidental deletion or system bugs
• Maintains referential integrity in related tables

Audit Trail Design

File Access Logs table captures:

• file_id: Which file was accessed
• user_id: Who accessed it
• action: What operation was performed (view, download, delete)
• accessed_at: When it occurred

Data Archival Strategy and Retention Policy

Archival Strategy:

• Files marked as "deleted" after 30 days are moved to archival storage
• Metadata is preserved but file content is archived to cheaper storage (e.g., Glacier)
• Archival metadata includes original file details and archive location

Retention Policy:

• 30 days: Active files (accessible)
• 90 days: Soft-deleted files (can be recovered)
• 365 days: Archived files (permanent storage with metadata retention)

public enum FileStatus {
    UPLOADING,
    UPLOADED,
    PROCESSING,
    AVAILABLE,
    FAILED
}

4. Class Design & Service Layer

classDiagram
    class FileUploadService {
        +uploadFile(UploadRequest) UploadResponse
        +getFileMetadata(String) FileMetadata
        +deleteFile(String) boolean
        +validateFile(UploadRequest) ValidationResult
    }
    
    class FileStorageService {
        +storeFile(FileData, String) StorageResult
        +retrieveFile(String) FileData
        +deleteFile(String) boolean
    }
    
    class FileValidator {
        +validate(UploadRequest) ValidationResult
        +validateFileSize(long, long) boolean
        +validateFileType(String, List~String~) boolean
    }
    
    class FileMetadataService {
        +createMetadata(FileMetadata) FileMetadata
        +updateMetadata(String, Map~String, Object~) FileMetadata
        +getMetadata(String) FileMetadata
    }
    
    class EventPublisher {
        +publish(FileUploadedEvent)
    }
    
    class FileUploadProcessor {
        +process(UploadRequest) UploadResponse
        +handleFileProcessing(UploadRequest) FileProcessingResult
    }
    
    class FileProcessingResult {
        +fileId: String
        +status: ProcessingStatus
        +errors: List~String~
    }
    
    class UploadRequest {
        +fileName: String
        +fileSize: long
        +contentType: String
        +fileData: byte[]
        +userId: String
    }
    
    class FileMetadata {
        +id: String
        +fileName: String
        +fileSize: long
        +contentType: String
        +userId: String
        +uploadTime: Instant
        +status: FileStatus
    }
    
    class UploadResponse {
        +fileId: String
        +status: String
        +message: String
    }
    
    class ValidationResult {
        +isValid: boolean
        +errors: List~String~
    }
    
    class StorageResult {
        +storagePath: String
        +success: boolean
        +error: String
    }
    
    class FileUploadedEvent {
        +fileId: String
        +userId: String
        +fileName: String
        +uploadTime: Instant
    }
    
    FileUploadService --> FileStorageService : uses
    FileUploadService --> FileValidator : validates
    FileUploadService --> FileMetadataService : manages metadata
    FileUploadService --> EventPublisher : publishes events
    FileUploadService --> FileUploadProcessor : processes files
    
    FileUploadProcessor --> FileStorageService : stores
    FileUploadProcessor --> FileMetadataService : persists metadata
    
    FileValidator --> UploadRequest : validates
    FileMetadataService --> FileMetadata : manages

stateDiagram-v2
    [*] --> Pending
    Pending --> Processing : upload initiated
    Processing --> Success : file stored successfully
    Processing --> Failed : validation error or storage failure
    Success --> Archived : file archived after retention period
    Failed --> [*]
    Archived --> [*]

Service Layer Architecture

The service layer follows a layered architecture pattern with clear separation of concerns:

1. FileUploadService - Main entry point for file upload operations
2. FileStorageService - Handles actual file storage operations
3. FileValidator - Validates file metadata and content
4. FileMetadataService - Manages file metadata persistence
5. EventPublisher - Handles event publishing for audit and async processing

Business Logic Patterns

Strategy Pattern Implementation

public interface FileValidationStrategy {
    ValidationResult validate(UploadRequest request);
}

@Component
public class SizeBasedFileValidator implements FileValidationStrategy {
    private final long maxFileSize;
    
    public SizeBasedFileValidator(@Value("${file.upload.max-size:10485760}") long maxFileSize) {
        this.maxFileSize = maxFileSize;
    }
    
    @Override
    public ValidationResult validate(UploadRequest request) {
        if (request.getFileSize() > maxFileSize) {
            return ValidationResult.failure("File size exceeds maximum limit");
        }
        return ValidationResult.success();
    }
}

@Component
public class TypeBasedFileValidator implements FileValidationStrategy {
    private final Set<String> allowedTypes;
    
    public TypeBasedFileValidator(@Value("${file.upload.allowed-types:}") String allowedTypes) {
        this.allowedTypes = Arrays.stream(allowedTypes.split(","))
            .map(String::trim)
            .collect(Collectors.toSet());
    }
    
    @Override
    public ValidationResult validate(UploadRequest request) {
        if (!allowedTypes.contains(request.getContentType())) {
            return ValidationResult.failure("File type not allowed");
        }
        return ValidationResult.success();
    }
}

Critical-Path Method Implementation

@Transactional
public UploadResponse uploadFile(UploadRequest request) {
    // Validate input
    ValidationResult validation = validateFile(request);
    if (!validation.isValid()) {
        throw new ValidationException("File validation failed", validation.getErrors());
    }
    
    // Generate unique file ID
    String fileId = UUID.randomUUID().toString();
    
    try {
        // Acquire exclusive lock for file ID to prevent race conditions
        String lockKey = "file-upload-lock:" + fileId;
        if (!lockManager.acquireLock(lockKey, 30, TimeUnit.SECONDS)) {
            throw new ConcurrentModificationException("File upload in progress");
        }
        
        // Store file to permanent storage
        StorageResult storageResult = fileStorageService.storeFile(
            new FileData(fileId, request.getFileData()), 
            fileId
        );
        
        if (!storageResult.isSuccess()) {
            throw new StorageException("Failed to store file", storageResult.getError());
        }
        
        // Create metadata record
        FileMetadata metadata = FileMetadata.builder()
            .id(fileId)
            .fileName(request.getFileName())
            .fileSize(request.getFileSize())
            .contentType(request.getContentType())
            .userId(request.getUserId())
            .uploadTime(Instant.now())
            .status(FileStatus.ACTIVE)
            .build();
            
        fileMetadataService.createMetadata(metadata);
        
        // Publish event for async processing and audit
        FileUploadedEvent event = new FileUploadedEvent(
            fileId, 
            request.getUserId(), 
            request.getFileName(),
            Instant.now()
        );
        
        eventPublisher.publish(event);
        
        // Return success response
        return UploadResponse.builder()
            .fileId(fileId)
            .status("SUCCESS")
            .message("File uploaded successfully")
            .build();
            
    } catch (Exception e) {
        // Rollback any partial operations
        cleanupFailedUpload(fileId);
        throw new FileUploadException("File upload failed", e);
    } finally {
        // Release lock
        lockManager.releaseLock(lockKey);
    }
}

private void cleanupFailedUpload(String fileId) {
    try {
        fileStorageService.deleteFile(fileId);
        fileMetadataService.deleteMetadata(fileId);
    } catch (Exception cleanupException) {
        // Log but don't fail the main operation
        log.warn("Failed to cleanup partial upload for file: {}", fileId, cleanupException);
    }
}

Transaction Boundary Design

The transaction boundary encompasses:

1. File storage operation (database + file system)
2. Metadata creation in database
3. Event publishing to message queue

Service Interface Contracts

public interface FileUploadService {
    UploadResponse uploadFile(UploadRequest request);
    FileMetadata getFileMetadata(String fileId);
    boolean deleteFile(String fileId);
    ValidationResult validateFile(UploadRequest request);
}

Saga Pattern for Distributed Transactions

For distributed file upload, we implement a saga pattern:

@Component
public class FileUploadSaga {
    private final EventPublisher eventPublisher;
    private final FileStorageService fileStorageService;
    private final FileMetadataService metadataService;
    
    public void executeUploadSaga(UploadRequest request) {
        String sagaId = UUID.randomUUID().toString();
        
        try {
            // Step 1: Validate file
            ValidationResult validation = validateFile(request);
            if (!validation.isValid()) {
                throw new ValidationException("Validation failed");
            }
            
            // Step 2: Store file in storage
            String fileId = UUID.randomUUID().toString();
            StorageResult storageResult = fileStorageService.storeFile(
                new FileData(fileId, request.getFileData()), 
                fileId
            );
            
            // Step 3: Create metadata record
            FileMetadata metadata = createMetadata(fileId, request);
            
            // Step 4: Publish event
            eventPublisher.publish(new FileUploadedEvent(fileId, request.getUserId()));
            
        } catch (Exception e) {
            // Compensating actions
            compensateSaga(sagaId);
            throw new FileUploadException("Saga failed", e);
        }
    }
    
    private void compensateSaga(String sagaId) {
        // Rollback operations in reverse order
        // Delete file from storage, remove metadata, etc.
    }
}

Domain Object Design

@Value
public class FileMetadata {
    String id;
    String fileName;
    long fileSize;
    String contentType;
    String userId;
    Instant uploadTime;
    FileStatus status;
    
    public static class Builder {
        private String id;
        private String fileName;
        private long fileSize;
        private String contentType;
        private String userId;
        private Instant uploadTime;
        private FileStatus status;
        
        public Builder id(String id) { this.id = id; return this; }
        public Builder fileName(String fileName) { this.fileName = fileName; return this; }
        public Builder fileSize(long fileSize) { this.fileSize = fileSize; return this; }
        public Builder contentType(String contentType) { this.contentType = contentType; return this; }
        public Builder userId(String userId) { this.userId = userId; return this; }
        public Builder uploadTime(Instant uploadTime) { this.uploadTime = uploadTime; return this; }
        public Builder status(FileStatus status) { this.status = status; return this; }
        
        public FileMetadata build() {
            return new FileMetadata(id, fileName, fileSize, contentType, userId, uploadTime, status);
        }
    }
}

Value Object Design

@Value
public class FileData {
    String fileId;
    byte[] content;
    long size;
    
    public static class Builder {
        private String fileId;
        private byte[] content;
        private long size;
        
        public Builder fileId(String fileId) { this.fileId = fileId; return this; }
        public Builder content(byte[] content) { this.content = content; return this; }
        public Builder size(long size) { this.size = size; return this; }
        
        public FileData build() {
            return new FileData(fileId, content, size);
        }
    }
}

5. Concurrency & Thread Safety

Concurrency Model

The file upload service operates in a multi-threaded environment with the following concurrency model:

• Thread pool executor for handling concurrent upload requests (typically 100-500 threads based on system capacity)
• Async I/O model using NIO for efficient file handling (non-blocking I/O operations)
• Single-threaded event loop pattern for processing upload metadata and state transitions
• Thread-safe components using synchronized collections, atomic references, and lock-free data structures

Race Condition Prevention Strategy

Race conditions are mitigated through:

1. Immutable file metadata objects to prevent concurrent modification during upload processing
2. Atomic operations for updating file status counters and metadata in database
3. Database-level constraints (e.g., unique constraints on file names) to prevent duplicate uploads
4. Optimistic locking using version numbers for concurrent metadata updates
5. Synchronized access to shared file system resources (e.g., temporary directories, storage buckets)

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected for the following reasons:

• High read-to-write ratio: Most operations are read-heavy (file listing, status checks)
• Low contention: Concurrent uploads typically target different file paths or names
• Better performance: Avoids blocking threads and reduces lock acquisition overhead
• Database-level conflict detection: PostgreSQL's UPDATE ... WHERE version = ? pattern handles conflicts gracefully

Deadlock Prevention Strategy

Deadlock prevention mechanisms include:

1. Consistent lock ordering: All database operations follow a consistent order (by file ID, then by timestamp)
2. Timeout configuration: Database transaction timeouts set to 30 seconds
3. Lock acquisition timeout: File system resource locks with 5-second timeout
4. Hierarchical locking: Acquire locks in a consistent hierarchy (metadata first, then file content)
5. Retry logic with exponential backoff: For retrying failed optimistic lock updates

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is handled through:

1. Idempotency key strategy using client-provided upload tokens or UUIDs
2. Database uniqueness constraints on upload tokens to prevent duplicate uploads
3. Cache-based idempotency check using Redis for fast duplicate request detection
4. State synchronization: Concurrent requests to same file path wait for the first operation to complete

public class FileUploadService {
    private final Map<String, CompletableFuture<Void>> ongoingUploads = new ConcurrentHashMap<>();
    
    public CompletableFuture<UploadResult> handleUploadRequest(
            UploadRequest request, String idempotencyKey) {
        return ongoingUploads.computeIfAbsent(idempotencyKey, key -> {
            CompletableFuture<UploadResult> future = CompletableFuture.supplyAsync(() -> {
                try {
                    // Check if this upload is already in progress
                    if (isDuplicateUpload(key)) {
                        return getExistingResult(key);
                    }
                    
                    // Process upload with optimistic locking
                    return processUploadWithLocking(request, key);
                } catch (Exception e) {
                    // Handle and propagate errors
                    throw new UploadProcessingException("Upload failed", e);
                }
            });
            
            // Cleanup after completion
            future.whenComplete((result, error) -> 
                ongoingUploads.remove(idempotencyKey));
            
            return future;
        });
    }
    
    private boolean isDuplicateUpload(String key) {
        // Check Redis cache or database for existing upload
        return fileMetadataRepository.existsByIdempotencyKey(key);
    }
    
    private UploadResult processUploadWithLocking(UploadRequest request, String key) {
        // Use database transaction with optimistic locking
        return fileMetadataRepository.withOptimisticLocking(() -> {
            // Validate and create file metadata
            FileMetadata metadata = createFileMetadata(request);
            
            // Process file content with thread-safe I/O
            processFileContent(metadata, request);
            
            // Update metadata with completion status
            return updateFileMetadata(metadata);
        });
    }
}

The implementation ensures that concurrent requests with the same idempotency key don't trigger duplicate processing, while maintaining thread safety for concurrent uploads with different keys.

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant Client
    participant Gateway
    participant AuthN
    participant FileService
    participant Storage
    participant EventPublisher

    Client->>Gateway: POST /upload
    Gateway->>AuthN: Validate JWT
    AuthN-->>Gateway: Authenticated User
    Gateway->>FileService: UploadFile(file, metadata)
    FileService->>Storage: Store file (S3/GCS)
    Storage-->>FileService: File ID
    FileService->>EventPublisher: Publish FileUploaded event
    EventPublisher-->>FileService: Event published
    FileService-->>Gateway: Response (fileId, url)
    Gateway-->>Client: 201 Created

sequenceDiagram
    participant Client
    participant Gateway
    participant AuthN
    participant FileService
    participant Storage
    participant EventPublisher

    Client->>Gateway: POST /upload with invalid file type
    Gateway->>AuthN: Validate JWT
    AuthN-->>Gateway: Authenticated User
    Gateway->>FileService: UploadFile(file, metadata)
    FileService-->>Gateway: 400 Bad Request (Invalid file type)
    Gateway-->>Client: 400 Bad Request

sequenceDiagram
    participant Client
    participant Gateway
    participant AuthN
    participant FileService
    participant Storage
    participant EventPublisher
    participant RetryableConsumer

    Client->>Gateway: POST /upload
    Gateway->>AuthN: Validate JWT
    AuthN-->>Gateway: Authenticated User
    Gateway->>FileService: UploadFile(file, metadata)
    FileService->>Storage: Store file (S3/GCS)
    Storage-->>FileService: File ID
    FileService->>EventPublisher: Publish FileUploaded event
    EventPublisher-->>FileService: Event published
    FileService->>RetryableConsumer: Process file metadata async
    RetryableConsumer-->>FileService: Success or retry queue
    FileService-->>Gateway: Response (fileId, url)
    Gateway-->>Client: 201 Created

7. Caching Design

Cache Strategy Selection

For file upload service, we'll use cache-aside pattern with write-through behavior for metadata and write-behind for content. This approach provides optimal balance between consistency and performance.

Cache Key Design

Cache keys will follow a structured namespace pattern:

file:{fileId}:metadata
file:{fileId}:content
file:{fileId}:thumbnail

TTL Strategy

Data Type	TTL	Reason
File Metadata	1 hour	Frequent reads, metadata changes infrequently
File Content (in-memory)	30 minutes	Content accessed during download, can be regenerated
Thumbnail Cache	24 hours	Thumbnails are expensive to generate, stable
Upload Session State	1 hour	Session data is volatile but needs to persist

Cache Eviction Policy

We'll use LRU (Least Recently Used) with TTL-based expiration as the primary eviction policy. This ensures:

• Most frequently accessed files stay in cache
• Memory pressure is managed automatically
• Cache warming is efficient for hot paths

Cache Stampede Prevention

To prevent cache stampedes:

1. Probabilistic Early Expiry: 20% of TTL for metadata cache
2. Distributed Locks: For expensive operations like thumbnail generation
3. Cache Warm-up Strategy: Pre-warm hot files during low-traffic periods

Distributed Cache Architecture

Using Redis Cluster with:

• 3 master nodes for data distribution
• 3 replica nodes for high availability
• Auto-failover capability
• Memory eviction policy set to allkeys-lru

Cache Invalidation Strategy

Event-driven invalidation for metadata and content:

1. When file is uploaded → invalidate metadata cache
2. When file is deleted → invalidate both metadata and content caches
3. When thumbnail is generated → update thumbnail cache
4. TTL-based invalidation for content (as needed)

public class FileCacheService {
    private final RedisTemplate<String, Object> redisTemplate;
    
    public void invalidateFileCache(String fileId) {
        String metadataKey = "file:" + fileId + ":metadata";
        String contentKey = "file:" + fileId + ":content";
        String thumbnailKey = "file:" + fileId + ":thumbnail";
        
        // Use pipeline for batch invalidation
        redisTemplate.executePipelined(
            (RedisCallback<Object>) connection -> {
                connection.del(metadataKey.getBytes());
                connection.del(contentKey.getBytes());
                connection.del(thumbnailKey.getBytes());
                return null;
            }
        );
    }
    
    public void cacheFileMetadata(String fileId, FileMetadata metadata) {
        String key = "file:" + fileId + ":metadata";
        // Use cache-aside with TTL
        redisTemplate.opsForValue().set(
            key, 
            metadata, 
            Duration.ofHours(1)
        );
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and persistence concerns. Each entity has its own repository interface with standardized CRUD operations.

public interface FileRepository {
    File save(File file);
    Optional<File> findById(UUID id);
    List<File> findByUserId(UUID userId);
    void deleteById(UUID id);
    boolean existsById(UUID id);
}

ORM vs. Raw Query Decisions

For the file upload service, we use JPA/Hibernate for most operations due to its type safety and ORM benefits. However, for high-volume bulk operations like file metadata updates or batch processing, raw SQL queries are preferred to avoid unnecessary object mapping overhead.

@Repository
public class FileRepositoryImpl implements FileRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    @Override
    public File save(File file) {
        if (file.getId() == null) {
            entityManager.persist(file);
        } else {
            entityManager.merge(file);
        }
        return file;
    }
    
    @Override
    public Optional<File> findById(UUID id) {
        return Optional.ofNullable(entityManager.find(File.class, id));
    }
    
    // Raw SQL for performance-critical batch operations
    @Override
    public void updateFileMetadata(List<UUID> fileIds, String status) {
        String sql = "UPDATE files SET status = :status WHERE id IN :ids";
        entityManager.createNativeQuery(sql)
            .setParameter("status", status)
            .setParameter("ids", fileIds)
            .executeUpdate();
    }
}

Read vs. Write Separation

We implement CQRS (Command Query Responsibility Segregation) pattern for read-heavy operations to optimize performance and scalability:

// Write model - used for mutations (save, update)
@Repository
public class FileWriteRepository {
    @PersistenceContext
    private EntityManager entityManager;
    
    public File save(File file) {
        entityManager.persist(file);
        return file;
    }
}

// Read model - optimized for querying
@Repository
public class FileReadRepository {
    @PersistenceContext
    private EntityManager entityManager;
    
    public List<FileMetadata> findRecentFilesByUser(UUID userId, int limit) {
        String jpql = "SELECT new com.example.dto.FileMetadata(f.id, f.name, f.size, f.status) " +
                     "FROM File f WHERE f.userId = :userId ORDER BY f.createdAt DESC";
        return entityManager.createQuery(jpql, FileMetadata.class)
                           .setParameter("userId", userId)
                           .setMaxResults(limit)
                           .getResultList();
    }
}

Connection Pool Configuration

Connection pool is configured using HikariCP with optimized settings for concurrent file upload operations:

spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      validation-timeout: 5000
      idle-timeout: 600000
      max-lifetime: 1800000
      leak-detection-threshold: 60000

N+1 Query Prevention Strategy

To prevent N+1 query issues, we use eager loading with JOIN FETCH and batch size optimization:

// Efficient query with eager loading of related entities
@Query("SELECT f FROM File f JOIN FETCH f.user u WHERE f.userId = :userId")
List<File> findByUserIdWithUser(@Param("userId") UUID userId);

// Batch loading for multiple files
@Query("SELECT f FROM File f WHERE f.id IN :ids")
List<File> findByIds(@Param("ids") List<UUID> ids);

Batch Operation Design

For bulk operations like file status updates or metadata synchronization, we implement batch processing:

@Service
@Transactional
public class BatchFileOperationService {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    public void updateFileStatusBatch(List<UUID> fileIds, String newStatus) {
        // Batch size of 1000 to avoid memory issues
        int batchSize = 1000;
        
        for (int i = 0; i < fileIds.size(); i += batchSize) {
            int endIndex = Math.min(i + batchSize, fileIds.size());
            List<UUID> batch = fileIds.subList(i, endIndex);
            
            String sql = "UPDATE files SET status = :status WHERE id IN (:ids)";
            Query query = entityManager.createNativeQuery(sql);
            query.setParameter("status", newStatus);
            query.setParameter("ids", batch);
            query.executeUpdate();
        }
    }
}

Transaction Management

All repository operations are wrapped in appropriate transaction boundaries:

@Service
@Transactional(readOnly = true)
public class FileService {
    
    @Autowired
    private FileWriteRepository writeRepo;
    
    @Autowired
    private FileReadRepository readRepo;
    
    @Transactional
    public File createFile(File file) {
        return writeRepo.save(file);
    }
    
    @Transactional
    public void updateFileStatus(UUID fileId, String status) {
        File file = writeRepo.findById(fileId)
                           .orElseThrow(() -> new FileNotFoundException(fileId));
        file.setStatus(status);
        writeRepo.save(file);
    }
}

Query Optimization Techniques

1. Indexing: Proper indexes on frequently queried fields
2. Query hints: Using @QueryHint for performance tuning
3. Result caching: For frequently accessed metadata

@QueryHints(value = {
    @QueryHint(name = "org.hibernate.fetchSize", value = "50"),
    @QueryHint(name = "org.hibernate.timeout", value = "30")
})
@Query("SELECT f FROM File f WHERE f.userId = :userId AND f.status = :status")
List<File> findByUserIdAndStatus(@Param("userId") UUID userId, @Param("status") String status);

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The file upload service will process uploads synchronously for immediate validation and metadata extraction, but will use asynchronous processing for time-intensive operations like virus scanning, content analysis, and thumbnail generation. The key boundary is that synchronous operations must return a response within 2-3 seconds to meet SLA requirements, while asynchronous operations can take minutes or hours.

Message schema design

{
  "version": "1.0",
  "event_type": "file.processed",
  "timestamp": "2023-11-15T14:30:00Z",
  "payload": {
    "file_id": "uuid-12345",
    "file_name": "document.pdf",
    "file_size": 1024000,
    "upload_timestamp": "2023-11-15T14:29:00Z",
    "storage_path": "/files/uuid-12345/document.pdf",
    "original_file_url": "https://storage.example.com/uuid-12345/document.pdf",
    "processing_metadata": {
      "virus_scan_status": "pending",
      "thumbnail_status": "pending",
      "content_analysis_status": "pending"
    }
  },
  "source": "file-upload-service",
  "correlation_id": "corr-abcde"
}

Dead letter queue strategy and poison message handling

All failed messages will be routed to a dead-letter queue (DLQ) with a maximum retry limit of 5 attempts. Poison messages will be flagged in the database and manually reviewed by operations team after 3 failed retries. Messages that fail with non-retryable errors (e.g., invalid file format) will be moved to DLQ immediately.

At-least-once vs. exactly-once delivery

At-least-once delivery is selected for this feature because:

• File processing operations are idempotent (reprocessing same file won't cause issues)
• The system can handle duplicate processing gracefully
• Simpler implementation and better performance

Retry policy for consumers

Retry policy uses exponential backoff with jitter:

• Max retries: 5 attempts
• Base delay: 1 second
• Backoff formula: delay = base_delay * (2^attempt + random(0, 1))
• Retryable errors: network timeouts, service unavailability, storage failures
• Non-retryable errors: invalid file format, permission denied

Outbox pattern for reliable event publishing

Outbox pattern will be implemented to ensure message delivery reliability:

public class FileEventPublisher {
    private final EventRepository eventRepository;
    private final MessageProducer messageProducer;

    public void publishFileProcessedEvent(File file) {
        // Create event in outbox table
        FileProcessEvent event = new FileProcessEvent();
        event.setFileId(file.getId());
        event.setStatus(EventStatus.PENDING);
        eventRepository.save(event);

        try {
            // Publish to message broker
            messageProducer.publish(event);
            
            // Update status to published
            event.setStatus(EventStatus.PUBLISHED);
            eventRepository.save(event);
        } catch (Exception e) {
            // Mark as failed for retry
            event.setStatus(EventStatus.FAILED);
            eventRepository.save(event);
            throw new EventPublishingException("Failed to publish file processed event", e);
        }
    }
}

Sequence diagram for async flows

sequenceDiagram
    participant Client
    participant FileUploadService
    participant StorageService
    participant EventPublisher
    participant MessageBroker
    participant VirusScannerConsumer
    participant ThumbnailGeneratorConsumer

    Client->>FileUploadService: Upload file
    FileUploadService->>StorageService: Store file
    FileUploadService->>EventPublisher: Publish file.processed event
    EventPublisher->>MessageBroker: Send to queue
    MessageBroker->>VirusScannerConsumer: Deliver to virus scanner
    MessageBroker->>ThumbnailGeneratorConsumer: Deliver to thumbnail generator

    loop Retry logic
        VirusScannerConsumer->>MessageBroker: Retry on failure
        ThumbnailGeneratorConsumer->>MessageBroker: Retry on failure
    end

    MessageBroker->>VirusScannerConsumer: Deliver to virus scanner
    VirusScannerConsumer->>EventPublisher: Update file status
    EventPublisher->>MessageBroker: Publish file.scanned event
    MessageBroker->>FileUploadService: Deliver to status update handler

    MessageBroker->>ThumbnailGeneratorConsumer: Deliver to thumbnail generator
    ThumbnailGeneratorConsumer->>EventPublisher: Update file status
    EventPublisher->>MessageBroker: Publish file.thumbnail.generated event
    MessageBroker->>FileUploadService: Deliver to status update handler

10. Error Handling Design

Error Classification Hierarchy

Error
├── ValidationException (400)
│   ├── InvalidFileFormatException
│   ├── FileSizeExceededException
│   └── MissingRequiredFieldException
├── ResourceNotFoundException (404)
│   ├── FileNotfoundException
│   └── UserNotFoundException
├── ConflictException (409)
│   ├── FileAlreadyExistsException
│   └── DuplicateUploadException
├── InternalServerException (500)
│   ├── StorageException
│   ├── ProcessingException
│   └── ExternalServiceException
└── UnauthorizedException (401)
    └── InvalidTokenException

Error Propagation Strategy

• Domain-level errors: Caught and converted to appropriate HTTP status codes at service boundary
• Infrastructure errors: Bubble up to global error handler for consistent logging and response formatting
• Validation errors: Handled at API layer with specific error messages to client
• Business logic errors: Translated to domain exceptions and re-thrown for proper HTTP status handling

Retry Strategy with Exponential Backoff and Jitter

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;

    public <T> T executeWithRetry(Supplier<T> operation) {
        int attempt = 0;
        while (true) {
            try {
                return operation.get();
            } catch (StorageException | ExternalServiceException e) {
                if (attempt >= MAX_RETRIES) {
                    throw new RetryExhaustedException("Operation failed after " + MAX_RETRIES + " attempts", e);
                }
                
                long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempt) + 
                                   (Math.random() - 0.5) * 2 * JITTER_FACTOR * BASE_DELAY_MS);
                try {
                    Thread.sleep(delay);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new RetryExhaustedException("Retry interrupted", ie);
                }
                attempt++;
            }
        }
    }
}

Circuit Breaker Design

public class FileUploadCircuitBreaker {
    private final CircuitBreaker circuitBreaker = CircuitBreaker.ofDefaults("file-upload");
    
    public <T> T executeWithCircuitBreaker(Supplier<T> operation) {
        return circuitBreaker.executeSupplier(operation);
    }
    
    // Custom configuration for file upload service
    public static CircuitBreakerConfig createConfig() {
        return CircuitBreakerConfig.custom()
            .failureThreshold(5)
            .waitDurationInOpenState(Duration.ofSeconds(30))
            .permittedNumberOfCallsInHalfOpenState(3)
            .slidingWindowSize(10)
            .build();
    }
}

Fallback Behavior Design

public class FileUploadFallbacks {
    
    public CompletableFuture<UploadResult> fallbackUpload(
            UploadRequest request, 
            Throwable cause) {
        
        if (cause instanceof StorageException || 
            cause instanceof ExternalServiceException) {
            
            // Fallback to local temporary storage with async processing
            return CompletableFuture.completedFuture(
                UploadResult.fallback("Using fallback storage", cause));
        }
        
        // For other errors, rethrow to let higher-level handler deal with it
        return CompletableFuture.failedFuture(cause);
    }
}

Error Logging Format and Alerting Thresholds

public class FileUploadErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(FileUploadErrorLogger.class);
    
    public void logError(String operation, Exception error, Map<String, Object> context) {
        String traceId = MDC.get("trace_id");
        String userId = MDC.get("user_id");
        
        Map<String, Object> logContext = new HashMap<>();
        logContext.put("operation", operation);
        logContext.put("error_code", getErrorCode(error));
        logContext.put("error_message", error.getMessage());
        logContext.put("trace_id", traceId);
        logContext.put("user_id", userId);
        logContext.put("timestamp", Instant.now());
        logContext.putAll(context);
        
        logger.error("File upload error occurred", 
            new ObjectMapper().valueToTree(logContext));
        
        // Alerting logic - send to monitoring system if error rate exceeds threshold
        if (shouldAlert(error)) {
            sendAlertToMonitoringSystem(logContext);
        }
    }
    
    private boolean shouldAlert(Exception error) {
        // Alert if storage errors exceed 10% of total uploads in last 5 minutes
        return error instanceof StorageException;
    }
}

Alerting Thresholds

Metric	Warning Level	Critical Level	Escalation Path
Storage service availability	< 95%	< 90%	On-call engineer
Upload failure rate	> 2%	> 5%	SRE team
External service timeout rate	> 1%	> 3%	Platform engineering
Retry exhaustion rate	> 0.5%	> 1%	SRE team

Error Response Format

{
  "error": {
    "code": "FILE_SIZE_EXCEEDED",
    "message": "File size exceeds maximum allowed limit of 10MB",
    "timestamp": "2023-12-01T10:30:00Z",
    "trace_id": "abc123def456"
  }
}

11. Security Design

Input Validation and Sanitization Strategy

All file upload requests undergo strict input validation using Apache Commons Validator for MIME type checking and size limits. Custom sanitization rules prevent path traversal attacks by rejecting filenames containing .. or / characters. The validation pipeline includes:

• File size limits (max 100MB)
• MIME type whitelist (image/jpeg, image/png, application/pdf)
• Filename length limits (max 255 characters)
• Extension validation against allowed list

SQL Injection Prevention

All database interactions use parameterized prepared statements exclusively. No dynamic SQL string concatenation occurs. The file metadata is stored in a structured format with no raw user input in SQL queries.

Authentication Middleware Design

Authentication is handled via JWT tokens passed in the Authorization header. The middleware validates token signature, expiration, and checks for required scopes (file:upload, file:read). Token refresh is handled via a separate endpoint with proper session management.

Authorization Enforcement

Role-Based Access Control (RBAC) is implemented at the service layer:

• USER role can upload files to their own namespace
• ADMIN role can upload to any namespace and view all files
• File ownership is enforced at service level using the authenticated user ID

Sensitive Data Encryption at Rest and in Transit

• In transit: All uploads use HTTPS (TLS 1.3+) with HSTS header
• At rest: Files are encrypted using AES-256-GCM before storage in object storage (S3-compatible)
• Metadata is stored in encrypted database fields using AES-256-GCM with a key rotation policy every 90 days

PII Handling and Masking Strategy

• User identifiers in filenames are hashed using SHA-256 with salt
• File metadata does not contain PII except for user ID (hashed)
• All file content is treated as potentially sensitive and encrypted

Secrets Management

• API keys are managed via AWS Secrets Manager with automatic rotation every 30 days
• Database credentials are fetched from AWS Secrets Manager at startup
• Encryption keys are rotated using AWS KMS with key rotation policy

Audit Logging Design

All file operations are logged to a structured audit trail:

{
  "timestamp": "2023-10-15T14:30:45Z",
  "user_id": "hashed_user_id",
  "operation": "UPLOAD_FILE",
  "file_id": "uuid",
  "filename": "sanitized_filename",
  "size_bytes": 1024,
  "ip_address": "192.168.1.1",
  "user_agent": "Mozilla/5.0...",
  "status": "SUCCESS"
}

Logging follows the principle of least privilege - sensitive fields are masked in logs. Audit logs are retained for 90 days with daily backups to secure storage.

Security Headers and CORS Configuration

• Content Security Policy (CSP) headers prevent XSS attacks
• X-Content-Type-Options: nosniff to prevent MIME type sniffing
• X-Frame-Options: DENY to prevent clickjacking
• CORS policy restricts origins to whitelisted domains only

Rate Limiting and DDoS Protection

• Per-user rate limiting: 100 uploads/hour
• Per-IP rate limiting: 500 uploads/day
• Burst protection using token bucket algorithm
• Automated DDoS mitigation through AWS WAF integration

Vulnerability Scanning and Dependency Management

• All dependencies are scanned for known vulnerabilities using OWASP Dependency Check
• Docker images built with minimal base OS and security patches applied
• Regular security audits of third-party libraries and frameworks

12. Testing Strategy

Unit Testing Scope and Coverage Targets

Unit testing will target 85% code coverage for business logic classes, with a minimum of 90% coverage for critical paths. Each service class will have dedicated unit tests covering:

• Happy path scenarios with valid inputs
• Boundary condition testing (empty files, max size limits)
• Error condition handling (invalid file types, corrupted files)
• Concurrent access scenarios for shared resources
• Integration with external dependencies (storage, messaging)

Integration Testing Strategy

Integration testing will utilize testcontainers for database and storage system validation:

• Database: PostgreSQL 13+ container with schema migration testing
• Storage: MinIO container for S3-compatible object storage validation
• Messaging: Kafka or RabbitMQ containers for event-driven flow testing
• External Services: WireMock for third-party API fakes (e.g., virus scanning service)

Contract Testing

Contract testing will be implemented using Pact for consumer-driven contracts:

• API Gateway: Consumer contract with frontend services
• Storage Service: Provider contract with file upload service
• Notification Service: Consumer contract for status update events

Performance Testing Tool and Key Scenarios

Performance testing will use k6 with the following key scenarios:

• Concurrent Uploads: 1000 simultaneous uploads of various file sizes (1KB, 1MB, 10MB)
• Throughput Testing: 500 concurrent uploads at 100MB total volume
• Latency Testing: P95 response time under 200ms for 1MB files
• Stress Testing: 2000 concurrent uploads to identify breaking points

Test Data Management Strategy

Test data will be managed through:

• Factories: Java-based factory classes for generating valid test entities (FileMetadata, UploadSession)
• Fixtures: JSON/YAML files for complex test scenarios and edge cases
• Seeding: Database seed scripts for initial test state setup
• Randomization: Faker library for generating realistic but non-sensitive test data

// Example factory pattern for test data generation
public class FileMetadataFactory {
    public static FileMetadata createValidFileMetadata() {
        return FileMetadata.builder()
                .id(UUID.randomUUID())
                .fileName("test-file.txt")
                .fileSize(1024L)
                .contentType("text/plain")
                .status(FileStatus.UPLOADED)
                .createdAt(Instant.now())
                .build();
    }
    
    public static FileMetadata createLargeFileMetadata() {
        return FileMetadata.builder()
                .id(UUID.randomUUID())
                .fileName("large-file.zip")
                .fileSize(1024 * 1024 * 100L) // 100MB
                .contentType("application/zip")
                .status(FileStatus.PROCESSING)
                .createdAt(Instant.now())
                .build();
    }
}

13. Observability Design

Structured Logging Schema

All logs must follow a structured JSON format to enable programmatic parsing and analysis. The schema includes:

{
  "timestamp": "2023-10-27T14:30:00.000Z",
  "level": "INFO|WARN|ERROR",
  "service": "file-upload-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user-123",
  "request_id": "req-456",
  "operation": "uploadFile",
  "file_id": "file-789",
  "file_size_bytes": 1024,
  "status_code": 201,
  "duration_ms": 150,
  "error_code": "FILE_SIZE_EXCEEDED",
  "error_message": "File size exceeds maximum limit of 10MB",
  "stack_trace": "...",
  "ip_address": "192.168.1.100",
  "user_agent": "Mozilla/5.0...",
  "method": "POST",
  "path": "/api/v1/files"
}

PII masking rules:

• User IDs are hashed before logging
• IP addresses are masked to CIDR /24 range (e.g., 192.168.1.0/24)
• User agent strings are truncated to first 50 characters

Distributed Tracing Instrumentation

Using OpenTelemetry with custom spans:

Key Operations to Trace:

1. file.upload.start - Entry point of upload flow
2. file.validation - Validation logic (size, type, etc.)
3. file.storage.write - Actual file write to storage
4. file.metadata.create - Metadata persistence
5. file.processing.async - Async processing triggers (if applicable)

Span Attributes:

• file.id: UUID of the file
• file.name: Original filename (sanitized)
• file.size.bytes: Size in bytes
• file.mime.type: MIME type detected
• storage.provider: S3, GCS, local disk, etc.
• upload.method: Direct upload, chunked, multipart
• user.id: Authenticated user ID (hashed)

Metrics Design

RED Method Metrics:

Metric Name	Type	Labels	Description
file_upload_requests_total	Counter	method, status_code, user_id	Total file upload requests
file_upload_errors_total	Counter	error_type, status_code	Errors during file upload
file_upload_duration_seconds	Histogram	method, user_id	Upload duration in seconds
file_upload_size_bytes	Histogram	method, user_id	File sizes uploaded
storage_write_duration_seconds	Histogram	storage_provider, file_type	Time to write file to storage
storage_read_duration_seconds	Histogram	storage_provider, file_type	Time to read file from storage
active_uploads_count	Gauge	user_id	Concurrent uploads per user

Additional Metrics:

• file_storage_used_bytes (Gauge): Total storage used by files
• file_upload_rate_limit_exceeded_total (Counter): Rate limit hits
• file_validation_failures_total (Counter): Validation errors

Health Check Endpoint Design

/health:
  - Checks database connectivity (500 if failed)
  - Checks storage provider connectivity (500 if failed)
  - Returns 200 OK with status details

/ready:
  - Checks all dependencies (DB, Storage, Cache)
  - Returns 200 OK when service is ready to serve traffic
  - Returns 503 Service Unavailable if any dependency fails

/live:
  - Checks basic service liveness (no DB or storage checks)
  - Returns 200 OK when process is running
  - Returns 503 if process appears hung

/startup:
  - Monitors startup sequence completion
  - Returns 200 OK once all components initialized

Alerting Thresholds

Metric	Warning Level	Critical Level	Alerting Channel
file_upload_errors_total	> 10 errors per minute	> 50 errors per minute	Slack/Email
file_upload_duration_seconds (p99)	> 5 seconds	> 10 seconds	PagerDuty
storage_write_duration_seconds (p99)	> 3 seconds	> 8 seconds	PagerDuty
file_upload_requests_total (rate)	> 10k req/min	> 50k req/min	Slack
active_uploads_count (per user)	> 5 concurrent uploads	> 10 concurrent uploads	Slack
file_storage_used_bytes (utilization)	> 80% of quota	> 95% of quota	PagerDuty

On-call escalation path:

1. Primary engineer (5 min response)
2. Secondary engineer (15 min response)
3. Team lead (30 min response)
4. Engineering manager (60 min response)

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• File retention policy: Should files be deleted after X days/weeks/months or kept indefinitely? What's the SLA for file availability?
• Storage tier selection: Should we use object storage (S3, GCS) or block storage for different file types? What's the cost vs. performance trade-off?
• File size limits: What's the maximum file size we support? How do we handle files that exceed this limit?
• Metadata storage strategy: Should all metadata be stored in database or only essential fields, with file details in object storage?

Known trade-offs and justifications

• Storage vs. Compute Trade-off: Storing file metadata in database vs. object storage - database provides ACID properties but increases query complexity, while object storage offers better scalability and cost efficiency for large files.
• Consistency vs. Availability Trade-off: Using eventual consistency for file metadata vs. strong consistency - eventual consistency provides better availability and performance but may cause temporary inconsistency during concurrent operations.
• Caching vs. Storage Trade-off: Caching file metadata vs. storing in database - caching improves read performance but increases complexity and memory usage.
• Single vs. Multi-tenant Storage: Single storage backend vs. per-tenant storage - single backend reduces complexity but increases security risks and data isolation concerns.

Risks and mitigation strategies

• Storage Exhaustion Risk: Risk of running out of storage space due to unlimited file uploads. Mitigation: Implement storage quotas per user/tenant with alerts and automatic cleanup policies.
• Security Risk: Unauthorized file access or malicious file uploads. Mitigation: Implement content type validation, virus scanning, and proper access control mechanisms.
• Performance Risk: High concurrent upload traffic causing system degradation. Mitigation: Implement rate limiting, connection pooling, and horizontal scaling capabilities.
• Data Loss Risk: File corruption or deletion due to system failures. Mitigation: Implement redundant storage, backup strategies, and proper error handling with retry mechanisms.

Technical debt acknowledged

• File Format Support: Current design assumes generic file handling without specific format validation or processing. Future enhancement needed for content-type specific handling.
• Metadata Indexing: Basic metadata indexing strategy may not scale well with complex search requirements. Consider implementing full-text search capabilities.
• Cross-Region Replication: No built-in cross-region replication strategy for disaster recovery or global availability. This would require additional infrastructure and complexity.
• File Versioning: No support for file versioning or content history tracking. This could be a future enhancement for compliance or collaboration use cases.
• Batch Operations: No support for batch file operations. Future design should consider bulk upload and delete capabilities to improve performance for large-scale operations.