File Metadata Service Low-Level Design

This document presents a production-ready low-level design for a file metadata service that efficiently handles file indexing, search capabilities, and storage optimization while maintaining high availability and scalability across distributed systems.

1. Problem Statement & Requirements

Clarifying questions

1. What scale expectations are there for the file metadata service?

   • How many files are expected to be stored and indexed?
   • What is the peak concurrent read/write operations target?

2. What consistency requirements exist for file metadata?

   • Should reads always return the latest version of metadata?
   • Are there any scenarios where eventual consistency is acceptable?

3. What are the key entities and their relationships?

   • Are files grouped into folders or buckets?
   • What are the primary use cases for file metadata (e.g., search, sharing, versioning)?

4. What is the read/write ratio for metadata operations?

   • Is there a higher volume of reads vs. writes?
   • Are metadata updates frequent or infrequent?

5. What are the latency sensitivity requirements?

   • What are the p50/p95/p99 latency targets for metadata queries?
   • Is sub-100ms response time critical?

6. What authentication and authorization model is in place?

   • Is there an existing auth system (e.g., OAuth2, JWT)?
   • Are files scoped to users or organizations?

7. What external systems integrate with this service?

   • Does the system interact with storage systems (e.g., S3, GCS)?
   • Are there external systems that consume file metadata events?

Functional requirements

• Store and retrieve metadata for files (name, size, type, creation date, owner, tags)
• Support search and filtering by metadata fields (e.g., file type, owner, last modified date)
• Support versioning of file metadata
• Allow tagging and categorization of files
• Support soft deletion of files with metadata retention for audit purposes
• Provide file size and content type information
• Allow querying file metadata by user or organization

Non-functional requirements

• Latency Targets:

  • p50: <50ms
  • p95: <100ms
  • p99: <200ms

• Throughput:

  • 10,000 read operations/second
  • 1,000 write operations/second

• Availability:

  • 99.9% uptime (2.5 min downtime/month)

• Scalability:

  • Support for 100M+ files with horizontal scaling capability

Constraints and assumptions

• Assumption: File content is stored in external storage (e.g., S3, GCS) and metadata is stored separately
• Assumption: Metadata will not include file content or binary data
• Constraint: Metadata must be queryable with sub-second response times
• Constraint: No external dependencies for metadata service (e.g., no database replication or cross-service transactions)
• Assumption: Users will not modify metadata concurrently for the same file

Out of scope

• File content storage or retrieval
• File content encryption or decryption
• Real-time file change notifications (e.g., via webhooks)
• File sharing or permission management (handled by separate services)
• Metadata replication across regions or data centers
• File version restore or rollback functionality
• Metadata synchronization with external systems (e.g., file system sync)

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for the File Metadata service due to:

• Strong typing and schema evolution support with versioning
• Efficient binary serialization for metadata operations
• Built-in support for streaming and bidirectional communication
• Native support for cross-language development (Java, Go, Python)
• Better performance characteristics for bulk metadata operations

Endpoint Definitions

Method	Path	Description	Auth
CreateFileMetadata	/v1/file-metadata	Creates new file metadata entry	JWT
GetFileMetadata	/v1/file-metadata/{fileId}	Retrieves file metadata by ID	JWT
UpdateFileMetadata	/v1/file-metadata/{fileId}	Updates existing file metadata	JWT
DeleteFileMetadata	/v1/file-metadata/{fileId}	Soft-deletes file metadata	JWT
ListFileMetadata	/v1/file-metadata	Lists files with pagination	JWT
SearchFiles	/v1/file-metadata/search	Searches files by metadata fields	JWT
GetFileContent	/v1/file-content/{fileId}	Retrieves file content (streamed)	JWT
UploadFileContent	/v1/file-content/{fileId}	Uploads file content (streamed)	JWT

Request and Response Schemas

CreateFileMetadataRequest

{
  "name": "document.pdf",
  "size": 1024000,
  "contentType": "application/pdf",
  "metadata": {
    "author": "John Doe",
    "tags": ["important", "draft"],
    "category": "documents"
  },
  "ownerId": "user-123",
  "parentId": "folder-456"
}

CreateFileMetadataResponse

{
  "fileId": "file-789",
  "name": "document.pdf",
  "size": 1024000,
  "contentType": "application/pdf",
  "metadata": {
    "author": "John Doe",
    "tags": ["important", "draft"],
    "category": "documents"
  },
  "ownerId": "user-123",
  "parentId": "folder-456",
  "createdAt": "2023-10-15T14:30:00Z",
  "updatedAt": "2023-10-15T14:30:00Z"
}

SearchFilesRequest

{
  "query": {
    "name": "document",
    "contentType": "application/pdf",
    "ownerId": "user-123"
  },
  "sort": {
    "field": "createdAt",
    "direction": "DESC"
  },
  "page": {
    "limit": 20,
    "cursor": "eyJmaWxlSWQiOiJmaWxlLTc4OSJ9"
  }
}

SearchFilesResponse

{
  "files": [
    {
      "fileId": "file-789",
      "name": "document.pdf",
      "size": 1024000,
      "contentType": "application/pdf",
      "metadata": {
        "author": "John Doe",
        "tags": ["important", "draft"],
        "category": "documents"
      },
      "ownerId": "user-123",
      "parentId": "folder-456",
      "createdAt": "2023-10-15T14:30:00Z",
      "updatedAt": "2023-10-15T14:30:00Z"
    }
  ],
  "nextCursor": "eyJmaWxlSWQiOiJmaWxlLTc4OSJ9"
}

HTTP Status Codes and Error Response Format

Status Code	Description
200	Success
201	Created
400	Bad Request (validation errors)
401	Unauthorized
403	Forbidden
404	Not Found
409	Conflict (duplicate)
429	Rate Limited
500	Internal Server Error

Standardized error response:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "File name must not be empty",
    "details": [
      {
        "field": "name",
        "message": "File name must not be empty"
      }
    ]
  }
}

API Versioning Strategy

Versioned using path prefix: /v1/file-metadata and /v1/file-content

Pagination Design

Cursor-based pagination is selected because:

• More consistent and predictable than offset-based
• Performs better with large datasets
• Handles concurrent modifications gracefully
• Provides better performance for deep pagination

Authentication and Authorization

All endpoints require JWT Bearer token with scopes:

• file:read - for read operations
• file:write - for write operations

Rate Limiting

Per-user rate limiting with:

• 1000 requests/minute for read operations
• 100 requests/minute for write operations
• Implementation using Redis with sliding window algorithm

Idempotency Design

Idempotency keys are supported for write operations:

• Header: X-Idempotency-Key
• Value: UUID string
• Implementation uses Redis to store operation state for 5 minutes

gRPC Service Definition (Protocol Buffer)

syntax = "proto3";

package filemetadata;

service FileMetadataService {
  rpc CreateFileMetadata(CreateFileMetadataRequest) returns (CreateFileMetadataResponse);
  rpc GetFileMetadata(GetFileMetadataRequest) returns (GetFileMetadataResponse);
  rpc UpdateFileMetadata(UpdateFileMetadataRequest) returns (UpdateFileMetadataResponse);
  rpc DeleteFileMetadata(DeleteFileMetadataRequest) returns (DeleteFileMetadataResponse);
  rpc ListFileMetadata(ListFileMetadataRequest) returns (ListFileMetadataResponse);
  rpc SearchFiles(SearchFilesRequest) returns (SearchFilesResponse);
  rpc UploadFileContent(stream UploadFileContentRequest) returns (UploadFileContentResponse);
  rpc GetFileContent(GetFileContentRequest) returns (stream GetFileContentResponse);
}

message FileMetadata {
  string file_id = 1;
  string name = 2;
  int64 size = 3;
  string content_type = 4;
  map<string, string> metadata = 5;
  string owner_id = 6;
  string parent_id = 7;
  google.protobuf.Timestamp created_at = 8;
  google.protobuf.Timestamp updated_at = 9;
}

message CreateFileMetadataRequest {
  string name = 1;
  int64 size = 2;
  string content_type = 3;
  map<string, string> metadata = 4;
  string owner_id = 5;
  string parent_id = 6;
  string idempotency_key = 7;
}

message CreateFileMetadataResponse {
  FileMetadata file_metadata = 1;
}

message GetFileMetadataRequest {
  string file_id = 1;
}

message GetFileMetadataResponse {
  FileMetadata file_metadata = 1;
}

message UpdateFileMetadataRequest {
  string file_id = 1;
  string name = 2;
  int64 size = 3;
  string content_type = 4;
  map<string, string> metadata = 5;
  string owner_id = 6;
  string parent_id = 7;
  string idempotency_key = 8;
}

message UpdateFileMetadataResponse {
  FileMetadata file_metadata = 1;
}

message DeleteFileMetadataRequest {
  string file_id = 1;
  string idempotency_key = 2;
}

message DeleteFileMetadataResponse {
  bool deleted = 1;
}

message ListFileMetadataRequest {
  int32 page_size = 1;
  string page_token = 2;
}

message ListFileMetadataResponse {
  repeated FileMetadata files = 1;
  string next_page_token = 2;
}

message SearchFilesRequest {
  message Query {
    string name = 1;
    string content_type = 2;
    string owner_id = 3;
  }
  
  Query query = 1;
  message Sort {
    string field = 1;
    string direction = 2; // ASC or DESC
  }
  
  Sort sort = 2;
  int32 limit = 3;
  string cursor = 4;
}

message SearchFilesResponse {
  repeated FileMetadata files = 1;
  string next_cursor = 2;
}

message UploadFileContentRequest {
  string file_id = 1;
  bytes content = 2;
}

message UploadFileContentResponse {
  bool success = 1;
}

message GetFileContentRequest {
  string file_id = 1;
}

message GetFileContentResponse {
  bytes content = 1;
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    USER {
        string id PK
        string email
        string name
        timestamp created_at
        timestamp updated_at
    }

    FILE {
        string id PK
        string name
        string path
        string mime_type
        int64 size_bytes
        string owner_id FK
        string storage_key
        timestamp created_at
        timestamp updated_at
        timestamp deleted_at
    }

    FILE_TAG {
        string file_id FK
        string tag_name
        timestamp created_at
    }

    TAG {
        string name PK
        string description
        timestamp created_at
    }

    FILE_VERSION {
        string id PK
        string file_id FK
        string storage_key
        int64 size_bytes
        timestamp created_at
    }

    USER ||--o{ FILE : "owns"
    FILE ||--o{ FILE_TAG : "has"
    TAG ||--o{ FILE_TAG : "tagged"
    FILE ||--o{ FILE_VERSION : "has"

Table Schema Definitions

Users Table

CREATE TABLE users (
    id VARCHAR(36) PRIMARY KEY,
    email VARCHAR(255) UNIQUE NOT NULL,
    name VARCHAR(255) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    
    INDEX idx_email (email),
    INDEX idx_created_at (created_at)
);

Files Table

CREATE TABLE files (
    id VARCHAR(36) PRIMARY KEY,
    name VARCHAR(512) NOT NULL,
    path VARCHAR(1024) NOT NULL,
    mime_type VARCHAR(128),
    size_bytes BIGINT UNSIGNED,
    owner_id VARCHAR(36) NOT NULL,
    storage_key VARCHAR(255) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    deleted_at TIMESTAMP NULL,
    
    INDEX idx_owner_id (owner_id),
    INDEX idx_path (path),
    INDEX idx_created_at (created_at),
    INDEX idx_deleted_at (deleted_at),
    INDEX idx_storage_key (storage_key),
    
    FOREIGN KEY (owner_id) REFERENCES users(id) ON DELETE CASCADE
);

Tags Table

CREATE TABLE tags (
    name VARCHAR(128) PRIMARY KEY,
    description TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_created_at (created_at)
);

File-Tag Junction Table

CREATE TABLE file_tags (
    file_id VARCHAR(36),
    tag_name VARCHAR(128),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    PRIMARY KEY (file_id, tag_name),
    INDEX idx_tag_name (tag_name),
    
    FOREIGN KEY (file_id) REFERENCES files(id) ON DELETE CASCADE,
    FOREIGN KEY (tag_name) REFERENCES tags(name) ON DELETE CASCADE
);

File Versions Table

CREATE TABLE file_versions (
    id VARCHAR(36) PRIMARY KEY,
    file_id VARCHAR(36) NOT NULL,
    storage_key VARCHAR(255) NOT NULL,
    size_bytes BIGINT UNSIGNED,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_file_id (file_id),
    INDEX idx_created_at (created_at),
    
    FOREIGN KEY (file_id) REFERENCES files(id) ON DELETE CASCADE
);

Primary Key Strategy

UUID v4 is selected as the primary key strategy for all entities due to:

• Global uniqueness without coordination
• No sequential patterns that could reveal system information
• Support for distributed systems and sharding
• Better performance in high-concurrency environments (no locking on auto-increment)
• Easier to implement cross-service references

Index Design

Index Name	Fields	Type	Purpose
idx_owner_id	owner_id	Single	Fast user file lookup
idx_path	path	Single	Fast file location queries
idx_created_at	created_at	Single	Time-based filtering and sorting
idx_deleted_at	deleted_at	Single	Efficient soft-delete queries
idx_storage_key	storage_key	Single	Fast lookup by storage reference
idx_tag_name	tag_name	Single	Fast tag-based file queries
idx_file_id	file_id	Single	Fast version lookups
idx_owner_path	owner_id, path	Composite	Efficient user file path queries

Data Partitioning / Sharding Strategy

Shard by owner_id (user-based sharding):

• Files are partitioned by user ID to ensure data locality
• Each user's files reside on the same shard for efficient querying
• Shard key: owner_id (UUID)
• Sharding algorithm: Consistent hashing with 64 shards
• Migration strategy: User-based batch migration during maintenance windows

Soft Delete vs. Hard Delete

Soft delete is chosen because:

• Data recovery capability needed for compliance and user errors
• Audit trail preservation required for security and compliance
• Ability to restore files if accidentally deleted
• Support for data retention policies

Audit Trail Design

All metadata changes are logged in an audit table:

CREATE TABLE file_audit_log (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    file_id VARCHAR(36),
    operation_type ENUM('CREATE', 'UPDATE', 'DELETE', 'TAG_ADD', 'TAG_REMOVE') NOT NULL,
    changed_fields JSON,
    old_values JSON,
    new_values JSON,
    user_id VARCHAR(36),
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_file_id (file_id),
    INDEX idx_timestamp (timestamp),
    INDEX idx_user_id (user_id)
);

Data Archival Strategy

Archival Policy:

• Files older than 1 year with no tags or access logs are archived
• Archived files are moved to cold storage (S3 Glacier)
• Metadata is retained in database but marked as archived
• Archive retention: 5 years for compliance

CREATE TABLE file_archive_log (
    file_id VARCHAR(36) PRIMARY KEY,
    archived_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    archive_storage_key VARCHAR(255),
    original_size_bytes BIGINT UNSIGNED,
    archived_by_user_id VARCHAR(36),
    
    INDEX idx_archived_at (archived_at)
);

4. Class Design & Service Layer

classDiagram
    class FileMetadataService {
        +createFileMetadata(FileMetadataRequest) FileMetadataResponse
        +getFileMetadata(String) FileMetadataResponse
        +updateFileMetadata(String, FileUpdateRequest) FileMetadataResponse
        +deleteFileMetadata(String) FileMetadataResponse
        +searchFiles(FileSearchRequest) FileSearchResponse
    }
    
    class FileMetadataRepository {
        +save(FileMetadata) FileMetadata
        +findById(String) Optional~FileMetadata~
        +findByPath(String) Optional~FileMetadata~
        +findAllByOwner(String) List~FileMetadata~
        +deleteById(String)
    }
    
    class FileMetadata {
        -id: String
        -name: String
        -path: String
        -owner: String
        -size: Long
        -contentType: String
        -createdAt: Instant
        -updatedAt: Instant
        -version: Long
        +getId(): String
        +getName(): String
        +getPath(): String
        +getOwner(): String
        +getSize(): Long
        +getContentType(): String
        +getCreatedAt(): Instant
        +getUpdatedAt(): Instant
        +getVersion(): Long
    }
    
    class FileMetadataEventPublisher {
        +publish(FileMetadataEvent)
    }
    
    class FileMetadataEvent {
        -type: String
        -fileId: String
        -filePath: String
        -timestamp: Instant
    }
    
    class FileMetadataValidator {
        +validateCreateRequest(FileMetadataRequest) ValidationResult
        +validateUpdateRequest(FileUpdateRequest) ValidationResult
    }
    
    class FileMetadataIndexer {
        +indexFileMetadata(FileMetadata)
    }
    
    class FileMetadataSearchService {
        +search(FileSearchRequest) List~FileMetadata~
    }
    
    FileMetadataService --> FileMetadataRepository : uses
    FileMetadataService --> FileMetadataEventPublisher : publishes
    FileMetadataService --> FileMetadataValidator : validates
    FileMetadataService --> FileMetadataIndexer : indexes
    FileMetadataService --> FileMetadataSearchService : searches
    
    FileMetadataValidator --> FileMetadata : validates
    FileMetadataIndexer --> FileMetadata : indexes
    
    class FileMetadataRequest {
        -name: String
        -path: String
        -owner: String
        -size: Long
        -contentType: String
    }
    
    class FileUpdateRequest {
        -name: String
        -size: Long
        -contentType: String
    }
    
    class FileSearchRequest {
        -owner: String
        -contentType: String
        -name: String
        -limit: Integer
        -offset: Integer
    }
    
    class FileMetadataResponse {
        -id: String
        -name: String
        -path: String
        -owner: String
        -size: Long
        -contentType: String
        -createdAt: Instant
        -updatedAt: Instant
    }
    
    class FileSearchResponse {
        -files: List~FileMetadataResponse~
        -total: Long
    }
    
    class ValidationResult {
        -valid: Boolean
        -errors: List~String~
    }

stateDiagram-v2
    [*] --> Created
    Created --> Indexed : file indexed
    Indexed --> Available : searchable
    Available --> Deleted : soft deleted
    Deleted --> [*] : permanently removed
    
    state "Available" {
        [*] --> Indexed
        Indexed --> Searchable : indexed and searchable
        Searchable --> Updated : file updated
        Updated --> Indexed : re-indexed
    }
    
    state "Deleted" {
        [*] --> SoftDeleted
        SoftDeleted --> HardDeleted : after retention period
    }

Service Layer Design

The File Metadata service follows a clean architecture pattern with clear separation of concerns:

1. FileMetadataService - Primary service handling business logic and orchestration
2. FileMetadataRepository - Data access layer abstraction
3. FileMetadataValidator - Input validation and business rule enforcement
4. FileMetadataEventPublisher - Event publishing for async processing
5. FileMetadataIndexer - Search index management (e.g., Elasticsearch)
6. FileMetadataSearchService - Search query handling

Business Logic Encapsulation Patterns

Strategy Pattern Implementation for Validation

public interface ValidationStrategy {
    ValidationResult validate(Object request);
}

@Component
public class CreateFileValidationStrategy implements ValidationStrategy {
    private final FileMetadataValidator validator;
    
    @Override
    public ValidationResult validate(Object request) {
        if (!(request instanceof FileMetadataRequest)) {
            return ValidationResult.failure("Invalid request type");
        }
        
        FileMetadataRequest req = (FileMetadataRequest) request;
        List<String> errors = new ArrayList<>();
        
        if (StringUtils.isEmpty(req.getName())) {
            errors.add("File name is required");
        }
        
        if (StringUtils.isEmpty(req.getPath())) {
            errors.add("File path is required");
        }
        
        if (req.getSize() == null || req.getSize() < 0) {
            errors.add("File size must be positive");
        }
        
        if (StringUtils.isEmpty(req.getOwner())) {
            errors.add("Owner is required");
        }
        
        return ValidationResult.builder()
                .valid(errors.isEmpty())
                .errors(errors)
                .build();
    }
}

@Component
public class FileMetadataValidator {
    private final Map<String, ValidationStrategy> strategies;
    
    public FileMetadataValidator(List<ValidationStrategy> strategies) {
        this.strategies = strategies.stream()
                .collect(Collectors.toMap(
                    s -> s.getClass().getSimpleName(),
                    Function.identity()
                ));
    }
    
    public ValidationResult validateCreateRequest(FileMetadataRequest request) {
        return strategies.get("CreateFileValidationStrategy")
                .validate(request);
    }
}

Critical-Path Method Implementation

@Transactional
public FileMetadataResponse createFileMetadata(FileMetadataRequest request) {
    // Validate input
    ValidationResult validation = fileMetadataValidator.validateCreateRequest(request);
    if (!validation.isValid()) {
        throw new ValidationException("Invalid file metadata", validation.getErrors());
    }
    
    // Check for existing file at same path
    Optional<FileMetadata> existing = fileMetadataRepository.findByPath(request.getPath());
    if (existing.isPresent()) {
        throw new DuplicateFileException("File already exists at path: " + request.getPath());
    }
    
    // Create new file metadata
    FileMetadata file = FileMetadata.builder()
            .id(UUID.randomUUID().toString())
            .name(request.getName())
            .path(request.getPath())
            .owner(request.getOwner())
            .size(request.getSize())
            .contentType(request.getContentType())
            .createdAt(Instant.now())
            .updatedAt(Instant.now())
            .version(1L)
            .build();
    
    // Save to database
    FileMetadata saved = fileMetadataRepository.save(file);
    
    // Index for search
    fileMetadataIndexer.indexFileMetadata(saved);
    
    // Publish event for async processing
    FileMetadataEvent event = FileMetadataEvent.builder()
            .type("FILE_CREATED")
            .fileId(saved.getId())
            .filePath(saved.getPath())
            .timestamp(Instant.now())
            .build();
    
    fileMetadataEventPublisher.publish(event);
    
    // Return response
    return FileMetadataResponse.builder()
            .id(saved.getId())
            .name(saved.getName())
            .path(saved.getPath())
            .owner(saved.getOwner())
            .size(saved.getSize())
            .contentType(saved.getContentType())
            .createdAt(saved.getCreatedAt())
            .updatedAt(saved.getUpdatedAt())
            .build();
}

Transaction Boundary Design

• Single transaction scope: All operations within a single business operation (create, update, delete) are wrapped in a single database transaction
• Consistency guarantees: All data mutations happen atomically with proper locking to prevent race conditions
• Distributed transaction boundaries: Events published are handled asynchronously and don't block the primary transaction

Service Interface Contracts

public interface FileMetadataService {
    FileMetadataResponse createFileMetadata(FileMetadataRequest request);
    FileMetadataResponse getFileMetadata(String id);
    FileMetadataResponse updateFileMetadata(String id, FileUpdateRequest request);
    FileMetadataResponse deleteFileMetadata(String id);
    FileSearchResponse searchFiles(FileSearchRequest request);
}

Saga Pattern for Distributed Transactions

For complex operations involving external systems (e.g., file storage system), a saga pattern would be implemented:

public class FileMetadataSaga {
    private final List<SagaStep> steps;
    
    public void execute(SagaContext context) {
        for (SagaStep step : steps) {
            try {
                step.execute(context);
            } catch (Exception e) {
                // Compensate previous steps
                compensateSteps(steps, step);
                throw new SagaExecutionException("Saga failed", e);
            }
        }
    }
    
    private void compensateSteps(List<SagaStep> steps, SagaStep failedStep) {
        // Reverse the effects of completed steps
        for (int i = steps.indexOf(failedStep) - 1; i >= 0; i--) {
            steps.get(i).compensate();
        }
    }
}

Domain Object Design

@Builder
@EqualsAndHashCode(onlyExplicitlyIncluded = true)
public class FileMetadata {
    @EqualsAndHashCode.Include
    private String id;
    
    private String name;
    private String path;
    private String owner;
    private Long size;
    private String contentType;
    private Instant createdAt;
    private Instant updatedAt;
    private Long version;
    
    // Getters and setters...
    
    public boolean isDeleted() {
        return false; // Soft delete implementation
    }
}

The design emphasizes:

• Single Responsibility Principle: Each class has a clear, focused purpose
• Testability: Services are easily unit testable with mocked dependencies
• Scalability: Event-driven architecture enables horizontal scaling
• Maintainability: Clear separation between validation, persistence, and search logic
• Idempotency: Proper handling of concurrent operations through database constraints and optimistic locking

5. Concurrency & Thread Safety

Concurrency Model

The File Metadata service will use a thread pool-based concurrency model with Java's CompletableFuture and ExecutorService for handling asynchronous operations. The service will leverage the Spring Boot's default TaskExecutor with a configurable thread pool size to handle concurrent requests efficiently while avoiding resource exhaustion.

Race Condition Prevention Strategy

Race conditions will be prevented through:

1. Database-level atomic operations using optimistic locking via version fields
2. Application-level synchronization where necessary (e.g., file upload completion tracking)
3. Idempotency checks to handle duplicate requests gracefully
4. Transaction boundaries to ensure atomicity of operations

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected for the following reasons:

• High read-to-write ratio: Metadata service primarily handles reads (file listings, search) with fewer writes
• Scalability: Avoids blocking operations and improves throughput under concurrent access
• Conflict resolution: Conflicts are rare and can be handled gracefully with retry logic
• Simplicity: No need to manage lock acquisition/release lifecycle

Implementation approach:

// Using @Version field in entity for optimistic locking
@Entity
public class FileMetadata {
    @Id
    private String id;
    
    @Version
    private Long version; // Used for optimistic locking
    
    // other fields...
}

Deadlock Prevention Strategy

Deadlock prevention will be implemented through:

1. Consistent ordering of operations: All database operations follow a consistent order (e.g., always acquire locks in ID-sorted order)
2. Timeout configuration: Set reasonable timeouts (5 seconds) for lock acquisition
3. Hierarchical locking: Use a single global lock for metadata operations to prevent cross-entity deadlocks
4. Retry with exponential backoff: Implement retry logic for lock acquisition failures

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency will be handled at the API layer using:

1. Idempotency keys: Clients provide unique keys for idempotent operations
2. Cache-based deduplication: Store recent operation results in Redis with TTL to prevent reprocessing
3. Database uniqueness constraints: Enforce uniqueness at database level where applicable

@Service
public class FileMetadataService {
    
    private final IdempotencyCache idempotencyCache;
    
    @PostMapping("/files")
    public ResponseEntity<FileMetadata> createFile(
            @RequestHeader("Idempotency-Key") String idempotencyKey,
            @RequestBody CreateFileRequest request) {
        
        // Check if this operation was already processed
        String cacheKey = "file_create:" + idempotencyKey;
        FileMetadata cachedResult = idempotencyCache.get(cacheKey);
        
        if (cachedResult != null) {
            return ResponseEntity.ok(cachedResult);
        }
        
        // Process the operation
        FileMetadata result = processCreateFile(request);
        
        // Cache the result for 1 hour
        idempotencyCache.put(cacheKey, result, Duration.ofHours(1));
        
        return ResponseEntity.status(HttpStatus.CREATED).body(result);
    }
}

Thread Safety in Service Layer

All service methods will be thread-safe through:

1. Immutable objects: Value objects and DTOs are immutable
2. Stateless design: Service methods do not maintain state between calls
3. Thread-local storage: For per-request context (e.g., user ID, trace ID)
4. Synchronized collections: When necessary for temporary state management

Concurrency Control in Repository Layer

Repository methods will be designed to:

1. Use transactional boundaries properly to avoid concurrent modification issues
2. Implement proper error handling for optimistic lock failures
3. Use batch operations where applicable to reduce concurrent access points

@Repository
public class FileMetadataRepository {
    
    @Modifying
    @Transactional
    public void updateFileMetadata(FileMetadata metadata) {
        // Database operation with optimistic locking via @Version field
        entityManager.merge(metadata);
    }
    
    @Query("SELECT f FROM FileMetadata f WHERE f.ownerId = :ownerId")
    public List<FileMetadata> findByOwnerId(@Param("ownerId") String ownerId);
}

Retry Strategy for Concurrent Operations

For concurrent operations that fail due to optimistic locking:

1. Exponential backoff: 100ms, 200ms, 400ms, 800ms, 1600ms
2. Max retries: 3 attempts before failing
3. Jitter: Add randomization to prevent thundering herd

public class RetryableOperation {
    
    public <T> T executeWithRetry(Supplier<T> operation, int maxRetries) {
        int attempts = 0;
        while (attempts < maxRetries) {
            try {
                return operation.get();
            } catch (OptimisticLockException e) {
                attempts++;
                if (attempts >= maxRetries) throw e;
                
                // Exponential backoff with jitter
                long delay = (100L * Math.pow(2, attempts)) + ThreadLocalRandom.current().nextInt(50);
                Thread.sleep(delay);
            }
        }
        throw new RuntimeException("Operation failed after retries");
    }
}

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant U as User
    participant API as FileMetadataServiceAPI
    participant Auth as AuthMiddleware
    participant Service as FileMetadataService
    participant DB as Database
    participant Cache as RedisCache
    participant Storage as ObjectStorage

    U->>API: POST /files/upload
    API->>Auth: Validate JWT
    Auth-->>API: Valid token
    API->>Service: uploadFile(request)
    Service->>Cache: Check if file exists (cache-aside)
    Cache-->>Service: Miss
    Service->>DB: Save file metadata (transaction)
    DB-->>Service: Success
    Service->>Storage: Store file blob
    Storage-->>Service: Success
    Service->>Cache: Cache metadata (write-through)
    Cache-->>Service: Success
    Service-->>API: File ID and metadata
    API-->>U: 201 Created with file metadata

sequenceDiagram
    participant U as User
    participant API as FileMetadataServiceAPI
    participant Auth as AuthMiddleware
    participant Service as FileMetadataService
    participant DB as Database
    participant Cache as RedisCache

    U->>API: GET /files/{fileId}
    API->>Auth: Validate JWT
    Auth-->>API: Valid token
    API->>Service: getFileById(fileId)
    Service->>Cache: Get file metadata
    Cache-->>Service: Hit (found in cache)
    Service-->>API: Return cached metadata
    API-->>U: 200 OK with file metadata

sequenceDiagram
    participant U as User
    participant API as FileMetadataServiceAPI
    participant Auth as AuthMiddleware
    participant Service as FileMetadataService
    participant DB as Database
    participant Cache as RedisCache
    participant Storage as ObjectStorage

    U->>API: DELETE /files/{fileId}
    API->>Auth: Validate JWT
    Auth-->>API: Valid token
    API->>Service: deleteFile(fileId)
    Service->>DB: Mark file as deleted (soft delete)
    DB-->>Service: Success
    Service->>Storage: Delete file blob
    Storage-->>Service: Success
    Service->>Cache: Invalidate cache entry
    Cache-->>Service: Success
    Service-->>API: Return success response
    API-->>U: 204 No Content

7. Caching Design

Cache Strategy Selection

For the File Metadata service, we'll use a cache-aside pattern with LRU eviction policy. This approach provides the best balance between performance and consistency for metadata operations where reads significantly outnumber writes.

Cache Key Design

Cache keys will follow a structured namespace pattern:

file-metadata:{fileId}:{version}
file-metadata:by-path:{filePath}:{version}
file-metadata:by-owner:{ownerId}:{version}

TTL Strategy

Data Type	TTL	Reason
File metadata (primary)	5 minutes	Frequent reads, acceptable staleness
File metadata (recently updated)	1 minute	Recent changes require faster propagation
File statistics	30 minutes	Less volatile, expensive to recalculate
User file lists	15 minutes	User session-based, moderate volatility

Cache Eviction Policy

We'll implement LRU (Least Recently Used) eviction with probabilistic early expiry to prevent cache stampedes:

• Primary LRU eviction for memory pressure
• 15% probabilistic early expiry to reduce stampede risk
• Cache warming for hot files during peak hours

Distributed Cache Architecture

Using Redis Cluster with:

• 3 master nodes, 3 replica nodes for HA
• 200MB per node cache size limit
• Redis Sentinel for automatic failover
• Cache sharding based on file ID hash modulo 100

Cache Invalidation Strategy

Event-driven invalidation with the following rules:

1. Write Operations: Immediate cache invalidation for affected keys
2. File Deletion: Remove all related cache entries
3. Metadata Updates: Update TTL to 1 minute for hot files, invalidate old keys
4. Batch Operations: Use cache tags with TTL-based invalidation

Cache Stampede Prevention

Implement distributed locks using Redis SETNX with TTL:

public boolean acquireCacheLock(String key, String lockValue, int ttlSeconds) {
    return redisTemplate.opsForValue().setIfAbsent(
        "lock:" + key, 
        lockValue, 
        Duration.ofSeconds(ttlSeconds)
    );
}

Cache Read-Through Pattern

public FileMetadata getCachedFileMetadata(String fileId) {
    String cacheKey = "file-metadata:" + fileId;
    
    // Try to get from cache first
    FileMetadata metadata = cache.get(cacheKey);
    if (metadata != null) {
        return metadata;
    }
    
    // Acquire distributed lock to prevent stampede
    String lockKey = "lock:" + cacheKey;
    String lockValue = UUID.randomUUID().toString();
    
    if (acquireCacheLock(lockKey, lockValue, 5)) {
        try {
            // Double-check cache after lock acquisition
            metadata = cache.get(cacheKey);
            if (metadata != null) {
                return metadata;
            }
            
            // Load from database and populate cache
            metadata = fileMetadataRepository.findById(fileId);
            if (metadata != null) {
                cache.put(cacheKey, metadata, Duration.ofMinutes(5));
            }
            return metadata;
        } finally {
            releaseCacheLock(lockKey, lockValue);
        }
    }
    
    // Fallback to cache read if lock acquisition fails
    return cache.get(cacheKey);
}

Cache Hit/Miss Metrics

Track cache performance with:

• cache.hits (counter) - increment on cache hit
• cache.misses (counter) - increment on cache miss
• cache.evictions (counter) - increment on LRU eviction
• cache.latency (histogram) - time to get from cache

Cache Warm-up Strategy

Implement background warming for:

• Hot files (accessed > 100 times per hour)
• Recently updated files
• Files accessed in last 5 minutes

This approach ensures optimal performance while maintaining data consistency and preventing cache stampedes.

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and persistence concerns. Each entity has its dedicated repository interface and implementation.

public interface FileMetadataRepository {
    Optional<FileMetadata> findById(UUID id);
    List<FileMetadata> findByUserId(UUID userId);
    List<FileMetadata> findByUserIdAndName(UUID userId, String name);
    FileMetadata save(FileMetadata fileMetadata);
    void delete(UUID id);
    boolean existsById(UUID id);
}

@Repository
public class FileMetadataRepositoryImpl implements FileMetadataRepository {
    private final JdbcTemplate jdbcTemplate;
    private final ObjectMapper objectMapper;

    public FileMetadataRepositoryImpl(JdbcTemplate jdbcTemplate, ObjectMapper objectMapper) {
        this.jdbcTemplate = jdbcTemplate;
        this.objectMapper = objectMapper;
    }

    @Override
    public Optional<FileMetadata> findById(UUID id) {
        String sql = "SELECT * FROM file_metadata WHERE id = ?";
        try {
            return Optional.ofNullable(jdbcTemplate.queryForObject(sql, new FileMetadataRowMapper(), id));
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }

    @Override
    public List<FileMetadata> findByUserId(UUID userId) {
        String sql = "SELECT * FROM file_metadata WHERE user_id = ? ORDER BY created_at DESC";
        return jdbcTemplate.query(sql, new FileMetadataRowMapper(), userId);
    }

    @Override
    public List<FileMetadata> findByUserIdAndName(UUID userId, String name) {
        String sql = "SELECT * FROM file_metadata WHERE user_id = ? AND name = ? ORDER BY created_at DESC";
        return jdbcTemplate.query(sql, new FileMetadataRowMapper(), userId, name);
    }

    @Override
    public FileMetadata save(FileMetadata fileMetadata) {
        if (fileMetadata.getId() == null) {
            return createFileMetadata(fileMetadata);
        } else {
            return updateFileMetadata(fileMetadata);
        }
    }

    private FileMetadata createFileMetadata(FileMetadata fileMetadata) {
        String sql = """
            INSERT INTO file_metadata (
                id, user_id, name, size, mime_type, path, created_at, updated_at
            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
            """;
        UUID id = UUID.randomUUID();
        fileMetadata.setId(id);
        jdbcTemplate.update(sql, 
            id, 
            fileMetadata.getUserId(), 
            fileMetadata.getName(),
            fileMetadata.getSize(),
            fileMetadata.getMimeType(),
            fileMetadata.getPath(),
            fileMetadata.getCreatedAt(),
            fileMetadata.getUpdatedAt()
        );
        return fileMetadata;
    }

    private FileMetadata updateFileMetadata(FileMetadata fileMetadata) {
        String sql = """
            UPDATE file_metadata SET 
                name = ?, size = ?, mime_type = ?, path = ?, updated_at = ?
            WHERE id = ?
            """;
        jdbcTemplate.update(sql,
            fileMetadata.getName(),
            fileMetadata.getSize(),
            fileMetadata.getMimeType(),
            fileMetadata.getPath(),
            fileMetadata.getUpdatedAt(),
            fileMetadata.getId()
        );
        return fileMetadata;
    }

    @Override
    public void delete(UUID id) {
        String sql = "DELETE FROM file_metadata WHERE id = ?";
        jdbcTemplate.update(sql, id);
    }

    @Override
    public boolean existsById(UUID id) {
        String sql = "SELECT COUNT(*) FROM file_metadata WHERE id = ?";
        return jdbcTemplate.queryForObject(sql, Integer.class, id) > 0;
    }
}

ORM vs. Raw Query Decisions

For this service, raw SQL queries are preferred over ORM for several reasons:

1. Performance: Raw queries provide better performance with minimal overhead
2. Control: Direct control over query execution and optimization
3. Complexity: Simple CRUD operations don't benefit from ORM complexity
4. Maintainability: Clear, readable SQL queries are easier to debug and optimize

Read vs. Write Separation

The service uses a single repository for both read and write operations, but with database read replicas for read-heavy workloads:

@Component
public class FileMetadataReadRepository {
    private final JdbcTemplate readJdbcTemplate;

    public FileMetadataReadRepository(JdbcTemplate readJdbcTemplate) {
        this.readJdbcTemplate = readJdbcTemplate;
    }

    public List<FileMetadata> findByUserId(UUID userId) {
        String sql = "SELECT * FROM file_metadata WHERE user_id = ? ORDER BY created_at DESC";
        return readJdbcTemplate.query(sql, new FileMetadataRowMapper(), userId);
    }

    public Optional<FileMetadata> findById(UUID id) {
        String sql = "SELECT * FROM file_metadata WHERE id = ?";
        try {
            return Optional.ofNullable(readJdbcTemplate.queryForObject(sql, new FileMetadataRowMapper(), id));
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }
}

Connection Pool Configuration

Connection pool is configured with:

spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      validation-timeout: 5000
      leak-detection-threshold: 60000

N+1 Query Prevention Strategy

To prevent N+1 query issues, eager loading is used in batch operations:

public class FileMetadataBatchService {
    public List<FileMetadata> getFilesWithUserDetails(List<UUID> fileIds) {
        String sql = """
            SELECT f.*, u.email as user_email 
            FROM file_metadata f 
            JOIN users u ON f.user_id = u.id 
            WHERE f.id IN (?)
            """;
        return jdbcTemplate.query(sql, new FileMetadataWithUserRowMapper(), fileIds);
    }
}

Batch Operation Design

Batch operations are implemented to handle bulk operations efficiently:

public class FileMetadataBatchRepository {
    public void batchSave(List<FileMetadata> fileMetadataList) {
        String sql = """
            INSERT INTO file_metadata (
                id, user_id, name, size, mime_type, path, created_at, updated_at
            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
            """;
        
        jdbcTemplate.batchUpdate(sql, fileMetadataList, fileMetadataList.size(),
            (ps, fileMetadata) -> {
                ps.setObject(1, fileMetadata.getId());
                ps.setObject(2, fileMetadata.getUserId());
                ps.setString(3, fileMetadata.getName());
                ps.setLong(4, fileMetadata.getSize());
                ps.setString(5, fileMetadata.getMimeType());
                ps.setString(6, fileMetadata.getPath());
                ps.setTimestamp(7, Timestamp.valueOf(fileMetadata.getCreatedAt()));
                ps.setTimestamp(8, Timestamp.valueOf(fileMetadata.getUpdatedAt()));
            });
    }
}

Transaction Management

All repository operations are wrapped in proper transaction boundaries:

@Service
@Transactional
public class FileMetadataService {
    private final FileMetadataRepository fileMetadataRepository;
    
    public FileMetadata save(FileMetadata fileMetadata) {
        return fileMetadataRepository.save(fileMetadata);
    }
    
    @Transactional(readOnly = true)
    public List<FileMetadata> findByUserId(UUID userId) {
        return fileMetadataRepository.findByUserId(userId);
    }
}

Query Optimization

Database queries are optimized with proper indexing and query structure:

public class OptimizedFileMetadataRepository {
    // Query with proper index usage
    public List<FileMetadata> findRecentFilesByUser(UUID userId, LocalDateTime afterDate) {
        String sql = """
            SELECT * FROM file_metadata 
            WHERE user_id = ? AND created_at >= ? 
            ORDER BY created_at DESC 
            LIMIT 100
            """;
        return jdbcTemplate.query(sql, new FileMetadataRowMapper(), userId, afterDate);
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The File Metadata service will use asynchronous processing for operations that don't require immediate response to the client, particularly around file processing and metadata extraction. The key boundaries are:

1. File upload completion → trigger metadata extraction pipeline
2. Metadata extraction success/failure → update file status and notify subscribers
3. File lifecycle events (archive, delete) → trigger cleanup operations

Synchronous operations remain for direct read/write of metadata to ensure immediate consistency and response to client requests.

Message schema design

{
  "version": "1.0",
  "event_type": "file.metadata.extracted",
  "timestamp": "2023-10-15T14:30:00Z",
  "payload": {
    "file_id": "uuid-12345",
    "file_path": "/uploads/12345/document.pdf",
    "metadata": {
      "size_bytes": 1024000,
      "mime_type": "application/pdf",
      "created_at": "2023-10-15T14:30:00Z",
      "modified_at": "2023-10-15T14:30:00Z",
      "extracted_fields": {
        "title": "Annual Report 2023",
        "author": "John Doe"
      }
    },
    "status": "SUCCESS|FAILED",
    "error_details": {
      "code": "METADATA_EXTRACTION_ERROR",
      "message": "Failed to parse PDF metadata"
    }
  }
}

Dead letter queue strategy and poison message handling

For failed message processing, we implement a dead-letter queue (DLQ) with:

• Max retry attempts: 3
• Retry backoff: Exponential (1s, 5s, 15s)
• DLQ retention: 7 days
• Poison message detection: Messages that fail 3 times within 1 hour are routed to DLQ

At-least-once vs. exactly-once delivery

We choose at-least-once delivery for file metadata operations because:

• Metadata extraction is idempotent (re-running won't change the result)
• The system handles duplicate events gracefully through state checks
• Performance overhead of exactly-once is not justified for this use case

Retry policy for consumers

public class RetryPolicy {
    private static final int MAX_RETRIES = 3;
    private static final long INITIAL_DELAY_MS = 1000;
    private static final long MAX_DELAY_MS = 30000;
    
    public boolean shouldRetry(int attempt, Exception exception) {
        if (attempt >= MAX_RETRIES) return false;
        
        // Don't retry on certain error types
        if (exception instanceof ValidationException || 
            exception instanceof IllegalArgumentException) {
            return false;
        }
        
        return true;
    }
    
    public long getDelay(int attempt) {
        return Math.min(INITIAL_DELAY_MS * (1L << attempt), MAX_DELAY_MS);
    }
}

Outbox pattern for reliable event publishing

We do implement the outbox pattern to ensure event delivery reliability:

@Entity
@Table(name = "event_outbox")
public class EventOutbox {
    @Id
    private UUID id;
    
    @Column(name = "event_type")
    private String eventType;
    
    @Lob
    @Column(name = "payload")
    private String payload;
    
    @Column(name = "status")
    private EventStatus status = EventStatus.PENDING;
    
    @Column(name = "retry_count")
    private int retryCount = 0;
    
    @Column(name = "created_at")
    private LocalDateTime createdAt = LocalDateTime.now();
    
    @Column(name = "next_retry_at")
    private LocalDateTime nextRetryAt;
}

Sequence diagram for async flows

sequenceDiagram
    participant Client
    participant FileService
    participant EventPublisher
    participant MetadataExtractor
    participant EventConsumer
    
    Client->>FileService: Upload file
    FileService->>EventPublisher: Publish FILE_UPLOADED event
    EventPublisher->>MetadataExtractor: Trigger metadata extraction
    MetadataExtractor-->>EventPublisher: Publish METADATA_EXTRACTED event
    EventPublisher->>EventConsumer: Deliver to subscribers

10. Error Handling Design

Error Classification Hierarchy

The error handling strategy follows a hierarchical classification to enable proper routing and response generation:

Error
├── DomainError (business logic violations)
│   ├── ValidationException
│   ├── BusinessLogicException
│   └── ResourceNotFoundException
├── InfrastructureError (system-level issues)
│   ├── DatabaseException
│   ├── ExternalServiceException
│   └── StorageException
└── ClientError (request-level issues)
    ├── BadRequestException
    ├── UnauthorizedException
    └── ForbiddenException

Error Propagation Strategy

• Domain and Infrastructure errors are caught at service boundaries and converted to appropriate API-level exceptions
• Client errors (4xx) are immediately returned to the client without logging as errors
• Infrastructure and domain errors that propagate up are logged at ERROR level for monitoring and alerting

Retry Strategy with Exponential Backoff and Jitter

For external service calls and transient database issues:

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;

    public <T> T executeWithRetry(Supplier<T> operation) {
        int attempts = 0;
        while (true) {
            try {
                return operation.get();
            } catch (Exception e) {
                if (attempts >= MAX_RETRIES || !isRetryable(e)) {
                    throw new RetryExhaustedException("Operation failed after " + MAX_RETRIES + " attempts", e);
                }
                long delay = calculateBackoffDelay(attempts);
                try {
                    Thread.sleep(delay);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new RetryExhaustedException("Retry interrupted", ie);
                }
                attempts++;
            }
        }
    }

    private long calculateBackoffDelay(int attempt) {
        long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempt));
        // Add jitter to prevent thundering herd
        long jitter = (long) (delay * JITTER_FACTOR * (Math.random() - 0.5) * 2);
        return delay + jitter;
    }

    private boolean isRetryable(Exception e) {
        return e instanceof DatabaseException || 
               e instanceof ExternalServiceException ||
               e instanceof StorageException;
    }
}

Circuit Breaker Design

For external dependencies with high failure rate:

public class CircuitBreaker {
    private final int failureThreshold;
    private final long timeoutMs;
    private final int successThreshold;

    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private long lastFailureTime = 0;

    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }

    public <T> T execute(Supplier<T> operation) throws Exception {
        switch (state) {
            case CLOSED:
                return executeClosed(operation);
            case OPEN:
                return executeOpen(operation);
            case HALF_OPEN:
                return executeHalfOpen(operation);
            default:
                throw new IllegalStateException("Unknown circuit state");
        }
    }

    private <T> T executeClosed(Supplier<T> operation) throws Exception {
        try {
            T result = operation.get();
            onSuccess();
            return result;
        } catch (Exception e) {
            onFailure(e);
            throw e;
        }
    }

    private <T> T executeOpen(Supplier<T> operation) throws Exception {
        if (System.currentTimeMillis() - lastFailureTime > timeoutMs) {
            state = CircuitState.HALF_OPEN;
            return executeHalfOpen(operation);
        }
        throw new CircuitBreakerOpenException("Circuit is open");
    }

    private <T> T executeHalfOpen(Supplier<T> operation) throws Exception {
        try {
            T result = operation.get();
            state = CircuitState.CLOSED;
            failureCount = 0;
            return result;
        } catch (Exception e) {
            state = CircuitState.OPEN;
            lastFailureTime = System.currentTimeMillis();
            throw e;
        }
    }

    private void onSuccess() {
        if (state == CircuitState.HALF_OPEN) {
            failureCount = 0;
        }
    }

    private void onFailure(Exception e) {
        if (state == CircuitState.CLOSED) {
            failureCount++;
            lastFailureTime = System.currentTimeMillis();
            if (failureCount >= failureThreshold) {
                state = CircuitState.OPEN;
            }
        }
    }
}

Fallback Behavior Design

For external services that may be unavailable:

public class FallbackService {
    public FileMetadata getMetadataWithFallback(String fileId) {
        try {
            return fileStorageService.getMetadata(fileId);
        } catch (ExternalServiceException e) {
            // Log fallback trigger
            logger.warn("Falling back to cached metadata for file ID: {}", fileId);
            return metadataCache.get(fileId);
        }
    }

    public List<FileMetadata> listFilesWithFallback(String userId, int limit) {
        try {
            return fileStorageService.listFiles(userId, limit);
        } catch (ExternalServiceException e) {
            logger.warn("Falling back to cached file list for user ID: {}", userId);
            return metadataCache.listCachedFiles(userId, limit);
        }
    }
}

Error Logging Format and Alerting Thresholds

Structured logging with required fields:

{
  "timestamp": "2023-10-27T14:30:00.000Z",
  "trace_id": "abc123def456",
  "span_id": "xyz789uvw012",
  "user_id": "user-12345",
  "entity_id": "file-67890",
  "error_code": "STORAGE_ERROR",
  "error_message": "Failed to read from storage backend",
  "error_type": "InfrastructureError",
  "stack_trace": "...",
  "http_status": 500,
  "service_name": "file-metadata-service"
}

Alerting thresholds:

Error Type	Alert Threshold	Severity
DatabaseException	> 5 occurrences in 5 minutes	CRITICAL
ExternalServiceException	> 10 occurrences in 10 minutes	WARNING
StorageException	> 3 occurrences in 5 minutes	CRITICAL
ValidationException	> 100 occurrences in 1 hour	INFO

Error Response Format

Standardized JSON response for API errors:

{
  "error": {
    "code": "FILE_NOT_FOUND",
    "message": "File with ID 'file-12345' not found",
    "details": {
      "fileId": "file-12345"
    },
    "timestamp": "2023-10-27T14:30:00.000Z"
  }
}

Error Handling in Service Layer

@Service
public class FileMetadataService {
    private final FileStorageService fileStorageService;
    private final MetadataValidator validator;

    public FileMetadata getMetadata(String fileId) {
        try {
            // Validate input
            validator.validateFileId(fileId);
            
            // Attempt to get metadata from storage
            FileMetadata metadata = fileStorageService.getMetadata(fileId);
            
            if (metadata == null) {
                throw new ResourceNotFoundException("File metadata not found", fileId);
            }
            
            return metadata;
        } catch (DatabaseException e) {
            logger.error("Database error while fetching file metadata: {}", fileId, e);
            throw new InfrastructureError("Failed to retrieve file metadata", e);
        } catch (StorageException e) {
            logger.error("Storage error while fetching file metadata: {}", fileId, e);
            throw new InfrastructureError("Failed to read file metadata from storage", e);
        } catch (ValidationException e) {
            logger.warn("Validation error for file ID: {}", fileId, e);
            throw new BadRequestException("Invalid file ID format", e);
        }
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All file metadata inputs undergo strict validation using Apache Commons Validator for format checks and OWASP Java HTML Sanitizer for preventing XSS in metadata fields like descriptions, tags, and file names. Custom validation rules enforce:

• File name length ≤ 255 characters
• File size limits (max 10GB for uploaded files)
• Allowed MIME types from predefined whitelist
• Tag count ≤ 50 with each tag ≤ 100 characters

SQL Injection Prevention

All database queries use parameterized prepared statements exclusively. ORM framework (Hibernate) is configured with strict validation and automatic escaping of user inputs. Stored procedures are avoided for metadata operations due to complexity in maintaining input sanitization.

Authentication Middleware Design

Authentication is handled via JWT tokens with the following flow:

1. OAuth2 authorization code flow for user login
2. JWT token generation with 15-minute expiration and refresh token rotation
3. Middleware validates JWT signature, expiration, and user session state
4. Token refresh endpoint requires client secret validation

Authorization Enforcement

Role-based access control (RBAC) is implemented with the following permissions:

• FILE_READ: Read file metadata and content (for public files)
• FILE_WRITE: Create/update/delete file metadata
• FILE_ADMIN: Full access including sharing and permissions management

Permission checks occur at service layer using a centralized authorization service that validates user roles against file ownership and sharing policies.

Sensitive Data Encryption

• At-rest encryption: AES-256-GCM using AWS KMS for file metadata and content storage
• In-transit encryption: TLS 1.3 with HSTS header enforcement for all API endpoints
• PII encryption: Sensitive fields like user IDs and file paths are encrypted using AES-GCM with key rotation every 90 days

PII Handling and Masking Strategy

• User identifiers in metadata are hashed using SHA-256 with salt for anonymization
• File paths and names containing PII are redacted in logs and audit trails
• Personal identifiers are never stored in metadata tables; only referenced via hashed IDs

Secrets Management

Secrets (API keys, database passwords, encryption keys) are managed through AWS Secrets Manager with:

• Automatic rotation every 90 days for database credentials
• IAM roles with least-privilege access to secrets
• Client-side encryption of secrets before storage in memory

Audit Logging Design

All security-relevant events are logged with:

• Event type (login, file access, metadata change)
• Timestamp and user identifier (hashed)
• IP address and session ID
• Action outcome (success/failure)

Log format follows structured JSON schema with fields:

{
  "timestamp": "2023-10-27T14:30:00Z",
  "user_id": "hashed_user_id",
  "event_type": "FILE_ACCESS",
  "ip_address": "192.168.1.100",
  "action": "READ",
  "result": "SUCCESS",
  "file_id": "uuid"
}

Retention policy: audit logs stored for 180 days with daily backups to secure S3 bucket.

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

• Service Layer: 85% code coverage, focusing on business logic and edge cases
• Repository Layer: 90% coverage with mock database interactions
• Domain Objects: 100% coverage for value objects and immutable entities
• Exception Handling: 100% coverage for all error paths in service methods
• Validation Logic: 95% coverage for input validation and constraints

Integration Testing Strategy

• Database Integration: Test with embedded H2 in-memory database for fast execution and isolation
• External Service Mocks: Use WireMock or Mockito to simulate file storage service (e.g., S3, GCS) interactions
• Event Publisher Integration: Validate event publishing to Kafka/RabbitMQ using embedded message brokers or test containers
• File System Operations: Use temporary file system for real I/O operations during integration tests

Contract Testing

• API Contract Tests: Use Pact for consumer-driven contract testing between File Metadata service and file storage service
• Event Schema Contracts: Validate event payloads using JSON Schema or Avro schema registry with contract testing tools
• Version Compatibility: Test backward compatibility of API endpoints and event formats

Performance Testing Tool and Key Scenarios

• Load Testing: Use k6 for load testing with concurrent user simulations (1000+ concurrent users)
• Stress Testing: JMeter for stress testing under high throughput (5000+ requests/sec)
• Key Scenarios:
  • Concurrent file metadata creation (1000 concurrent requests)
  • Bulk metadata retrieval (1000 files in single query)
  • File metadata search with filters (by size, type, date range)
  • High-volume file upload and metadata indexing pipeline

Test Data Management Strategy

• Fixtures: Predefined JSON fixtures for common file metadata structures (e.g., image, video, document)
• Factories: Factory classes for generating realistic test data (e.g., FileMetadataFactory with random but valid values)
• Seeding: Use database seed scripts for initial test data population in test environments
• Data Isolation: Each test runs in isolated database transaction with automatic rollback
• Test Data Generation: Use libraries like Faker or JavaPoet to generate realistic but non-sensitive test data

// Example unit test for metadata validation
@Test
public void testFileMetadataValidation_InvalidSize() {
    FileMetadata metadata = new FileMetadata();
    metadata.setSize(-1L); // Invalid negative size
    
    assertThrows(ValidationException.class, () -> {
        fileMetadataService.createMetadata(metadata);
    });
}

// Example integration test with embedded database
@Test
public void testCreateFileMetadata_Success() {
    FileMetadata metadata = FileMetadataFactory.createValidMetadata();
    
    FileMetadata created = fileMetadataService.createMetadata(metadata);
    
    assertThat(created.getId()).isNotNull();
    assertThat(created.getCreatedAt()).isNotNull();
    assertThat(created.getName()).isEqualTo(metadata.getName());
}

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective search, filtering, and correlation. The structured logging format includes:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO",
  "service": "file-metadata-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user-12345",
  "entity_id": "file-98765",
  "operation": "createFileMetadata",
  "request_id": "req-abc123",
  "message": "File metadata created successfully",
  "duration_ms": 45,
  "error_code": null,
  "error_message": null,
  "stack_trace": null
}

PII masking rules:

• User IDs and entity IDs are logged as obfuscated (e.g., user-12345 → user-***)
• File content is never logged
• Sensitive fields like file paths should be redacted or masked

Distributed Tracing Instrumentation

We use OpenTelemetry for distributed tracing to track requests across service boundaries.

Custom Spans:

• file_metadata.create - for file creation flow
• file_metadata.get - for file retrieval
• file_metadata.update - for metadata updates
• file_metadata.delete - for soft-delete operations

Attributes captured per span:

span.setAttribute("file_id", fileMetadata.getId());
span.setAttribute("user_id", userId);
span.setAttribute("file_size_bytes", fileMetadata.getSize());
span.setAttribute("file_type", fileMetadata.getType());
span.setAttribute("operation", "create");

Metrics Design

We implement RED (Rate, Errors, Duration) monitoring at service level:

Counters:

• file_metadata_requests_total - Counter for all file metadata operations (labels: operation, status)
• file_metadata_errors_total - Counter for failed operations (labels: operation, error_type)

Histograms:

• file_metadata_duration_seconds - Histogram for operation durations (labels: operation)
• file_metadata_size_bytes - Histogram for file sizes (labels: operation)

Gauges:

• file_metadata_count - Gauge for current number of metadata entries (labels: type)

Label Design Examples:

// Rate metric with labels
Counter<Long> fileRequests = meter.counterBuilder("file_metadata_requests_total")
    .setDescription("Total file metadata operations")
    .build();

// Histogram with operation labels
Histogram<Double> fileDuration = meter.histogramBuilder("file_metadata_duration_seconds")
    .setDescription("Duration of file metadata operations in seconds")
    .build();

Health Check Endpoint Design

We implement three standard health check endpoints:

/health - Application liveness

• Checks if the service is running and responding to requests
• Returns 200 OK with payload: {"status": "healthy", "timestamp": "..."}

/ready - Readiness check

• Verifies that service is ready to serve traffic (DB connected, cache available)
• Returns 200 OK when ready, 503 Service Unavailable otherwise

/live - Readiness check (optional)

• Checks if service is live and not in a degraded state
• Returns 200 OK with payload: {"status": "alive", "timestamp": "..."}

/startup - Startup validation

• Validates that all required dependencies are available and functional
• Returns 200 OK when successful, 503 Service Unavailable on failure

Alerting Thresholds

Critical Alerts (on-call escalation):

• file_metadata_errors_total > 10 per minute for 5 minutes → Critical (escalate to on-call)
• file_metadata_duration_seconds p99 > 5 seconds → Warning (escalate to on-call)
• file_metadata_requests_total rate drops below 50% of baseline for 10 minutes → Warning (escalate to on-call)

Warning Alerts:

• file_metadata_errors_total > 5 per minute for 2 minutes → Warning (notify team)
• file_metadata_duration_seconds p99 > 3 seconds → Warning (notify team)
• Cache hit ratio < 80% for 5 minutes → Warning (notify team)

Service Level Indicators:

• SLI: 99.9% of file metadata operations complete within 500ms
• SLO: 99.9% availability for file metadata service
• Error Budget: 0.1% of operations allowed to fail

All alerts are configured with Slack integration and escalation policies that notify on-call engineers within 5 minutes of threshold breach.

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• File format support scope: Should we support all file types or limit to specific formats (e.g., images, documents, videos)? This impacts storage strategy and metadata extraction complexity.
• Metadata enrichment strategy: Should we enrich metadata with external services (e.g., OCR for images, EXIF data extraction) or keep it minimal? External integrations increase latency and failure surface.
• Storage backend choice: Should we use object storage (S3-compatible) or local filesystem? This affects scalability, cost, and data durability requirements.
• Versioning strategy: Should we support file versioning with metadata tracking? This adds complexity to the data model and API design.
• Cross-region replication: Should metadata be replicated across regions for low-latency access? This increases infrastructure cost and consistency complexity.

Known trade-offs and justifications

Trade-off	Option 1	Option 2	Justification
Storage vs. Compute	Store raw metadata in DB, compute derived fields	Pre-compute and store all derived metadata	Option 1 reduces storage cost but increases query complexity; Option 2 improves read performance at cost of storage and compute overhead
Consistency vs. Availability	Strong consistency for all metadata operations	Eventual consistency with optimistic locking	Option 1 provides better data integrity but may impact availability under high load; Option 2 improves availability at cost of potential race conditions
Indexing vs. Storage	Index all metadata fields for fast querying	Index only frequently queried fields	Option 1 provides better query performance but increases storage and write overhead; Option 2 reduces storage costs but may require full table scans for some queries
Caching vs. Freshness	Cache metadata with TTL-based invalidation	Cache metadata with event-driven invalidation	Option 1 provides better performance but risks serving stale data; Option 2 ensures freshness but increases complexity of cache invalidation logic

Risks and mitigation strategies

1. Metadata inconsistency risk: Risk of metadata becoming inconsistent with actual file content due to asynchronous processing or concurrent updates.

   • Mitigation: Implement write-through caching, validate metadata against file content on read, and use transactional operations where possible.

2. High storage cost risk: Risk of excessive metadata storage due to file versioning, rich metadata fields, or inefficient data types.

   • Mitigation: Implement storage quotas, use compression for metadata fields, and design a data retention policy with automatic archival.

3. Performance degradation risk: Risk of slow query performance as metadata volume grows.

   • Mitigation: Implement proper indexing strategy, use read replicas for heavy query loads, and design pagination for large result sets.

4. External dependency failure risk: Risk of downstream services (e.g., OCR, file type detection) failing and blocking metadata operations.

   • Mitigation: Implement circuit breaker patterns, fallback to basic metadata when external services fail, and use asynchronous processing for non-critical enrichment.

5. Data loss risk: Risk of losing metadata due to system failures or accidental deletion.

   • Mitigation: Implement robust backup strategies, audit logging, and soft-delete design with recovery mechanisms.

Technical debt acknowledged

1. Hardcoded metadata extraction logic: Current implementation assumes specific file type handlers, making future extensibility difficult.
2. Limited metadata field validation: Basic validation is implemented but more comprehensive validation rules are not yet in place.
3. No metadata schema evolution support: Changes to metadata structure require careful migration handling without breaking existing clients.
4. Inconsistent error handling across service boundaries: Some operations return detailed error messages while others provide generic responses.
5. Lack of comprehensive audit trail: While basic audit logging exists, more granular tracking of metadata changes and user actions is not yet implemented.
6. No support for hierarchical metadata: Current design does not support nested or hierarchical metadata structures that may be needed in future use cases.