Document Management System LLD Design

Comprehensive low-level design for a scalable document management system with version control, search, and access management capabilities.

PublishedJuly 27, 2025
Length8,812 words
Reading time45 min read

Backend LLD

This document presents a detailed low-level design for a scalable document management system that handles document storage, versioning, search functionality, and fine-grained access control across distributed systems.

1. Problem Statement & Requirements

Clarifying questions

What scale and throughput expectations are we targeting?
- How many concurrent users?
- What's the expected document size distribution (e.g., 1KB to 100MB+)?
- What's the peak ingestion rate (documents per second)?
What consistency and availability requirements exist?
- Is strong consistency required for document metadata (e.g., versioning, access control)?
- What is the acceptable downtime for the system?
- Are there any compliance or audit requirements?
What are the key entities and their relationships?
- Do users have roles (admin, editor, viewer)?
- Are documents part of folders/hierarchical structures?
- Is there a concept of document sharing or collaboration?
What is the read/write ratio and latency sensitivity?
- Is the system primarily used for document creation or retrieval?
- What are the latency targets (e.g., p50 < 100ms, p99 < 2s)?
- Are there real-time collaboration or live preview requirements?
What authentication and authorization models are in use?
- Is the system integrated with SSO (e.g., OAuth2, SAML)?
- Are permissions granular (e.g., per-document or per-folder)?
- Is multi-tenancy required?
What external integrations or dependencies exist?
- Is there integration with storage systems (e.g., S3, GCS)?
- Are there third-party document processing services (e.g., OCR, conversion)?
- Is there an existing identity provider or directory service?
What features are out of scope?
- Are document versioning, history tracking, or search capabilities in-scope?
- Is there a requirement for document preview (PDF, image) or metadata extraction?

Functional requirements

Users can upload documents to the system
Users can organize documents into folders (hierarchical structure)
Users can search for documents by name, content, or metadata
Users can view and download documents
Users can edit document metadata (e.g., title, description)
Users can delete documents (soft or hard delete)
Users can share documents with other users or groups
System supports versioning of documents (optional)
Users can set access permissions (read, write, admin) per document or folder

Non-functional requirements

Latency targets:
- p50: < 100ms for metadata operations
- p95: < 500ms for metadata operations
- p99: < 2s for metadata operations
Throughput:
- 10,000 concurrent users
- 500 document uploads/second
- 10,000 document downloads/second
Availability:
- 99.9% uptime (5-minute SLA)
- Support for multi-region deployment

Constraints and assumptions

Documents are stored in a cloud object storage (e.g., AWS S3 or GCP Cloud Storage)
Metadata is stored in a relational database (e.g., PostgreSQL)
Authentication handled by an external identity provider (e.g., Auth0 or Okta)
Users are authenticated via JWT tokens
Document content is not stored in the database, only metadata and references

Out of scope

Full-text search engine integration (e.g., Elasticsearch or Solr)
Real-time collaborative editing of documents
Document encryption at rest (handled by storage provider)
Multi-language support for UI or content
Document workflow automation or approval processes
Mobile app integration (only web and API)

2. API Design

API Style Selection

REST is selected for this document management system due to its simplicity, wide adoption, and clear semantics for CRUD operations. The system will support standard HTTP methods (GET, POST, PUT, DELETE) with JSON payloads for maximum interoperability and ease of integration.

Endpoint Definitions

Method	Path	Description	Auth
GET	`/api/v1/documents`	List documents with filtering and pagination	JWT
POST	`/api/v1/documents`	Create new document	JWT
GET	`/api/v1/documents/{id}`	Get specific document by ID	JWT
PUT	`/api/v1/documents/{id}`	Update document metadata	JWT
DELETE	`/api/v1/documents/{id}`	Delete document (soft delete)	JWT
POST	`/api/v1/documents/{id}/versions`	Create new version of document	JWT
GET	`/api/v1/documents/{id}/versions`	List all versions of document	JWT
GET	`/api/v1/documents/{id}/versions/{versionId}`	Get specific version	JWT
POST	`/api/v1/documents/{id}/share`	Share document with user	JWT
GET	`/api/v1/documents/{id}/shares`	List shares for document	JWT
POST	`/api/v1/documents/{id}/lock`	Lock document for editing	JWT
DELETE	`/api/v1/documents/{id}/lock`	Release document lock	JWT

Request and Response Schemas

Document Creation Request

{
  "name": "Annual Report Q4.pdf",
  "description": "Quarterly financial report for 2023",
  "folderId": "f1234567-89ab-cdef-0123-456789abcdef",
  "fileType": "PDF",
  "size": 1024000,
  "metadata": {
    "author": "John Doe",
    "tags": ["finance", "annual-report"]
  }
}

Document Response

{
  "id": "d1234567-89ab-cdef-0123-456789abcdef",
  "name": "Annual Report Q4.pdf",
  "description": "Quarterly financial report for 2023",
  "folderId": "f1234567-89ab-cdef-0123-456789abcdef",
  "fileType": "PDF",
  "size": 1024000,
  "version": 1,
  "createdAt": "2023-10-15T14:30:00Z",
  "updatedAt": "2023-10-15T14:30:00Z",
  "createdBy": "user-123",
  "lastModifiedBy": "user-123",
  "metadata": {
    "author": "John Doe",
    "tags": ["finance", "annual-report"]
  },
  "lock": null,
  "shares": []
}

Document Version Response

{
  "id": "v1234567-89ab-cdef-0123-456789abcdef",
  "documentId": "d1234567-89ab-cdef-0123-456789abcdef",
  "version": 2,
  "name": "Annual Report Q4.pdf",
  "fileType": "PDF",
  "size": 1024000,
  "createdAt": "2023-10-15T14:30:00Z",
  "createdBy": "user-123"
}

{
  "userId": "user-456",
  "permission": "READ"
}

HTTP Status Codes and Error Response Format

Status Code	Description
200	Success
201	Created
400	Bad Request
401	Unauthorized
403	Forbidden
404	Not Found
409	Conflict (e.g., document already locked)
422	Unprocessable Entity (validation errors)
429	Rate Limited
500	Internal Server Error

Standardized error response format:

{
  "error": {
    "code": "DOCUMENT_NOT_FOUND",
    "message": "Document with ID d1234567-89ab-cdef-0123-456789abcdef not found",
    "details": {
      "documentId": "d1234567-89ab-cdef-0123-456789abcdef"
    }
  }
}

API Versioning Strategy

Versioning will be done via path prefix (/api/v1/) to allow for clean migration paths and backward compatibility. Major version changes will require explicit client-side updates.

Pagination Design

Cursor-based pagination is selected over offset-based due to better performance characteristics at scale, particularly with large datasets and frequent updates. It provides consistent results even when items are inserted or deleted between paginated requests.

Authentication and Authorization

JWT tokens will be used for authentication, with scopes defining access levels. RBAC will be implemented at the service layer to enforce permissions:

DOCUMENT_CREATE: Create documents
DOCUMENT_READ: Read documents
DOCUMENT_UPDATE: Update documents
DOCUMENT_DELETE: Delete documents
DOCUMENT_SHARE: Share documents

Rate Limiting

Rate limiting will be implemented per endpoint:

Document creation: 100 requests/minute
Document read/list: 500 requests/minute
Document update/delete: 50 requests/minute
Share operations: 20 requests/minute

Rate limiting will be enforced using a token bucket algorithm with Redis for distributed state management.

Idempotency Design

For mutation endpoints, idempotency keys will be supported via X-Idempotency-Key header. The system will store the result of the first request with a given key and return the same result for subsequent requests with the same key, ensuring that retries do not cause duplicate operations.

@PostMapping("/documents")
public ResponseEntity<Document> createDocument(
    @RequestHeader("X-Idempotency-Key") String idempotencyKey,
    @RequestBody CreateDocumentRequest request) {
    
    // Check if this idempotency key has been used before
    Document existing = documentService.findExistingDocument(idempotencyKey);
    if (existing != null) {
        return ResponseEntity.ok(existing);
    }
    
    // Process the request
    Document document = documentService.createDocument(request);
    
    // Store the result for future idempotent requests
    documentService.storeIdempotencyResult(idempotencyKey, document);
    
    return ResponseEntity.status(HttpStatus.CREATED).body(document);
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    USER {
        string id PK
        string email
        string name
        timestamp created_at
        timestamp updated_at
        boolean is_active
    }

    DOCUMENT {
        string id PK
        string title
        string description
        string file_key
        string file_url
        string file_type
        integer file_size
        string version
        string status
        string created_by FK
        string updated_by FK
        timestamp created_at
        timestamp updated_at
        timestamp deleted_at
    }

    DOCUMENT_VERSION {
        string id PK
        string document_id FK
        string file_key
        string file_url
        string file_type
        integer file_size
        string version
        timestamp created_at
        string created_by FK
    }

    FOLDER {
        string id PK
        string name
        string parent_folder_id FK
        string created_by FK
        timestamp created_at
        timestamp updated_at
    }

    DOCUMENT_FOLDER {
        string document_id FK
        string folder_id FK
        timestamp created_at
    }

    DOCUMENT_ACCESS {
        string id PK
        string document_id FK
        string user_id FK
        string permission_level
        timestamp created_at
        timestamp updated_at
    }

    DOCUMENT_TAG {
        string id PK
        string document_id FK
        string tag_name
        timestamp created_at
    }

    TAG {
        string id PK
        string name
        string created_by FK
        timestamp created_at
    }

    USER_TEAM {
        string user_id FK
        string team_id FK
        timestamp created_at
    }

    TEAM {
        string id PK
        string name
        string description
        timestamp created_at
        timestamp updated_at
    }

    USER {
        string id PK
        string email
        string name
        timestamp created_at
        timestamp updated_at
        boolean is_active
    }

    DOCUMENT ||--o{ DOCUMENT_VERSION : "has"
    DOCUMENT ||--o{ DOCUMENT_FOLDER : "belongs to"
    DOCUMENT ||--o{ DOCUMENT_ACCESS : "has access"
    DOCUMENT ||--o{ DOCUMENT_TAG : "has tag"
    DOCUMENT_FOLDER ||--o{ DOCUMENT_FOLDER : "nested"
    DOCUMENT_FOLDER ||--o{ DOCUMENT_FOLDER : "contains"
    DOCUMENT_ACCESS ||--o{ USER : "granted to"
    DOCUMENT_TAG ||--o{ TAG : "references"
    USER ||--o{ DOCUMENT_ACCESS : "granted access"
    USER ||--o{ DOCUMENT : "created"
    USER ||--o{ DOCUMENT_VERSION : "created"
    USER ||--o{ FOLDER : "created"
    USER ||--o{ TAG : "created"
    USER ||--o{ USER_TEAM : "belongs to"
    TEAM ||--o{ USER_TEAM : "has members"

Table Schema Definitions

-- Users table
CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email VARCHAR(255) UNIQUE NOT NULL,
    name VARCHAR(255) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    is_active BOOLEAN DEFAULT TRUE
);

-- Teams table
CREATE TABLE teams (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL,
    description TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- User-Team mapping table
CREATE TABLE user_teams (
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    team_id UUID REFERENCES teams(id) ON DELETE CASCADE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (user_id, team_id)
);

-- Folders table
CREATE TABLE folders (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL,
    parent_folder_id UUID REFERENCES folders(id) ON DELETE CASCADE,
    created_by UUID REFERENCES users(id) ON DELETE SET NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Documents table
CREATE TABLE documents (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    title VARCHAR(500) NOT NULL,
    description TEXT,
    file_key VARCHAR(500) NOT NULL,
    file_url TEXT NOT NULL,
    file_type VARCHAR(100),
    file_size INTEGER,
    version VARCHAR(50) DEFAULT '1.0',
    status VARCHAR(50) DEFAULT 'ACTIVE',
    created_by UUID REFERENCES users(id) ON DELETE SET NULL,
    updated_by UUID REFERENCES users(id) ON DELETE SET NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    deleted_at TIMESTAMP
);

-- Document versions table
CREATE TABLE document_versions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID REFERENCES documents(id) ON DELETE CASCADE,
    file_key VARCHAR(500) NOT NULL,
    file_url TEXT NOT NULL,
    file_type VARCHAR(100),
    file_size INTEGER,
    version VARCHAR(50) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    created_by UUID REFERENCES users(id) ON DELETE SET NULL
);

-- Document-Folder mapping table
CREATE TABLE document_folders (
    document_id UUID REFERENCES documents(id) ON DELETE CASCADE,
    folder_id UUID REFERENCES folders(id) ON DELETE CASCADE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (document_id, folder_id)
);

-- Document access permissions table
CREATE TABLE document_access (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID REFERENCES documents(id) ON DELETE CASCADE,
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    permission_level VARCHAR(50) NOT NULL, -- 'READ', 'WRITE', 'MANAGE'
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Tags table
CREATE TABLE tags (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) UNIQUE NOT NULL,
    created_by UUID REFERENCES users(id) ON DELETE SET NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Document tags mapping table
CREATE TABLE document_tags (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID REFERENCES documents(id) ON DELETE CASCADE,
    tag_name VARCHAR(255) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Primary Key Strategy

UUIDs are used as primary keys for all entities:

Justification: UUIDs provide global uniqueness across distributed systems, eliminate the need for coordination between services, and support horizontal scaling without key conflicts.
Advantages:
- No need for centralized ID generation
- Support for distributed systems and microservices
- Better security (no sequential patterns)
Disadvantages:
- Larger storage footprint (~36 bytes vs. 8 bytes for integers)
- Potential performance impact on indexing (but mitigated with proper indexing)

Index Design

-- Indexes for performance optimization

-- Users table indexes
CREATE INDEX idx_users_email ON users(email);
CREATE INDEX idx_users_active ON users(is_active);

-- Folders table indexes
CREATE INDEX idx_folders_parent ON folders(parent_folder_id);
CREATE INDEX idx_folders_created_by ON folders(created_by);

-- Documents table indexes
CREATE INDEX idx_documents_created_by ON documents(created_by);
CREATE INDEX idx_documents_updated_by ON documents(updated_by);
CREATE INDEX idx_documents_status ON documents(status);
CREATE INDEX idx_documents_created_at ON documents(created_at);
CREATE INDEX idx_documents_title ON documents(title);
CREATE INDEX idx_documents_file_key ON documents(file_key);

-- Document versions table indexes
CREATE INDEX idx_document_versions_document_id ON document_versions(document_id);
CREATE INDEX idx_document_versions_version ON document_versions(version);

-- Document-Folder mapping indexes
CREATE INDEX idx_document_folders_folder_id ON document_folders(folder_id);

-- Document access table indexes
CREATE INDEX idx_document_access_document_id ON document_access(document_id);
CREATE INDEX idx_document_access_user_id ON document_access(user_id);

-- Tags table indexes
CREATE INDEX idx_tags_name ON tags(name);
CREATE INDEX idx_document_tags_document_id ON document_tags(document_id);
CREATE INDEX idx_document_tags_tag_name ON document_tags(tag_name);

-- Composite indexes for common queries
CREATE INDEX idx_documents_created_by_status ON documents(created_by, status);
CREATE INDEX idx_document_access_user_document ON document_access(user_id, document_id);

Data Partitioning / Sharding Strategy

Sharding by User ID (Hash-based):

Strategy: Documents are partitioned by user ID hash modulo N shards
Justification:
- Ensures even distribution of data across shards
- Supports horizontal scaling without hotspots
- Maintains consistency for user-specific operations

Implementation:

-- Example sharding key derivation (in application logic)
-- shard_id = hash(user_id) % NUMBER_OF_SHARDS

Soft Delete vs. Hard Delete Decision

Soft Delete is used:

Justification:
- Enables audit trail and recovery capability
- Maintains referential integrity for analytics and reporting
- Supports version control and history tracking
Implementation:
- deleted_at timestamp field on documents table
- Soft-delete queries filter out records where deleted_at IS NOT NULL

Audit Trail Design

Audit Fields:

All tables include:
- created_at (timestamp)
- updated_at (timestamp)
- created_by (user ID reference)
- updated_by (user ID reference)

Audit Log Table (Optional):

CREATE TABLE audit_logs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    operation VARCHAR(50) NOT NULL, -- CREATE, UPDATE, DELETE
    table_name VARCHAR(100) NOT NULL,
    record_id UUID NOT NULL,
    user_id UUID REFERENCES users(id),
    changes JSONB,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Data Archival Strategy and Retention Policy

Archival Strategy:

Retention Policy:
- Documents older than 2 years are archived to cold storage (e.g., S3 Glacier)
- Archive metadata is retained in database for search and retrieval
Archival Trigger:
- Scheduled job runs daily to identify documents older than 2 years
- Archive process updates document status to 'ARCHIVED'
Archive State:
- file_url points to archived location in S3
- status changes to 'ARCHIVED'
- Original file is deleted from primary storage (if applicable)

4. Class Design & Service Layer

UML Class Diagram

classDiagram
    class DocumentService {
        +createDocument(CreateDocumentRequest) Document
        +updateDocument(UpdateDocumentRequest) Document
        +deleteDocument(String) void
        +getDocument(String) Document
        +searchDocuments(SearchDocumentsRequest) List~Document~
        +uploadDocumentFile(String, InputStream) Document
        +downloadDocumentFile(String) InputStream
        +shareDocument(ShareDocumentRequest) DocumentShare
        +getDocumentShares(String) List~DocumentShare~
    }

    class Document {
        -id: String
        -name: String
        -size: Long
        -contentType: String
        -version: Integer
        -createdAt: Instant
        -updatedAt: Instant
        -ownerId: String
        -status: DocumentStatus
        -tags: List~String~
        -metadata: Map~String, String~
        +getId(): String
        +getName(): String
        +getSize(): Long
        +getContentType(): String
        +getVersion(): Integer
        +getCreatedAt(): Instant
        +getUpdatedAt(): Instant
        +getOwnerId(): String
        +getStatus(): DocumentStatus
        +getTags(): List~String~
        +getMetadata(): Map~String, String~
        +setStatus(DocumentStatus): void
        +setVersion(Integer): void
        +setUpdatedAt(Instant): void
    }

    class DocumentStatus {
        CREATED
        PROCESSING
        PROCESSED
        DELETED
    }

    class DocumentShare {
        -id: String
        -documentId: String
        -sharedBy: String
        -sharedWith: String
        -permission: PermissionType
        -createdAt: Instant
        -expiresAt: Instant
        +getId(): String
        +getDocumentId(): String
        +getSharedBy(): String
        +getSharedWith(): String
        +getPermission(): PermissionType
        +getCreatedAt(): Instant
        +getExpiresAt(): Instant
    }

    class PermissionType {
        READ
        WRITE
        ADMIN
    }

    class DocumentRepository {
        +save(Document) Document
        +findById(String) Optional~Document~
        +findByOwnerId(String) List~Document~
        +deleteById(String) void
        +findAll() Iterator~Document~
    }

    class DocumentShareRepository {
        +save(DocumentShare) DocumentShare
        +findById(String) Optional~DocumentShare~
        +findByDocumentId(String) List~DocumentShare~
        +deleteById(String) void
    }

    class FileStorageService {
        +uploadFile(String, InputStream) String
        +downloadFile(String) InputStream
        +deleteFile(String) void
    }

    class EventPublisher {
        +publish(Event) void
    }

    class DocumentEvent {
        -type: String
        -documentId: String
        -timestamp: Instant
        -payload: Map~String, Object~
    }

    class DocumentValidator {
        +validateCreateRequest(CreateDocumentRequest) void
        +validateUpdateRequest(UpdateDocumentRequest) void
    }

    class DocumentIndexer {
        +indexDocument(Document) void
    }

    class SearchService {
        +search(SearchDocumentsRequest) List~Document~
    }

    DocumentService --> Document : aggregates
    DocumentService --> DocumentRepository : uses
    DocumentService --> FileStorageService : uses
    DocumentService --> EventPublisher : publishes
    DocumentService --> DocumentValidator : validates
    DocumentService --> SearchService : delegates
    DocumentService --> DocumentIndexer : indexes

    Document --> DocumentStatus : has
    Document --> DocumentShare : has many
    DocumentShare --> PermissionType : has

    DocumentRepository --> Document : persists
    DocumentShareRepository --> DocumentShare : persists

    EventPublisher --> DocumentEvent : publishes

    DocumentValidator --> CreateDocumentRequest : validates
    DocumentValidator --> UpdateDocumentRequest : validates

    SearchService --> DocumentIndexer : indexes

State Machine Diagram for Document Entity

stateDiagram-v2
    [*] --> CREATED
    CREATED --> PROCESSING : uploadFile()
    PROCESSING --> PROCESSED : processComplete()
    PROCESSED --> DELETED : deleteDocument()
    DELETED --> [*] : softDelete()

Service Class Structure and Responsibilities

The system follows a clean architecture with the following service layer structure:

DocumentService - Main orchestration service for document operations
DocumentValidator - Validation layer for request payloads
SearchService - Search and indexing operations
DocumentIndexer - Indexing logic for search capabilities

Interface / Abstract Class Design

public interface DocumentRepository {
    Document save(Document document);
    Optional<Document> findById(String id);
    List<Document> findByOwnerId(String ownerId);
    void deleteById(String id);
    Iterator<Document> findAll();
}

public interface DocumentShareRepository {
    DocumentShare save(DocumentShare documentShare);
    Optional<DocumentShare> findById(String id);
    List<DocumentShare> findByDocumentId(String documentId);
    void deleteById(String id);
}

public interface FileStorageService {
    String uploadFile(String documentId, InputStream content);
    InputStream downloadFile(String fileId);
    void deleteFile(String fileId);
}

public interface EventPublisher {
    void publish(Event event);
}

Business Logic Encapsulation Patterns

Strategy Pattern Implementation for Document Processing

public interface DocumentProcessingStrategy {
    void process(Document document, InputStream content);
}

@Component
public class PDFDocumentProcessingStrategy implements DocumentProcessingStrategy {
    @Override
    public void process(Document document, InputStream content) {
        // PDF-specific processing logic
        // Extract metadata, validate content, etc.
    }
}

@Component
public class ImageDocumentProcessingStrategy implements DocumentProcessingStrategy {
    @Override
    public void process(Document document, InputStream content) {
        // Image-specific processing logic
        // Extract EXIF data, validate dimensions, etc.
    }
}

@Service
public class DocumentProcessingService {
    private final Map<String, DocumentProcessingStrategy> strategies;

    public DocumentProcessingService(
            List<DocumentProcessingStrategy> strategyList) {
        this.strategies = strategyList.stream()
                .collect(Collectors.toMap(
                    s -> s.getClass().getSimpleName(),
                    Function.identity()));
    }

    public void processDocument(String contentType, Document document, InputStream content) {
        DocumentProcessingStrategy strategy = strategies.get(contentType);
        if (strategy == null) {
            throw new IllegalArgumentException("Unsupported content type: " + contentType);
        }
        strategy.process(document, content);
    }
}

Critical-Path Method Implementation

@Transactional
public Document uploadDocumentFile(String documentId, InputStream content) {
    // 1. Validate request and fetch existing document
    Document document = documentRepository.findById(documentId)
        .orElseThrow(() -> new DocumentNotFoundException("Document not found: " + documentId));
    
    // 2. Validate content type and size limits
    if (content == null) {
        throw new IllegalArgumentException("Content cannot be null");
    }
    
    // 3. Acquire exclusive lock for document to prevent concurrent updates
    // Using pessimistic locking with database-level row locking
    String lockKey = "document_lock_" + documentId;
    try {
        // In a real system, this would use Redis or database advisory locks
        acquireLock(lockKey);
        
        // 4. Process content and update document metadata
        String fileId = fileStorageService.uploadFile(documentId, content);
        document.setVersion(document.getVersion() + 1);
        document.setStatus(DocumentStatus.PROCESSING);
        document.setUpdatedAt(Instant.now());
        
        // 5. Update database with new state
        Document updatedDocument = documentRepository.save(document);
        
        // 6. Trigger asynchronous processing for content analysis
        eventPublisher.publish(new DocumentProcessingEvent(
            updatedDocument.getId(),
            fileId,
            updatedDocument.getContentType()
        ));
        
        // 7. Update search index
        documentIndexer.indexDocument(updatedDocument);
        
        return updatedDocument;
    } finally {
        releaseLock(lockKey);
    }
}

Transaction Boundary Design

Each document operation is wrapped in a single database transaction:

Create/Update/Delete: Single DB transaction with proper locking
Upload: Transaction includes metadata update and file storage operation
Share: Transaction ensures atomicity of share creation and event publishing

Service Interface Contracts

public interface DocumentService {
    Document createDocument(CreateDocumentRequest request);
    Document updateDocument(UpdateDocumentRequest request);
    void deleteDocument(String documentId);
    Document getDocument(String documentId);
    List<Document> searchDocuments(SearchDocumentsRequest request);
    Document uploadDocumentFile(String documentId, InputStream content);
    InputStream downloadDocumentFile(String documentId);
    DocumentShare shareDocument(ShareDocumentRequest request);
    List<DocumentShare> getDocumentShares(String documentId);
}

Saga / Orchestration Pattern

For complex operations like document sharing with email notifications:

@Service
public class DocumentSharingSaga {
    private final EventPublisher eventPublisher;
    
    public void shareDocumentWithNotification(ShareDocumentRequest request) {
        // Start saga
        DocumentShare share = createDocumentShare(request);
        
        // Publish event to trigger async notification
        eventPublisher.publish(new DocumentSharingEvent(
            share.getDocumentId(),
            share.getSharedWith(),
            share.getPermission()
        ));
        
        // If notification fails, rollback the share creation
        // This would be handled by a compensating transaction or retry mechanism
    }
}

Domain Object / Value Object Design

public class Document {
    private String id;
    private String name;
    private Long size;
    private String contentType;
    private Integer version = 0;
    private Instant createdAt;
    private Instant updatedAt;
    private String ownerId;
    private DocumentStatus status;
    private List<String> tags = new ArrayList<>();
    private Map<String, String> metadata = new HashMap<>();
    
    // Getters and setters with validation
    
    public void addTag(String tag) {
        if (tag != null && !tag.trim().isEmpty()) {
            this.tags.add(tag);
        }
    }
    
    public void setMetadata(Map<String, String> metadata) {
        this.metadata = metadata != null ? new HashMap<>(metadata) : new HashMap<>();
    }
}

public class DocumentShare {
    private String id;
    private String documentId;
    private String sharedBy;
    private String sharedWith;
    private PermissionType permission;
    private Instant createdAt;
    private Instant expiresAt;
    
    // Getters and setters
}

public enum DocumentStatus {
    CREATED, PROCESSING, PROCESSED, DELETED
}

public enum PermissionType {
    READ, WRITE, ADMIN
}

5. Concurrency & Thread Safety

Concurrency Model

The document management system will utilize a thread pool-based concurrency model with Java's CompletableFuture and ExecutorService for handling asynchronous operations. For core business logic, we'll use a single-threaded executor for critical sections to ensure atomicity and consistency. The system will leverage java.util.concurrent primitives like ReentrantLock, ReadWriteLock, and StampedLock for fine-grained control over shared resources.

Race Condition Prevention Strategy

Race conditions will be prevented through:

Lock-based synchronization for critical sections involving document metadata updates
Optimistic concurrency control (OCC) for document content operations using version numbers
Idempotency keys for external API calls to prevent duplicate processing
Database-level constraints to enforce uniqueness and referential integrity

Optimistic vs. Pessimistic Locking Selection

We will use optimistic locking for document content operations because:

Read-heavy workloads with infrequent writes
Low contention scenarios in typical usage patterns
Better performance under concurrent access
Documents are typically updated by a single user at a time

Pessimistic locking will be used for metadata operations that require immediate consistency and are infrequent.

Deadlock Prevention Strategy

Deadlock prevention will be implemented through:

Lock ordering - all operations acquire locks in a consistent order (e.g., by document ID)
Timeout-based acquisition - 5-second timeout on lock acquisition
Lock hierarchy - define clear lock levels (metadata locks before content locks)
Deadlock detection - periodic monitoring of lock wait chains

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency will be handled at the service layer using:

public class DocumentService {
    private final Map<String, CompletableFuture<Document>> pendingOperations = new ConcurrentHashMap<>();
    
    public CompletableFuture<Document> uploadDocument(String idempotencyKey, DocumentUploadRequest request) {
        return pendingOperations.computeIfAbsent(idempotencyKey, key -> {
            try {
                // Check if operation already completed
                Document existing = documentRepository.findByUploadIdempotencyKey(key);
                if (existing != null) {
                    return CompletableFuture.completedFuture(existing);
                }
                
                // Perform actual operation
                return CompletableFuture.supplyAsync(() -> {
                    Document document = processDocument(request);
                    // Publish events, update metadata
                    return document;
                });
            } catch (Exception e) {
                // Handle and propagate errors
                return CompletableFuture.failedFuture(e);
            }
        });
    }
}

Additionally, we'll implement a distributed idempotency store using Redis to handle cross-service idempotency:

public class DistributedIdempotencyService {
    private final RedisTemplate<String, String> redisTemplate;
    
    public boolean isIdempotent(String key) {
        String existing = redisTemplate.opsForValue().get(key);
        return existing != null;
    }
    
    public void markAsProcessed(String key, String result) {
        redisTemplate.opsForValue().set(key, result, 24, TimeUnit.HOURS);
    }
}

Thread Safety Patterns in Core Components

1. Document Metadata Updates

public class DocumentMetadataService {
    private final ReadWriteLock metadataLock = new ReentrantReadWriteLock();
    private final Map<String, DocumentMetadata> metadataCache = new ConcurrentHashMap<>();
    
    public void updateDocumentMetadata(String documentId, DocumentUpdateRequest request) {
        // Use write lock for metadata updates
        metadataLock.writeLock().lock();
        try {
            DocumentMetadata current = metadataCache.get(documentId);
            if (current == null) {
                throw new DocumentNotFoundException(documentId);
            }
            
            // Validate version for optimistic concurrency
            if (request.getVersion() != current.getVersion()) {
                throw new ConcurrencyException("Document has been modified");
            }
            
            DocumentMetadata updated = current.withUpdates(request);
            metadataCache.put(documentId, updated);
            
            // Publish metadata change event
            eventPublisher.publish(new MetadataUpdatedEvent(documentId, updated));
        } finally {
            metadataLock.writeLock().unlock();
        }
    }
}

2. Content Operations with Versioning

public class DocumentContentService {
    private final StampedLock contentLock = new StampedLock();
    
    public DocumentContent updateContent(String documentId, ContentUpdateRequest request) {
        long stamp = contentLock.readLock();
        try {
            DocumentContent current = contentRepository.load(documentId);
            
            // Validate version for optimistic concurrency
            if (request.getVersion() != current.getVersion()) {
                throw new ConcurrencyException("Content has been modified");
            }
            
            // Upgrade to write lock for update
            stamp = contentLock.tryConvertToWriteLock(stamp);
            if (stamp == 0) {
                stamp = contentLock.writeLock();
            }
            
            DocumentContent updated = current.withUpdates(request);
            contentRepository.save(updated);
            
            return updated;
        } finally {
            // Release appropriate lock
            contentLock.unlock(stamp);
        }
    }
}

Concurrency Control for Batch Operations

Batch operations will use a combination of:

Partitioned processing - Distribute batches across different threads based on document ID ranges
Work queue with bounded capacity - Prevent memory exhaustion during large batch uploads
Rate limiting per document type - Control concurrent processing of specific document categories

Thread Safety in Event Handling

public class DocumentEventConsumer {
    private final ExecutorService eventProcessingPool = 
        Executors.newFixedThreadPool(10, 
            new ThreadFactoryBuilder().setNameFormat("doc-event-%d").build());
    
    @Async
    public void handleDocumentCreatedEvent(DocumentCreatedEvent event) {
        // Process in separate thread to avoid blocking main processing
        try {
            processDocumentCreation(event);
        } catch (Exception e) {
            // Handle and log errors without blocking other events
            logger.error("Failed to process document creation event", e);
        }
    }
}

Performance Considerations

Use lock-free data structures where possible (e.g., ConcurrentHashMap for caches)
Implement batching strategies to reduce lock contention
Use asynchronous processing for non-critical operations to maintain responsiveness
Monitor and tune thread pool sizes based on actual workload patterns

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Auth as Auth Service
    participant DocSvc as Document Service
    participant Storage as Object Storage
    participant EventBus as Event Bus

    U->>API: Upload document request (multipart/form-data)
    API->>Auth: Validate JWT token
    Auth-->>API: Authenticated user (user_id: 123)
    API->>DocSvc: CreateDocument (document metadata)
    DocSvc->>DocSvc: Validate input, generate document_id
    DocSvc->>Storage: Store file (bucket: docs-123, key: doc-uuid)
    Storage-->>DocSvc: File stored successfully
    DocSvc->>DocSvc: Save document metadata to DB
    DocSvc->>EventBus: Publish DocumentCreated event
    EventBus-->>DocSvc: Event acknowledged
    DocSvc-->>API: Document created response (201)
    API-->>U: Created document (id: doc-uuid, name: sample.pdf)

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Auth as Auth Service
    participant DocSvc as Document Service
    participant Storage as Object Storage
    participant DB as Database

    U->>API: Get document by ID (doc-uuid)
    API->>Auth: Validate JWT token
    Auth-->>API: Authenticated user (user_id: 123)
    API->>DocSvc: GetDocument (doc-uuid)
    DocSvc->>DB: Fetch document metadata
    DB-->>DocSvc: Document found (status: ACTIVE)
    DocSvc->>Storage: Get file (bucket: docs-123, key: doc-uuid)
    Storage-->>DocSvc: File content (binary)
    DocSvc-->>API: Document response with file content
    API-->>U: Document content (PDF)

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Auth as Auth Service
    participant DocSvc as Document Service
    participant DB as Database
    participant EventBus as Event Bus

    U->>API: Delete document request (doc-uuid)
    API->>Auth: Validate JWT token
    Auth-->>API: Authenticated user (user_id: 123)
    API->>DocSvc: DeleteDocument (doc-uuid)
    DocSvc->>DB: Fetch document metadata
    DB-->>DocSvc: Document found (status: ACTIVE)
    DocSvc->>Storage: Delete file (bucket: docs-123, key: doc-uuid)
    Storage-->>DocSvc: File deleted
    DocSvc->>DB: Update document status to DELETED
    DocSvc->>EventBus: Publish DocumentDeleted event
    EventBus-->>DocSvc: Event acknowledged
    DocSvc-->>API: Success response (204)
    API-->>U: Document deleted successfully

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Auth as Auth Service
    participant DocSvc as Document Service
    participant Storage as Object Storage
    participant DB as Database
    participant Queue as Message Queue

    U->>API: Upload document with validation error
    API->>Auth: Validate JWT token
    Auth-->>API: Authenticated user (user_id: 123)
    API->>DocSvc: CreateDocument (invalid metadata)
    DocSvc->>DocSvc: Validate input
    DocSvc-->>API: Bad Request (400) - Invalid file size
    API-->>U: Error response (400, "File size exceeds 10MB limit")

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Auth as Auth Service
    participant DocSvc as Document Service
    participant Storage as Object Storage
    participant DB as Database
    participant EventBus as Event Bus

    U->>API: Upload document with storage failure
    API->>Auth: Validate JWT token
    Auth-->>API: Authenticated user (user_id: 123)
    API->>DocSvc: CreateDocument (document metadata)
    DocSvc->>Storage: Store file (bucket: docs-123, key: doc-uuid)
    Storage-->>DocSvc: Storage timeout (504)
    DocSvc->>DB: Rollback document metadata
    DB-->>DocSvc: Rollback successful
    DocSvc-->>API: Service Unavailable (503) - Storage timeout
    API-->>U: Error response (503, "Storage service unavailable")

7. Caching Design

Cache Strategy Selection

For a document management system, I'll use cache-aside pattern with LRU eviction policy. This approach provides the best balance between performance and consistency for a system where document content is read frequently but updated less often.

Cache Key Design

Cache keys will follow a structured namespace pattern:

{cache_namespace}:{entity_type}:{entity_id}:{version}

Example key structure:

doc:document:12345:latest
doc:folder:67890:1
doc:user:54321:permissions

TTL Strategy

Data Type	TTL	Reason
Document content	1 hour	Content changes infrequently, but should not be stale for too long
Document metadata	30 minutes	Metadata updates more frequently than content
User permissions	15 minutes	Permission changes require timely propagation
Folder structure	1 hour	Folder hierarchy is relatively stable
Search index	5 minutes	Search indexes need to be more responsive to changes

Cache Eviction Policy

LRU (Least Recently Used) with probabilistic early expiry:

Primary eviction: LRU-based
Secondary optimization: 10% probabilistic early expiry to prevent cache stampede on hot keys
Cache warming during system startup for frequently accessed documents

Cache Stampede Prevention

Probabilistic Early Expiry: 10% of cache entries expire 5-10 minutes before TTL
Distributed Locks: For hot keys, use Redis distributed locks to serialize cache population
Cache Entry Versioning: Use versioned keys to handle concurrent updates gracefully

Distributed Cache Architecture

Using Redis Cluster with:

3 master nodes, 3 replica nodes for high availability
1000+ keys per shard (sharding by document ID hash)
Redis Sentinel for automatic failover
Memory policy: allkeys-lru

Cache Invalidation Strategy

Event-driven invalidation with:

Document update: Invalidate document content and metadata keys
Folder move/delete: Invalidate folder structure and child document keys
Permission change: Invalidate user permission keys
Search index update: Invalidate search index keys

Cache Implementation Example

@Component
public class DocumentCacheService {
    private final RedisTemplate<String, Object> redisTemplate;
    private final ObjectMapper objectMapper;
    
    public DocumentContent getCachedDocument(String docId) {
        String key = "doc:document:" + docId + ":latest";
        DocumentContent cached = (DocumentContent) redisTemplate.opsForValue().get(key);
        
        if (cached != null) {
            // Update LRU position
            redisTemplate.expire(key, Duration.ofHours(1));
            return cached;
        }
        
        // Cache miss - populate cache with distributed lock
        String lockKey = "lock:doc:" + docId;
        if (redisTemplate.opsForValue().setIfAbsent(lockKey, "locked", 
            Duration.ofSeconds(5), TimeUnit.SECONDS)) {
            
            try {
                // Double-check pattern
                cached = (DocumentContent) redisTemplate.opsForValue().get(key);
                if (cached != null) {
                    return cached;
                }
                
                // Fetch from database and populate cache
                DocumentContent content = fetchDocumentFromDB(docId);
                redisTemplate.opsForValue().set(key, content, Duration.ofHours(1));
                return content;
            } finally {
                redisTemplate.delete(lockKey);
            }
        }
        
        // Wait briefly and retry
        Thread.sleep(50);
        return getCachedDocument(docId);
    }
    
    public void invalidateDocumentCache(String docId) {
        String contentKey = "doc:document:" + docId + ":latest";
        String metadataKey = "doc:document:" + docId + ":metadata";
        
        redisTemplate.delete(contentKey);
        redisTemplate.delete(metadataKey);
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and database operations. Each entity will have its own repository interface and concrete implementation.

// Repository Interface
public interface DocumentRepository {
    Optional<Document> findById(UUID id);
    List<Document> findByUserId(UUID userId);
    List<Document> findByFolderId(UUID folderId);
    Document save(Document document);
    void delete(UUID id);
    Page<Document> findByUserIdWithPagination(UUID userId, Pageable pageable);
}

// Repository Implementation
@Repository
public class DocumentRepositoryImpl implements DocumentRepository {
    private final JdbcTemplate jdbcTemplate;
    private final ObjectMapper objectMapper;

    public DocumentRepositoryImpl(JdbcTemplate jdbcTemplate, ObjectMapper objectMapper) {
        this.jdbcTemplate = jdbcTemplate;
        this.objectMapper = objectMapper;
    }

    @Override
    public Optional<Document> findById(UUID id) {
        String sql = "SELECT * FROM documents WHERE id = ?";
        try {
            Document document = jdbcTemplate.queryForObject(sql, new Object[]{id}, new DocumentRowMapper());
            return Optional.of(document);
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }

    @Override
    public List<Document> findByUserId(UUID userId) {
        String sql = "SELECT * FROM documents WHERE user_id = ? ORDER BY created_at DESC";
        return jdbcTemplate.query(sql, new Object[]{userId}, new DocumentRowMapper());
    }

    @Override
    public List<Document> findByFolderId(UUID folderId) {
        String sql = "SELECT * FROM documents WHERE folder_id = ? ORDER BY created_at DESC";
        return jdbcTemplate.query(sql, new Object[]{folderId}, new DocumentRowMapper());
    }

    @Override
    public Document save(Document document) {
        if (document.getId() == null) {
            return createDocument(document);
        } else {
            return updateDocument(document);
        }
    }

    private Document createDocument(Document document) {
        String sql = """
            INSERT INTO documents (id, user_id, folder_id, name, content_type, size, version, created_at, updated_at)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
            """;
        UUID id = UUID.randomUUID();
        document.setId(id);
        document.setCreatedAt(Instant.now());
        document.setUpdatedAt(Instant.now());

        jdbcTemplate.update(sql, 
            id,
            document.getUserId(),
            document.getFolderId(),
            document.getName(),
            document.getContentType(),
            document.getSize(),
            document.getVersion(),
            document.getCreatedAt(),
            document.getUpdatedAt());

        return document;
    }

    private Document updateDocument(Document document) {
        String sql = """
            UPDATE documents SET name = ?, content_type = ?, size = ?, version = ?, updated_at = ?
            WHERE id = ?
            """;
        document.setUpdatedAt(Instant.now());
        jdbcTemplate.update(sql,
            document.getName(),
            document.getContentType(),
            document.getSize(),
            document.getVersion(),
            document.getUpdatedAt(),
            document.getId());

        return document;
    }

    @Override
    public void delete(UUID id) {
        String sql = "DELETE FROM documents WHERE id = ?";
        jdbcTemplate.update(sql, id);
    }

    @Override
    public Page<Document> findByUserIdWithPagination(UUID userId, Pageable pageable) {
        String countSql = "SELECT COUNT(*) FROM documents WHERE user_id = ?";
        int total = jdbcTemplate.queryForObject(countSql, Integer.class, userId);

        String sql = """
            SELECT * FROM documents WHERE user_id = ? ORDER BY created_at DESC LIMIT ? OFFSET ?
            """;
        List<Document> content = jdbcTemplate.query(sql, 
            new Object[]{userId, pageable.getPageSize(), pageable.getOffset()}, 
            new DocumentRowMapper());

        return new PageImpl<>(content, pageable, total);
    }
}

ORM vs Raw Query Decisions

For this system, we use JDBC with raw queries rather than an ORM for several reasons:

Performance: Raw queries provide better performance for complex queries and reduce object mapping overhead
Control: Direct control over SQL allows for optimization and complex joins
Database portability: Explicit queries make it easier to handle database-specific features when needed

Read vs Write Separation

We implement CQRS pattern for read and write operations to optimize performance:

// Read-Only Repository for optimized queries
@Repository
public class DocumentReadRepository {
    private final JdbcTemplate jdbcTemplate;

    public DocumentReadRepository(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }

    public List<DocumentSummary> getDocumentSummaries(UUID userId) {
        String sql = """
            SELECT id, name, content_type, size, created_at 
            FROM documents 
            WHERE user_id = ? 
            ORDER BY created_at DESC
            """;
        return jdbcTemplate.query(sql, new Object[]{userId}, (rs, rowNum) -> 
            new DocumentSummary(
                rs.getObject("id", UUID.class),
                rs.getString("name"),
                rs.getString("content_type"),
                rs.getLong("size"),
                rs.getTimestamp("created_at").toInstant()
            )
        );
    }

    public DocumentMetadata getDocumentMetadata(UUID documentId) {
        String sql = """
            SELECT d.id, d.name, d.content_type, d.size, d.version, 
                   f.name as folder_name, u.username
            FROM documents d
            JOIN folders f ON d.folder_id = f.id
            JOIN users u ON d.user_id = u.id
            WHERE d.id = ?
            """;
        return jdbcTemplate.queryForObject(sql, new Object[]{documentId}, (rs, rowNum) -> 
            new DocumentMetadata(
                rs.getObject("id", UUID.class),
                rs.getString("name"),
                rs.getString("content_type"),
                rs.getLong("size"),
                rs.getInt("version"),
                rs.getString("folder_name"),
                rs.getString("username")
            )
        );
    }
}

Connection Pool Configuration

# application.yml
spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      validation-timeout: 5000
      leak-detection-threshold: 60000

N+1 Query Prevention Strategy

We implement eager loading and batch operations to prevent N+1 query problems:

// Batch operation for folder document retrieval
public List<Document> getDocumentsInFolderBatch(List<UUID> folderIds) {
    if (folderIds.isEmpty()) return Collections.emptyList();
    
    String sql = "SELECT * FROM documents WHERE folder_id IN (" + 
        String.join(",", Collections.nCopies(folderIds.size(), "?")) + ")";
    
    Object[] params = folderIds.toArray();
    return jdbcTemplate.query(sql, params, new DocumentRowMapper());
}

// Eager loading of related entities
public DocumentWithMetadata getDocumentWithMetadata(UUID documentId) {
    String sql = """
        SELECT d.*, f.name as folder_name, u.username 
        FROM documents d
        JOIN folders f ON d.folder_id = f.id
        JOIN users u ON d.user_id = u.id
        WHERE d.id = ?
        """;
    return jdbcTemplate.queryForObject(sql, new Object[]{documentId}, (rs, rowNum) -> 
        new DocumentWithMetadata(
            rs.getObject("id", UUID.class),
            rs.getString("name"),
            rs.getString("content_type"),
            rs.getLong("size"),
            rs.getTimestamp("created_at").toInstant(),
            rs.getString("folder_name"),
            rs.getString("username")
        )
    );
}

Batch Operation Design

// Batch create documents with optimized transaction handling
public List<Document> batchCreateDocuments(List<Document> documents) {
    String sql = """
        INSERT INTO documents (id, user_id, folder_id, name, content_type, size, version, created_at, updated_at)
        VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
        """;
    
    return jdbcTemplate.batchUpdate(sql, documents, documents.size(), 
        (ps, document) -> {
            ps.setObject(1, document.getId());
            ps.setObject(2, document.getUserId());
            ps.setObject(3, document.getFolderId());
            ps.setString(4, document.getName());
            ps.setString(5, document.getContentType());
            ps.setLong(6, document.getSize());
            ps.setInt(7, document.getVersion());
            ps.setTimestamp(8, Timestamp.from(document.getCreatedAt()));
            ps.setTimestamp(9, Timestamp.from(document.getUpdatedAt()));
        });
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The document management system will use a hybrid synchronous/asynchronous approach. Core operations like document upload, metadata update, and deletion will be handled synchronously to ensure immediate consistency for critical user actions. However, heavy processing operations such as content analysis, OCR extraction, virus scanning, and document format conversion will be offloaded to asynchronous workflows.

Key triggers for async processing:

Document upload completion → trigger content analysis and virus scanning
Document format change → trigger conversion to target format
Metadata update → trigger search index rebuild
User request for document sharing → trigger notification and access control updates

Message schema design with sample payload

{
  "version": "1.0",
  "eventId": "uuid-12345678-1234-1234-1234-123456789012",
  "timestamp": "2023-10-15T14:30:00Z",
  "eventType": "document.processed",
  "payload": {
    "documentId": "doc-12345678-1234-1234-1234-123456789012",
    "userId": "user-12345678-1234-1234-1234-123456789012",
    "action": "content_analysis_complete",
    "result": {
      "wordCount": 1250,
      "language": "en",
      "extractedText": "Sample extracted text...",
      "metadata": {
        "createdDate": "2023-10-15T14:30:00Z",
        "lastModified": "2023-10-15T14:30:00Z"
      }
    },
    "status": "success",
    "error": null
  }
}

Dead letter queue strategy and poison message handling

All async consumers will use dead-letter queues (DLQ) with a 3-retry policy. Messages that fail after 3 attempts will be moved to DLQ with a retention period of 7 days. Poison messages will be logged with detailed context including:

Original message payload
Error stack trace
Retry count and timestamps
Consumer service name and version

DLQ monitoring will trigger alerts when queue size exceeds 100 messages or when error rate exceeds 5% of total processed messages.

At-least-once vs. exactly-once delivery

The system uses at-least-once delivery model with idempotency keys to handle duplicate messages. This is chosen for simplicity and cost efficiency over the complexity of exactly-once semantics, which would require distributed transaction coordination.

Retry policy for consumers

Retry policy configuration:

Max attempts: 3
Backoff strategy: Exponential with jitter (1s, 2s, 4s)
Retryable errors: Network timeouts, service unavailability, temporary storage issues
Non-retryable errors: Invalid document format, user permission denied, content validation failures

Outbox pattern for reliable event publishing

The outbox pattern is implemented to ensure event delivery reliability. When a domain operation completes successfully, the system creates an Event record in the database with status PENDING. A separate background job processes these pending events and publishes them to the message broker. If publishing fails, the event status is updated back to PENDING for retry.

Sequence diagram for async flows

sequenceDiagram
    participant User
    participant API
    participant DocumentService
    participant EventPublisher
    participant MessageBroker
    participant ContentAnalyzer
    participant VirusScanner

    User->>API: Upload Document
    API->>DocumentService: processUpload()
    DocumentService->>EventPublisher: publish(documentUploaded)
    EventPublisher->>MessageBroker: queue event
    MessageBroker->>ContentAnalyzer: consume contentAnalysisEvent
    ContentAnalyzer->>DocumentService: updateContentMetadata()
    MessageBroker->>VirusScanner: consume virusScanEvent
    VirusScanner->>DocumentService: updateVirusStatus()

10. Error Handling Design

Error Classification Hierarchy

Errors are categorized into three primary levels to enable appropriate handling and alerting:

Domain Errors - Business logic violations (e.g., document not found, insufficient permissions)
Infrastructure Errors - System-level issues (e.g., database connection failure, storage timeout)
Validation Errors - Input data issues (e.g., invalid file format, missing required fields)

public class DocumentManagementException extends RuntimeException {
    private final ErrorType errorType;
    private final String errorCode;
    
    public DocumentManagementException(ErrorType errorType, String errorCode, String message) {
        super(message);
        this.errorType = errorType;
        this.errorCode = errorCode;
    }
    
    public DocumentManagementException(ErrorType errorType, String errorCode, String message, Throwable cause) {
        super(message, cause);
        this.errorType = errorType;
        this.errorCode = errorCode;
    }
    
    // Getters
}

public enum ErrorType {
    DOMAIN, INFRASTRUCTURE, VALIDATION
}

Error Propagation Strategy

Errors are propagated up the call stack to appropriate layers for handling:

Validation errors are caught at the API layer and converted to HTTP 400 responses
Domain errors are handled at service level and may be re-thrown or converted to appropriate HTTP responses
Infrastructure errors are logged at the service layer and may trigger circuit breaker logic or retry mechanisms

Retry Strategy with Exponential Backoff and Jitter

For transient infrastructure errors, a retry mechanism is implemented using exponential backoff with jitter:

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;

    public <T> T executeWithRetry(Supplier<T> operation) {
        int attempts = 0;
        while (attempts < MAX_RETRIES) {
            try {
                return operation.get();
            } catch (InfrastructureException e) {
                if (attempts >= MAX_RETRIES - 1) throw e;
                
                long delay = calculateBackoffDelay(attempts);
                try {
                    Thread.sleep(delay);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new InfrastructureException("Retry interrupted", ie);
                }
                attempts++;
            }
        }
        throw new IllegalStateException("Unexpected state in retry operation");
    }

    private long calculateBackoffDelay(int attempt) {
        long baseDelay = (long) Math.pow(2, attempt) * BASE_DELAY_MS;
        long jitter = (long) (baseDelay * JITTER_FACTOR * (Math.random() - 0.5) * 2);
        return baseDelay + jitter;
    }
}

Circuit Breaker Design

For external dependencies like storage services or document processing APIs, a circuit breaker pattern is implemented:

public class CircuitBreaker {
    private final int failureThreshold;
    private final long timeoutMs;
    private final AtomicInteger failureCount = new AtomicInteger(0);
    private final AtomicLong lastFailureTime = new AtomicLong(0);
    private volatile CircuitState state = CircuitState.CLOSED;

    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }

    public <T> T execute(Supplier<T> operation) throws Exception {
        switch (state) {
            case CLOSED:
                return executeClosed(operation);
            case OPEN:
                if (System.currentTimeMillis() - lastFailureTime.get() > timeoutMs) {
                    state = CircuitState.HALF_OPEN;
                    return executeHalfOpen(operation);
                }
                throw new ServiceUnavailableException("Circuit breaker is open");
            case HALF_OPEN:
                return executeHalfOpen(operation);
            default:
                throw new IllegalStateException("Unknown circuit state");
        }
    }

    private <T> T executeClosed(Supplier<T> operation) throws Exception {
        try {
            T result = operation.get();
            failureCount.set(0);
            return result;
        } catch (Exception e) {
            if (failureCount.incrementAndGet() >= failureThreshold) {
                state = CircuitState.OPEN;
                lastFailureTime.set(System.currentTimeMillis());
            }
            throw e;
        }
    }

    private <T> T executeHalfOpen(Supplier<T> operation) throws Exception {
        try {
            T result = operation.get();
            state = CircuitState.CLOSED;
            failureCount.set(0);
            return result;
        } catch (Exception e) {
            state = CircuitState.OPEN;
            lastFailureTime.set(System.currentTimeMillis());
            throw e;
        }
    }
}

Fallback Behavior Design

Fallbacks are implemented for critical operations:

public class DocumentService {
    private final StorageService storageService;
    private final DocumentProcessingService processingService;

    public DocumentMetadata processDocumentWithFallback(String documentId) {
        try {
            return processingService.processDocument(documentId);
        } catch (InfrastructureException e) {
            // Fallback to basic metadata extraction from storage
            return extractBasicMetadataFromStorage(documentId);
        }
    }

    private DocumentMetadata extractBasicMetadataFromStorage(String documentId) {
        // Basic metadata extraction logic
        return new DocumentMetadata(documentId, "fallback");
    }
}

Error Logging Format and Alerting Thresholds

Structured logging is implemented with consistent fields:

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);

    public void logError(String operation, String errorCode, String errorMessage, 
                        Map<String, Object> additionalContext) {
        Map<String, Object> logFields = new HashMap<>();
        logFields.put("timestamp", Instant.now().toString());
        logFields.put("operation", operation);
        logFields.put("error_code", errorCode);
        logFields.put("message", errorMessage);
        logFields.put("context", additionalContext);
        
        logger.error("Document management error occurred: {}", 
                    objectMapper.writeValueAsString(logFields));
    }
}

Alerting thresholds:

Error Type	Alert Threshold	Severity
Infrastructure errors (DB timeouts, storage failures)	> 5 occurrences in 5 minutes	Critical
Validation errors (invalid input)	> 100 occurrences in 1 hour	Warning
Domain errors (permissions, not found)	> 50 occurrences in 1 hour	Info

Error Response Format

Standardized error responses are returned to clients:

{
  "error": {
    "code": "DOCUMENT_NOT_FOUND",
    "message": "The requested document was not found",
    "timestamp": "2023-10-15T10:30:00Z",
    "requestId": "req-12345"
  }
}

11. Security Design

Input Validation and Sanitization Strategy

All inputs are validated using a combination of:

Library-based validation: Apache Commons Validator for email, URL, and format validation
Custom regex patterns: For document name sanitization (removes dangerous characters like ;, &, |)
Length limits: Document names capped at 255 characters, content size limited to 100MB
Content type validation: MIME type checking using Tika library to prevent file type confusion attacks

SQL Injection Prevention

Prepared statements: All database queries use parameterized prepared statements with JDBC/MyBatis
No dynamic query building: All queries are static templates with parameter binding
Input escaping: Special characters in filenames and metadata are escaped using database-specific escaping functions

Authentication Middleware Design

JWT-based authentication: Stateless JWT tokens with 24-hour expiration for API access
OAuth2 integration: For enterprise SSO support (Google Workspace, Microsoft Azure AD)
Token refresh mechanism: 12-hour refresh tokens with automatic session invalidation on logout
Authentication middleware: Spring Security filter chain intercepting all requests and validating JWT tokens

Authorization Enforcement

Role-Based Access Control (RBAC):
- Admin (full access)
- Editor (read/write to owned documents)
- Viewer (read-only access to shared documents)
Permission checking:
- Document ownership verification before write operations
- Group membership validation for shared document access
- Resource-level authorization using Spring Security annotations

Sensitive Data Encryption

At-rest encryption: AES-256-GCM encryption using AWS KMS for document content and metadata
In-transit encryption: TLS 1.3 enforced for all API communications (HTTPS)
Key rotation: Automatic key rotation every 90 days with seamless migration

PII Handling and Masking Strategy

PII classification: Document metadata fields containing personal data (author, last modified by) are flagged
Masking in logs: PII fields in audit logs are masked (e.g., "john.doe@example.com" → "j***.e@***.com")
Data retention: PII data is deleted after 3 years of inactivity or upon explicit user request

Secrets Management

AWS Secrets Manager: For storing database credentials, encryption keys, and third-party API tokens
Secret rotation: Automatic rotation every 30 days with zero-downtime deployment
Access control: IAM policies limiting access to secrets to only necessary service roles

Audit Logging Design

Log fields:
- timestamp, user_id, action_type, document_id, ip_address, user_agent, request_id
Event types:
- Document creation, modification, deletion, sharing, access
Retention policy: 1 year for audit logs with daily backups to S3 Glacier
Compliance: Logs are structured in JSON format and support SIEM integration for regulatory compliance (GDPR, HIPAA)

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

Core domain logic: 85% code coverage with 100% branch coverage for business rules and validation logic
Service layer methods: 90% coverage with focus on edge cases and error paths
Repository layer operations: 80% coverage for CRUD operations with parameterized test cases
Value object validation: 100% coverage for all validation constraints and boundary conditions
Exception handling paths: 100% coverage for all expected error scenarios and fallback logic

Integration Testing Strategy

Database integration tests: Use testcontainers with PostgreSQL 14+ for end-to-end persistence validation, including transaction boundary testing and concurrent access scenarios
External service mocks: WireMock for third-party document processing API, AWS S3 mock with MinIO for storage integration
Cross-service communication: Embedded Redis and Kafka for async event handling testing, ensuring proper message serialization and consumer behavior
Authentication integration: Mock OAuth2 server using Spring Security's test utilities to validate JWT token processing and permission checks

Contract Testing

API contract testing: Implement Pact for consumer-driven contracts between document service and dependent services (e.g., user service, notification service)
Message contract validation: Use Avro schema registry with schema versioning to validate event payload structure and backward compatibility for document upload events
External API contracts: Define and test against documented OpenAPI specs using Swagger UI integration tests to ensure contract compliance

Performance Testing Tool and Key Scenarios

Load testing: Use k6 for concurrent document upload and retrieval scenarios with 1000+ concurrent users
Stress testing: JMeter for simulating peak document processing loads (5000 concurrent uploads)
Key performance scenarios:
- Document upload with 10MB file: p95 < 2s, throughput > 500 ops/sec
- Document search query: p95 < 500ms, throughput > 1000 ops/sec
- Concurrent document retrieval: p95 < 1s, throughput > 2000 ops/sec
Latency targets: p50 < 100ms, p95 < 500ms, p99 < 2s for all primary operations

Test Data Management Strategy

Test fixtures: YAML-based fixtures with predefined document metadata, user roles, and permission sets for consistent test environments
Factory pattern: Implement Factory classes for generating valid document objects with various content types, sizes, and ownership scenarios
Database seeding: Use Liquibase changelogs for test data setup with clean state management between test runs
Data isolation: Each test suite runs in isolated database schema with automatic cleanup to prevent cross-test contamination
Sample data generation: Use libraries like Java Faker for generating realistic but non-sensitive test content (e.g., document titles, descriptions)

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective search, filtering, and correlation across services:

{
  "timestamp": "2023-10-15T14:30:45.123Z",
  "level": "INFO|WARN|ERROR",
  "service": "document-management-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user-123",
  "entity_id": "doc-456",
  "operation": "uploadDocument",
  "message": "Document uploaded successfully",
  "duration_ms": 125,
  "error_code": "DOCUMENT_UPLOAD_FAILED",
  "error_message": "Failed to process document",
  "stack_trace": "...",
  "request_id": "req-789"
}

PII masking rules:

User emails and names are masked in logs
Document content is never logged (only metadata)
Sensitive fields like API keys are redacted

Distributed Tracing Instrumentation

Using OpenTelemetry for distributed tracing with custom spans:

Key Operations to Trace:

DocumentUpload - from API entry point to storage
DocumentSearch - query processing and result aggregation
VersionHistory - version creation and retrieval
AccessControlCheck - permission validation

Span Attributes:

document.id: Document identifier
document.type: File type (PDF, DOCX, etc.)
user.id: Authenticated user ID
request.duration.ms: End-to-end operation time
storage.backend: Storage system used (S3, local disk)
access.level: Permission level (read, write, admin)

Metrics Design

RED Method Metrics:

Metric Name	Type	Labels	Description
`document_uploads_total`	Counter	status, user_type	Total document uploads
`document_upload_duration_seconds`	Histogram	status, storage_backend	Upload processing time
`document_search_requests_total`	Counter	status, query_type	Search operation count
`document_search_duration_seconds`	Histogram	status, backend	Search query processing time
`active_document_versions`	Gauge	document_id	Current version count per doc
`storage_used_bytes`	Gauge	storage_type	Total storage consumed

Additional Service Metrics:

cache_hit_rate (Counter + Gauge)
api_request_count (Counter with method, status labels)
db_connection_pool_active (Gauge)
failed_auth_attempts (Counter)

Health Check Endpoint Design

GET /health
GET /ready
GET /live
GET /startup

Response Format:

{
  "status": "healthy|degraded|unhealthy",
  "checks": {
    "database": {"status": "healthy", "details": "Connected to primary DB"},
    "storage": {"status": "healthy", "details": "S3 connection OK"},
    "cache": {"status": "healthy", "details": "Redis cluster healthy"},
    "auth_service": {"status": "healthy", "details": "OAuth2 service reachable"}
  },
  "timestamp": "2023-10-15T14:30:45.123Z",
  "uptime_seconds": 3600
}

Health Check Logic:

/live - Basic service liveness (process running)
/ready - Service readiness (DB, cache, auth ready)
/health - Comprehensive health including all dependencies
/startup - Startup validation (config loaded, initial connections established)

Alerting Thresholds

Critical Alerts (Escalation to On-Call):

document_upload_duration_seconds{p99} > 5s - Alert if 99th percentile exceeds 5 seconds
document_upload_failures_total{status="error"} > 10 in 5m - Alert if 10+ uploads fail in 5 minutes
cache_hit_rate < 70% - Alert if cache efficiency drops below 70%
db_connection_pool_active > 90% - Alert if pool usage exceeds 90%

Warning Alerts:

document_search_duration_seconds{p99} > 2s - Alert if search latency exceeds 2 seconds
storage_used_bytes > 80% - Alert if storage usage exceeds 80%
api_request_count{status="5xx"} > 5 in 1m - Alert if 5xx errors occur frequently

On-Call Escalation Path:

First level: Service team (5 minutes)
Second level: SRE team (30 minutes)
Third level: Platform engineering (2 hours)

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

File format support matrix: Which document formats need to be supported natively (PDF, DOCX, XLSX, PPTX) vs. converted to a common format? This impacts storage strategy and processing complexity.
Version control granularity: Should versioning be at document level or at individual file chunks? This affects storage efficiency and restore capabilities.
Retention policy enforcement: How strictly should we enforce data retention policies? Should they be configurable per organization or global?
Cross-document search capabilities: What level of cross-document search (full-text, metadata-based) is required? This impacts indexing strategy and query complexity.

Known trade-offs and justifications

Storage vs. Performance Trade-off: Storing multiple file versions in the same bucket vs. separate buckets for performance. The former reduces complexity but increases retrieval time and storage costs.
Consistency vs. Availability Trade-off: Strong consistency for document metadata vs. eventual consistency for file content operations. Metadata consistency is critical for search and access control, while content can tolerate eventual consistency.
Caching vs. Memory Usage Trade-off: Caching large document content vs. keeping only metadata in memory. This trade-off affects system responsiveness and resource consumption.
Indexing vs. Storage Cost Trade-off: Comprehensive indexing for search features vs. storage overhead and maintenance cost of indices.

Risks and mitigation strategies

Data Loss Risk: Risk of losing documents due to storage corruption or deletion. Mitigation includes implementing multi-region backups, version control, and audit logging.
Scalability Risk: Risk of system scaling issues with large document volumes. Mitigation includes sharding by organization, implementing efficient indexing strategies, and using distributed storage.
Security Risk: Risk of unauthorized access to sensitive documents. Mitigation includes robust RBAC, encryption at rest and in transit, and comprehensive audit logging.
Performance Degradation Risk: Risk of slow search and retrieval operations as document count grows. Mitigation includes implementing proper indexing, caching strategies, and query optimization.

Technical debt acknowledged

Legacy format support: Supporting older document formats (e.g., .doc, .xls) may require additional dependencies and maintenance overhead that could become technical debt.
Custom search implementation: Building custom search capabilities instead of leveraging existing solutions like Elasticsearch or Solr introduces maintenance burden and potential performance limitations.
Inconsistent metadata handling: Different document types may have varying metadata structures, leading to complexity in normalization and querying.
File size handling: Large file handling (e.g., >1GB) may require special processing pipelines and could introduce complexity in concurrent access patterns.