Pastebin Service Low-Level Design - Scalable Text Storage System

This document presents a scalable low-level design for a pastebin service that handles text storage, retrieval, and expiration with distributed caching and efficient database interactions.

1. Problem Statement & Requirements

Clarifying questions

1. What scale expectations are there for this pastebin service?

   • How many paste documents are expected to be created per second?
   • What is the anticipated peak concurrent read/write operations?

2. What consistency requirements exist for paste retrieval?

   • Should reads always return the latest version, or is eventual consistency acceptable?
   • Are there any specific SLA requirements for read latency?

3. What are the key entities and their relationships?

   • Is there a user model, or is it anonymous paste creation?
   • Do pastes have expiration policies or are they persistent?

4. What is the read/write ratio?

   • Is it predominantly reads (e.g., 90% read, 10% write) or more balanced?
   • How does this impact caching and data access patterns?

5. How sensitive is the system to latency?

   • What are the p50/p95/p99 latency targets for paste creation and retrieval?

6. What authentication model is required?

   • Should users be authenticated, or is it a completely anonymous service?

7. Are there any external integrations or dependencies?

   • Should pastes be exportable to other systems or have webhook support?
   • Is there a need for content moderation or virus scanning?

Functional requirements

• Users can create pastes with content and optional metadata (e.g., title, language, expiration time).
• Users can retrieve pastes by unique identifier.
• Users can optionally delete pastes (if they are owner or have admin rights).
• Pastes may have expiration times (e.g., 1 hour, 1 day, etc.).
• Pastes can be shared via a unique URL.
• Anonymous paste creation is supported (no authentication required).
• Optional: Paste can be edited by owner within a time window.
• Optional: Paste history or versioning support.

Non-functional requirements

• Latency targets:

  • p50: < 50ms
  • p95: < 200ms
  • p99: < 500ms

• Throughput:

  • 10,000 paste creations/sec (peak)
  • 50,000 paste reads/sec (peak)

• Availability:

  • 99.9% uptime (5 minutes of downtime/month max)

• Storage:

  • Support for pastes up to 10MB in size
  • Data durability and replication (e.g., 3x replication)

Constraints and assumptions

• Pastes are stored in a single database (e.g., PostgreSQL or MongoDB) for simplicity.
• No complex content analysis or moderation is required initially.
• Authentication and authorization are optional features (can be added later).
• The service is designed for a single region with no cross-region replication.
• Paste content is stored in plaintext (no encryption at rest or in transit for now).
• No external dependencies like CDN or file storage systems are required.

Out of scope

• Paste content encryption or decryption.
• Content moderation, virus scanning, or spam detection.
• Paste versioning or history tracking (can be added later).
• Multi-region replication or geographic distribution.
• Paste sharing with specific users or groups (public/private paste model).
• Analytics dashboard or paste statistics.
• API rate limiting or user quotas (can be added later).

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for the Pastebin service due to:

• Strong typing and schema validation at compile time
• Efficient binary serialization for high-throughput scenarios
• Built-in support for streaming and bidirectional communication
• Native support for versioning and schema evolution
• Better performance characteristics compared to REST for this use case

Endpoint Definitions

Method	Path	Description	Auth	Rate Limit
CreatePaste	POST /v1/pastes	Create a new paste	Optional JWT	100/hour
GetPaste	GET /v1/pastes/{id}	Retrieve a paste by ID	Optional JWT	1000/hour
UpdatePaste	PUT /v1/pastes/{id}	Update paste content	JWT required	50/hour
DeletePaste	DELETE /v1/pastes/{id}	Delete a paste	JWT required	20/hour
ListPastes	GET /v1/pastes	List user's pastes	JWT required	50/hour
GetPasteStats	GET /v1/pastes/{id}/stats	Get paste view statistics	Optional JWT	200/hour

Request and Response Schemas

CreatePasteRequest

{
  "content": "string",
  "title": "string",
  "expiration_hours": 24,
  "is_private": false,
  "language": "plaintext"
}

CreatePasteResponse

{
  "id": "uuid",
  "url": "string",
  "created_at": "timestamp",
  "expires_at": "timestamp"
}

GetPasteResponse

{
  "id": "uuid",
  "content": "string",
  "title": "string",
  "created_at": "timestamp",
  "expires_at": "timestamp",
  "is_private": false,
  "language": "plaintext",
  "view_count": 0
}

UpdatePasteRequest

{
  "content": "string",
  "title": "string",
  "expiration_hours": 24,
  "is_private": false
}

ListPastesResponse

{
  "pastes": [
    {
      "id": "uuid",
      "title": "string",
      "created_at": "timestamp",
      "expires_at": "timestamp",
      "is_private": false,
      "view_count": 0
    }
  ],
  "next_cursor": "string"
}

HTTP Status Codes and Error Response Format

Standard HTTP status codes:

• 200: Success
• 201: Created
• 400: Bad Request
• 401: Unauthorized
• 403: Forbidden
• 404: Not Found
• 429: Rate Limited
• 500: Internal Server Error

Standardized error response:

{
  "error": {
    "code": "string",
    "message": "string",
    "details": "object"
  }
}

API Versioning Strategy

Versioned using path prefix (/v1/) with:

• Backward compatibility maintained for same major version
• Breaking changes in new major versions (v2, v3)
• Deprecation policy: 6-month notice before removal

Pagination Design

Cursor-based pagination selected because:

• More stable than offset-based for large datasets
• Consistent performance regardless of page size
• Handles concurrent modifications better
• Supports efficient forward/backward navigation

Authentication and Authorization

JWT-based authentication with:

• Optional for read operations (public pastes)
• Required for write/delete operations
• User roles: user, admin
• JWT claims: user_id, role, exp

Rate Limiting

Per-endpoint rate limiting:

• Create: 100/hour
• Get: 1000/hour
• Update/Delete: 50/hour
• List: 50/hour
• Stats: 200/hour

Implementation using Redis with sliding window algorithm.

Idempotency Design

Idempotency key strategy:

• Clients provide X-Idempotency-Key header
• Server stores operation result by key
• Repeated requests with same key return cached result
• Key TTL: 1 hour

gRPC Service Definition (Protocol Buffer)

syntax = "proto3";

package pastebin;

service PasteService {
  rpc CreatePaste(CreatePasteRequest) returns (CreatePasteResponse);
  rpc GetPaste(GetPasteRequest) returns (GetPasteResponse);
  rpc UpdatePaste(UpdatePasteRequest) returns (UpdatePasteResponse);
  rpc DeletePaste(DeletePasteRequest) returns (DeletePasteResponse);
  rpc ListPastes(ListPastesRequest) returns (ListPastesResponse);
  rpc GetPasteStats(GetPasteStatsRequest) returns (GetPasteStatsResponse);
}

message CreatePasteRequest {
  string content = 1;
  string title = 2;
  int32 expiration_hours = 3;
  bool is_private = 4;
  string language = 5;
}

message CreatePasteResponse {
  string id = 1;
  string url = 2;
  string created_at = 3;
  string expires_at = 4;
}

message GetPasteRequest {
  string id = 1;
}

message GetPasteResponse {
  string id = 1;
  string content = 2;
  string title = 3;
  string created_at = 4;
  string expires_at = 5;
  bool is_private = 6;
  string language = 7;
  int64 view_count = 8;
}

message UpdatePasteRequest {
  string id = 1;
  string content = 2;
  string title = 3;
  int32 expiration_hours = 4;
  bool is_private = 5;
}

message UpdatePasteResponse {
  string id = 1;
  string updated_at = 2;
}

message DeletePasteRequest {
  string id = 1;
}

message DeletePasteResponse {
  bool deleted = 1;
}

message ListPastesRequest {
  string user_id = 1;
  int32 limit = 2;
  string cursor = 3;
}

message ListPastesResponse {
  repeated PasteSummary pastes = 1;
  string next_cursor = 2;
}

message PasteSummary {
  string id = 1;
  string title = 2;
  string created_at = 3;
  string expires_at = 4;
  bool is_private = 5;
  int64 view_count = 6;
}

message GetPasteStatsRequest {
  string id = 1;
}

message GetPasteStatsResponse {
  int64 view_count = 1;
  int64 unique_views = 2;
  string last_viewed_at = 3;
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    PASTE {
        string id PK
        string content
        string title
        string author_id
        string language
        boolean is_private
        integer expiration_hours
        timestamp created_at
        timestamp updated_at
        timestamp expires_at
        string hash
    }

    USER {
        string id PK
        string email
        string username
        timestamp created_at
    }

    PASTE_TAG {
        string paste_id FK
        string tag_name
    }

    TAG {
        string name PK
        timestamp created_at
    }

    PASTE -> USER : author
    PASTE_TAG ||--o{ PASTE : has
    PASTE_TAG ||--o{ TAG : tagged_with

Table Schema Definitions

-- Main paste table
CREATE TABLE pastes (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    content TEXT NOT NULL,
    title VARCHAR(255),
    author_id UUID REFERENCES users(id) ON DELETE CASCADE,
    language VARCHAR(50),
    is_private BOOLEAN DEFAULT FALSE,
    expiration_hours INTEGER NOT NULL DEFAULT 24,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    expires_at TIMESTAMP WITH TIME ZONE,
    hash VARCHAR(64) UNIQUE NOT NULL,
    
    -- Indexes
    INDEX idx_paste_author (author_id),
    INDEX idx_paste_hash (hash),
    INDEX idx_paste_expires_at (expires_at),
    INDEX idx_paste_created_at (created_at)
);

-- User table
CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email VARCHAR(255) UNIQUE NOT NULL,
    username VARCHAR(100) UNIQUE NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    
    INDEX idx_user_email (email),
    INDEX idx_user_username (username)
);

-- Paste tags junction table
CREATE TABLE paste_tags (
    paste_id UUID REFERENCES pastes(id) ON DELETE CASCADE,
    tag_name VARCHAR(100) NOT NULL,
    
    PRIMARY KEY (paste_id, tag_name),
    INDEX idx_paste_tags_tag (tag_name)
);

-- Tags table for tag management
CREATE TABLE tags (
    name VARCHAR(100) PRIMARY KEY,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

Primary Key Strategy

UUID Strategy: Using UUIDs as primary keys for the paste entity provides several advantages:

• Global uniqueness across distributed systems without coordination
• No hotspots in database sharding scenarios since UUIDs are randomly generated
• No sequential patterns that could be exploited for enumeration attacks
• Better performance under high-concurrency write scenarios

Index Design

Index Name	Columns	Type	Purpose
idx_paste_author	author_id	Standard	Efficient user paste retrieval
idx_paste_hash	hash	Unique	Fast lookup by hash (URL shortening)
idx_paste_expires_at	expires_at	Standard	Efficient cleanup of expired pastes
idx_paste_created_at	created_at	Standard	Efficient time-based queries
idx_paste_tags_tag	tag_name	Standard	Fast tag-based filtering

Data Partitioning / Sharding Strategy

Time-based sharding by creation date:

• Partition pastes by month/year to enable efficient time-range queries
• Use composite partition key: (created_at::date / 30) to create monthly partitions
• Allows for easy cleanup of expired pastes and efficient time-range queries

Soft Delete vs. Hard Delete Decision

Soft delete strategy with expires_at field:

• Pastes are marked as expired rather than deleted to maintain audit trail
• Allows for potential recovery of content if needed
• Enables analytics on paste lifecycle and popularity metrics
• Reduces risk of data loss from accidental deletion or system errors

Audit Trail Design

Audit logging approach:

• Track paste creation, modification, and deletion events
• Log: user_id, paste_id, action_type, timestamp, IP address (if available)
• Store in separate audit table or as part of paste metadata
• Include content hash for content verification

Data Archival Strategy and Retention Policy

Archival approach:

• Move pastes older than 1 year to cold storage (e.g., S3 Glacier)
• Keep metadata in primary database for search and reference
• Archive content hash and metadata only to reduce storage costs
• Implement retention policy: 1 year active, 5 years archived, then permanently deleted

// Sample Java class for paste entity with audit fields
public class Paste {
    private String id;
    private String content;
    private String title;
    private String authorId;
    private String language;
    private boolean isPrivate;
    private Integer expirationHours;
    private LocalDateTime createdAt;
    private LocalDateTime updatedAt;
    private LocalDateTime expiresAt;
    private String hash;
    
    // Constructor, getters, setters...
}

4. Class Design & Service Layer

classDiagram
    class PasteService {
        +createPaste(CreatePasteRequest) PasteResponse
        +getPaste(String) PasteResponse
        +updatePaste(String, UpdatePasteRequest) PasteResponse
        +deletePaste(String) DeleteResponse
        +listPastes(String, int, int) PasteListResponse
    }
    
    class PasteRepository {
        +save(Paste) Paste
        +findById(String) Optional~Paste~
        +findByUserAndId(String, String) Optional~Paste~
        +deleteById(String) void
        +findAllByUser(String, int, int) List~Paste~
    }
    
    class Paste {
        -id: String
        -content: String
        -userId: String
        -createdAt: Instant
        -expiresAt: Instant
        -isPublic: boolean
        -accessCount: long
        +incrementAccessCount()
    }
    
    class PasteValidator {
        +validateCreateRequest(CreatePasteRequest) ValidationResult
        +validateUpdateRequest(UpdatePasteRequest) ValidationResult
    }
    
    class PasteEventPublisher {
        +publishPasteCreated(Paste) void
        +publishPasteUpdated(Paste) void
        +publishPasteDeleted(String) void
    }
    
    class PasteMetrics {
        +recordPasteCreation() void
        +recordPasteAccess(String) void
        +recordPasteDeletion() void
    }
    
    class PasteCleanupService {
        +cleanupExpiredPastes() int
    }
    
    PasteService --> PasteRepository : uses
    PasteService --> PasteValidator : validates
    PasteService --> PasteEventPublisher : publishes events
    PasteService --> PasteMetrics : records metrics
    PasteService --> PasteCleanupService : schedules cleanup
    
    class CreatePasteRequest {
        +content: String
        +userId: String
        +isPublic: boolean
        +expirationHours: int
    }
    
    class PasteResponse {
        +id: String
        +content: String
        +userId: String
        +createdAt: Instant
        +expiresAt: Instant
        +isPublic: boolean
        +accessCount: long
    }
    
    class UpdatePasteRequest {
        +content: String
        +isPublic: boolean
        +expirationHours: int
    }
    
    class PasteListResponse {
        +pastes: List~PasteResponse~
        +total: long
        +page: int
        +size: int
    }
    
    class DeleteResponse {
        +success: boolean
        +message: String
    }

stateDiagram-v2
    [*] --> Active
    Active --> Expired : expiration time reached
    Active --> Deleted : deleted by user or admin
    Expired --> [*]
    Deleted --> [*]

Service Layer Overview

The PasteService is the primary orchestrator for paste operations, handling business logic and coordinating between repositories, validators, and external systems. It implements the core paste lifecycle management including creation, retrieval, updates, deletion, and listing.

Business Logic Patterns

Strategy Pattern for Expiration Handling

public interface ExpirationStrategy {
    Instant calculateExpiration(Instant now, int hours);
}

@Component
public class DefaultExpirationStrategy implements ExpirationStrategy {
    @Override
    public Instant calculateExpiration(Instant now, int hours) {
        return now.plus(hours, ChronoUnit.HOURS);
    }
}

@Component
public class NeverExpireStrategy implements ExpirationStrategy {
    @Override
    public Instant calculateExpiration(Instant now, int hours) {
        return now.plus(100, ChronoUnit.YEARS); // effectively never
    }
}

Critical-Path Method Implementation

@Transactional
public PasteResponse createPaste(CreatePasteRequest request) {
    // Validate input
    ValidationResult validation = pasteValidator.validateCreateRequest(request);
    if (!validation.isValid()) {
        throw new ValidationException("Invalid paste creation request", validation.getErrors());
    }

    // Generate unique ID
    String pasteId = UUID.randomUUID().toString();
    
    // Calculate expiration time
    Instant now = Instant.now();
    Instant expiresAt = expirationStrategy.calculateExpiration(now, request.getExpirationHours());
    
    // Create paste object
    Paste paste = Paste.builder()
        .id(pasteId)
        .content(request.getContent())
        .userId(request.getUserId())
        .createdAt(now)
        .expiresAt(expiresAt)
        .isPublic(request.isPublic())
        .accessCount(0L)
        .build();
    
    // Save to database
    Paste savedPaste = pasteRepository.save(paste);
    
    // Publish event for monitoring and analytics
    pasteEventPublisher.publishPasteCreated(savedPaste);
    
    // Record metrics
    pasteMetrics.recordPasteCreation();
    
    // Return response
    return PasteResponse.builder()
        .id(savedPaste.getId())
        .content(savedPaste.getContent())
        .userId(savedPaste.getUserId())
        .createdAt(savedPaste.getCreatedAt())
        .expiresAt(savedPaste.getExpiresAt())
        .isPublic(savedPaste.isPublic())
        .accessCount(savedPaste.getAccessCount())
        .build();
}

Transaction Boundary Design

The transaction boundary encompasses:

1. Paste creation in database
2. Event publishing (using outbox pattern for reliability)
3. Metrics recording

All operations within this method are atomic, ensuring consistency between data persistence and event publication.

Service Interface Contracts

public interface PasteService {
    PasteResponse createPaste(CreatePasteRequest request);
    PasteResponse getPaste(String pasteId);
    PasteResponse updatePaste(String pasteId, UpdatePasteRequest request);
    DeleteResponse deletePaste(String pasteId);
    PasteListResponse listPastes(String userId, int page, int size);
}

Saga Pattern for Distributed Transactions

For complex operations involving external systems (e.g., analytics service, notification service), a saga pattern would be implemented using a state machine that coordinates multiple steps with compensation logic.

Domain Object Design

@Builder
@Data
@NoArgsConstructor
@AllArgsConstructor
@Entity
@Table(name = "pastes")
public class Paste {
    @Id
    @Column(name = "id", nullable = false)
    private String id;
    
    @Lob
    @Column(name = "content", nullable = false)
    private String content;
    
    @Column(name = "user_id", nullable = false)
    private String userId;
    
    @Column(name = "created_at", nullable = false)
    private Instant createdAt;
    
    @Column(name = "expires_at")
    private Instant expiresAt;
    
    @Column(name = "is_public", nullable = false)
    private boolean isPublic;
    
    @Column(name = "access_count", nullable = false)
    private long accessCount;
    
    public void incrementAccessCount() {
        this.accessCount++;
    }
}

Key Design Decisions

1. Service Layer Isolation: Business logic is encapsulated in PasteService to ensure clean separation of concerns
2. Immutable Responses: Response objects are built with immutable fields to prevent accidental mutation
3. Event-Driven Architecture: Events are published for external systems to react to paste operations
4. Metrics Integration: Built-in metrics collection for operational monitoring and performance tracking
5. Idempotency Consideration: Paste IDs are UUIDs to support idempotent creation operations
6. Transaction Strategy: Single database transaction for all paste operations to ensure consistency

5. Concurrency & Thread Safety

Concurrency Model

The Pastebin service will use a thread-safe, asynchronous processing model with Java's CompletableFuture and ExecutorService for handling concurrent paste creation and retrieval operations. The service will leverage a fixed thread pool (ForkJoinPool.commonPool()) for parallel processing of independent operations like paste creation, content retrieval, and metadata updates.

Race Condition Prevention Strategy

Race conditions will be prevented through:

1. Atomic operations for paste ID generation using AtomicLong
2. Database-level constraints for paste uniqueness and soft-delete consistency
3. Read-Write locks for metadata operations during content updates
4. Optimistic locking with version fields in the paste entity

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected because:

• Paste creation and retrieval are primarily read-heavy operations
• Content updates are rare and typically happen in a single transaction
• The system expects low contention for paste operations
• Pessimistic locking would introduce unnecessary latency and reduce throughput

Deadlock Prevention Strategy

Deadlock prevention is achieved through:

1. Fixed ordering of resource acquisition - all operations acquire locks in consistent order (e.g., by paste ID)
2. Timeout-based lock acquisition - operations will fail fast if locks cannot be acquired within 50ms
3. Hierarchical lock management - no cross-domain locking (e.g., never lock paste and user simultaneously)
4. Transaction isolation level set to READ_COMMITTED to prevent phantom reads

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is handled through:

1. Idempotency keys in request headers (e.g., X-Idempotency-Key)
2. Cache-based deduplication for recent paste creation requests
3. Database constraint enforcement - unique constraints on paste identifiers and content hashes

public class PasteService {
    private final Map<String, CompletableFuture<Paste>> idempotencyCache = new ConcurrentHashMap<>();
    private final ExecutorService executor = Executors.newFixedThreadPool(20);
    
    public CompletableFuture<Paste> createPasteAsync(PasteRequest request) {
        String idempotencyKey = extractIdempotencyKey(request);
        
        // Check for existing operation with same idempotency key
        return idempotencyCache.computeIfAbsent(idempotencyKey, key -> {
            CompletableFuture<Paste> future = CompletableFuture.supplyAsync(() -> {
                try {
                    // Check if paste already exists by hash or identifier
                    Paste existing = pasteRepository.findByContentHash(request.getContentHash());
                    if (existing != null) {
                        return existing;
                    }
                    
                    // Create new paste with optimistic locking
                    Paste paste = createPasteWithVersion(request);
                    return pasteRepository.save(paste);
                } catch (Exception e) {
                    throw new RuntimeException("Paste creation failed", e);
                }
            }, executor);
            
            // Remove from cache after completion to prevent memory leak
            future.thenRun(() -> idempotencyCache.remove(idempotencyKey));
            return future;
        });
    }
    
    private String extractIdempotencyKey(PasteRequest request) {
        return Optional.ofNullable(request.getIdempotencyKey())
                      .orElse(UUID.randomUUID().toString());
    }
}

Thread Safety of Core Components

1. PasteRepository - Thread-safe through database transaction isolation and proper connection pooling
2. PasteContentService - Uses thread-safe concurrent collections for caching content metadata
3. PasteMetricsService - Uses atomic counters and thread-safe metrics registry
4. PasteCleanupService - Single-threaded execution with proper synchronization for cleanup tasks

Concurrency Control in Write Operations

Write operations will use:

1. Database transaction boundaries with proper isolation levels
2. Version fields in paste entities for optimistic concurrency control
3. Batch processing for bulk content operations to reduce lock contention

Performance Considerations

• Thread pool size tuned to handle peak concurrent paste creation (20-50 threads)
• Read-heavy operations will not block write operations through proper database indexing
• Content retrieval operations will use read replicas to avoid write lock contention

Testing Strategy for Concurrency

• Stress testing with concurrent paste creation (1000+ simultaneous requests)
• Race condition detection using JMH microbenchmarks
• Thread dump analysis for deadlocks during load testing
• Integration tests with concurrent database operations using testcontainers

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant Auth as Auth Service
    participant PasteService as Paste Service
    participant Storage as Storage Layer
    participant Cache as Redis Cache

    U->>API: POST /pastes
    API->>Auth: Validate JWT
    Auth-->>API: Valid token
    API->>PasteService: CreatePaste(payload)
    PasteService->>Cache: Check if paste exists (cache-aside)
    Cache-->>PasteService: Miss
    PasteService->>Storage: Save paste to DB
    Storage-->>PasteService: Paste ID
    PasteService->>Cache: Cache paste with TTL
    Cache-->>PasteService: Cached
    PasteService-->>API: Return paste ID and URL
    API-->>U: 201 Created with paste URL

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant PasteService as Paste Service
    participant Storage as Storage Layer
    participant Cache as Redis Cache

    U->>API: GET /pastes/{id}
    API->>PasteService: GetPaste(id)
    PasteService->>Cache: Check if paste exists
    Cache-->>PasteService: Hit
    PasteService-->>API: Return paste content
    API-->>U: 200 OK with content

    Note over PasteService,Cache: Cache miss fallback
    PasteService->>Cache: Check if paste exists
    Cache-->>PasteService: Miss
    PasteService->>Storage: Fetch paste from DB
    Storage-->>PasteService: Paste content
    PasteService->>Cache: Cache paste with TTL
    Cache-->>PasteService: Cached
    PasteService-->>API: Return paste content
    API-->>U: 200 OK with content

sequenceDiagram
    participant U as User
    participant API as API Gateway
    participant PasteService as Paste Service
    participant Storage as Storage Layer
    participant EventPublisher as Event Publisher
    participant NotificationService as Notification Service

    U->>API: POST /pastes
    API->>PasteService: CreatePaste(payload)
    PasteService->>Storage: Save paste to DB
    Storage-->>PasteService: Paste ID
    PasteService->>EventPublisher: Publish PasteCreatedEvent
    EventPublisher-->>PasteService: Event published
    PasteService-->>API: Return paste ID and URL
    API-->>U: 201 Created with paste URL

    Note over PasteService,EventPublisher: Async notification flow
    EventPublisher->>NotificationService: Send paste created notification
    NotificationService-->>EventPublisher: Acknowledged

7. Caching Design

Cache Strategy Selection

For a Pastebin service, we'll use cache-aside pattern with LRU eviction policy. This approach provides optimal control over cache behavior, allows for easy cache warming, and supports the typical read-heavy workload pattern where most operations are reads of existing pastes.

Cache Key Design

Cache keys will follow a structured namespace pattern:

paste:{pasteId}:{version}
user:{userId}:{pasteId}

TTL Strategy

Data Type	TTL	Reason
Paste content	24h	Content is immutable once created, but may be accessed infrequently
Paste metadata	1h	Metadata changes less frequently than content
User recent pastes	30m	User activity is dynamic and should not be cached too long
Popular paste stats	5m	Stats change frequently and need to be fresh

Cache Eviction Policy

We'll implement LRU (Least Recently Used) with probabilistic early expiry to prevent cache stampedes:

• Primary eviction: LRU
• Secondary: TTL-based cleanup every 5 minutes

Cache Stampede Prevention

To prevent thundering herd on cache misses:

1. Probabilistic early expiry: Set TTL to 2x actual TTL and use random offset for cache refresh
2. Distributed locks: Use Redis distributed locks when cache miss occurs to ensure only one request computes the value
3. Cache warming: Pre-populate popular pastes during off-peak hours

Distributed Cache Architecture

We'll use Redis Cluster with:

• 3 master nodes for data distribution
• 3 replica nodes per shard for high availability
• Redis Sentinel for automatic failover
• Cache client with connection pooling (min 10, max 50 connections)

Cache Invalidation Strategy

We'll use event-driven invalidation:

1. When paste is created/updated → invalidate related cache keys
2. When paste is deleted → invalidate all associated cache entries
3. Periodic cleanup of expired keys using Redis's built-in TTL mechanism

public class PasteCacheService {
    private final RedisTemplate<String, String> redisTemplate;
    private final ObjectMapper objectMapper;
    
    public PasteCacheService(RedisTemplate<String, String> redisTemplate, 
                           ObjectMapper objectMapper) {
        this.redisTemplate = redisTemplate;
        this.objectMapper = objectMapper;
    }
    
    public Paste getCachedPaste(String pasteId) {
        String cacheKey = "paste:" + pasteId;
        String cachedValue = redisTemplate.opsForValue().get(cacheKey);
        
        if (cachedValue != null) {
            try {
                return objectMapper.readValue(cachedValue, Paste.class);
            } catch (IOException e) {
                // Log error but continue with cache miss
                log.error("Failed to deserialize cached paste: {}", pasteId, e);
            }
        }
        
        return null;
    }
    
    public void cachePaste(String pasteId, Paste paste) {
        String cacheKey = "paste:" + pasteId;
        try {
            String value = objectMapper.writeValueAsString(paste);
            redisTemplate.opsForValue().set(
                cacheKey, 
                value, 
                Duration.ofHours(24)
            );
        } catch (IOException e) {
            log.error("Failed to serialize paste for caching: {}", pasteId, e);
        }
    }
    
    public void invalidatePasteCache(String pasteId) {
        String[] keys = {
            "paste:" + pasteId,
            "user:" + pasteId
        };
        
        redisTemplate.delete(Arrays.asList(keys));
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and persistence concerns. Each entity has its own repository interface with standardized CRUD operations.

public interface PasteRepository {
    Paste save(Paste paste);
    Optional<Paste> findById(String id);
    List<Paste> findByUserId(String userId, Pageable pageable);
    void deleteById(String id);
    boolean existsById(String id);
}

public interface PasteStatsRepository {
    PasteStats save(PasteStats stats);
    Optional<PasteStats> findByPasteId(String pasteId);
}

ORM vs. Raw Query Decisions

For the main paste entity, raw SQL queries are preferred over ORM for performance and control, particularly for read-heavy operations. The ORM is used only for complex joins or when the query becomes unwieldy.

@Repository
public class PasteRepositoryImpl implements PasteRepository {
    
    private final JdbcTemplate jdbcTemplate;
    private final ObjectMapper objectMapper;
    
    public PasteRepositoryImpl(JdbcTemplate jdbcTemplate, ObjectMapper objectMapper) {
        this.jdbcTemplate = jdbcTemplate;
        this.objectMapper = objectMapper;
    }
    
    @Override
    public Paste save(Paste paste) {
        String sql = """
            INSERT INTO pastes (id, content, user_id, created_at, expires_at, is_public, metadata)
            VALUES (?, ?, ?, ?, ?, ?, ?)
            """;
            
        jdbcTemplate.update(sql, paste.getId(), paste.getContent(), paste.getUserId(),
                           paste.getCreatedAt(), paste.getExpiresAt(), paste.isPublic(),
                           objectMapper.writeValueAsString(paste.getMetadata()));
        
        return paste;
    }
    
    @Override
    public Optional<Paste> findById(String id) {
        String sql = """
            SELECT id, content, user_id, created_at, expires_at, is_public, metadata
            FROM pastes WHERE id = ? AND (expires_at IS NULL OR expires_at > NOW())
            """;
            
        try {
            return jdbcTemplate.queryForObject(sql, new Object[]{id}, (rs, rowNum) -> {
                Paste paste = new Paste();
                paste.setId(rs.getString("id"));
                paste.setContent(rs.getString("content"));
                paste.setUserId(rs.getString("user_id"));
                paste.setCreatedAt(rs.getTimestamp("created_at").toInstant());
                paste.setExpiresAt(rs.getTimestamp("expires_at") != null ? 
                                  rs.getTimestamp("expires_at").toInstant() : null);
                paste.setPublic(rs.getBoolean("is_public"));
                paste.setMetadata(objectMapper.readValue(rs.getString("metadata"), Map.class));
                return paste;
            });
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        } catch (JsonProcessingException e) {
            throw new RuntimeException("Failed to deserialize metadata", e);
        }
    }
}

Read vs. Write Separation

The system implements CQRS (Command Query Responsibility Segregation) pattern with separate data paths for reads and writes:

// Write model - used for mutations (save, delete)
@Repository
public class PasteWriteRepository {
    // Implements full CRUD operations for write-heavy operations
    
    @Transactional
    public Paste save(Paste paste) {
        // Write to primary database
        return pasteRepository.save(paste);
    }
}

// Read model - optimized for queries (find by user, search)
@Repository
public class PasteReadRepository {
    // Uses materialized views or denormalized tables for faster reads
    
    public List<PasteSummary> findPublicPastes(Pageable pageable) {
        String sql = """
            SELECT id, user_id, created_at, is_public
            FROM pastes 
            WHERE is_public = true AND (expires_at IS NULL OR expires_at > NOW())
            ORDER BY created_at DESC
            LIMIT ? OFFSET ?
            """;
            
        return jdbcTemplate.query(sql, new Object[]{
            pageable.getPageSize(), 
            pageable.getPageNumber() * pageable.getPageSize()
        }, (rs, rowNum) -> new PasteSummary(
            rs.getString("id"),
            rs.getString("user_id"),
            rs.getTimestamp("created_at").toInstant()
        ));
    }
}

Connection Pool Configuration

Connection pool is configured with optimal settings for high-throughput, low-latency operations:

# application.yml
spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      validation-timeout: 5000
      leak-detection-threshold: 60000

N+1 Query Prevention Strategy

To prevent N+1 query problems, eager loading is used with batch operations:

@Repository
public class PasteRepositoryImpl {
    
    public List<Paste> findByIds(List<String> ids) {
        if (ids.isEmpty()) return Collections.emptyList();
        
        String placeholders = String.join(",", Collections.nCopies(ids.size(), "?"));
        String sql = "SELECT * FROM pastes WHERE id IN (" + placeholders + ")";
        
        return jdbcTemplate.query(sql, ids.toArray(), (rs, rowNum) -> {
            // Map result to Paste object
        });
    }
    
    public List<PasteWithStats> findPasteWithStats(List<String> pasteIds) {
        // Use JOIN to fetch paste and stats in single query
        String sql = """
            SELECT p.id, p.content, p.user_id, p.created_at, 
                   ps.view_count, ps.last_viewed_at
            FROM pastes p
            LEFT JOIN paste_stats ps ON p.id = ps.paste_id
            WHERE p.id IN (%s)
            """.formatted(String.join(",", Collections.nCopies(pasteIds.size(), "?")));
            
        return jdbcTemplate.query(sql, pasteIds.toArray(), (rs, rowNum) -> 
            new PasteWithStats(
                rs.getString("id"),
                rs.getString("content"),
                rs.getString("user_id"),
                rs.getInt("view_count")
            ));
    }
}

Batch Operation Design

Batch operations are implemented for bulk inserts and updates:

@Repository
public class PasteBatchRepository {
    
    public void batchSave(List<Paste> pastes) {
        String sql = """
            INSERT INTO pastes (id, content, user_id, created_at, expires_at, is_public, metadata)
            VALUES (?, ?, ?, ?, ?, ?, ?)
            """;
            
        jdbcTemplate.batchUpdate(sql, pastes, pastes.size(), (ps, paste) -> {
            ps.setString(1, paste.getId());
            ps.setString(2, paste.getContent());
            ps.setString(3, paste.getUserId());
            ps.setTimestamp(4, Timestamp.from(paste.getCreatedAt()));
            ps.setTimestamp(5, paste.getExpiresAt() != null ? 
                          Timestamp.from(paste.getExpiresAt()) : null);
            ps.setBoolean(6, paste.isPublic());
            ps.setString(7, objectMapper.writeValueAsString(paste.getMetadata()));
        });
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The Pastebin service will use a hybrid synchronous/asynchronous approach. The core paste creation and retrieval operations must be synchronous to ensure immediate response to users. However, secondary operations like analytics tracking, content moderation, and backup processes will be handled asynchronously to avoid impacting user experience.

Key asynchronous triggers:

• Paste creation → trigger analytics event and content moderation
• Paste deletion → trigger backup archival and analytics update
• Paste expiration → trigger cleanup and notification

Message schema design with sample payload

{
  "version": "1.0",
  "event_type": "paste.created",
  "timestamp": "2023-11-15T14:30:00Z",
  "payload": {
    "paste_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "user_id": "user_12345",
    "content_length": 1024,
    "created_at": "2023-11-15T14:30:00Z",
    "language": "java",
    "is_private": false
  },
  "source": "pastebin-service"
}

Dead letter queue strategy and poison message handling

Messages that fail to process after maximum retries will be moved to a dead letter queue (DLQ) for manual inspection. The DLQ will use a separate topic with retention policy of 30 days and alerting when threshold is exceeded.

Poison message handling:

• Messages failing more than 5 times will be routed to DLQ
• Failed messages will include retry count and error details in metadata
• DLQ monitoring dashboard with auto-alerts for thresholds

At-least-once vs. exactly-once delivery

At-least-once delivery is selected for this feature due to:

• Paste creation and deletion are idempotent operations
• Content moderation is a best-effort process that doesn't require exact delivery
• Simpler implementation and better performance characteristics

Retry policy for consumers

Retry configuration:

• Max attempts: 5
• Backoff strategy: Exponential with jitter (1s, 2s, 4s, 8s, 16s)
• Retryable errors: Network timeouts, service unavailability
• Non-retryable errors: Invalid paste data, malformed content

Outbox pattern for reliable event publishing

Outbox pattern is implemented to ensure eventual consistency between database writes and message publishing:

public class PasteEventPublisher {
    private final EventRepository eventRepository;
    private final KafkaTemplate<String, PasteEvent> kafkaTemplate;

    public void publishPasteCreatedEvent(Paste paste) {
        // 1. Create event in database
        PasteEvent event = PasteEvent.builder()
            .pasteId(paste.getId())
            .userId(paste.getUserId())
            .eventType("paste.created")
            .timestamp(Instant.now())
            .payload(PasteEventPayload.builder()
                .pasteId(paste.getId())
                .userId(paste.getUserId())
                .contentLength(paste.getContent().length())
                .language(paste.getLanguage())
                .isPrivate(paste.isPrivate())
                .build())
            .build();

        // 2. Save to outbox table
        eventRepository.save(event);

        // 3. Publish to Kafka (will be processed by separate consumer)
        kafkaTemplate.send("paste-events", event.getPasteId(), event);
    }
}

Sequence diagram for async flows

sequenceDiagram
    participant Client
    participant PasteService
    participant DB
    participant OutboxRepo
    participant Kafka
    participant AnalyticsConsumer
    participant ModerationConsumer

    Client->>PasteService: Create Paste
    PasteService->>DB: Save Paste
    PasteService->>OutboxRepo: Insert Event to Outbox
    PasteService->>Kafka: Publish to paste-events topic
    Kafka->>AnalyticsConsumer: Deliver paste.created event
    Kafka->>ModerationConsumer: Deliver paste.created event
    AnalyticsConsumer-->>Kafka: Acknowledge
    ModerationConsumer-->>Kafka: Acknowledge

10. Error Handling Design

Error Classification Hierarchy

The error handling strategy follows a hierarchical classification to enable proper routing and response generation:

Error
├── ValidationException (400)
│   ├── InvalidInputException
│   └── MissingRequiredFieldException
├── BusinessLogicException (400)
│   ├── PasteNotFoundException
│   └── PasteExpiredException
├── InfrastructureException (500)
│   ├── DatabaseException
│   ├── CacheException
│   └── ExternalServiceException
└── AuthorizationException (403)
    └── UnauthorizedAccessException

Error Propagation Strategy

Errors propagate up the call stack with appropriate wrapping and context addition:

1. Domain layer: Throws specific business exceptions (e.g., PasteNotFoundException)
2. Service layer: Wraps domain errors with context and re-throws as BusinessLogicException
3. Controller layer: Converts to appropriate HTTP response with proper status codes

Retry Strategy

For transient infrastructure errors, exponential backoff with jitter is implemented:

public class RetryConfig {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;
    
    public static long calculateDelay(int attempt) {
        long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempt));
        double jitter = ThreadLocalRandom.current().nextDouble(-JITTER_FACTOR, JITTER_FACTOR);
        return (long) (delay * (1 + jitter));
    }
}

Circuit Breaker Design

For external service dependencies, a circuit breaker pattern is implemented:

public class CircuitBreaker {
    private final int failureThreshold = 5;
    private final long timeoutMs = 60000;
    private final int resetTimeoutMs = 30000;
    
    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private long lastFailureTime = 0;
    
    public boolean canExecute() {
        if (state == CircuitState.OPEN) {
            if (System.currentTimeMillis() - lastFailureTime > resetTimeoutMs) {
                state = CircuitState.HALF_OPEN;
            } else {
                return false;
            }
        }
        return true;
    }
    
    public void recordSuccess() {
        if (state == CircuitState.HALF_OPEN) {
            state = CircuitState.CLOSED;
        }
    }
    
    public void recordFailure() {
        if (state == CircuitState.CLOSED) {
            failureCount++;
            lastFailureTime = System.currentTimeMillis();
            
            if (failureCount >= failureThreshold) {
                state = CircuitState.OPEN;
            }
        } else if (state == CircuitState.HALF_OPEN) {
            state = CircuitState.OPEN;
        }
    }
}

Fallback Behavior Design

For external dependencies, fallbacks provide graceful degradation:

public class ExternalServiceFallback {
    public PasteData getFallbackPaste(String pasteId) {
        // Return default or cached content if available
        return new PasteData("Default content", "text/plain", 
                           System.currentTimeMillis(), false);
    }
    
    public List<PasteSummary> getFallbackPastes(String userId) {
        // Return empty list or cached summaries
        return Collections.emptyList();
    }
}

Error Logging Format

Structured logging includes all relevant context:

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);
    
    public void logError(String operation, Exception ex, Map<String, Object> context) {
        logger.error(
            "Error in operation: {} | Error Code: {} | User ID: {} | Paste ID: {} | " +
            "Timestamp: {} | Stack Trace: {}",
            operation,
            getErrorCode(ex),
            context.get("userId"),
            context.get("pasteId"),
            System.currentTimeMillis(),
            ex.getMessage()
        );
    }
    
    private String getErrorCode(Exception ex) {
        if (ex instanceof ValidationException) return "VALIDATION_ERROR";
        if (ex instanceof BusinessLogicException) return "BUSINESS_LOGIC_ERROR";
        if (ex instanceof InfrastructureException) return "INFRASTRUCTURE_ERROR";
        return "UNKNOWN_ERROR";
    }
}

Alerting Thresholds

Critical errors trigger immediate alerts:

Error Type	Alert Threshold	Severity
Database connection failures	> 5 errors in 5 minutes	Critical
Cache unavailability	> 10 errors in 10 minutes	High
External service timeouts	> 3 failures in 1 minute	High
Business logic validation errors	> 100 errors in 5 minutes	Medium

Error Response Format

Standardized JSON response for all error cases:

{
  "error": {
    "code": "PASTE_NOT_FOUND",
    "message": "Paste with ID 'abc123' was not found",
    "timestamp": "2023-10-15T14:30:00Z",
    "requestId": "req-1234567890"
  }
}

Retryable vs Non-Retryable Errors

Error Type	Retryable	Reason
DatabaseException	Yes	Transient network issues
CacheException	Yes	Temporary cache unavailability
ValidationException	No	Input validation issues
PasteNotFoundException	No	Resource does not exist
AuthorizationException	No	Permission denied

Error Budget Considerations

For SLO compliance, error budget is tracked at service level:

public class ErrorBudgetTracker {
    private final double errorBudgetPercentage = 0.05; // 5% of traffic
    private volatile double consumedBudget = 0.0;
    
    public boolean isWithinErrorBudget() {
        return consumedBudget < errorBudgetPercentage;
    }
    
    public void recordError(String errorType) {
        consumedBudget += calculateErrorImpact(errorType);
    }
    
    private double calculateErrorImpact(String errorType) {
        switch (errorType) {
            case "VALIDATION_ERROR": return 0.001;
            case "CACHE_ERROR": return 0.005;
            case "DATABASE_ERROR": return 0.01;
            default: return 0.002;
        }
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All input data undergoes strict validation using a combination of:

• Regex-based validation for format constraints (e.g., paste ID pattern, content length limits)
• Length limits to prevent oversized payloads (max 1MB paste content, 100 character title)
• Content sanitization using OWASP Java HTML Sanitizer for user-generated content to prevent XSS
• Rate limiting per IP to prevent abuse and DoS attacks

SQL Injection Prevention

All database queries are constructed using parameterized prepared statements exclusively. No dynamic query building or string concatenation is allowed. ORM framework (Hibernate) is configured with strict validation to prevent injection vectors.

Authentication Middleware Design

• No authentication required for public paste creation or retrieval (as per pastebin model)
• Optional authentication for user-specific paste management (delete, update) via JWT tokens
• Authentication middleware validates token signature and expiration using JWK Set endpoint for public keys
• JWT claims are validated (iss, aud, exp, iat) and custom claims (user_id, role) are extracted

Authorization Enforcement

• Role-Based Access Control (RBAC) implemented at service layer:
  • Public paste access: READ_PUBLIC_PASTE (any authenticated user)
  • User paste management: MANAGE_USER_PASTE (requires valid JWT with matching user_id)
• Authorization checks are performed before any domain operation using a centralized permission service
• For paste deletion, the requesting user must match the paste owner or have admin privileges

Sensitive Data Encryption

• At-rest encryption: AES-256-GCM using AWS KMS-managed keys for paste content storage
• In-transit encryption: TLS 1.3 enforced for all API endpoints (HTTPS only)
• Key rotation policy: 90-day automatic key rotation with seamless transition

PII Handling and Masking Strategy

• Paste content is treated as potentially sensitive user data
• No PII extraction or logging occurs in application logs or metrics
• User identifiers (if present) are masked in logs using hash-based obfuscation
• Audit trail only records minimal metadata (paste ID, timestamp, operation type)

Secrets Management

• AWS Secrets Manager used for storing:
  • Database credentials (rotated every 90 days)
  • JWT signing keys (rotated every 30 days)
  • Third-party API credentials (e.g., analytics, monitoring)
• Secrets are accessed via secure IAM roles with least privilege principle
• No hardcoded secrets in source code or configuration files

Audit Logging Design

• Event types logged: paste creation, paste retrieval, paste deletion, paste update
• Log format includes:
  • timestamp: ISO8601 UTC
  • trace_id: OpenTelemetry trace identifier
  • user_id: hashed for privacy (if authenticated)
  • operation_type: CREATE, READ, UPDATE, DELETE
  • paste_id: obfuscated for privacy
  • ip_address: masked to CIDR block (e.g., 192.168.1.0/24)
  • status_code: HTTP response code
• Retention policy: 90 days for audit logs, archived to S3 with lifecycle policy
• Alerting: Security incidents (e.g., 10+ paste deletions in 5 minutes) trigger immediate alert to security team

12. Testing Strategy

Unit Testing Scope and Coverage Targets

Target coverage: 85% line coverage, 90% branch coverage for core business logic.

Scope breakdown:

• Service layer methods (90%+ coverage)
• Repository layer operations (85%+ coverage)
• Domain object validation logic (100% coverage)
• Business rule implementations (95%+ coverage)
• Exception handling paths (100% coverage)

Testing approach:

// Example unit test for paste creation with validation
@Test
public void testCreatePasteWithValidContent() {
    Paste paste = new Paste("Hello World", "text/plain", 3600);
    Paste created = pasteService.createPaste(paste);
    
    assertThat(created.getId()).isNotNull();
    assertThat(created.getContent()).isEqualTo("Hello World");
    assertThat(created.getExpiresAt()).isAfter(Instant.now());
}

@Test
public void testCreatePasteWithInvalidContent() {
    Paste paste = new Paste("", "text/plain", 3600);
    
    assertThatThrownBy(() -> pasteService.createPaste(paste))
        .isInstanceOf(ValidationException.class)
        .hasMessageContaining("Content cannot be empty");
}

Integration Testing Strategy

Database integration tests:

• Use testcontainers with PostgreSQL for real database testing
• Isolated test databases per test suite using Docker containers
• Test data cleanup between tests to ensure isolation

External service integration:

• WireMock for external API fakes (e.g., rate limiter, analytics)
• Embedded Redis for cache integration testing
• Mocked Kafka consumers for async processing validation

@ExtendWith(SpringExtension.class)
@Testcontainers
public class PasteServiceIntegrationTest {
    
    @Container
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:13")
            .withDatabaseName("testdb")
            .withUsername("test")
            .withPassword("test");

    @DynamicPropertySource
    static void configureProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.datasource.url", postgres::getJdbcUrl);
    }

    @Test
    public void testPasteCreationAndRetrieval() {
        Paste paste = pasteService.createPaste(
            new Paste("test content", "text/plain", 3600));
        
        Paste retrieved = pasteService.getPaste(paste.getId());
        assertThat(retrieved.getContent()).isEqualTo("test content");
    }
}

Contract Testing

Consumer-driven contract testing using Pact:

• Define contracts between Paste service and client applications
• Validate that the API responses match expected formats
• Test with consumer-side stubs to ensure backward compatibility

Example Pact contract:

PactProvider("paste-service")
class PasteContractTest {
    
    @Pact(consumer = "client-app")
    RequestResponsePact createPact(PactDslWithProvider builder) {
        return builder
            .given("a paste exists")
            .uponReceiving("a request to create a paste")
            .method("POST")
            .path("/pastes")
            .body('{"content":"Hello World","expiresIn":3600}')
            .willRespondWith()
            .status(201)
            .headers { headers ->
                headers.header("Content-Type", "application/json")
            }
            .body('{"id":"12345678-1234-1234-1234-123456789012","content":"Hello World"}')
            .toPact()
    }
}

Performance Testing Strategy

Load testing tools:

• k6 for load and stress testing
• JMeter for functional performance testing

Key performance scenarios:

1. Concurrent paste creation (500 concurrent requests)
2. Paste retrieval under load (1000 concurrent reads)
3. Rate limit exhaustion testing
4. Cache warm-up and hit ratio validation

Performance test example:

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  vus: 100,
  duration: '30s',
};

export default function () {
  const payload = JSON.stringify({
    content: 'Test paste content',
    expiresIn: 3600
  });
  
  const res = http.post('http://paste-service/api/pastes', payload, {
    headers: { 'Content-Type': 'application/json' }
  });
  
  check(res, {
    'status is 201': (r) => r.status === 201,
    'response time < 200ms': (r) => r.timings.duration < 200
  });
  
  sleep(1);
}

Test Data Management Strategy

Test data factories:

public class PasteFactory {
    public static Paste createValidPaste() {
        return new Paste(
            "Sample content",
            "text/plain",
            3600,
            UUID.randomUUID(),
            Instant.now().plusSeconds(3600)
        );
    }
    
    public static Paste createPasteWithExpiry(int seconds) {
        return new Paste(
            "Content with expiry",
            "text/plain",
            seconds,
            UUID.randomUUID(),
            Instant.now().plusSeconds(seconds)
        );
    }
}

Test data seeding:

• Use database migrations to seed test data
• Implement fixtures for common test scenarios (expired pastes, public/private pastes)
• Use data builders to create realistic test data structures

Test data lifecycle:

• Per-test database transaction rollback (for clean state)
• Test container cleanup after each test run
• Use of test-specific database schemas to prevent data pollution

Test isolation:

• Each test runs in isolated environment
• Use of unique identifiers for test data to prevent collisions
• Mocking external dependencies to ensure deterministic behavior

Test maintenance:

• Regular cleanup of test data to prevent database bloat
• Automated test data validation to ensure consistency
• Test data versioning for regression testing scenarios

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective search, filtering, and correlation:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO|WARN|ERROR",
  "service": "pastebin-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user-123",
  "entity_id": "paste-456",
  "request_id": "req-789",
  "method": "POST /pastes",
  "endpoint": "/pastes",
  "duration_ms": 45,
  "message": "Paste created successfully",
  "error_code": "PASTE_001",
  "error_message": "Paste content exceeds maximum length",
  "stack_trace": "...",
  "http_status": 201,
  "payload_size_bytes": 1024
}

PII masking rules:

• User IDs in logs are masked to user-***
• Paste content is truncated or masked for logs
• IP addresses are masked to 192.168.***

Distributed Tracing Instrumentation

Using OpenTelemetry for end-to-end traceability:

Custom spans to add:

1. paste.create - from request start to paste creation
2. paste.retrieve - from request to content fetch
3. cache.miss - when cache lookup fails
4. db.query - for all database operations

Attributes to capture per key operation:

Span span = tracer.spanBuilder("paste.create")
    .setAttribute("paste.id", pasteId)
    .setAttribute("paste.size_bytes", content.length())
    .setAttribute("user.id", userId)
    .setAttribute("is_public", isPublic)
    .startSpan();

Metrics Design

RED Method metrics (Rate, Errors, Duration):

Metric Name	Type	Labels	Description
pastebin.paste.count	Counter	status, method	Paste creation count
pastebin.paste.errors	Counter	error_type, method	Paste operation errors
pastebin.paste.duration_ms	Histogram	method, status	Paste operation latency
pastebin.cache.hits	Counter	type, source	Cache hit counts
pastebin.cache.misses	Counter	type, source	Cache miss counts
pastebin.db.query.duration_ms	Histogram	operation, status	Database query times
pastebin.http.request.duration_ms	Histogram	method, path, status_code	HTTP request latencies

Gauge metrics:

• pastebin.cache.size - Current cache size in entries
• pastebin.db.connection.active - Active DB connections

Health Check Endpoint Design

GET /health
GET /ready
GET /live
GET /startup

Response format:

{
  "status": "healthy|unhealthy",
  "checks": {
    "database": {"status": "healthy", "message": "Connected to primary DB"},
    "cache": {"status": "healthy", "message": "Redis connected"},
    "disk_space": {"status": "healthy", "message": "Available: 10GB"}
  },
  "timestamp": "2023-10-27T14:30:45.123Z"
}

What each checks:

• /health - Core dependencies (DB, cache)
• /ready - All services ready to serve traffic
• /live - Service is running (basic liveness)
• /startup - Startup completion status

Alerting Thresholds

Metric	Warning Level	Critical Level	Alerting Path
pastebin.paste.errors rate (5m)	> 10 errors/min	> 50 errors/min	On-call engineer
pastebin.paste.duration_ms p99	> 500ms	> 2s	SRE team
pastebin.cache.misses rate (5m)	> 1000 misses/min	> 5000 misses/min	Cache team
pastebin.db.query.duration_ms p99	> 100ms	> 500ms	DBA team
pastebin.http.request.duration_ms p99	> 1s	> 3s	Backend team
pastebin.db.connection.active	> 80% of pool	> 95% of pool	DBA team

On-call escalation path:

1. Primary engineer (5 min)
2. SRE team (30 min)
3. Engineering leadership (1 hour)

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• Content expiration policy: Should pastes automatically expire after a fixed time (e.g., 1 hour, 1 day) or be configurable per paste? This affects storage costs and user experience.
• Public vs. private paste visibility: How should the system handle public pastes that are later made private? Should we allow re-encryption or just delete and recreate?
• Rate limiting granularity: Should rate limits be applied at the user level, IP level, or both? What's the acceptable threshold for abuse prevention?
• Storage vs. bandwidth trade-off: Should we prioritize storing compressed content or optimize for faster retrieval by keeping uncompressed copies?

Known trade-offs and justifications

• Storage vs. performance trade-off: Storing compressed content reduces storage costs but increases CPU overhead during read operations. We're choosing to prioritize performance for better user experience.
• Caching vs. consistency trade-off: Using eventual consistency in cache layer provides better performance but introduces potential staleness. We accept this for improved latency and throughput.
• Read-heavy vs. write-heavy optimization: The system is optimized for read-heavy workloads with caching and indexing, but this introduces complexity in maintaining cache consistency.
• Single-region vs. multi-region trade-off: Single region deployment simplifies architecture but introduces single point of failure and latency for global users. We're implementing single region for MVP.

Risks and mitigation strategies

• Content abuse risk: Malicious content (malware, spam) could be uploaded. Mitigation includes content scanning integration, user reputation scoring, and manual review queue for flagged items.
• High write volume risk: If paste creation spikes significantly, we might overwhelm database. Mitigation includes rate limiting, async processing queue, and read replica scaling.
• Cache invalidation failure: Cache inconsistency could serve stale content. Mitigation includes TTL-based expiration, event-driven invalidation, and cache warming strategies.
• Storage cost growth: Unbounded paste size could lead to exponential storage costs. Mitigation includes content size limits (5MB max) and automatic archiving of older pastes.

Technical debt acknowledged

• Monolithic architecture: The entire system is built as a single service, which makes future horizontal scaling more difficult. This is acceptable for MVP but needs architectural refactoring for production scale.
• Limited search capabilities: Basic text search is implemented with simple substring matching. Advanced search features (full-text, syntax highlighting) are deferred to future iterations.
• No content versioning: Pastes cannot be reverted to previous versions. This is acceptable for a simple pastebin but limits advanced use cases.
• Basic authentication model: Currently uses simple token-based auth, not OAuth2 integration. This simplifies implementation but limits third-party integrations.
• No content moderation system: Manual review and flagging system is not implemented, which could lead to inappropriate content being hosted. This is a known limitation for the MVP release.
• Hardcoded limits: Content size, rate limits, and cache TTLs are hardcoded rather than dynamically configurable via feature flags or configuration service.