Search Service Low-Level Design for Scalable Text Retrieval

This document presents a scalable search service architecture that handles large-scale text indexing, efficient query execution, and distributed processing patterns essential for modern search applications.

1. Problem Statement & Requirements

Clarifying questions

1. What is the scale of data and expected query volume?

   • Is this a global search across billions of documents or limited to a specific domain (e.g., 10M products)?
   • Expected QPS: 10k, 100k, or 1M+?

2. What are the consistency and latency requirements?

   • Should search results be strongly consistent or eventually consistent?
   • What are the target latencies (e.g., p50 < 100ms, p99 < 500ms)?

3. What are the key entities and their relationships?

   • Is search limited to a single document type (e.g., products) or multi-type (e.g., products, users, articles)?
   • Are there hierarchical or nested fields to consider?

4. What is the read/write ratio and query patterns?

   • Is search mostly read-heavy or does it involve frequent updates (e.g., real-time indexing)?
   • Are queries mostly full-text, structured, or faceted?

5. How sensitive is the system to latency and availability?

   • Is sub-100ms response critical for business (e.g., e-commerce) or can it tolerate higher latency?
   • What is the SLA for availability (e.g., 99.9% uptime)?

6. What authentication and authorization models are in place?

   • Should search results be filtered based on user permissions?
   • Is the system expected to integrate with OAuth2, JWT, or API keys?

7. Are there external integrations or data sources to consider?

   • Will search results pull from external systems (e.g., third-party APIs)?
   • Is there a need to support real-time data ingestion or batch processing?

Functional requirements

• Full-text search: Support searching across text fields with fuzzy matching and relevance scoring.
• Structured filtering: Allow filtering by categories, price ranges, tags, or other metadata fields.
• Faceted search: Enable users to refine results by facets like brand, color, or size.
• Autocomplete suggestions: Provide real-time search term suggestions.
• Search result ranking: Support custom ranking logic (e.g., popularity, recency, relevance).
• Search result pagination: Support efficient cursor-based pagination.
• Search analytics: Track search queries, popular terms, and result click-through rates.

Non-functional requirements

• Latency targets:
  • p50: <100ms
  • p95: <300ms
  • p99: <500ms
• Throughput: Support 10k+ QPS with horizontal scaling capability.
• Availability: 99.9% uptime (99.99% for critical search paths).
• Scalability: Support 100M+ indexed documents with sub-second search performance.
• Indexing latency: Index new or updated content within 10 seconds.

Constraints and assumptions

• Assumption: Search service will be built using a modern search engine (e.g., Elasticsearch or Solr) for indexing and querying.
• Constraint: The system will not handle real-time content generation or user-generated content indexing in the same service (e.g., content ingestion handled by a separate pipeline).
• Assumption: Search results will be cached to improve performance for frequently searched terms.
• Constraint: The system assumes that the search index is updated via a separate indexing pipeline or event-driven architecture.

Out of scope

• Content generation: Creating or modifying content within the search service (e.g., user-generated reviews).
• Real-time indexing: Real-time content ingestion and indexing within the search service (assumed to be handled by a separate pipeline).
• Search result personalization: Customizing search results based on user behavior or preferences (e.g., A/B testing).
• Multi-tenancy: Isolation of search results or indexes for different tenants (assumed to be handled by the application layer).
• Search result highlighting: Returning matched text snippets or highlighted search terms in results.
• Advanced NLP features: Named entity recognition, sentiment analysis, or semantic search.

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for the Search Service due to:

• Strong typing and schema evolution support
• Efficient binary serialization with lower network overhead than REST
• Built-in support for streaming and bidirectional communication
• Native support for versioning through proto file management
• Better performance characteristics for search-heavy workloads

Endpoint Definitions

Method	Path	Description	Auth	Rate Limit
Search	/search.v1.SearchService/Search	Execute full-text search query	JWT	1000 req/min
Suggest	/search.v1.SearchService/Suggest	Auto-complete suggestions	JWT	5000 req/min
IndexDocument	/search.v1.SearchService/IndexDocument	Index new document	Service Token	100 req/sec
DeleteDocument	/search.v1.SearchService/DeleteDocument	Remove document from index	Service Token	100 req/sec
GetStats	/search.v1.SearchService/GetStats	Retrieve search service metrics	Service Token	50 req/min

Request and Response Schemas

SearchRequest

message SearchRequest {
  string query = 1;
  int32 page = 2;
  int32 page_size = 3;
  repeated Filter filters = 4;
  Sort sort = 5;
  repeated string fields_to_return = 6;
  string search_context = 7;
}

message Filter {
  string field = 1;
  string operator = 2; // eq, neq, gt, gte, lt, lte, in, not_in, exists, not_exists
  oneof value {
    string string_value = 3;
    int64 int_value = 4;
    double double_value = 5;
    bool bool_value = 6;
    repeated string string_list = 7;
  }
}

message Sort {
  string field = 1;
  bool ascending = 2;
}

SearchResponse

message SearchResponse {
  repeated SearchResult results = 1;
  int32 total_hits = 2;
  int32 page = 3;
  int32 page_size = 4;
  string next_cursor = 5;
  repeated string facets = 6;
}

message SearchResult {
  string id = 1;
  double score = 2;
  map<string, any> document = 3;
  repeated string highlighted_fields = 4;
}

SuggestRequest

message SuggestRequest {
  string prefix = 1;
  int32 max_suggestions = 2;
  repeated string fields = 3;
}

SuggestResponse

message SuggestResponse {
  repeated string suggestions = 1;
}

HTTP Status Codes and Error Response

Code	Description
200	Success
400	Invalid request parameters
401	Authentication required
403	Insufficient permissions
429	Rate limit exceeded
500	Internal server error
503	Service unavailable

Standard Error Response

{
  "error": {
    "code": "INVALID_ARGUMENT",
    "message": "Query parameter is required",
    "details": [
      {
        "field": "query",
        "message": "Query parameter is required"
      }
    ]
  }
}

API Versioning Strategy

Versioned using gRPC package naming: search.v1 for current version. Future versions will use search.v2, etc.

Pagination Design

Cursor-based pagination is used because:

• More stable than offset-based pagination when data changes
• Better performance for large datasets
• Consistent results across concurrent operations

Authentication and Authorization

All endpoints require JWT tokens with scopes:

• search:read for search and suggest operations
• search:write for indexing and deletion operations

Rate Limiting

Rate limiting implemented using token bucket algorithm:

• Search: 1000 requests/minute
• Suggest: 5000 requests/minute
• Index/Delete: 100 requests/second
• Stats: 50 requests/minute

Idempotency Design

For mutation endpoints (IndexDocument, DeleteDocument):

• Accept idempotency_key in request header
• Store operation in idempotency store with TTL of 5 minutes
• If same key exists and operation is identical, return cached result
• Use Redis for distributed idempotency store

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    SEARCH_INDEX ||--o{ DOCUMENT : contains
    DOCUMENT ||--o{ DOCUMENT_FIELD : has
    DOCUMENT_FIELD ||--o{ SEARCH_RESULT : maps_to
    SEARCH_INDEX ||--o{ SEARCH_QUERY : indexed_by
    SEARCH_QUERY ||--o{ SEARCH_RESULT : produces
    
    SEARCH_INDEX {
        string id PK
        string name
        string description
        timestamp created_at
        timestamp updated_at
        boolean is_active
    }
    
    DOCUMENT {
        string id PK
        string search_index_id FK
        string title
        string content
        string url
        timestamp created_at
        timestamp updated_at
        boolean is_deleted
    }
    
    DOCUMENT_FIELD {
        string id PK
        string document_id FK
        string field_name
        string field_value
        timestamp created_at
    }
    
    SEARCH_QUERY {
        string id PK
        string query_text
        timestamp created_at
        string user_id
    }
    
    SEARCH_RESULT {
        string id PK
        string search_query_id FK
        string document_id FK
        float score
        timestamp created_at
    }

Table Schema Definitions

SearchIndex Table

CREATE TABLE search_index (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL,
    description TEXT,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    is_active BOOLEAN DEFAULT TRUE,
    
    INDEX idx_search_index_active (is_active),
    INDEX idx_search_index_name (name)
);

Document Table

CREATE TABLE document (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    search_index_id UUID NOT NULL REFERENCES search_index(id) ON DELETE CASCADE,
    title TEXT NOT NULL,
    content TEXT NOT NULL,
    url TEXT,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    is_deleted BOOLEAN DEFAULT FALSE,
    
    INDEX idx_document_search_index (search_index_id),
    INDEX idx_document_deleted (is_deleted),
    INDEX idx_document_created_at (created_at),
    
    -- Full-text search index
    INDEX idx_document_fts (to_tsvector('english', title || ' ' || content))
);

DocumentField Table

CREATE TABLE document_field (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE,
    field_name VARCHAR(100) NOT NULL,
    field_value TEXT NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    
    INDEX idx_document_field_document (document_id),
    INDEX idx_document_field_name (field_name),
    INDEX idx_document_field_value (field_value)
);

SearchQuery Table

CREATE TABLE search_query (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    query_text TEXT NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    user_id UUID,
    
    INDEX idx_search_query_created_at (created_at),
    INDEX idx_search_query_user (user_id)
);

SearchResult Table

CREATE TABLE search_result (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    search_query_id UUID NOT NULL REFERENCES search_query(id) ON DELETE CASCADE,
    document_id UUID NOT NULL REFERENCES document(id),
    score FLOAT NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    
    INDEX idx_search_result_query (search_query_id),
    INDEX idx_search_result_document (document_id),
    INDEX idx_search_result_score (score DESC)
);

Primary Key Strategy

UUIDs are used as primary keys for the following reasons:

• Distributed system compatibility: UUIDs eliminate conflicts when merging data from multiple sources
• No sequential patterns: Prevents predictable ID enumeration attacks
• Scalability: No need to coordinate ID generation across services
• Cross-service referential integrity: UUIDs can be generated client-side and safely used in distributed systems

Index Design

Index Name	Fields	Type	Purpose
idx_search_index_active	is_active	Standard	Fast filtering of active indices
idx_search_index_name	name	Standard	Fast lookup by index name
idx_document_search_index	search_index_id	Standard	Fast document retrieval by index
idx_document_deleted	is_deleted	Standard	Efficient soft-delete filtering
idx_document_created_at	created_at	Standard	Time-based document queries
idx_document_fts	to_tsvector('english', title		' '
idx_document_field_document	document_id	Standard	Fast field lookups per document
idx_document_field_name	field_name	Standard	Fast field name filtering
idx_document_field_value	field_value	Standard	Fast field value lookups
idx_search_query_created_at	created_at	Standard	Query history tracking
idx_search_result_query	search_query_id	Standard	Fast result retrieval by query
idx_search_result_document	document_id	Standard	Fast document lookup from results
idx_search_result_score	score DESC	Standard	Fast sorting by relevance

Data Partitioning / Sharding Strategy

Shard by SearchIndex ID: Documents are partitioned by their search_index_id to:

• Enable horizontal scaling of search indices
• Allow independent maintenance and backup per index
• Support multi-tenant scenarios where each tenant has a dedicated search index

Partitioning Strategy:

-- Example partitioning by search_index_id (PostgreSQL)
CREATE TABLE document_01 PARTITION OF document
FOR VALUES IN ('index-uuid-01');

CREATE TABLE document_02 PARTITION OF document
FOR VALUES IN ('index-uuid-02');

Soft Delete vs. Hard Delete Decision

Soft delete is used because:

• Auditability: Historical search results and queries need to be preserved for analytics
• Recovery capability: Documents can be restored if accidentally deleted
• Search result consistency: Search results should not disappear when documents are soft-deleted
• Compliance: Regulatory requirements may mandate data retention

Audit Trail Design

Audit fields added to all tables:

• created_at (TIMESTAMP WITH TIME ZONE)
• updated_at (TIMESTAMP WITH TIME ZONE)
• created_by (UUID) - if user tracking is required
• updated_by (UUID) - if user tracking is required

Data Archival Strategy and Retention Policy

Archival Strategy:

• Documents older than 1 year are archived to cold storage (e.g., S3 Glacier)
• Search query history older than 6 months is archived
• SearchResult records older than 30 days are purged

Retention Policy:

Entity	Retention Period	Archival Destination
Document	1 year	S3 Glacier
SearchQuery	6 months	S3 Standard
SearchResult	30 days	Database

-- Example archival job (runs daily)
DELETE FROM search_result 
WHERE created_at < NOW() - INTERVAL '30 days';

-- Archive documents to cold storage (pseudo-code)
SELECT * FROM document 
WHERE created_at < NOW() - INTERVAL '1 year'
AND is_deleted = TRUE
-- Move to S3 Glacier

4. Class Design & Service Layer

classDiagram
    class SearchService {
        +search(SearchRequest) SearchResponse
        +suggest(SearchSuggestionRequest) SearchSuggestionResponse
        +indexDocument(Document) void
        +deleteDocument(String) void
    }
    
    class SearchEngine {
        +search(SearchRequest) SearchResult
        +indexDocument(Document) void
        +deleteDocument(String) void
    }
    
    class Document {
        +id: String
        +title: String
        +content: String
        +tags: List~String~
        +createdAt: Instant
        +updatedAt: Instant
    }
    
    class SearchRequest {
        +query: String
        +filters: Map~String, Object~
        +sort: SortOrder
        +page: int
        +size: int
    }
    
    class SearchResponse {
        +results: List~SearchResult~
        +totalHits: long
        +page: int
        +size: int
    }
    
    class SearchSuggestionRequest {
        +prefix: String
        +maxSuggestions: int
    }
    
    class SearchSuggestionResponse {
        +suggestions: List~String~
    }
    
    class SearchResult {
        +documentId: String
        +score: double
        +matchedFields: List~String~
    }
    
    class IndexManager {
        +createIndex(String) void
        +deleteIndex(String) void
        +refreshIndex(String) void
    }
    
    class DocumentProcessor {
        +processDocument(Document) ProcessedDocument
        +validateDocument(Document) boolean
    }
    
    class EventPublisher {
        +publishSearchEvent(SearchEvent) void
    }
    
    class SearchEvent {
        +type: SearchEventType
        +documentId: String
        +timestamp: Instant
    }
    
    class SearchEventType {
        +INDEXED
        +DELETED
    }
    
    SearchService --> SearchEngine : delegates to
    SearchService --> IndexManager : manages indexes
    SearchService --> DocumentProcessor : processes documents
    SearchService --> EventPublisher : publishes events
    
    SearchEngine --> Document : operates on
    SearchEngine --> SearchResult : returns results
    
    DocumentProcessor --> Document : transforms
    EventPublisher --> SearchEvent : publishes
    
    class SearchEngineImpl {
        -searchEngine: SearchEngine
        -indexManager: IndexManager
        -documentProcessor: DocumentProcessor
        -eventPublisher: EventPublisher
        +search(SearchRequest) SearchResponse
        +indexDocument(Document) void
        +deleteDocument(String) void
    }
    
    SearchEngineImpl --> SearchEngine : delegates to
    SearchEngineImpl --> IndexManager : manages indexes
    SearchEngineImpl --> DocumentProcessor : processes documents
    SearchEngineImpl --> EventPublisher : publishes events
    
    class SearchService {
        +search(SearchRequest) SearchResponse
        +suggest(SearchSuggestionRequest) SearchSuggestionResponse
        +indexDocument(Document) void
        +deleteDocument(String) void
    }
    
    SearchService --> SearchEngineImpl : implements

stateDiagram-v2
    [*] --> Idle
    Idle --> Processing : search()
    Processing --> Idle : completed
    Processing --> Error : exception
    Idle --> Indexing : indexDocument()
    Indexing --> Idle : completed
    Indexing --> Error : exception
    Idle --> Deleting : deleteDocument()
    Deleting --> Idle : completed
    Deleting --> Error : exception
    Error --> Idle : retry or fail

Service Class Structure and Responsibilities

The SearchService is the primary entry point for search functionality, handling both search queries and document management operations. It delegates to specialized components:

1. SearchEngineImpl: Core search engine integration, handling query execution and result processing
2. IndexManager: Index lifecycle management (creation, deletion, refresh)
3. DocumentProcessor: Document validation and preprocessing before indexing
4. EventPublisher: Event publishing for search activity tracking

Interface / Abstract Class Design

public interface SearchEngine {
    SearchResult search(SearchRequest request);
    void indexDocument(Document document);
    void deleteDocument(String documentId);
}

public interface IndexManager {
    void createIndex(String indexName);
    void deleteIndex(String indexName);
    void refreshIndex(String indexName);
}

public interface DocumentProcessor {
    ProcessedDocument processDocument(Document document);
    boolean validateDocument(Document document);
}

public interface EventPublisher {
    void publishSearchEvent(SearchEvent event);
}

Business Logic Encapsulation Patterns

Strategy Pattern Implementation: Search query processing uses a strategy pattern to support different search algorithms.

public interface SearchStrategy {
    SearchResult execute(SearchRequest request);
}

@Component
public class FuzzySearchStrategy implements SearchStrategy {
    @Override
    public SearchResult execute(SearchRequest request) {
        // Implementation for fuzzy search logic
        return new SearchResult();
    }
}

@Component
public class ExactSearchStrategy implements SearchStrategy {
    @Override
    public SearchResult execute(SearchRequest request) {
        // Implementation for exact match search logic
        return new SearchResult();
    }
}

@Component
public class SearchStrategyFactory {
    private final Map<String, SearchStrategy> strategies;
    
    public SearchStrategyFactory(List<SearchStrategy> searchStrategies) {
        this.strategies = searchStrategies.stream()
            .collect(Collectors.toMap(
                strategy -> strategy.getClass().getSimpleName(),
                Function.identity()
            ));
    }
    
    public SearchStrategy getStrategy(String type) {
        return strategies.getOrDefault(type, strategies.values().iterator().next());
    }
}

Critical-Path Method Implementation

@Transactional
public SearchResponse search(SearchRequest request) {
    // Validate input parameters
    if (request.getQuery() == null || request.getQuery().trim().isEmpty()) {
        throw new IllegalArgumentException("Search query cannot be null or empty");
    }
    
    // Apply rate limiting for the search endpoint
    if (rateLimiter.isRateLimited(request.getQuery())) {
        throw new RateLimitExceededException("Search rate limit exceeded");
    }
    
    // Build search query with filters and sorting
    SearchQuery searchQuery = buildSearchQuery(request);
    
    // Execute search with timeout protection
    SearchResult result;
    try {
        result = searchEngine.search(searchQuery);
    } catch (SearchException e) {
        logger.error("Search failed for query: {}", request.getQuery(), e);
        throw new SearchServiceException("Search operation failed", e);
    }
    
    // Transform search results to response format
    SearchResponse response = new SearchResponse();
    response.setResults(result.getDocuments().stream()
        .map(this::toSearchResult)
        .collect(Collectors.toList()));
    response.setTotalHits(result.getTotalHits());
    response.setPage(request.getPage());
    response.setSize(request.getSize());
    
    // Publish search event for analytics
    SearchEvent searchEvent = new SearchEvent();
    searchEvent.setType(SearchEventType.SEARCH);
    searchEvent.setQuery(request.getQuery());
    searchEvent.setTimestamp(Instant.now());
    eventPublisher.publishSearchEvent(searchEvent);
    
    return response;
}

Transaction Boundary Design

The search service operates within a single database transaction for consistency when:

1. Indexing documents with related metadata updates
2. Performing batch operations on search indexes

For read-only operations like search queries, no transaction is required as they're idempotent and don't modify state.

Service Interface Contracts

public interface SearchService {
    SearchResponse search(SearchRequest request) throws SearchServiceException;
    SearchSuggestionResponse suggest(SearchSuggestionRequest request) throws SearchServiceException;
    void indexDocument(Document document) throws SearchServiceException;
    void deleteDocument(String documentId) throws SearchServiceException;
}

Saga / Orchestration Pattern

For complex multi-step search operations involving external systems, a saga pattern would be implemented:

public class SearchSaga {
    private final List<SagaStep> steps;
    
    public void execute(SearchRequest request) {
        for (SagaStep step : steps) {
            try {
                step.execute();
            } catch (Exception e) {
                // Compensate for failed steps
                compensateSteps();
                throw new SearchSagaException("Search saga failed", e);
            }
        }
    }
}

Domain Object / Value Object Design

public class Document {
    private String id;
    private String title;
    private String content;
    private List<String> tags;
    private Instant createdAt;
    private Instant updatedAt;
    
    // Getters and setters
}

public class SearchRequest {
    private String query;
    private Map<String, Object> filters;
    private SortOrder sort;
    private int page;
    private int size;
    
    // Getters and setters
}

public class SearchResult {
    private String documentId;
    private double score;
    private List<String> matchedFields;
    
    // Getters and setters
}

5. Concurrency & Thread Safety

Concurrency Model

The search service will use a thread pool model with Java's ExecutorService for handling concurrent search requests. Each incoming request will be processed by a worker thread from the thread pool, allowing for efficient handling of multiple concurrent queries without blocking the main request processing thread. For high-concurrency scenarios, we'll use a fixed-size thread pool with size tuned based on CPU cores and I/O wait times.

Race Condition Prevention Strategy

Race conditions in search service primarily occur during:

1. Index updates: Concurrent writes to search index (e.g., adding/removing documents)
2. Query result caching: Concurrent cache reads/writes for same query
3. Search statistics updates: Concurrent access to metrics counters

We implement race condition prevention using:

• Read-Write locks for index operations (ReentrantReadWriteLock)
• Atomic reference updates for cache entries and statistics
• Synchronized blocks for critical sections in search result aggregation

Optimistic vs. Pessimistic Locking Selection

For the search service, we use optimistic locking for index updates because:

• Search index updates are infrequent and batched
• Concurrent reads are far more frequent than writes
• The cost of conflict resolution is lower than continuous locking overhead

We implement optimistic locking using version numbers in document metadata. When updating a document, we check that the current version matches expected version before applying changes.

Deadlock Prevention Strategy

Deadlock prevention in search service includes:

1. Lock ordering: All threads acquire locks in consistent order (e.g., by document ID hash)
2. Timeout-based acquisition: Use tryLock() with timeout to prevent indefinite blocking
3. Hierarchical lock management: Index locks are acquired before query result locks

Idempotency Implementation for Concurrent Duplicate Requests

For search requests that may be retried (e.g., due to network issues), we implement idempotency using:

1. Idempotency keys in request headers (e.g., X-Idempotency-Key)
2. Cache-based deduplication: Store recent search requests with their results
3. Request fingerprinting: Hash query parameters to detect duplicates

public class SearchService {
    private final Map<String, SearchResult> idempotencyCache = new ConcurrentHashMap<>();
    private final ReadWriteLock cacheLock = new ReentrantReadWriteLock();
    
    public SearchResult search(SearchRequest request) {
        String idempotencyKey = request.getIdempotencyKey();
        
        // Check if this exact search was already processed
        if (idempotencyKey != null) {
            SearchResult cached = idempotencyCache.get(idempotencyKey);
            if (cached != null) {
                return cached;
            }
        }
        
        // Perform actual search
        SearchResult result = performSearch(request);
        
        // Cache the result for idempotency
        if (idempotencyKey != null) {
            idempotencyCache.put(idempotencyKey, result);
            
            // Schedule cache eviction after TTL
            ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
            scheduler.schedule(() -> idempotencyCache.remove(idempotencyKey), 
                              30, TimeUnit.MINUTES);
        }
        
        return result;
    }
}

Thread Safety in Search Index Updates

Search index operations are made thread-safe using:

public class SearchIndex {
    private final ReadWriteLock indexLock = new ReentrantReadWriteLock();
    private final Map<String, Document> documents = new ConcurrentHashMap<>();
    
    public void updateDocument(Document doc) {
        // Use write lock for index updates
        indexLock.writeLock().lock();
        try {
            String docId = doc.getId();
            
            // Check version for optimistic locking
            Document existing = documents.get(docId);
            if (existing != null && existing.getVersion() != doc.getVersion()) {
                throw new ConcurrentModificationException("Document version mismatch");
            }
            
            documents.put(docId, doc);
            updateSearchIndex(doc);
        } finally {
            indexLock.writeLock().unlock();
        }
    }
    
    public List<Document> search(SearchQuery query) {
        // Use read lock for concurrent reads
        indexLock.readLock().lock();
        try {
            return performSearch(query);
        } finally {
            indexLock.readLock().unlock();
        }
    }
}

Synchronization Strategy for Concurrent Search Results

Search results aggregation uses atomic operations:

public class SearchResultAggregator {
    private final AtomicLong totalHits = new AtomicLong(0);
    private final AtomicReference<List<Document>> results = 
        new AtomicReference<>(Collections.emptyList());
    
    public void aggregate(SearchResult partialResult) {
        // Update total hits atomically
        totalHits.addAndGet(partialResult.getTotalHits());
        
        // Atomic reference update for results list
        results.updateAndGet(current -> {
            List<Document> merged = new ArrayList<>(current);
            merged.addAll(partialResult.getDocuments());
            return merged;
        });
    }
}

Thread Safety in Caching Layer

The caching layer uses concurrent collections and atomic operations:

public class SearchCache {
    private final Map<String, CachedResult> cache = new ConcurrentHashMap<>();
    private final AtomicInteger hitCount = new AtomicInteger(0);
    private final AtomicInteger missCount = new AtomicInteger(0);
    
    public CachedResult get(String key) {
        CachedResult result = cache.get(key);
        if (result != null) {
            hitCount.incrementAndGet();
            return result;
        }
        missCount.incrementAndGet();
        return null;
    }
    
    public void put(String key, CachedResult result) {
        cache.put(key, result);
    }
    
    public int getHitRate() {
        int hits = hitCount.get();
        int misses = missCount.get();
        return (hits + misses) > 0 ? (int) ((double) hits / (hits + misses) * 100) : 0;
    }
}

6. Sequence Diagrams for Key Flows

Happy Path Search Flow

sequenceDiagram
    participant Client
    participant Gateway
    participant SearchService
    participant Cache
    participant Indexer
    participant Database

    Client->>Gateway: GET /search?q=java&size=10
    Gateway->>SearchService: Forward request
    SearchService->>Cache: Check if results exist
    Cache-->>SearchService: Return cached results
    SearchService->>Client: Return cached search results

    Note over Client,SearchService: Cache miss path
    SearchService->>Indexer: Query search index
    Indexer-->>SearchService: Return document IDs
    SearchService->>Database: Fetch full documents by IDs
    Database-->>SearchService: Return full documents
    SearchService->>Cache: Store results
    Cache-->>SearchService: Cache stored
    SearchService->>Client: Return search results

Error Handling Flow

sequenceDiagram
    participant Client
    participant Gateway
    participant SearchService
    participant Indexer
    participant Database

    Client->>Gateway: GET /search?q=invalid&size=1000
    Gateway->>SearchService: Forward request
    SearchService->>Indexer: Query search index
    Indexer-->>SearchService: Index timeout error
    SearchService->>Database: Fallback to database query
    Database-->>SearchService: Query failed
    SearchService->>Client: Return 500 Internal Server Error

Async Indexing Flow

sequenceDiagram
    participant Producer
    participant MessageBroker
    participant IndexerService
    participant SearchIndex
    participant Database

    Producer->>MessageBroker: Publish document event
    MessageBroker->>IndexerService: Deliver to consumer queue
    IndexerService->>Database: Fetch document from DB
    Database-->>IndexerService: Return document data
    IndexerService->>SearchIndex: Update search index
    SearchIndex-->>IndexerService: Index updated successfully
    IndexerService->>MessageBroker: Acknowledge successful processing

7. Caching Design

Cache Strategy Selection

For the search service, we'll use a cache-aside pattern with write-through behavior for critical search results. This pattern provides the best balance between cache consistency and performance for a search service where freshness is important but not mission-critical.

Cache Key Design

Cache keys will follow a structured namespace pattern:

search:{query}:{filters}:{sort}:{page}:{size}:{timestamp}

Example key format:

search:coffee+latte:category=beverage&price_range=10-20:sort=popularity&page=1:size=20:1698765432

TTL Strategy

Data Type	TTL	Reason
Search Results	5 minutes	Balances freshness with performance
Popular Queries	1 hour	High-value queries don't change frequently
Category/Filter Metadata	30 minutes	Stable data, infrequent updates
User Search History	24 hours	Personalized but not time-sensitive
Search Suggestions	15 minutes	Evolving based on trending terms

Cache Eviction Policy

We'll use LRU (Least Recently Used) with a time-based soft TTL to handle hot keys and prevent cache stampedes. The LRU eviction will be implemented at the application level with a concurrent LinkedHashMap.

Cache Stampede Prevention

We'll implement probabilistic early expiry with a 20% buffer on TTLs:

public class CacheStampedePrevention {
    private static final double EARLY_EXPIRY_BUFFER = 0.20;
    
    public String getWithStampedeProtection(String key, Supplier<String> loader) {
        String cached = cache.get(key);
        if (cached != null) {
            return cached;
        }
        
        // Use distributed lock to prevent stampede
        if (distributedLock.tryAcquire(key, 100)) {
            try {
                String fresh = loader.get();
                // Set TTL with early expiry
                int ttl = calculateTTL(key) * (int)(1 - EARLY_EXPIRY_BUFFER);
                cache.put(key, fresh, ttl);
                return fresh;
            } finally {
                distributedLock.release(key);
            }
        }
        
        // Fallback to loading from cache or returning stale data
        return cache.get(key);
    }
}

Distributed Cache Architecture

We'll use Redis Cluster with:

• 3 master nodes, 3 replica nodes for high availability
• 100% sharding with consistent hashing for key distribution
• Redis Sentinel for automatic failover monitoring

Cache Invalidation Strategy

We'll implement event-driven cache invalidation using a publish-subscribe model:

1. When search index is updated, publish search.index.updated event
2. Search service consumers listen for these events and invalidate relevant cache entries
3. For immediate consistency, we'll use write-through for critical search results

@Component
public class SearchCacheInvalidator {
    
    @EventListener
    public void handleIndexUpdate(SearchIndexUpdatedEvent event) {
        // Invalidate cache entries for affected search terms
        String query = event.getQuery();
        String keyPattern = "search:" + query + ":*";
        
        // Use Redis SCAN to find and invalidate matching keys
        Set<String> keys = redisTemplate.keys(keyPattern);
        redisTemplate.delete(keys);
        
        // Also invalidate popular query cache if needed
        redisTemplate.delete("popular:" + query);
    }
}

Cache Hit/Miss Metrics

We'll track cache performance with:

• search.cache.hits (counter)
• search.cache.misses (counter)
• search.cache.hit_ratio (gauge)
• search.cache.evictions (counter)

Cache Warm-up Strategy

For cold starts, we'll implement a background warm-up for:

• Top N most popular search queries
• Category-based search terms
• Recent trending searches

@Component
public class CacheWarmer {
    
    @Scheduled(fixedDelay = 300000) // Every 5 minutes
    public void warmPopularQueries() {
        List<String> popularQueries = searchService.getPopularQueries(100);
        
        for (String query : popularQueries) {
            // Pre-populate cache with recent results
            searchService.search(query, null, 1, 20);
        }
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain objects and persistence concerns. Each entity has its own repository interface with standardized CRUD operations.

public interface SearchDocumentRepository {
    SearchDocument save(SearchDocument document);
    Optional<SearchDocument> findById(String id);
    List<SearchDocument> findByIds(List<String> ids);
    Page<SearchDocument> findByQuery(SearchQuery query, Pageable pageable);
    void deleteById(String id);
    long count();
}

public interface SearchIndexRepository {
    void createIndex(SearchIndex index);
    void updateIndex(SearchIndex index);
    SearchIndex getIndex(String indexName);
    List<SearchIndex> getAllIndexes();
    void deleteIndex(String indexName);
}

ORM vs. Raw Query Decisions

For the search service, we use a hybrid approach:

• ORM: For simple CRUD operations and domain object mapping (Hibernate/JPA)
• Raw Queries: For complex search queries and aggregations (via native SQL or Elasticsearch DSL)

This approach provides:

1. Type safety and developer productivity for basic operations
2. Performance optimization for search-heavy queries
3. Flexibility to leverage database-specific features

Read vs. Write Separation

We implement CQRS (Command Query Responsibility Segregation) for the search service:

public class SearchCQRSRepository {
    private final SearchDocumentReadRepository readRepo;
    private final SearchDocumentWriteRepository writeRepo;
    
    public SearchDocument save(SearchDocument document) {
        SearchDocument saved = writeRepo.save(document);
        // Publish to event bus for async indexing
        eventPublisher.publish(new DocumentIndexedEvent(saved.getId()));
        return saved;
    }
    
    public Page<SearchDocument> search(SearchQuery query, Pageable pageable) {
        return readRepo.search(query, pageable);
    }
}

Connection Pool Configuration

Connection pool is configured with:

• Pool Size: 20-50 connections based on concurrent request load
• Timeout: 30 seconds for connection acquisition
• Validation: 30 seconds heartbeat validation
• Max Lifetime: 30 minutes to prevent connection leaks

@Configuration
public class DatabaseConfig {
    @Bean
    public HikariDataSource dataSource() {
        HikariConfig config = new HikariConfig();
        config.setJdbcUrl("jdbc:postgresql://localhost:5432/search_service");
        config.setMaximumPoolSize(30);
        config.setConnectionTimeout(30000);
        config.setIdleTimeout(60000);
        config.setMaxLifetime(1800000);
        config.setLeakDetectionThreshold(60000);
        return new HikariDataSource(config);
    }
}

N+1 Query Prevention Strategy

To prevent N+1 queries, we implement eager loading and batch operations:

@Repository
public class SearchDocumentRepositoryImpl implements SearchDocumentRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    // Eager loading for related entities
    @Override
    public List<SearchDocument> findByIds(List<String> ids) {
        String jpql = """
            SELECT d FROM SearchDocument d 
            LEFT JOIN FETCH d.tags t 
            LEFT JOIN FETCH d.categories c
            WHERE d.id IN :ids
            """;
        return entityManager.createQuery(jpql, SearchDocument.class)
                           .setParameter("ids", ids)
                           .getResultList();
    }
    
    // Batch operations for bulk indexing
    @Override
    public void batchSave(List<SearchDocument> documents) {
        String sql = """
            INSERT INTO search_documents (id, title, content, tags, created_at) 
            VALUES (?, ?, ?, ?, ?)
            """;
            
        try (Connection conn = dataSource.getConnection();
             PreparedStatement stmt = conn.prepareStatement(sql)) {
            
            for (SearchDocument doc : documents) {
                stmt.setString(1, doc.getId());
                stmt.setString(2, doc.getTitle());
                stmt.setString(3, doc.getContent());
                stmt.setArray(4, createTagArray(doc.getTags()));
                stmt.setTimestamp(5, Timestamp.valueOf(doc.getCreatedAt()));
                stmt.addBatch();
            }
            
            stmt.executeBatch();
        } catch (SQLException e) {
            throw new DataAccessException("Failed to batch save documents", e);
        }
    }
}

Batch Operation Design

Batch operations are implemented for high-volume indexing:

@Component
public class BatchSearchIndexer {
    
    private static final int BATCH_SIZE = 1000;
    
    public void indexDocuments(List<SearchDocument> documents) {
        List<List<SearchDocument>> batches = Lists.partition(documents, BATCH_SIZE);
        
        for (List<SearchDocument> batch : batches) {
            // Use bulk indexing API for Elasticsearch or database bulk operations
            performBulkIndex(batch);
        }
    }
    
    private void performBulkIndex(List<SearchDocument> batch) {
        // Implementation depends on search backend (Elasticsearch, Solr, etc.)
        // Example for Elasticsearch:
        BulkRequest bulkRequest = new BulkRequest();
        
        for (SearchDocument doc : batch) {
            IndexRequest request = new IndexRequest("search_documents")
                .id(doc.getId())
                .source(doc.toMap());
            bulkRequest.add(request);
        }
        
        // Execute bulk operation
        elasticsearchClient.bulk(bulkRequest, RequestOptions.DEFAULT);
    }
}

Read-Write Splitting

For high-throughput scenarios, we implement read-write splitting:

public class ReadWriteSearchRepository {
    private final SearchDocumentReadRepository readRepo;
    private final SearchDocumentWriteRepository writeRepo;
    
    @Transactional(readOnly = true)
    public Page<SearchDocument> search(SearchQuery query, Pageable pageable) {
        return readRepo.search(query, pageable);
    }
    
    @Transactional
    public SearchDocument save(SearchDocument document) {
        return writeRepo.save(document);
    }
}

Transaction Management

Transaction boundaries are carefully defined to ensure data consistency:

@Service
@Transactional
public class SearchDocumentService {
    
    @Autowired
    private SearchDocumentRepository repository;
    
    public SearchDocument createDocument(SearchDocument document) {
        // Transaction boundary starts here
        SearchDocument saved = repository.save(document);
        
        // Publish event for async processing
        eventPublisher.publish(new DocumentCreatedEvent(saved));
        
        return saved;
        // Transaction commits here
    }
}

Data Access Layer Testing Strategy

The repository layer is tested with:

1. Unit tests for individual repository methods using Mockito
2. Integration tests with embedded database (H2) for complex queries
3. Performance tests to validate batch operations and connection pooling

@ExtendWith(MockitoExtension.class)
public class SearchDocumentRepositoryTest {
    
    @Mock
    private EntityManager entityManager;
    
    @InjectMocks
    private SearchDocumentRepositoryImpl repository;
    
    @Test
    public void testFindByIds() {
        // Given
        List<String> ids = Arrays.asList("1", "2", "3");
        
        // When
        List<SearchDocument> result = repository.findByIds(ids);
        
        // Then
        verify(entityManager).createQuery(anyString(), any());
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The search service will be designed with a hybrid synchronous/asynchronous approach. The core search functionality (indexing, querying) will remain synchronous to ensure low-latency responses for real-time user queries. However, the service will leverage asynchronous processing for:

1. Indexing operations: When new content is added or updated, the indexing process will be asynchronous to avoid blocking the main request path.
2. Content reprocessing: When content schema changes or re-ranking logic is updated, asynchronous reprocessing ensures system availability.
3. Analytics and reporting: Usage analytics, search trend analysis, and reporting will be handled asynchronously to decouple from the core search experience.

Message schema design

We'll use Avro for message schema definition with versioning strategy:

{
  "type": "record",
  "name": "ContentIndexedEvent",
  "fields": [
    {"name": "contentId", "type": "string"},
    {"name": "title", "type": "string"},
    {"name": "content", "type": "string"},
    {"name": "tags", "type": {"type": "array", "items": "string"}},
    {"name": "indexedAt", "type": "long"},
    {"name": "version", "type": "int", "default": 1}
  ]
}

Versioning will follow semantic versioning with backward compatibility guarantees. Schema registry (e.g., Confluent Schema Registry) will be used to manage and validate schemas.

Dead letter queue strategy and poison message handling

Poison messages will be handled with a dead-letter queue (DLQ) strategy:

• Messages that fail more than 3 times will be moved to a DLQ topic
• DLQ messages will be logged with full context and sent to alerting system
• Manual reprocessing capability will be available via a dashboard or admin API

At-least-once vs. exactly-once delivery

At-least-once delivery will be used for content indexing events because:

• Duplicate indexing is acceptable (idempotent operation)
• The system can handle reprocessed content gracefully
• Simpler implementation and better performance

Retry policy for consumers

Retry configuration:

• Max attempts: 3
• Backoff strategy: Exponential with jitter (1s, 2s, 4s)
• Retryable errors: Network timeouts, transient storage errors
• Non-retryable errors: Invalid content format, authorization failures

Outbox pattern for reliable event publishing

The outbox pattern will be implemented to ensure eventual consistency between database writes and message publishing:

@Service
public class ContentIndexingService {
    
    @Autowired
    private EventOutboxRepository eventOutboxRepository;
    
    @Autowired
    private MessagePublisher messagePublisher;
    
    @Transactional
    public void indexContent(Content content) {
        // 1. Save content to database
        Content savedContent = contentRepository.save(content);
        
        // 2. Create outbox event
        EventOutbox event = EventOutbox.builder()
            .eventId(UUID.randomUUID().toString())
            .eventType("CONTENT_INDEXED")
            .payload(mapper.writeValueAsString(savedContent))
            .status(EventStatus.PENDING)
            .createdAt(Instant.now())
            .build();
            
        eventOutboxRepository.save(event);
        
        // 3. Publish to message queue (in same transaction)
        messagePublisher.publish(event);
        
        // 4. Update status after successful publish
        event.setStatus(EventStatus.PROCESSED);
        eventOutboxRepository.save(event);
    }
}

Sequence diagram for async flows

sequenceDiagram
    participant Client
    participant SearchService
    participant ContentRepository
    participant EventOutbox
    participant MessageBroker
    participant IndexingService
    
    Client->>SearchService: Submit content
    SearchService->>ContentRepository: Save content
    SearchService->>EventOutbox: Create event record
    SearchService->>MessageBroker: Publish content indexed event
    MessageBroker-->>IndexingService: ContentIndexedEvent
    IndexingService->>SearchService: Update search index
    SearchService-->>Client: Return success response

10. Error Handling Design

Error Classification Hierarchy

The search service implements a hierarchical error classification system to enable targeted handling and monitoring:

public abstract class SearchServiceException extends RuntimeException {
    public SearchServiceException(String message) { super(message); }
    public SearchServiceException(String message, Throwable cause) { super(message, cause); }
}

public class ValidationException extends SearchServiceException {
    public ValidationException(String message) { super(message); }
}

public class SearchIndexingException extends SearchServiceException {
    public SearchIndexingException(String message) { super(message); }
}

public class SearchQueryException extends SearchServiceException {
    public SearchQueryException(String message) { super(message); }
}

public class InfrastructureException extends SearchServiceException {
    public InfrastructureException(String message, Throwable cause) { super(message, cause); }
}

Error Propagation Strategy

Errors follow a layered propagation pattern:

• Service Layer: Converts infrastructure and domain errors into service-specific exceptions
• Controller Layer: Translates service exceptions to appropriate HTTP responses
• Global Exception Handler: Handles uncaught exceptions with standardized error responses

Retry Strategy with Exponential Backoff and Jitter

For transient infrastructure issues (e.g., search engine timeouts, database connection failures):

public class RetryableSearchOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;

    public <T> T executeWithRetry(Supplier<T> operation) {
        int attempts = 0;
        while (true) {
            try {
                return operation.get();
            } catch (InfrastructureException e) {
                if (++attempts >= MAX_RETRIES) {
                    throw new SearchServiceException("Operation failed after " + MAX_RETRIES + " attempts", e);
                }
                
                long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempts - 1));
                delay += (long) (delay * JITTER_FACTOR * (Math.random() - 0.5) * 2);
                
                try {
                    Thread.sleep(delay);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new SearchServiceException("Retry interrupted", ie);
                }
            }
        }
    }
}

Circuit Breaker Design

For external search engine dependencies:

public class CircuitBreaker {
    private final int failureThreshold = 5;
    private final long timeoutMs = 60000;
    private final int resetTimeoutMs = 30000;
    
    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private long lastFailureTime = 0;
    
    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }
    
    public <T> T execute(Supplier<T> operation) {
        switch (state) {
            case CLOSED:
                try {
                    T result = operation.get();
                    reset();
                    return result;
                } catch (Exception e) {
                    recordFailure();
                    throw new SearchServiceException("Circuit breaker open", e);
                }
                
            case OPEN:
                if (System.currentTimeMillis() - lastFailureTime > resetTimeoutMs) {
                    state = CircuitState.HALF_OPEN;
                    return execute(operation);
                }
                throw new SearchServiceException("Circuit breaker is OPEN");
                
            case HALF_OPEN:
                try {
                    T result = operation.get();
                    reset();
                    return result;
                } catch (Exception e) {
                    state = CircuitState.OPEN;
                    throw new SearchServiceException("Circuit breaker re-opened", e);
                }
        }
        throw new SearchServiceException("Unexpected circuit state");
    }
    
    private void recordFailure() {
        failureCount++;
        lastFailureTime = System.currentTimeMillis();
        if (failureCount >= failureThreshold) {
            state = CircuitState.OPEN;
        }
    }
    
    private void reset() {
        failureCount = 0;
        state = CircuitState.CLOSED;
    }
}

Fallback Behavior Design

For critical search operations, fallbacks provide graceful degradation:

public class SearchService {
    private final SearchEngine searchEngine;
    private final FallbackSearchEngine fallbackEngine;
    
    public SearchResult search(SearchRequest request) {
        try {
            return searchEngine.search(request);
        } catch (InfrastructureException e) {
            // Fallback to cached results or basic search
            return fallbackEngine.search(request);
        }
    }
}

Error Logging Format and Alerting Thresholds

Structured logging follows this schema:

{
  "timestamp": "2023-10-27T14:30:00Z",
  "trace_id": "abc123def456",
  "span_id": "xyz789uvw012",
  "user_id": "user-12345",
  "error_code": "SEARCH_INDEXING_ERROR",
  "error_message": "Failed to index document",
  "error_type": "InfrastructureException",
  "request_id": "req-67890",
  "service_name": "search-service",
  "severity": "ERROR"
}

Alerting thresholds:

• Critical: >100 errors per minute in search operations
• Warning: >50 errors per minute in search operations
• Info: >10 errors per minute in search operations

Error Response Format

Standardized JSON error responses:

{
  "error": {
    "code": "SEARCH_QUERY_ERROR",
    "message": "Invalid search query parameters",
    "details": {
      "field": "query",
      "reason": "Query must not be empty"
    },
    "timestamp": "2023-10-27T14:30:00Z",
    "request_id": "req-67890"
  }
}

Error Handling in API Layer

@RestController
@RequestMapping("/api/search")
public class SearchController {
    
    @ExceptionHandler(ValidationException.class)
    @ResponseStatus(HttpStatus.BAD_REQUEST)
    public ErrorResponse handleValidation(ValidationException e) {
        return new ErrorResponse("INVALID_SEARCH_QUERY", 
                                e.getMessage(), 
                                "Query parameter validation failed");
    }
    
    @ExceptionHandler(SearchIndexingException.class)
    @ResponseStatus(HttpStatus.INTERNAL_SERVER_ERROR)
    public ErrorResponse handleIndexing(SearchIndexingException e) {
        return new ErrorResponse("INDEXING_ERROR", 
                                "Search index operation failed", 
                                "Please try again later");
    }
    
    @ExceptionHandler(InfrastructureException.class)
    @ResponseStatus(HttpStatus.SERVICE_UNAVAILABLE)
    public ErrorResponse handleInfrastructure(InfrastructureException e) {
        return new ErrorResponse("INFRASTRUCTURE_ERROR", 
                                "Search service temporarily unavailable", 
                                "Please retry in a moment");
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using a combination of:

• Regex patterns for format validation (e.g., email, phone number)
• Length limits to prevent buffer overflow attacks
• Whitelisting for allowed characters in search terms and filters
• Custom validators for business logic constraints (e.g., date ranges, numeric bounds)
• OWASP Java HTML Sanitizer for preventing XSS in text fields

SQL Injection Prevention

• Prepared statements with parameterized queries for all database operations
• ORM usage (Hibernate/MyBatis) with strict mapping rules to prevent raw SQL injection
• Search term escaping using database-specific escape sequences when dynamic queries are unavoidable

Authentication Middleware Design

• JWT-based authentication with short-lived access tokens (15 min) and refresh tokens (7 days)
• OAuth2 integration for third-party authentication providers (Google, GitHub)
• Authentication middleware that intercepts all requests and validates JWT tokens
• Token revocation endpoint to invalidate session tokens on logout or token compromise

Authorization Enforcement

• Role-Based Access Control (RBAC) with predefined roles: ANONYMOUS, USER, ADMIN
• Permission checking at service layer using @PreAuthorize annotations in Spring Security
• Resource-level authorization for search results (e.g., users can only see their own search history)
• Fine-grained permissions for admin actions (e.g., clearing search index, modifying search configuration)

Sensitive Data Encryption

• At-rest encryption: AES-256 encryption for search index data and user preferences using AWS KMS
• In-transit encryption: TLS 1.3 for all API communications (enforced via HTTPS)
• Search result encryption: For PII fields in search results, AES-256 with key rotation every 90 days

PII Handling and Masking Strategy

• PII fields in search data (user names, emails) are masked in logs and responses
• Search query logging strips PII from search terms before logging (e.g., "john.doe@example.com" → "user@domain.com")
• Search result masking for sensitive fields using regex-based redaction patterns
• Audit trail logs only non-sensitive metadata (timestamp, user ID, query length) for compliance

Secrets Management

• AWS Secrets Manager for storing API keys, database credentials, and encryption keys
• Secret rotation policy: 90-day automatic rotation for all secrets with 30-day grace period
• Environment-specific secret paths to isolate dev/staging/prod credentials
• Least privilege access for services using IAM roles with minimal required permissions

Audit Logging Design

• Event types: SEARCH_QUERY, SEARCH_RESULT_VIEWED, SEARCH_HISTORY_MODIFIED, CONFIGURATION_CHANGED
• Log format: JSON with structured fields including:

  {
    "timestamp": "2023-10-27T14:30:00Z",
    "user_id": "uuid-12345",
    "event_type": "SEARCH_QUERY",
    "query_text": "user@domain.com",
    "ip_address": "192.168.1.100",
    "session_id": "session-abcde"
  }

• Retention policy: 90 days for audit logs with archiving to S3 Glacier
• Compliance support: Logs are timestamped and signed for GDPR/CCPA compliance

// Example of input validation in search service
public class SearchRequestValidator {
    private static final Pattern EMAIL_PATTERN = 
        Pattern.compile("^[A-Za-z0-9+_.-]+@([A-Za-z0-9.-]+\\.[A-Za-z]{2,})$");
    
    public void validateSearchRequest(SearchRequest request) {
        if (request.getQuery() == null || request.getQuery().trim().isEmpty()) {
            throw new ValidationException("Search query cannot be empty");
        }
        
        if (request.getQuery().length() > 1000) {
            throw new ValidationException("Search query exceeds maximum length");
        }
        
        // Sanitize query to prevent injection
        String sanitizedQuery = sanitizeInput(request.getQuery());
        
        // Validate no PII in query (basic check)
        if (EMAIL_PATTERN.matcher(sanitizedQuery).find()) {
            throw new ValidationException("PII detected in search query");
        }
    }
    
    private String sanitizeInput(String input) {
        return HtmlSanitizer.sanitize(input);
    }
}

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

• Service Layer: 85% code coverage, focusing on business logic and edge cases
• Repository Layer: 90% coverage with mock database interactions
• Domain Objects: 100% coverage for value objects and immutable entities
• Exception Handling: 100% coverage for all error paths and validation failures
• Core Algorithms: 95% coverage including search result ranking, filtering, and pagination logic

Integration Testing Strategy

• Database Integration: Use Testcontainers with PostgreSQL/MySQL for real database testing with schema migrations
• External Service Mocking: WireMock for third-party search index services (Elasticsearch, Solr)
• Event System: Embedded Kafka or RabbitMQ for async flow validation
• Caching Layer: Redis Testcontainers to validate cache behavior and invalidation flows
• Authentication Integration: Mock OAuth2 server with JWT generation for auth flow testing

Contract Testing

• API Contracts: Pact framework for consumer-driven contract testing between search service and dependent services (e.g., product catalog, user service)
• Message Contracts: Schema Registry integration with Avro schema validation for event payloads
• Version Compatibility: Contract tests to ensure backward compatibility across API versions

Performance Testing Tool and Key Scenarios

• Load Testing: k6 for high-concurrency search scenarios with 10,000+ concurrent users
• Stress Testing: Gatling for sustained load testing with 50,000+ requests to validate search service under pressure
• Key Scenarios:
  • High-volume keyword search (10k+ queries/sec)
  • Complex multi-field filtering (50% of queries with 3+ filters)
  • Pagination performance (100ms p95 for page 100 of 10k results)
  • Search with facets and aggregations (500ms p95 for top 100 results with 20 facets)
  • Cache hit rate validation (target: 85%+ cache hits)

Test Data Management Strategy

• Fixtures: YAML-based fixtures for common search data patterns (product categories, tags, user preferences)
• Factories: Factory pattern for generating realistic search data with configurable attributes (e.g., random text generation, synthetic timestamps)
• Seeding: Automated data seeding scripts using database migrations and test data builders for integration tests
• Data Isolation: Per-test database schema or transactional isolation to prevent test interference
• Synthetic Data Generation: Use libraries like Faker.java for generating realistic but non-sensitive test content

// Example unit test for search query validation
@Test
public void shouldThrowExceptionForInvalidSearchQuery() {
    SearchRequest request = new SearchRequest.Builder()
        .withQuery("")
        .withPageSize(0)
        .build();
    
    assertThatThrownBy(() -> searchService.search(request))
        .isInstanceOf(InvalidSearchQueryException.class)
        .hasMessageContaining("query cannot be empty");
}

13. Observability Design

Structured Logging Schema

All logs follow a structured JSON format with consistent fields to enable efficient querying and analysis:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO",
  "service": "search-service",
  "trace_id": "1234567890abcdef1234567890abcdef",
  "span_id": "abcdef1234567890",
  "user_id": "user-123",
  "request_id": "req-456",
  "operation": "search",
  "query": "laptop",
  "duration_ms": 150,
  "result_count": 24,
  "error_code": null,
  "error_message": null,
  "stack_trace": null
}

PII fields (user_id, email) are masked in logs using a configurable redaction pattern. For example, user-123 becomes user-***.

Distributed Tracing Instrumentation

Using OpenTelemetry with custom spans for key operations:

• Search Operation Span: Captures entire search flow including query parsing, index lookup, and result formatting.
• Index Query Span: Tracks individual database or search engine queries.
• External API Call Span: For any external service calls (e.g., analytics, user service).

Key attributes captured:

• search.query - original search query
• search.results.count - number of results returned
• search.duration.ms - total time in milliseconds
• search.index.type - type of index used (elastic, postgres, etc.)
• search.user.id - user identifier when available

Metrics Design

RED Method metrics for service health:

Metric Name	Type	Labels	Description
search_requests_total	Counter	operation, status, error_code	Total search requests
search_duration_seconds	Histogram	operation, index_type	Response time distribution
search_errors_total	Counter	error_code, operation	Error counts by type
search_results_count	Histogram	operation	Distribution of result counts
search_cache_hit_ratio	Gauge	cache_type	Cache hit rate percentage

Health Check Endpoint Design

GET /health
GET /ready
GET /live
GET /startup

/health: Comprehensive check including database connectivity, cache health, and external service availability.

{
  "status": "healthy",
  "checks": {
    "database": {"status": "healthy", "duration_ms": 5},
    "cache": {"status": "healthy", "duration_ms": 2},
    "external_api": {"status": "healthy", "duration_ms": 15}
  }
}

/ready: Service is ready to serve traffic (database and cache healthy). /live: Basic liveness check (process running). /startup: Startup validation including config loading and initial index load.

Alerting Thresholds

Metric	Warning Level	Critical Level	Escalation
search_duration_seconds p95	> 200ms	> 500ms	On-call engineer
search_errors_total rate	> 10/minute	> 50/minute	On-call engineer
search_cache_hit_ratio	< 70%	< 50%	On-call engineer
search_requests_total rate	> 1000/second	> 5000/second	On-call engineer
search_duration_seconds p99	> 500ms	> 1s	On-call engineer

All critical alerts trigger Slack notifications and escalate to on-call engineers after 5 minutes of sustained alert state.

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• Search relevance algorithm: Should we use Elasticsearch's built-in ranking or implement custom scoring logic? This decision will impact both search quality and maintenance overhead.
• Search result freshness: How often should we reindex data? Real-time indexing increases latency and complexity, while batch indexing introduces staleness.
• Search result deduplication: Should we deduplicate results across different content types or sources? This affects both user experience and system complexity.
• Search analytics scope: What metrics should we track? Should we focus on query performance, user behavior, or content engagement?

Known trade-offs and justifications

Trade-off	Option 1	Option 2	Justification
Indexing strategy	Real-time indexing	Batch indexing	Real-time provides better UX but increases write latency and system complexity. Batch indexing reduces load but introduces data staleness.
Search result format	Full document retrieval	Partial field retrieval	Full retrieval provides richer context but increases bandwidth and memory usage. Partial retrieval is more efficient but requires additional lookups.
Caching strategy	In-memory cache	Distributed cache	In-memory is faster but limited to single node. Distributed cache scales better but introduces network latency and complexity.
Search query parsing	Simple keyword matching	Advanced NLP processing	Simple matching is fast and simple but lacks semantic understanding. NLP provides better relevance but increases complexity and latency.

Risks and mitigation strategies

• Indexing performance degradation: High write volume can cause index rebuilds to take too long, impacting search availability. Mitigation: Implement incremental indexing with background reindexing and read-only index fallbacks.

• Search result inconsistency: Data inconsistencies between search index and database can cause incorrect results. Mitigation: Implement strict consistency policies with eventual consistency boundaries clearly defined.

• High query latency: Complex queries or large result sets can cause performance issues. Mitigation: Implement query optimization, result pagination, and early termination for slow queries.

• Resource exhaustion: Search service can consume excessive memory or CPU during peak loads. Mitigation: Implement query timeouts, resource quotas, and circuit breaker patterns.

Technical debt acknowledged

• Search index synchronization: Current design assumes perfect synchronization between database and search index, which may not hold in distributed environments. This creates technical debt around eventual consistency handling.

• Limited search query flexibility: The current API design prioritizes performance over query flexibility, which may require future refactoring to support advanced search features like faceted search or geospatial queries.

• Hardcoded search configuration: Search parameters and ranking algorithms are currently hardcoded, making it difficult to tune or A/B test different configurations without code changes.

• Monolithic search index: Single search index for all content types creates a single point of failure and makes it harder to scale specific content types independently. Future evolution should consider content-type-specific indexes.

• Limited analytics integration: Search analytics are not deeply integrated into the system, making it difficult to understand user behavior and optimize search relevance without additional infrastructure.