Semantic Search System Design: AI-Powered Content Retrieval Architecture

Comprehensive system design for a semantic search platform leveraging embeddings, vector databases, and neural ranking models for intelligent content retrieval.

PublishedSeptember 2, 2025
Length7,256 words
Reading time37 min read

AI/ML System Design

Modern semantic search systems leverage advanced NLP techniques and vector similarity matching to deliver intelligent content retrieval, transforming traditional keyword-based search into context-aware information discovery.

1. Problem Statement & ML Task Definition

Clarifying questions

What's the expected scale of documents/items to be indexed (e.g., millions vs billions)?
What's the target latency for search queries (e.g., <100ms vs <500ms)?
Is this a batch or real-time search system (e.g., user queries vs. pre-computed recommendations)?
What's the primary business use case (e.g., content discovery, product search, code search)?
What's the expected QPS for the system (e.g., 1000 vs 100,000 queries/sec)?
What's the acceptable trade-off between search accuracy and response time?
How important is the ability to handle multi-lingual queries?

Business use case and success criteria

The semantic search system aims to provide users with relevant content by understanding query semantics rather than keyword matching. This enables more intuitive search experiences where users can find information using natural language queries.

Success criteria include:

Precision@K (e.g., 0.8+ for top 10 results)
NDCG@K (e.g., >0.9 for top 10 ranking)
User engagement metrics (e.g., click-through rate, time spent on results)
Search satisfaction score (user surveys or implicit feedback)

ML problem formulation

This is a ranking problem with the following characteristics:

Task: Learning to rank documents based on semantic similarity to query
Input: Query text and document text (or embeddings)
Output: Ranked list of documents with relevance scores
Justification: Users want to find semantically similar content, not just keyword matches. The system needs to understand the meaning behind queries and match that meaning with relevant content.

ML evaluation metrics

Offline metrics:

AUC-ROC for binary relevance classification
NDCG@K (Normalized Discounted Cumulative Gain)
Precision@K (e.g., P@5, P@10)
Recall@K
BLEU score for generation tasks (if applicable)

Online metrics:

CTR (Click-Through Rate) on search results
Engagement time per result page
Revenue impact from improved search (e.g., conversion rate improvement)
User satisfaction scores

Scale and constraints

Back-of-envelope estimates:

Training data size: 10M query-document pairs (50GB text)
Number of features: 10K+ embedding dimensions for BERT-based models
Embedding matrix memory footprint: ~4GB (300M embeddings @ 128 bytes)
Serving QPS math: Assuming 50ms latency target, system needs to support 20K QPS for 100ms p99 latency

Constraints:

Latency: <200ms for 95th percentile response time
QPS: 10K+ queries/sec (peak)
Data freshness: Daily updates for content indexing
Storage: 100GB+ of indexed documents

Non-ML baseline strategy

Rule-based fallback:

Keyword matching with TF-IDF as fallback when semantic model fails
Exact match on title/keywords for cold start scenarios
Hybrid approach combining keyword and semantic matching

Cold start handling:

For new users: Use popularity-based ranking (most viewed, most liked)
For new content: Use content similarity to recently added items
Fallback to category-based search when no semantic matches found

# Example fallback logic in Python
def semantic_search_fallback(query, semantic_model, keyword_model, threshold=0.7):
    try:
        semantic_score = semantic_model.score(query)
        if semantic_score > threshold:
            return semantic_model.rank_documents(query)
        else:
            # Fallback to keyword matching
            return keyword_model.rank_documents(query)
    except Exception as e:
        # Complete fallback to rule-based
        return keyword_model.rank_documents(query)

2. Data Strategy

Training Data Sources

Explicit feedback: User queries with relevant document titles/URLs from search logs, user click-through data, and manual relevance judgments
Implicit signals: Query-document interaction patterns (time spent on page, scroll depth, bounce rate), user session behavior, and navigation paths
Third-party data: Publicly available document collections (Wikipedia, news articles, technical documentation), structured datasets from domain experts
Synthetic data: Generated query-document pairs using rule-based templates, paraphrasing engines, and domain-specific knowledge graphs

Labeling Pipeline Design

Human annotation: For high-quality training data, use expert annotators to label query-document relevance (relevance scores 0-4) with inter-annotator agreement > 85%
Weak supervision: Apply heuristics-based labeling (e.g., keyword matching, TF-IDF similarity) for initial data curation and to bootstrap labeling
Programmatic labeling: Use pre-trained semantic similarity models (e.g., Sentence-BERT) to generate initial labels, followed by human review for quality control

Class Imbalance Handling Strategy

Justification: Most queries have a single highly relevant document, while many others have no relevant documents or multiple equally relevant documents
Approach: Use stratified sampling during training to ensure balanced representation of query-document pairs across relevance levels. Apply weighted loss functions with higher weights for rare relevance classes (e.g., 0 and 4) to prevent model bias toward majority classes

Negative Sampling Approach

Strategy:
- Hard negative sampling: Select documents that are semantically similar but not relevant to the query using pre-trained embeddings and cosine similarity
- Random negative sampling: Sample documents from a pool of non-relevant documents to ensure diversity in training
- Negative query-document pairs: For each query, generate negative samples by swapping document titles or using paraphrased queries
Implementation:

def sample_negative_pairs(queries, documents, model, k=5):
    negative_samples = []
    for query in queries:
        # Get top-k most similar documents (excluding the positive)
        similarities = model.compute_similarity(query, documents)
        sorted_docs = sorted(enumerate(similarities), key=lambda x: x[1], reverse=True)
        # Select top-k hard negatives
        hard_negatives = [doc_id for doc_id, _ in sorted_docs[:k]]
        negative_samples.extend([(query, doc_id) for doc_id in hard_negatives])
    return negative_samples

Data Freshness and Update Frequency Requirements

Freshness:
- Real-time: Query logs updated every 15 minutes for active queries and user behavior
- Daily: Document corpus updates once daily to include new content
- Weekly: Relevance labels refreshed weekly to incorporate user feedback and new domain knowledge
Update frequency:
- High-volume queries: Updated every hour to capture recent trends
- Low-volume queries: Updated once a week or bi-weekly to reduce computational overhead

Privacy Compliance

GDPR/CCPA:
- Anonymization: Remove all personally identifiable information (PII) from queries and documents before training
- Data retention: Retain only the last 30 days of query logs and document metadata for compliance
- Consent management: Implement opt-out mechanisms for users to remove their data from training sets

Data Quality Validation and Monitoring

Validation checks:
- Query quality: Filter out queries with less than 3 characters or excessive special characters
- Document quality: Validate document content for spam, duplicate content, and low-quality text
- Relevance consistency: Ensure that labeled relevance scores are consistent across annotators with >90% agreement
Monitoring:
- Drift detection: Monitor for changes in query distribution and document content using statistical tests (e.g., Kolmogorov-Smirnov test)
- Feedback loop: Implement a system to collect user feedback on search results for continuous model improvement

def validate_query(query):
    if len(query) < 3 or not query.isprintable():
        return False
    return True

def validate_document(doc):
    if not doc or len(doc) < 100:
        return False
    # Additional checks for spam, duplicates, etc.
    return True

3. Feature Engineering

Feature Taxonomy

Features are categorized into four main types:

User Features: User demographics, behavior patterns, historical interactions, session data
Item Features: Content metadata (title, description, category), item popularity, freshness indicators
Context Features: Time of day, device type, geographic location, user session context
Interaction Features: User-item click history, dwell time, engagement signals, recency scores

Embedding Design

Entity Embeddings:

User ID embeddings: 128-dim, trained via click-through prediction
Item ID embeddings: 256-dim, learned from content and interaction signals

Text Embeddings:

Title + description: 512-dim BERT embeddings (base model)
Content processing pipeline: text cleaning → tokenization → embedding generation

Graph Embeddings:

User-item interaction graph: 128-dim Node2Vec embeddings
Content co-occurrence graph: 64-dim GraphSAGE embeddings

Real-time vs Batch Feature Computation

Feature Type	Computation Mode	Reason
User session features	Real-time	Immediate context relevance
Item popularity scores	Batch	Computationally expensive, updates hourly
Historical engagement	Batch	Pre-computed, stable over time
Content embeddings	Batch	Expensive to compute, updated daily

Feature Store Design

Online Store:

Low-latency KV store (Redis/FAISS) for real-time serving
TTL: 5 minutes for session features, 1 hour for user/item embeddings

Offline Store:

Data warehouse (Snowflake/BigQuery) for training and batch processing
Partitioned by date, optimized for SQL queries

Feature Freshness and TTL

Feature	Source	Update Frequency	Store Type
User session	Real-time stream	Every 30 seconds	Online
Item embeddings	Batch job	Daily	Online
User preferences	Real-time stream	Every 5 minutes	Online
Content metadata	CMS	Manual update	Offline
Popularity scores	Batch job	Hourly	Online

Cross Features and Feature Interaction Strategy

Cross features are generated through:

Multiplicative interactions: User embedding × Item embedding (dot product)
Concatenation features: User + Item feature vectors for dense layers
Temporal cross features: User-item interaction timestamp × item freshness

Feature Importance and Selection Methodology

Importance Metrics:

SHAP values from tree-based models (XGBoost)
Permutation importance from trained neural networks
Information gain for feature selection

Selection Strategy:

Remove features with SHAP mean absolute value < 0.01
Keep top 500 features by permutation importance for training
Apply L1 regularization (Lasso) in model training to enforce sparsity

# Feature engineering pipeline example
import numpy as np
from sklearn.preprocessing import StandardScaler

class SemanticSearchFeatureExtractor:
    def __init__(self):
        self.scaler = StandardScaler()
        
    def extract_user_features(self, user_data):
        """Extract user session and behavioral features"""
        features = {
            'user_session_length': len(user_data['sessions']),
            'user_engagement_score': np.mean(user_data['engagement_scores']),
            'user_recency_days': (datetime.now() - user_data['last_interaction']).days
        }
        return features
    
    def extract_item_features(self, item_data):
        """Extract content and metadata features"""
        features = {
            'item_content_length': len(item_data['content']),
            'item_category_id': item_data['category_id'],
            'item_popularity_score': item_data['popularity'],
            'item_freshness_days': (datetime.now() - item_data['created_at']).days
        }
        return features
    
    def compute_cross_features(self, user_emb, item_emb):
        """Compute interaction features"""
        # Dot product for semantic similarity
        dot_product = np.dot(user_emb, item_emb)
        
        # Concatenation for dense layer input
        concat_features = np.concatenate([user_emb, item_emb])
        
        # Feature engineering for temporal context
        temporal_feature = np.abs(np.dot(user_emb, item_emb) - np.mean(item_emb))
        
        return {
            'semantic_similarity': dot_product,
            'concatenated_features': concat_features,
            'temporal_context': temporal_feature
        }

4. Model Architecture

Model Selection with Justification

For semantic search, I'll select a two-tower Siamese Transformer architecture with the following rationale:

Transformer-based models provide superior semantic understanding compared to traditional BM25 or TF-IDF approaches
Two-tower architecture enables efficient candidate retrieval and scoring while maintaining strong semantic alignment between query and document representations
Fine-tuning pre-trained models (e.g., Sentence-BERT) offers strong baseline performance with minimal training overhead
Scalable inference through efficient attention mechanisms and model compression

The architecture will use a Siamese Transformer with shared weights for query and document encoders, enabling joint embedding space learning.

Architecture Diagram

graph TD
    A[Query Input] --> B[Query Encoder (Transformer)]
    C[Document Input] --> D[Document Encoder (Transformer)]
    B --> E[Query Embedding]
    D --> F[Document Embedding]
    E --> G[Similarity Score]
    F --> G
    G --> H[Ranking/Relevance Score]

Two-Tower / Multi-task / Ensemble Design

The system will implement a two-tower architecture:

Query encoder (Q) and document encoder (D) with shared transformer backbone
Both encoders map to same-dimensional embeddings (768-d for SBERT)
Final scoring uses cosine similarity between query and document embeddings

For production, we'll implement multi-task learning with:

Main semantic similarity task (primary)
Auxiliary tasks: relevance classification, reranking scores

Pre-trained Model Selection and Fine-tuning Strategy

Pre-trained model: Sentence-BERT (SBERT) - all-MiniLM-L6-v2 (768-d embeddings)

Fine-tuning approach:

Use contrastive loss with margin-based ranking
Dataset: MS MARCO passage retrieval dataset (88M passages)
Training: 10 epochs with batch size 64, learning rate 2e-5
Freeze base layers and fine-tune only the final projection layer

Model Complexity vs Latency Trade-off Analysis

Model	Parameters	Inference Time (ms)	Memory Usage (GB)	Accuracy (NDCG@10)
SBERT-base	100M	25ms	1.2GB	0.78
SBERT-large	350M	45ms	2.5GB	0.82
Distilled SBERT	25M	15ms	0.4GB	0.72
Quantized SBERT	100M	18ms	0.6GB	0.75

Offline Evaluation Baseline Framing

Baseline: BM25 + TF-IDF approach with TF-IDF vectorizer and cosine similarity Expected baseline performance: NDCG@10 = 0.65, Precision@10 = 0.42

Proposed model: Siamese Transformer with contrastive fine-tuning Target performance: NDCG@10 = 0.82, Precision@10 = 0.58

Core Model Code Snippet

import torch
import torch.nn as nn
from transformers import AutoTokenizer, AutoModel
from torch.nn.functional import cosine_similarity

class SemanticSearchModel(nn.Module):
    def __init__(self, model_name='sentence-transformers/all-MiniLM-L6-v2'):
        super().__init__()
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.encoder = AutoModel.from_pretrained(model_name)
        self.pooling = 'mean'  # mean pooling for sentence embeddings
        
    def forward(self, query_input_ids, doc_input_ids):
        """
        Forward pass for semantic search model
        Returns: similarity scores between query and documents
        """
        # Encode query
        query_outputs = self.encoder(
            input_ids=query_input_ids,
            attention_mask=query_input_ids != 0
        )
        
        # Encode document
        doc_outputs = self.encoder(
            input_ids=doc_input_ids,
            attention_mask=doc_input_ids != 0
        )
        
        # Apply mean pooling to get sentence embeddings
        query_embeddings = self._mean_pooling(query_outputs, query_input_ids != 0)
        doc_embeddings = self._mean_pooling(doc_outputs, doc_input_ids != 0)
        
        # Compute cosine similarity for ranking
        similarities = cosine_similarity(
            query_embeddings.unsqueeze(1), 
            doc_embeddings.unsqueeze(0)
        )
        
        return similarities
    
    def _mean_pooling(self, model_output, attention_mask):
        """Mean pooling of token embeddings to get sentence embedding"""
        token_embeddings = model_output.last_hidden_state
        input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
        
        # Sum token embeddings weighted by attention mask
        sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
        
        # Normalize by sum of attention mask values
        sum_mask = torch.sum(input_mask_expanded, 1)
        mean_embeddings = sum_embeddings / sum_mask
        
        return mean_embeddings
    
    def compute_loss(self, query_embeddings, doc_embeddings, labels=None):
        """
        Compute contrastive loss for training
        """
        if labels is None:
            # For inference, return similarity scores directly
            return cosine_similarity(
                query_embeddings.unsqueeze(1), 
                doc_embeddings.unsqueeze(0)
            )
        
        # Contrastive loss implementation
        batch_size = query_embeddings.size(0)
        similarities = cosine_similarity(
            query_embeddings.unsqueeze(1), 
            doc_embeddings.unsqueeze(0)
        )
        
        # Create positive mask (diagonal elements)
        pos_mask = torch.eye(batch_size, dtype=torch.bool, device=similarities.device)
        
        # Compute loss using InfoNCE-style contrastive loss
        exp_sim = torch.exp(similarities / 0.1)  # Temperature scaling
        pos_sim = torch.diag(exp_sim)
        
        # Negative samples sum
        neg_sum = torch.sum(exp_sim, dim=1) - pos_sim
        
        # Loss computation
        loss = -torch.log(pos_sim / neg_sum).mean()
        
        return loss

# Example usage for inference
def semantic_search_inference(model, query_text, document_texts, top_k=10):
    """
    Example inference function for semantic search
    """
    # Tokenize inputs
    query_tokens = model.tokenizer(
        query_text, 
        return_tensors='pt', 
        padding=True, 
        truncation=True,
        max_length=512
    )
    
    doc_tokens = model.tokenizer(
        document_texts, 
        return_tensors='pt', 
        padding=True, 
        truncation=True,
        max_length=512
    )
    
    # Get embeddings and compute similarities
    with torch.no_grad():
        similarities = model(
            query_tokens['input_ids'],
            doc_tokens['input_ids']
        )
    
    # Get top-k matches
    top_k_scores, top_k_indices = torch.topk(similarities[0], k=top_k)
    
    return [
        {'document_index': idx.item(), 
         'similarity_score': score.item()} 
        for idx, score in zip(top_k_indices, top_k_scores)
    ]

This implementation provides:

Complete transformer-based semantic encoding pipeline
Contrastive loss for training with proper attention masking
Mean pooling for sentence-level embeddings
Efficient inference path for real-time semantic search
Ready-to-use training and inference functions with proper tensor operations

5. Serving & Inference Architecture

System Diagram

graph TD
    A[Query Input] --> B[Query Preprocessing]
    B --> C[Embedding Model]
    C --> D[Embedding Store]
    D --> E[Vector Index]
    E --> F[Ranking Model]
    F --> G[Result Ranking]
    G --> H[Response Formatter]
    H --> I[API Response]

    subgraph "Online Serving"
        C
        D
        E
        F
        G
    end

    subgraph "Feature Store"
        D
        E
    end

Online vs Batch Serving Decision

The semantic search system requires real-time online serving with sub-100ms latency requirements for user queries. Batch serving would introduce unacceptable delays for interactive search experiences, where users expect near-instantaneous results.

Two-Stage Pipeline

The system implements a two-stage pipeline:

Candidate Retrieval: Fast approximate nearest neighbor search using vector indexes (e.g., FAISS, ScaNN)
Scoring & Re-ranking: Fine-grained scoring with dense embeddings and reranking models

Serving Infrastructure

Primary serving stack:

TorchServe for embedding model inference (supports multiple models, built-in metrics)
FAISS for efficient vector similarity search (optimized for large-scale approximate nearest neighbor)
Redis for embedding cache and metadata storage
gRPC + REST API for low-latency query serving

Justification:

TorchServe provides production-grade model serving with built-in batching and monitoring
FAISS offers optimal performance for large-scale vector search with configurable recall
Redis provides fast caching of frequently accessed embeddings and metadata

Latency Budget Breakdown

Stage	p50 Latency	p99 Latency	Budget
Query preprocessing	1ms	3ms	5ms
Embedding generation	20ms	40ms	50ms
Vector search	5ms	15ms	20ms
Re-ranking	10ms	25ms	30ms
Response formatting	2ms	5ms	10ms
Total	38ms	83ms	115ms

Model Compression Strategy

Quantization approach:

INT8 quantization for embedding models (reduces memory by 4x, ~5% accuracy loss)
Pruning for ranking models (removes 30% of weights, ~2% accuracy loss)
Knowledge distillation from larger models to smaller student models

Accuracy trade-off:

Quantized models maintain >95% of original performance on held-out test sets
Pruned models show <2% degradation in recall@K metrics

Caching Strategy

Prediction cache:

Query embedding cache: 100K most frequent queries with pre-computed embeddings
Result cache: 50K recent search results with TTL of 1 hour

Embedding cache:

Item embedding cache: 1M most frequent documents with pre-computed embeddings
KV cache: For LLM-based query expansion (if used)

Cache invalidation:

TTL-based eviction for freshness
Cache warming during model updates

Auto-scaling and Capacity Planning

Horizontal scaling strategy:

Query embedding service: Scale based on QPS (target 5000 QPS per instance)
Vector search service: Scale based on index size and query volume
Ranking service: Scale based on re-ranking queue depth

Capacity planning:

# Back-of-envelope calculation for 1M items
- Embedding size: 768 dimensions × 4 bytes = 3KB per item
- Total embedding memory: 1M × 3KB = 3GB
- Vector index size: ~5GB (with IVF indexing)
- Expected QPS: 1000
- Target latency: <100ms

Auto-scaling thresholds:

CPU utilization >85% → scale up by 1 instance
Queue depth >500 → scale up by 2 instances
Memory usage >90% → scale up by 1 instance

Failover strategy:

Primary embedding model fails → fallback to cached embeddings
Vector index unavailable → return basic keyword search results
Ranking model fails → return top-k results from vector search

6. Evaluation Framework

Offline Evaluation Methodology

For semantic search, we use temporal holdout splits to evaluate model performance. We split our dataset chronologically with 80% for training, 10% for validation, and 10% for testing. This ensures that our model is evaluated on truly unseen content and prevents data leakage from future content influencing training.

We implement cross-validation with time-based splits for robustness, particularly for handling temporal drift in user behavior and content evolution.

Slice-Based Evaluation

We define critical slices to ensure model fairness and robustness:

Slice	Definition	Rationale
User Segments	New vs. returning users, high-traffic vs. low-traffic users	Different search behaviors and content preferences
Content Types	Text, image, video, mixed media	Varying semantic richness and search requirements
Geographic Regions	US, EU, APAC	Language differences, content availability
Device Types	Mobile, desktop, tablet	Different search patterns and query lengths
Time of Day	Morning vs. evening	User intent varies by time

A/B Test Design

We design a multi-armed bandit A/B test with the following specifications:

Randomization Unit: User ID (to prevent leakage between users)

Sample Size Calculation:

Target effect size: 5% improvement in click-through rate
Power: 80%
Significance level: 5%
Estimated baseline CTR: 2.5%
Sample size per variant: ~13,000 users

Test Duration: 4 weeks (to capture seasonal and weekly patterns)

Primary Metric: Click-through rate (CTR) on search results

Secondary Metrics:

Average time to first click
Search satisfaction score (user survey)
Revenue per user (if applicable)

Guardrail Metrics and Blocking Conditions

Guardrail Metrics:

Precision@K (must not drop below 0.75)
NDCG@K (must maintain >95% of baseline)
Query coverage rate (must not fall below 90%)

Blocking Conditions:

If any slice shows >10% degradation in CTR, test is paused
If precision drops below threshold, immediate rollback to baseline
If model latency exceeds 500ms p99, automatic scaling trigger

Shadow Mode Rollout Plan

Week 1: 5% shadow traffic to new model with no action taken
Week 2: 15% shadow traffic, begin logging predictions and features
Week 3: 50% shadow traffic with real-time performance monitoring
Week 4: Full production rollout if all guardrails pass

Long-Term Metric Measurement

We track delayed conversion metrics to ensure semantic search systems drive long-term engagement:

Metric	Definition	Target
7-day conversion rate	Users who perform additional searches or actions within 7 days	+5%
Engagement decay	Time between search and next engagement	-10%
Session duration	Average time spent in search session	+15%

We implement a 30-day tracking window for comprehensive impact assessment, with weekly reporting to stakeholders.

# Example evaluation code for semantic search model

import numpy as np
from sklearn.metrics import ndcg_score, precision_score

class SemanticSearchEvaluator:
    def __init__(self):
        self.k = 10  # Top-k results to evaluate
        
    def evaluate_ranking_performance(self, predictions, ground_truth):
        """
        Evaluate semantic search ranking performance
        
        Args:
            predictions: List of predicted relevance scores for each document
            ground_truth: List of binary labels (0/1) indicating relevance
        
        Returns:
            dict: Evaluation metrics
        """
        # Calculate NDCG@K
        ndcg = ndcg_score([ground_truth], [predictions], k=self.k)
        
        # Calculate Precision@K
        pred_rankings = np.argsort(predictions)[::-1][:self.k]
        pred_labels = [ground_truth[i] for i in pred_rankings]
        precision_at_k = sum(pred_labels) / self.k
        
        # Calculate MRR (Mean Reciprocal Rank)
        relevant_positions = [i for i, label in enumerate(ground_truth) if label == 1]
        if relevant_positions:
            min_relevant_pos = min(relevant_positions) + 1  # +1 for 1-based indexing
            mrr = 1.0 / min_relevant_pos
        else:
            mrr = 0.0
            
        return {
            'ndcg@k': ndcg,
            'precision@k': precision_at_k,
            'mrr': mrr
        }
    
    def evaluate_query_coverage(self, queries_processed, queries_with_results):
        """Calculate query coverage rate"""
        return queries_with_results / queries_processed if queries_processed > 0 else 0.0

# Example usage
evaluator = SemanticSearchEvaluator()
sample_predictions = [0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0.05]
sample_ground_truth = [1, 0, 1, 0, 0, 1, 0, 0, 0, 0]

metrics = evaluator.evaluate_ranking_performance(sample_predictions, sample_ground_truth)
print(f"Performance Metrics: {metrics}")

7. Training Pipeline

Training Data Pipeline

The training pipeline processes raw text data through ETL workflows to generate training examples. Raw documents are ingested from content management systems, user interaction logs, and third-party sources. The pipeline includes:

Data Extraction: Parse documents from various formats (PDF, HTML, Markdown) using libraries like pdfminer, BeautifulSoup, and markdown
Preprocessing: Normalize text, remove special characters, apply tokenization using sentencepiece or BPE
Deduplication: Use MinHash LSH for near-duplicate detection with 0.95 Jaccard similarity threshold
Shuffling: Implement reservoir sampling for large-scale shuffling across distributed workers

import hashlib
from typing import List, Tuple
import numpy as np

class TrainingDataPipeline:
    def __init__(self):
        self.tokenizer = sentencepiece.SentencePieceProcessor(model_file="tokenizer.model")
    
    def preprocess_text(self, text: str) -> str:
        # Normalize whitespace and remove special chars
        return re.sub(r'[^\w\s]', '', text.lower()).strip()
    
    def generate_training_examples(self, documents: List[str]) -> List[Tuple[str, str]]:
        examples = []
        for doc in documents:
            # Create query-document pairs with various transformations
            processed_doc = self.preprocess_text(doc)
            query = self.generate_query(processed_doc)
            examples.append((query, processed_doc))
        return examples

    def generate_query(self, doc_text: str) -> str:
        # Sample 3-5 random sentences from document as query
        sentences = doc_text.split('.')
        return '. '.join(np.random.choice(sentences, min(3, len(sentences)), replace=False)) + '.'

Distributed Training Strategy

For large-scale training, we use data parallelism with model parallelism for transformer models:

Data Parallelism: Split training batches across multiple GPUs using PyTorch's DistributedDataParallel
Model Parallelism: For models > 10B parameters, split across multiple GPUs using tensor parallelism
Pipeline Parallelism: For very large models, use pipeline stages with gradient accumulation

Training cluster configuration:

8x A100 GPUs (80GB VRAM each)
256GB total memory capacity
128GB system RAM per node
InfiniBand network for inter-node communication

Training Infrastructure & Timeline

Training infrastructure consists of:

GPU Cluster: 8 nodes, 8 GPUs each (64 total GPUs)
Storage: NVMe SSDs for fast data loading (2TB per node)
Memory: 128GB RAM per training node

Estimated training timeline:

Model size: ~10B parameters (Llama 2 variant)
Training data: 50M document-query pairs
Epochs: 3 epochs
Training time: ~24 hours on 64 GPUs with mixed precision

Hyperparameter Tuning Approach

We use Bayesian optimization for hyperparameter tuning:

Key hyperparameters:

Learning rate: 1e-4 to 5e-4 (log scale)
Batch size: 16 to 128 (linear scale)
Warmup steps: 1000 to 5000 (linear scale)
Weight decay: 0.01 to 0.1 (log scale)
Dropout rate: 0.1 to 0.3 (linear scale)

Experiment Tracking & Reproducibility

We implement MLflow tracking with:

Experiment logging: All hyperparameters, metrics, and model artifacts
Version control: Git integration for codebase versioning
Model registry: Centralized model version management with semantic versioning
Parameter sweep: Automated grid search for key parameter combinations

Model Versioning and Registry Strategy

Model versioning follows semantic versioning (v1.0.0, v1.1.0) with:

Version tags: Based on performance improvements and feature additions
Model artifacts: Full model checkpoints, tokenizer files, configuration JSONs
Registry integration: Docker containers with model dependencies pre-installed

Continual / Online Learning Design

For online learning, we implement:

Incremental training: Periodic retraining with new data samples (daily)
Adaptive learning rate: Reduce learning rate by 50% when drift detected
Fine-tuning strategy: Use new user feedback to update model weights with 0.1 learning rate
A/B testing: New models are tested against baseline in shadow mode before production deployment

class OnlineLearningManager:
    def __init__(self, model_path: str):
        self.model = self.load_model(model_path)
        self.feedback_buffer = []
        self.drift_threshold = 0.05
        
    def process_feedback(self, query: str, relevant_doc: str, user_rating: float):
        # Store feedback for batch training
        self.feedback_buffer.append({
            'query': query,
            'doc': relevant_doc,
            'rating': user_rating
        })
        
        # Check for drift every 1000 samples
        if len(self.feedback_buffer) % 1000 == 0:
            drift_detected = self.detect_drift()
            if drift_detected:
                self.fine_tune_model()
                
    def detect_drift(self):
        # Simple drift detection using embedding similarity
        recent_samples = self.feedback_buffer[-1000:]
        # Implementation of drift detection logic
        return False  # Placeholder for actual drift detection
        
    def fine_tune_model(self):
        # Apply fine-tuning with new feedback data
        pass

8. Monitoring & Observability

Data Drift Detection

Data drift detection is crucial for maintaining model performance in semantic search systems. We implement drift detection using Population Stability Index (PSI) for categorical features and Kolmogorov-Smirnov test for numerical features.

import numpy as np
from scipy import stats
from sklearn.metrics import pairwise_distances

class DriftDetector:
    def __init__(self, threshold=0.1):
        self.threshold = threshold
        
    def psi_drift(self, old_data, new_data, bins=10):
        """Calculate PSI for categorical features"""
        # Normalize data to 0-1 range
        old_hist, _ = np.histogram(old_data, bins=bins, density=False)
        new_hist, _ = np.histogram(new_data, bins=bins, density=False)
        
        # Avoid zero division
        old_hist = np.clip(old_hist, 1e-6, None)
        new_hist = np.clip(new_hist, 1e-6, None)
        
        psi = np.sum((new_hist - old_hist) * np.log(new_hist / old_hist))
        return psi
    
    def ks_drift(self, old_data, new_data):
        """Kolmogorov-Smirnov test for numerical features"""
        ks_stat, p_value = stats.ks_2samp(old_data, new_data)
        return ks_stat, p_value

# Example usage for feature monitoring
detector = DriftDetector()
psi_score = detector.psi_drift(old_user_features, new_user_features)

Prediction Drift and Label Drift Monitoring

Prediction drift monitors changes in model outputs, while label drift monitors actual user feedback patterns.

class PredictionDriftMonitor:
    def __init__(self, window_size=1000):
        self.window_size = window_size
        self.predictions_history = []
        
    def detect_prediction_drift(self, current_predictions, reference_predictions):
        """Detect drift in model predictions using cosine similarity"""
        if len(self.predictions_history) < self.window_size:
            return False
            
        # Calculate cosine similarity between recent predictions and reference
        recent_avg = np.mean(self.predictions_history[-self.window_size:], axis=0)
        current_avg = np.mean(current_predictions, axis=0)
        
        similarity = 1 - pairwise_distances([recent_avg], [current_avg], metric='cosine')[0][0]
        return similarity < 0.95  # Threshold for drift detection
    
    def update_history(self, new_predictions):
        self.predictions_history.extend(new_predictions)
        if len(self.predictions_history) > self.window_size * 2:
            self.predictions_history = self.predictions_history[-self.window_size:]

# Example usage
monitor = PredictionDriftMonitor()
drift_detected = monitor.detect_prediction_drift(current_embeddings, reference_embeddings)

Model Performance Degradation Alerting

We implement automated alerting for performance degradation using sliding window statistics.

class PerformanceMonitor:
    def __init__(self, alert_thresholds={'ndcg@10': 0.85, 'recall@10': 0.75}):
        self.alert_thresholds = alert_thresholds
        self.performance_history = []
        
    def check_performance_degradation(self, current_metrics):
        """Check if performance has degraded beyond threshold"""
        self.performance_history.append(current_metrics)
        
        # Keep only last 100 evaluations
        if len(self.performance_history) > 100:
            self.performance_history = self.performance_history[-100:]
            
        # Calculate rolling averages
        rolling_avg = {}
        for metric in current_metrics:
            values = [h[metric] for h in self.performance_history]
            rolling_avg[metric] = np.mean(values[-50:])  # Last 50 evaluations
            
        # Alert if any metric drops below threshold
        alerts = []
        for metric, threshold in self.alert_thresholds.items():
            if rolling_avg.get(metric, 1.0) < threshold:
                alerts.append({
                    'metric': metric,
                    'current_value': rolling_avg[metric],
                    'threshold': threshold,
                    'alert_type': 'performance_degradation'
                })
                
        return alerts

# Example usage
monitor = PerformanceMonitor()
alerts = monitor.check_performance_degradation({
    'ndcg@10': 0.72,
    'recall@10': 0.68
})

Retraining Trigger Strategy

We implement a multi-trigger retraining strategy combining drift, performance, and scheduled triggers.

class RetrainingTrigger:
    def __init__(self, drift_threshold=0.15, performance_window=100):
        self.drift_threshold = drift_threshold
        self.performance_window = performance_window
        
    def should_retrain(self, drift_scores, performance_metrics, time_since_last_retrain):
        """Determine if model should be retrained"""
        # Drift-based trigger
        drift_detected = any(score > self.drift_threshold for score in drift_scores.values())
        
        # Performance degradation trigger
        performance_degraded = self._check_performance_degradation(performance_metrics)
        
        # Scheduled trigger (every 30 days)
        scheduled_retrain = time_since_last_retrain > 30 * 24 * 60 * 60  # 30 days in seconds
        
        return drift_detected or performance_degraded or scheduled_retrain
    
    def _check_performance_degradation(self, metrics):
        """Check if performance has degraded"""
        # Simple implementation - in practice would use more sophisticated drift detection
        return False  # Placeholder for actual logic

# Example usage
trigger = RetrainingTrigger()
retrain_needed = trigger.should_retrain(
    drift_scores={'user_features': 0.12, 'item_features': 0.08},
    performance_metrics={'ndcg@10': 0.72, 'recall@10': 0.68},
    time_since_last_retrain=25 * 24 * 60 * 60
)

Logging Design

Comprehensive logging for prediction, feature, and feedback collection.

import json
import time
from datetime import datetime

class PredictionLogger:
    def __init__(self, log_file='prediction_logs.json'):
        self.log_file = log_file
        
    def log_prediction(self, query_id, user_id, query_text, 
                      top_k_items, scores, latency, timestamp=None):
        """Log prediction results for monitoring"""
        if timestamp is None:
            timestamp = datetime.now().isoformat()
            
        log_entry = {
            'timestamp': timestamp,
            'query_id': query_id,
            'user_id': user_id,
            'query_text': query_text,
            'top_k_items': top_k_items,
            'scores': scores.tolist() if hasattr(scores, 'tolist') else scores,
            'latency_ms': latency,
            'model_version': 'v1.2.3'
        }
        
        with open(self.log_file, 'a') as f:
            f.write(json.dumps(log_entry) + '\n')
            
    def log_feature_update(self, feature_name, update_time, feature_values):
        """Log feature updates for drift monitoring"""
        log_entry = {
            'timestamp': datetime.now().isoformat(),
            'feature_name': feature_name,
            'update_time': update_time.isoformat(),
            'feature_values': feature_values.tolist() if hasattr(feature_values, 'tolist') else feature_values
        }
        
        with open('feature_logs.json', 'a') as f:
            f.write(json.dumps(log_entry) + '\n')

# Example usage
logger = PredictionLogger()
logger.log_prediction(
    query_id='q12345',
    user_id='user_789',
    query_text='semantic search system design',
    top_k_items=['item_1', 'item_2', 'item_3'],
    scores=[0.95, 0.87, 0.76],
    latency=45.2
)

Feedback Loop for Continuous Improvement

We implement a feedback collection system to capture user interactions and improve model performance.

class FeedbackCollector:
    def __init__(self, feedback_storage):
        self.feedback_storage = feedback_storage
        
    def collect_feedback(self, query_id, user_id, clicked_items, 
                        relevance_scores, timestamp=None):
        """Collect user feedback for model improvement"""
        feedback_entry = {
            'query_id': query_id,
            'user_id': user_id,
            'clicked_items': clicked_items,
            'relevance_scores': relevance_scores,
            'timestamp': timestamp or datetime.now().isoformat(),
            'feedback_type': 'click' if clicked_items else 'no_click'
        }
        
        # Store feedback for retraining
        self.feedback_storage.append(feedback_entry)
        
        return feedback_entry
    
    def get_feedback_for_retraining(self, time_window_hours=24):
        """Get recent feedback for retraining"""
        cutoff_time = datetime.now() - timedelta(hours=time_window_hours)
        
        recent_feedback = [
            fb for fb in self.feedback_storage 
            if datetime.fromisoformat(fb['timestamp']) > cutoff_time
        ]
        
        return recent_feedback

# Example usage
feedback_collector = FeedbackCollector([])
feedback = feedback_collector.collect_feedback(
    query_id='q12345',
    user_id='user_789',
    clicked_items=['item_1', 'item_2'],
    relevance_scores=[5, 4]
)

Key Monitoring Components

Component	Frequency	Alert Threshold	Description
Data Drift	Hourly	PSI > 0.15	Detects feature distribution shifts
Prediction Drift	Hourly	Cosine Similarity < 0.95	Monitors model output consistency
Performance Metrics	Daily	NDCG@10 < 0.85	Tracks model effectiveness
Retraining Triggers	Weekly	Scheduled + Drift	Determines when to retrain
Feature Logs	Real-time	N/A	Captures feature evolution

Alerting Configuration

We implement a multi-level alerting system:

Critical Alerts: Immediate retraining trigger when drift exceeds threshold
Warning Alerts: Performance degradation detected (NDCG < 0.85)
Info Alerts: Scheduled retraining reminders
Drift Alerts: Feature drift monitoring with detailed reports

The system uses Slack/Email notifications for critical alerts and maintains detailed dashboards in Grafana for ongoing monitoring of key metrics like drift scores, performance trends, and retraining frequency.

9. Responsible AI & Safety

Bias Analysis

Training data audit: Conduct comprehensive audit of training corpus to identify representation gaps in domains, demographics, and content types. Use statistical sampling to detect over/under-representation of specific categories.
Slice-based fairness evaluation: Evaluate model performance across user segments (age groups, geographic regions, content categories) to detect potential bias in search relevance. Monitor precision/recall differences across slices.
Content diversity assessment: Measure whether search results disproportionately favor certain content types or sources, potentially creating echo chamber effects.

Fairness Metrics and Constraints

Demographic parity: Ensure equal representation of content from different user demographics in top-k results.
Equalized odds: Maintain consistent recall rates across different content categories and user segments.
Decision rationale: Prioritize demographic parity for user experience fairness, but implement equalized odds as secondary constraint to prevent systematic underrepresentation of minority content.

Explainability Requirements and Approach

SHAP values: Implement SHAP-based feature importance analysis for search relevance scores to provide interpretable explanations for why specific documents rank highly.
Attention visualization: For transformer-based models, provide attention weight heatmaps showing which query terms influenced document ranking.
User feedback integration: Build feedback loop to capture user reactions to explanations and adjust model accordingly.

For LLM Systems: Safety Guardrails

Content filtering: Implement content safety filters to block harmful, illegal, or inappropriate results (e.g., hate speech, misinformation, explicit content).
Output validation: Validate that search results adhere to content policies and do not violate terms of service.
Rate limiting: Enforce rate limits on queries to prevent abuse and ensure system stability.
Red teaming: Conduct adversarial testing with diverse user queries to identify edge cases and safety gaps.

Model Card Documentation Plan

Model metadata: Document model version, training data sources, preprocessing steps, and performance benchmarks.
Usage guidelines: Specify intended use cases, limitations, and recommended deployment configurations.
Performance characteristics: Include detailed performance metrics across different user segments and content types.

Incident Response Plan for Model Failures or Harmful Outputs

Failure detection: Monitor for sudden drops in model performance, drift in result quality, or emergence of harmful content in search results.
Emergency rollback: Implement quick rollback mechanism to revert to previous model versions if safety issues are detected.
User reporting system: Provide users with easy mechanism to flag inappropriate results or safety concerns.
Incident escalation: Define clear escalation paths for security and compliance teams when harmful outputs are detected.

# Example safety filtering logic in Python
def filter_harmful_content(query, search_results):
    """
    Filter out harmful content from search results based on safety rules.
    
    Args:
        query (str): User search query
        search_results (list): List of document objects with content fields
    
    Returns:
        filtered_results (list): Search results with harmful content removed
    """
    
    # Safety rules dictionary
    safety_rules = {
        "hate_speech_keywords": ["hate", "discriminate", "racist"],
        "explicit_content_indicators": ["sex", "nude", "adult"],
        "misinformation_indicators": ["fake news", "conspiracy"]
    }
    
    filtered_results = []
    
    for doc in search_results:
        content = doc.get("content", "")
        title = doc.get("title", "")
        
        # Check for harmful content indicators
        is_harmful = any(keyword in content.lower() or keyword in title.lower()
                        for keyword in safety_rules["hate_speech_keywords"])
        
        if not is_harmful:
            filtered_results.append(doc)
    
    return filtered_results

# Example usage
query = "hate speech examples"
search_results = [
    {"title": "Hate speech analysis", "content": "This content contains hate speech"},
    {"title": "Legal definitions", "content": "Hate speech is illegal in many countries"}
]

filtered_results = filter_harmful_content(query, search_results)
print(f"Filtered {len(search_results) - len(filtered_results)} harmful results")

10. Scalability & Reliability

Bottleneck Analysis per Pipeline Stage

Training Pipeline:

Data processing: 20% of total training time spent on ETL and preprocessing
Model training: 70% of training time, dominated by GPU memory constraints (128GB limit)
Model serialization: 5% of training time, bottlenecked by disk I/O (1GB/s)

Serving Pipeline:

Embedding computation: 30% of latency, CPU-bound with 20ms p99 target
Similarity search: 40% of latency, memory-bound with 50ms p99 target
Feature retrieval: 20% of latency, database query time with 15ms p99 target
Response formatting: 10% of latency, JSON serialization with 5ms p99 target

Back-of-envelope estimates:

Training data size: 10M documents → 50GB raw text + 5GB embeddings
Feature dimensionality: 128M features → 512MB embedding matrix (4 bytes × 128M)
Serving QPS: 10,000 requests/sec → 50ms p99 latency target
Memory footprint: 2GB embedding cache + 1GB model weights = 3GB per serving node

Scale-up Strategy

Horizontal Scaling:

Feature store: Sharding by user/item ID with consistent hashing (32 shards)
Model serving: Auto-scaling groups with 100 nodes, each handling 100 QPS
Indexing: Distributed inverted index across 64 nodes with 256GB RAM each

Distributed Feature Computation:

Real-time features: 200 Kafka partitions, 1000 concurrent consumers
Batch features: Spark cluster with 50 executors, 256GB RAM each

Fault Tolerance

Model Serving Fallbacks:

Primary model failure (5% error rate): Fallback to cached predictions with 1-hour TTL
Secondary model failure (0.5% error rate): Rule-based fallback with 95% accuracy
Cache miss: Pre-computed embeddings from feature store with 10ms latency

Stale Model Handling:

Model versioning with 30-day retention policy
Rolling updates with 5% traffic ramp-up per hour
Health checks every 30 seconds, automatic rollback on >10% error rate

Cold Start Handling Strategy

New Users:

Default to popular items with 80% confidence
Fallback to collaborative filtering with 50% confidence after 10 queries
User embedding initialized from content preferences

New Items:

Content-based similarity using TF-IDF features (10ms latency)
Fallback to item category embeddings with 75% confidence
Hybrid ranking combining content and popularity scores

Multi-region Serving Strategy

Primary Region (us-west-2):

300 serving nodes with 100 QPS each
200GB SSD storage for embeddings and cache
100ms p99 latency target

Secondary Region (eu-central-1):

150 serving nodes with 100 QPS each
100GB SSD storage for embeddings and cache
200ms p99 latency target (acceptable degradation)

Cross-region replication:

2-hour sync window for feature store updates
15-minute sync window for model weights
Geo-distributed load balancing with 20% traffic routing to secondary region

# Example of distributed embedding retrieval logic for cold start handling
class ColdStartHandler:
    def __init__(self):
        self.default_items = ["popular_item_1", "popular_item_2"]
        self.category_embeddings = {}
        
    def get_user_embeddings(self, user_id):
        if not self.is_known_user(user_id):
            # Cold start: return default embeddings
            return self.default_embeddings()
        return self.fetch_user_embedding(user_id)
    
    def get_item_embeddings(self, item_id):
        if not self.is_known_item(item_id):
            # Cold start: content-based fallback
            return self.content_based_embedding(item_id)
        return self.fetch_item_embedding(item_id)
    
    def default_embeddings(self):
        # Return popular item embeddings with confidence scores
        return {
            "items": self.default_items,
            "scores": [0.9, 0.8]
        }

11. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering / policy alignment

Query interpretation ambiguity: Should we prioritize exact keyword matching or semantic understanding for ambiguous queries? This affects both model training and user experience expectations.
Content filtering policy: What level of content filtering should be applied to search results? This requires alignment between ML and policy teams regarding safety thresholds and acceptable risk levels.
Cross-domain semantic consistency: Should we allow semantic matching across different content domains (e.g., news articles and product descriptions), or constrain to same-domain similarity?
User intent vs content relevance trade-off: Should we prioritize understanding user's explicit intent (e.g., "buy iPhone 15") or pure content similarity?

Key ML risks

Risk	Impact	Likelihood	Mitigation
Semantic drift	High - model performance degrades over time as content evolves	Medium - frequent content updates	Implement periodic retraining with drift detection
Query ambiguity misinterpretation	Medium - poor relevance for ambiguous queries	High - 30-40% of queries are ambiguous	Implement multi-sense embedding approach and query expansion
Cold start for new content	Medium - poor retrieval for newly added items	High - continuous content ingestion	Use hybrid approach with content metadata and embedding fallback
Cross-domain contamination	Medium - semantic similarity across unrelated domains	Medium - content mixing in training	Implement domain-aware training and filtering
Data quality degradation	High - poor quality training data leads to poor model performance	Medium - labeling errors in large datasets	Implement robust data validation and quality monitoring

Trade-offs made with justification

Trade-off	Justification
Accuracy vs latency	Prioritize sub-100ms response time over marginal accuracy gains (e.g., 2% improvement) to maintain user experience. Use model distillation and caching for latency optimization
Freshness vs cost	Accept 24-hour freshness window for embeddings to reduce compute costs. Use incremental updates with batch retraining schedule
Model size vs quality	Optimize for 100MB model size to support edge deployment and reduce serving costs, accepting ~5% quality degradation
Online vs batch training	Use batch training with online feedback loops to reduce operational complexity while maintaining model freshness
Precision vs recall	Prioritize recall for search (ensure no relevant content is missed) over precision, as users can filter results themselves

Alternatives considered and rejected

Alternative	Reason for rejection
Transformer-based dense retrieval	Too computationally expensive for real-time serving at scale; replaced with efficient two-tower architecture
Full LLM reranking	Not suitable for semantic search due to high latency and cost; replaced with lightweight reranker
Multi-modal embeddings	Not required for current use case and would increase complexity; focus on text-only semantic search
Real-time query expansion	Would add significant latency and complexity; use pre-computed query variants instead
Cross-encoder model	Higher accuracy but too slow for production use; replaced with bi-encoder + cross-encoder hybrid

Technical debt acknowledged

Limited query understanding capabilities: Current model treats queries as pure text without explicit intent recognition, which will require future architectural changes
No content filtering integration: Content safety and filtering logic is handled at the application layer rather than being embedded in model training
Hardcoded embedding dimensions: Model architecture assumes fixed embedding sizes, requiring future refactoring for dynamic dimensionality
Limited cross-domain handling: Current architecture assumes content homogeneity, requiring future extension for multi-domain support
No feedback loop integration: User interaction data is not yet integrated into model retraining pipeline, creating a gap between user behavior and model adaptation

12. Phased Implementation Roadmap

Phase 1: MVP (Weeks 1-6)

Objective: Establish baseline semantic search with rule-based fallback

Deliverables:

Rule-based search engine (TF-IDF + BM25) as fallback
Basic embedding model (e.g., Sentence-BERT) for document retrieval
Simple online serving pipeline with 100ms p99 latency target
Basic feature store (Redis-based) for user/item metadata

Dependencies:

Data pipeline ready with 10M documents
Basic feature engineering completed
Model training infrastructure available

Critical Path:

Data ingestion pipeline (Week 1-2)
Basic embedding model training (Week 2-3)
Simple serving stack (Week 3-4)
Rule-based fallback integration (Week 4-5)
Basic A/B testing infrastructure (Week 5-6)

Phase 2: Scale (Weeks 7-12)

Objective: Distributed serving and feature engineering at scale

Deliverables:

Distributed model serving (TorchServe + Ray)
Feature store with online/offline sync
Real-time feature computation pipeline
Multi-stage retrieval (candidate generation → scoring)
Production-grade monitoring and alerting

Dependencies:

Phase 1 completed
Distributed training infrastructure ready
Feature store with proper TTL management

Critical Path:

Feature store architecture (Week 7-8)
Distributed model serving (Week 8-9)
Multi-stage retrieval pipeline (Week 9-10)
Production monitoring stack (Week 10-11)
A/B testing infrastructure (Week 11-12)

Phase 3: Excellence (Weeks 13-18)

Objective: Advanced models and continuous learning

Deliverables:

Transformer-based ranking model (e.g., BERT-based)
Real-time feature updates and caching
Online learning pipeline for model retraining
Multi-modal embedding support (text + image)
Advanced evaluation and A/B testing

Dependencies:

Phase 2 completed
Continuous learning infrastructure ready
Multi-modal data pipeline established

Critical Path:

Advanced model training (Week 13-14)
Online learning pipeline (Week 14-15)
Multi-modal support (Week 15-16)
Advanced evaluation framework (Week 16-17)
Production rollout (Week 17-18)