Web Search Ranking System Design

Comprehensive AI/ML system design for search result ranking, covering machine learning models, feature engineering, and production deployment strategies.

PublishedAugust 26, 2025
Length8,010 words
Reading time41 min read

AI/ML System Design

A web search ranking system must efficiently process billions of queries while delivering highly relevant results through sophisticated machine learning models and real-time feature processing.

1. Problem Statement & ML Task Definition

Clarifying questions

What is the expected scale of queries and documents in our search system? (e.g., 10^9 queries/day, 10^8 documents)
What is the latency budget for search ranking? (e.g., 100ms p99 for web search)
Is this a batch or online ranking system? (e.g., real-time query processing vs. pre-computed rankings)
What is the primary business metric we're optimizing for? (e.g., CTR, revenue from ads, user engagement)
What is the expected data freshness requirement? (e.g., real-time updates vs. daily batch processing)
What is the acceptable trade-off between model accuracy and serving latency? (e.g., can we accept 5% accuracy drop for 50ms latency improvement?)
What is the expected query volume and distribution? (e.g., 50% long-tail queries, 20% high-volume queries)

Business use case and success criteria

The web search ranking system aims to improve user experience by presenting the most relevant and useful search results for each query. The system must balance multiple objectives:

User satisfaction: Higher click-through rates on top results (CTR)
Business revenue: Ad impression quality and monetization
Content diversity: Preventing echo chamber effects and promoting quality content
Query intent matching: Accurately interpreting user search intent

Success criteria include:

15% improvement in average CTR over baseline
20% increase in ad revenue per search session
10% reduction in user bounce rate on search results page
5% improvement in time-to-relevance (user finds desired content within top 3 results)

ML problem formulation

This is a ranking task with the following characteristics:

Task type: Learning to rank (LTR) - multi-class ranking problem where the model learns to order a set of documents for a given query
Justification: Users expect results ordered by relevance, not random or arbitrary order. The system must learn complex relationships between query-document features and user behavior signals

The ranking problem is further characterized as:

Multi-task: Handles both organic search results and sponsored content
Dynamic: Results must adapt to user context, time, and query evolution
Multi-objective: Balances relevance, diversity, freshness, and business goals

ML evaluation metrics

Offline evaluation metrics:

NDCG@K: Normalized Discounted Cumulative Gain for top-K ranking quality
Precision@K: Proportion of relevant results in top-K predictions
MAP@K: Mean Average Precision for ranked lists
MRR: Mean Reciprocal Rank for first relevant result position

Online evaluation metrics:

CTR: Click-through rate on search results (primary metric)
Engagement time: Average time spent on clicked results
Revenue per search: Ad impression value from search sessions
User retention: Follow-up searches after initial session

Scale and constraints

Back-of-envelope estimates:

Training data: 100B query-document interactions (50M unique queries × 2K docs/query)
Features: 10,000+ engineered features (user, item, context, interaction)
Embedding matrix: 10M documents × 384-dim embeddings = ~15GB memory footprint
Serving QPS: 10,000 QPS (500ms p99 latency target) → 50M requests/day

Latency budget breakdown:

Query processing: 10ms
Feature retrieval: 20ms
Model inference: 50ms
Result composition: 10ms
Total: 90ms (p99 target)

Non-ML baseline strategy

Rule-based fallback:

Default ranking based on document freshness, domain authority, and keyword match score
Fallback to TF-IDF or BM25 for queries with insufficient training data
Cold start handling: Use popularity-based ranking for new users/items

Cold start strategy:

New queries: Use query expansion with popular related queries
New documents: Apply content-based similarity to existing high-quality documents
New users: Use demographic and geographic matching with similar users

2. Data Strategy

Training Data Sources

The training data for a web search ranking system comes from multiple sources:

Explicit Feedback: Click-through rate (CTR) data, dwell time, scroll depth, and explicit user ratings from search result interactions
Implicit Signals: Query logs, session data, navigation patterns, and user behavior sequences (e.g., "back to search" or "try different query")
Third-Party Data: Content quality signals from web crawlers, domain authority scores, freshness indicators, and content freshness metrics
Search Logs: Query-URL pairs with user session information, query intent classification, and search result impressions
Human Annotation: Expert-labeled relevance judgments for query-result pairs (e.g., 0=irrelevant, 1=partially relevant, 2=highly relevant)

Labeling Pipeline Design

The labeling pipeline is designed to handle both high-quality human annotations and scalable weak supervision:

Human Annotation: For high-value query-result pairs, use expert annotators to label relevance scores (e.g., using Amazon Mechanical Turk or internal labeling teams)
Weak Supervision: Apply rule-based heuristics to label large-scale datasets:
- Query-URL matching scores (e.g., keyword overlap, title match)
- Content freshness thresholds (e.g., content published within last 30 days)
- Domain authority scores from external sources (e.g., Alexa rank, PageRank)
Programmatic Labeling: Use content analysis to extract features like:
- Text similarity between query and document (TF-IDF, BM25)
- Metadata quality indicators (e.g., presence of title, description, structured data)

Class Imbalance Handling Strategy

Search ranking systems suffer from extreme class imbalance where most queries have highly relevant results, but the majority of result-document pairs are labeled as "irrelevant" or "low relevance". To address this:

Sampling Strategy: Use stratified sampling to ensure representation of both relevant and irrelevant examples in training batches
Weighted Loss Function: Apply class weights inversely proportional to frequency (e.g., weight for "irrelevant" examples = 1 / (0.95 * total_samples))
Focal Loss: For multi-class relevance labeling, implement focal loss to focus training on hard-to-classify examples

Negative Sampling Approach

To improve model generalization and reduce training time:

Hard Negative Mining: Sample negative examples that are semantically similar to relevant ones but have low relevance scores (e.g., documents with high BM25 score but low CTR)
Query-Dependent Sampling: For each query, sample negative examples from the same domain or category to maintain semantic consistency
Negative-to-Positive Ratio: Use a 1:5 ratio for training, with the option to increase this ratio during early training phases

Data Freshness and Update Frequency Requirements

To maintain model relevance:

Real-time Signals: CTR, dwell time, and user feedback should be updated within 1 hour
Batch Updates: Content freshness, domain authority, and metadata should be updated daily or weekly
Query Logs: Historical query logs should be retained for at least 90 days to support temporal analysis and drift detection

Privacy Compliance

Anonymization: All user data should be stripped of PII (Personally Identifiable Information) before training, including IP addresses, user IDs, and session identifiers
Data Retention: Implement a 6-month data retention policy for search logs and user behavior data, in compliance with GDPR and CCPA
Consent Management: Ensure that user consent is collected for data usage in accordance with privacy regulations

Data Quality Validation and Monitoring

Data Validation: Implement automated checks for:
- Missing or malformed URLs
- Inconsistent relevance labels across annotators
- Duplicate query-result pairs
Data Drift Detection: Monitor for drift in:
- Query distribution (e.g., seasonal trends)
- Content freshness and quality metrics
- User behavior patterns (e.g., shift in dwell time)
Logging and Alerting: Log data quality metrics and alert on:
- Drop in label consistency (e.g., >10% variance in human annotation)
- Spike in malformed query-result pairs
- Change in distribution of relevance labels over time

# Example data validation function for training set
def validate_training_data(df):
    # Check for missing URLs or queries
    assert not df['query'].isnull().any(), "Missing query data"
    assert not df['url'].isnull().any(), "Missing URL data"
    
    # Check for valid relevance labels (0, 1, 2)
    assert df['relevance'].isin([0, 1, 2]).all(), "Invalid relevance labels"
    
    # Check for duplicate query-result pairs
    duplicates = df.duplicated(subset=['query', 'url'], keep=False)
    assert not duplicates.any(), "Duplicate query-result pairs found"
    
    return True

3. Feature Engineering

Feature Taxonomy

The web search ranking system requires a comprehensive feature set to understand user intent and document relevance:

User Features: Query history, user demographics, geographic location, device type, session duration, click patterns, dwell time
Item Features: Document title, URL structure, page content, metadata (publish date, domain authority), content embeddings, keyword density, freshness score
Context Features: Search query, timestamp, device type, geographic region, time of day, seasonal factors, trending topics
Interaction Features: Click-through rate, dwell time, bounce rate, scroll depth, query-document similarity scores, user-item affinity

Embedding Design

Entity Embeddings:

Query embedding (128-dim) using BERT-based model trained on query-document pairs
Document embedding (256-dim) from title + content using sentence-transformers
Domain embedding (64-dim) capturing topic-specific semantic patterns

Text Embeddings:

Query embeddings: 512-dim using transformer encoder with attention masking
Document content embeddings: 768-dim using RoBERTa model fine-tuned on web documents
Keyword embeddings: 128-dim using TF-IDF weighted word vectors

Graph Embeddings:

User-document interaction graph embeddings (128-dim) using Node2Vec
Query-document co-occurrence graph embeddings (64-dim) for semantic relationships

Real-time vs Batch Feature Computation

Feature Type	Computation Mode	Reasoning
Query embeddings	Real-time	Dynamic query understanding, must be fresh
Document content features	Batch	Static content, computed once per document update
User session features	Real-time	Session state changes frequently
Historical click patterns	Batch	Computed periodically, updated with new user data
Freshness scores	Real-time	Time-sensitive, must reflect current document age

Feature Store Design

Online Store:

Low-latency key-value store (Redis or DynamoDB)
Designed for sub-100ms response times
Supports real-time feature lookups during ranking

Offline Store:

Data warehouse (Snowflake or BigQuery)
Used for training data preparation and batch feature engineering
Supports complex aggregations and historical analysis

Feature Freshness and TTL

Feature	Source	Update Frequency	Store Type	TTL
Query embedding	Real-time API	Per-request	Online	0s
User click history	User logs	Hourly	Online	1d
Document content	Web crawl	Daily	Offline	30d
Domain authority	External API	Weekly	Offline	90d
Query popularity	Analytics	Daily	Online	7d

Cross Features and Feature Interactions

Cross features are crucial for capturing complex user-document relationships:

Query-document similarity: Cosine similarity between query and document embeddings
User-query relevance: User's historical performance with similar queries
Temporal query-document interaction: Time-weighted click patterns for query-document pairs
Geographic relevance: User location vs document geographic metadata

Feature Importance and Selection

Methodology:

SHAP values: For global feature importance analysis
Permutation importance: Using out-of-bag samples during training
Variance-based selection: Remove features with low variance across training data

Selection Strategy:

Top 200 most important features retained for model training
Features with SHAP importance < 0.01 excluded from training
Cross-feature interaction terms added selectively based on A/B test performance

class FeatureExtractor:
    def __init__(self):
        self.query_encoder = BertModel.from_pretrained('bert-base-uncased')
        self.doc_encoder = SentenceTransformer('all-MiniLM-L6-v2')
        
    def extract_query_features(self, query_text):
        """Extract real-time query features"""
        # Query embedding
        query_embedding = self.query_encoder(
            input_ids=self.tokenize(query_text),
            attention_mask=self.get_attention_mask(query_text)
        ).last_hidden_state.mean(dim=1)
        
        # Query length and keyword features
        query_length = len(query_text.split())
        keyword_density = self.calculate_keyword_density(query_text)
        
        return {
            'query_embedding': query_embedding,
            'query_length': query_length,
            'keyword_density': keyword_density
        }
    
    def extract_document_features(self, doc_content, doc_metadata):
        """Extract document features from static sources"""
        # Document embedding
        doc_embedding = self.doc_encoder.encode(doc_content)
        
        # Metadata features
        domain_authority = doc_metadata.get('domain_authority', 0)
        publish_date = doc_metadata.get('publish_date', None)
        content_length = len(doc_content.split())
        
        return {
            'doc_embedding': doc_embedding,
            'domain_authority': domain_authority,
            'content_length': content_length,
            'publish_age_days': self.calculate_age(publish_date)
        }
    
    def extract_interaction_features(self, user_id, query, doc_id):
        """Extract user-document interaction features"""
        # User session features
        user_session = self.get_user_session(user_id)
        
        # Historical click patterns
        historical_clicks = self.get_historical_clicks(user_id, query)
        
        # User-query-document affinity
        affinity_score = self.calculate_affinity(user_id, query, doc_id)
        
        return {
            'user_session_features': user_session,
            'historical_clicks': historical_clicks,
            'affinity_score': affinity_score
        }

4. Model Architecture

Model Selection with Justification

For a web search ranking system, I recommend a two-tower neural network architecture with multi-task learning capabilities. This approach offers the best balance of performance, interpretability, and scalability for search ranking:

Two-tower architecture: Separates user and item (document) representations, enabling efficient candidate scoring and retrieval
Multi-task learning: Simultaneously optimizes for relevance, diversity, and user engagement signals
Scalable: Can be extended to handle large-scale web search with proper distributed training

Alternatives considered and rejected:

Transformer-based models: While powerful, they're computationally expensive for real-time search ranking and may not scale well to web-scale
GBM models: Good for feature interactions but lack the scalability and real-time capabilities needed for web search
Logistic regression: Too simplistic to capture complex user-item interactions in search context

Architecture Diagram

graph TD
    A[User Query] --> B[User Tower]
    C[Document Content] --> D[Item Tower]
    B --> E[User Representation]
    D --> F[Document Representation]
    E --> G[Interaction Layer]
    F --> G
    G --> H[Ranking Score]
    G --> I[Click Prediction]
    G --> J[Engagement Score]

Two-Tower Architecture Design

The model uses two separate neural networks to encode user queries and document content, then computes a dot product similarity score:

import torch
import torch.nn as nn
import torch.nn.functional as F

class TwoTowerRanker(nn.Module):
    def __init__(self, 
                 query_vocab_size=50000,
                 doc_vocab_size=100000,
                 embedding_dim=384,
                 hidden_dim=256,
                 num_tasks=3):
        super().__init__()
        
        # Embedding layers
        self.query_embedding = nn.Embedding(query_vocab_size, embedding_dim)
        self.doc_embedding = nn.Embedding(doc_vocab_size, embedding_dim)
        
        # User tower (query encoding)
        self.user_encoder = nn.Sequential(
            nn.Linear(embedding_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim // 2)
        )
        
        # Document tower (content encoding)
        self.doc_encoder = nn.Sequential(
            nn.Linear(embedding_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim // 2)
        )
        
        # Multi-task head
        self.task_heads = nn.ModuleList([
            nn.Linear(hidden_dim // 2, 1) for _ in range(num_tasks)
        ])
        
        self.dropout = nn.Dropout(0.2)
        
    def forward(self, query_ids, doc_ids):
        # Embedding lookup
        query_emb = self.query_embedding(query_ids)  # [B, seq_len, emb_dim]
        doc_emb = self.doc_embedding(doc_ids)       # [B, seq_len, emb_dim]
        
        # Average pooling for sequence embeddings
        query_emb = query_emb.mean(dim=1)  # [B, emb_dim]
        doc_emb = doc_emb.mean(dim=1)      # [B, emb_dim]
        
        # Encode towers
        user_repr = self.user_encoder(query_emb)  # [B, hidden_dim/2]
        doc_repr = self.doc_encoder(doc_emb)      # [B, hidden_dim/2]
        
        # Interaction layer
        interaction = torch.mul(user_repr, doc_repr)  # [B, hidden_dim/2]
        
        # Multi-task predictions
        task_outputs = []
        for head in self.task_heads:
            output = head(interaction)
            task_outputs.append(output)
            
        return task_outputs  # [B, 1], [B, 1], [B, 1]
    
    def score(self, query_ids, doc_ids):
        """Compute ranking score for a query-document pair"""
        task_outputs = self.forward(query_ids, doc_ids)
        return task_outputs[0].squeeze(-1)  # [B]

# Example usage for scoring
def compute_ranking_score(model, query_tokens, doc_tokens):
    model.eval()
    with torch.no_grad():
        scores = model.score(query_tokens, doc_tokens)
        return scores.cpu().numpy()

# Loss function for multi-task training
def compute_loss(task_outputs, labels):
    """Compute weighted loss across multiple tasks"""
    losses = []
    weights = [0.6, 0.2, 0.2]  # Relevance, Click, Engagement
    
    for i, (output, label) in enumerate(zip(task_outputs, labels)):
        if i == 0:  # Relevance task - binary classification
            loss = F.binary_cross_entropy_with_logits(output, label)
        else:  # Other tasks - regression or binary classification
            loss = F.mse_loss(output, label) if i > 0 else F.binary_cross_entropy_with_logits(output, label)
        losses.append(loss * weights[i])
    
    return sum(losses)

Pre-trained Model Strategy

For foundation model integration:

Query encoder: Fine-tune a pre-trained BERT or T5 model on search query understanding
Document encoder: Use a passage transformer trained on web document collections
Cross-encoder: For fine-grained relevance scoring, use a dual-encoder with attention fusion

Model Complexity vs Latency Trade-off

Model	Parameters	P99 Latency (ms)	Memory Usage	Use Case
Two-Tower Small	12M	5	256MB	Production
Two-Tower Medium	48M	12	512MB	Production
Two-Tower Large	192M	30	2GB	Research

Offline Evaluation Baseline

Baseline model: Logistic Regression with TF-IDF features Expected performance:

AUC-ROC: ~0.75 (baseline)
NDCG@10: ~0.45 (baseline)

Proposed model: Two-Tower Neural Network Target performance:

AUC-ROC: ~0.85+ (improvement)
NDCG@10: ~0.65+ (improvement)

Core Model Forward Pass Logic

The model's core functionality is implemented in the forward method above, which:

Takes query and document token IDs as input
Embeds tokens using learned embeddings
Encodes user and document representations through neural towers
Computes interaction between representations using element-wise multiplication
Generates multi-task outputs for relevance, click probability, and engagement

This design enables efficient real-time scoring while maintaining the flexibility to optimize for multiple business objectives simultaneously. The model can be easily extended with additional features like user history, document metadata, or contextual signals through the multi-task architecture.

5. Serving & Inference Architecture

System Diagram

graph TD
    A[User Query] --> B[Query Processing & Normalization]
    B --> C[Feature Store Lookup]
    C --> D[Embedding Cache]
    D --> E[Model Serving Layer]
    E --> F[Scoring & Ranking]
    F --> G[Re-ranking & Final Results]
    G --> H[Response to Client]
    
    subgraph "Online Serving Infrastructure"
        E --> I[TorchServe/ONNX Runtime]
        I --> J[Model Server]
        J --> K[GPU/TPU Cluster]
    end
    
    subgraph "Caching Layer"
        D --> L[Embedding Cache]
        G --> M[Prediction Cache]
    end

Online vs Batch Serving Decision

For web search ranking, we require real-time online serving with sub-100ms latency SLA. Batch serving is unsuitable for search ranking because:

User queries are highly variable and unpredictable
Search results must be personalized in real-time
Immediate feedback loop required for relevance optimization
Interactive user experience demands millisecond response times

Two-Stage Pipeline Design

Stage 1: Candidate Retrieval (p50: 20ms, p99: 50ms)
- BM25 + embedding-based retrieval (e.g., FAISS)
- Top-K candidates (e.g., 1000 items)

Stage 2: Scoring & Re-ranking (p50: 15ms, p99: 40ms)
- ML model scoring (e.g., BERT-based ranking)
- Final re-ranking with business rules

Serving Infrastructure

We use TorchServe for serving our ranking models with the following justification:

Production-grade: Built for enterprise ML deployment
Multi-model support: Can handle multiple ranking models (e.g., for different search domains)
GPU acceleration: Native support for CUDA tensors
REST API: Easy integration with existing web services
Model versioning: Built-in model management capabilities

Latency Budget Breakdown

Stage	p50 Latency	p99 Latency	Budget	Notes
Query Processing	5ms	10ms	15ms	Tokenization, normalization
Feature Retrieval	10ms	25ms	35ms	From feature store
Model Inference	15ms	40ms	55ms	BERT-based ranking
Re-ranking	5ms	15ms	70ms	Business logic, filtering
Response	5ms	10ms	80ms	Serialization, caching
Total	40ms	100ms	80ms	Within SLA

Model Compression Strategy

We implement a multi-layer compression approach:

Quantization: INT8 quantization reduces model size by 4x with <2% accuracy loss
Pruning: Structured pruning removes 30% of model parameters while maintaining performance
Distillation: Student model trained to mimic teacher model with 2x faster inference

Caching Strategy

Prediction Cache (TTL: 30min)
- Stores top-K results for recent queries
- Reduces redundant computation for similar queries
- Eviction policy: LRU with size limit

Embedding Cache (TTL: 24h)
- Caches user/item embeddings for frequent access
- Reduces repeated embedding lookups
- Size: ~50GB for 1M users with 256-dim embeddings

Auto-scaling and Capacity Planning

Horizontal Scaling Strategy:

Target QPS: 10,000 queries/sec
Per-node capacity: 500 QPS (with 8x GPU instances)
Auto-scaling target: 20 nodes minimum, up to 100 nodes
Scaling trigger: CPU utilization > 80% or latency > 80ms

Capacity Planning Math:

Required nodes = (10,000 QPS * 0.08s) / (500 QPS/node)
Required nodes = 1600/500 = 3.2 → 4 nodes minimum

Fault Tolerance:

Model fallback to pre-trained baseline when primary model fails
Cache warming with warm-up queries during deployment
Health checks every 5 seconds with automatic failover

Performance Optimization Techniques

Batching: Process queries in batches of 32-64 for GPU utilization
Prefetching: Preload embeddings and features before model inference
Async processing: Non-blocking feature lookups during model computation
Model parallelism: Split large models across multiple GPUs for memory efficiency

Example Serving Flow

class SearchRankingService:
    def __init__(self, model_path, feature_store, cache_ttl=30*60):
        self.model = torch.load(model_path)
        self.feature_store = feature_store
        self.cache = {}
        self.cache_ttl = cache_ttl
        
    def rank_results(self, query, candidates):
        # 1. Query preprocessing
        processed_query = self._preprocess_query(query)
        
        # 2. Feature retrieval
        user_features = self.feature_store.get_user_features()
        query_features = self.feature_store.get_query_features(processed_query)
        
        # 3. Model inference
        scores = self._compute_scores(query_features, user_features, candidates)
        
        # 4. Re-ranking with business rules
        final_results = self._re_rank(scores, candidates)
        
        return final_results
    
    def _compute_scores(self, query_emb, user_emb, candidates):
        # BERT-based scoring logic
        scores = []
        for candidate in candidates:
            item_emb = self.feature_store.get_item_embedding(candidate.id)
            score = self.model.score(query_emb, user_emb, item_emb)
            scores.append((candidate, score))
        return sorted(scores, key=lambda x: x[1], reverse=True)

6. Evaluation Framework

Offline Evaluation Methodology

For offline evaluation, we use a temporal split approach to maintain the chronological order of queries and their relevance judgments. This prevents data leakage by ensuring that models trained on historical data are evaluated on future data points, simulating real-world deployment conditions.

We partition our dataset into three temporal segments:

Training: 60% of data (oldest)
Validation: 20% of data (middle)
Test: 20% of data (most recent)

This approach is preferred over random splits because it:

Preserves temporal dependencies in search behavior
Simulates realistic model deployment conditions
Enables proper evaluation of model drift over time

Slice-Based Evaluation

We evaluate model performance across critical user slices to ensure robustness and fairness:

Slice	Description	Rationale
User Demographics	Age groups, geographic regions	Ensures equitable performance across user segments
Query Types	Short-form vs long-form, informational vs navigational	Captures performance differences in query complexity
Device Types	Mobile, desktop, tablet	Validates cross-platform consistency
Time of Day	Morning vs evening usage patterns	Identifies temporal performance variations

We define slices based on business impact and ensure each slice contains sufficient samples for statistical significance (minimum 1000 queries per slice).

A/B Test Design

For online evaluation, we implement a randomized controlled experiment with the following specifications:

Randomization Unit: Individual search queries (not user sessions) to avoid query-level correlation bias.

Sample Size Calculation:

Target: 5% relative improvement in CTR
Power: 80%
Significance level: 5%
Baseline CTR: 2.5%
Estimated sample size: ~1.2 million queries per variant

Test Duration: 30 days to capture seasonal and temporal variations.

Primary Metric: Click-Through Rate (CTR) with 95% confidence interval.

Secondary Metrics:

Average position of top result (lower is better)
Session duration (higher is better)
Revenue per session (if applicable)

Guardrail Metrics and Blocking Conditions

To ensure system safety, we implement the following guardrails:

Metric	Threshold	Action
NDCG@10	Drop below 0.85 (20% from baseline)	Immediate A/B test pause
Query Volume	Drop below 90% of baseline	Alert and investigation
Error Rate	Exceed 1%	Immediate rollback
Latency P99	Exceed 500ms	Alert and capacity review

Shadow Mode Rollout Plan

Before full A/B testing, we deploy shadow mode for 7 days:

Traffic Split: 5% of live traffic routed to new model
Logging Only: All predictions logged for offline comparison
Performance Monitoring: Track all guardrail metrics in real-time
Feedback Loop: Collect user feedback from shadow queries

Long-Term Metric Measurement

We establish long-term tracking for delayed impact evaluation:

Metric	Time Horizon	Purpose
Delayed Conversion	7-30 days post-search	Measure business impact beyond immediate clicks
Engagement Decay	1-30 days	Track session retention and revisit behavior
User Satisfaction Score	30-day rolling average	Monitor long-term user experience quality
Revenue Impact	30-day cumulative	Evaluate monetization effectiveness

Long-term metrics are tracked using cohort analysis to understand user behavior patterns over time and identify any degradation in user experience or business outcomes.

# Example evaluation function for offline metric calculation
def calculate_ndcg_at_k(predictions, labels, k=10):
    """
    Calculate NDCG@K for ranked search results.
    
    Args:
        predictions: List of predicted relevance scores
        labels: List of true relevance labels (0, 1, 2, ...)
        k: Top-k results to consider
    
    Returns:
        ndcg_score: Normalized Discounted Cumulative Gain
    """
    # Calculate DCG@K
    dcg = 0.0
    for i in range(min(k, len(predictions))):
        rel = labels[i]
        dcg += (2 ** rel - 1) / math.log2(i + 2)
    
    # Calculate IDCG@K (optimal ranking)
    ideal_labels = sorted(labels, reverse=True)
    idcg = 0.0
    for i in range(min(k, len(ideal_labels))):
        rel = ideal_labels[i]
        idcg += (2 ** rel - 1) / math.log2(i + 2)
    
    # Return normalized DCG
    return dcg / idcg if idcg > 0 else 0.0

# Example A/B test significance calculation
def ab_test_significance(control_ctr, treatment_ctr, control_count, treatment_count):
    """
    Perform significance test for CTR A/B test.
    
    Returns:
        p_value: Statistical significance of difference
    """
    from scipy.stats import proportions_ztest
    
    # Convert to proportions and counts
    control_success = control_ctr * control_count
    treatment_success = treatment_ctr * treatment_count
    
    # Perform z-test for proportions
    stat, p_value = proportions_ztest(
        [control_success, treatment_success],
        [control_count, treatment_count],
        alternative='two-sided'
    )
    
    return p_value

7. Training Pipeline

Training Data Pipeline

The training pipeline begins with data ingestion from multiple sources:

Web crawl logs (structured click-through data, page metadata)
User interaction logs (search queries, click patterns, dwell time)
Third-party datasets (domain expertise, entity knowledge graphs)
Synthetic data generation for edge cases and rare query types

Data preprocessing includes:

Query normalization (lowercase, remove special characters)
Relevance label generation using click-through rate thresholds (CTR > 0.05 = relevant)
Negative sampling with 1:10 ratio of negative to positive examples
Feature extraction from document metadata (domain, page authority, freshness)
Data shuffling and batching for distributed training

import torch
from torch.utils.data import Dataset, DataLoader
import pandas as pd

class SearchRankingDataset(Dataset):
    def __init__(self, data_path, max_seq_length=512):
        self.data = pd.read_csv(data_path)
        self.max_seq_length = max_seq_length
        
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, idx):
        row = self.data.iloc[idx]
        query = row['query']
        document = row['document_text']
        label = row['relevance_score']  # 0 or 1
        
        # Tokenize query and document
        query_tokens = self.tokenize(query)
        doc_tokens = self.tokenize(document)
        
        # Truncate or pad to max sequence length
        query_tokens = query_tokens[:self.max_seq_length]
        doc_tokens = doc_tokens[:self.max_seq_length]
        
        return {
            'query': torch.tensor(query_tokens, dtype=torch.long),
            'document': torch.tensor(doc_tokens, dtype=torch.long),
            'label': torch.tensor(label, dtype=torch.float32)
        }

Distributed Training Strategy

For large-scale training, we use data parallelism with model parallelism for the transformer backbone:

Data Parallelism: Split training data across multiple GPUs (e.g., 8 GPUs per node)
Model Parallelism: Split transformer layers across multiple GPUs when model size exceeds GPU memory
Pipeline Parallelism: For very large models, pipeline stages can be distributed across nodes

Training infrastructure:

16 nodes with 8x A100 GPUs each (128 total GPUs)
Estimated training time: 48 hours for 50M samples
Batch size: 128 per GPU (1024 total batch size)
Learning rate: 5e-5 with cosine decay schedule

Hyperparameter Tuning Approach

We use Bayesian optimization for hyperparameter tuning:

Key hyperparameters:

Learning rate (5e-6 to 5e-4)
Batch size (32, 64, 128, 256)
Dropout rate (0.1 to 0.5)
Number of transformer layers (6, 8, 12)
Hidden dimension (256, 512, 768)

from skopt import gp_minimize
from skopt.space import Real, Integer

def objective(params):
    lr, batch_size, dropout, layers, hidden_dim = params
    
    # Initialize model with hyperparameters
    model = SearchRanker(
        num_layers=layers,
        hidden_dim=hidden_dim,
        dropout=dropout
    )
    
    # Train model and return validation AUC-ROC
    val_auc = train_and_evaluate(model, lr, batch_size)
    
    # Return negative AUC (since skopt minimizes)
    return -val_auc

# Bayesian optimization search space
search_space = [
    Real(1e-6, 1e-3, name='learning_rate'),
    Integer(32, 256, name='batch_size'),
    Real(0.1, 0.5, name='dropout_rate'),
    Integer(6, 12, name='num_layers'),
    Integer(256, 768, name='hidden_dim')
]

# Run optimization
best_params = gp_minimize(objective, search_space, n_calls=50, random_state=42)

Experiment Tracking and Reproducibility

We use MLflow for experiment tracking:

import mlflow
from mlflow.tracking import MlflowClient

# Log experiment parameters and metrics
with mlflow.start_run():
    mlflow.log_param("learning_rate", 5e-5)
    mlflow.log_param("batch_size", 128)
    mlflow.log_param("num_epochs", 10)
    
    # Log model artifacts
    mlflow.pytorch.log_model(model, "model")
    
    # Log metrics
    mlflow.log_metric("train_loss", train_loss)
    mlflow.log_metric("val_auc", val_auc)
    
    # Log feature importance
    mlflow.log_dict(feature_importance, "feature_importance.json")

Model Versioning and Registry Strategy

Versioning: Semantic versioning (v1.0.0, v1.1.0) based on major feature changes
Registry: MLflow Model Registry for model lifecycle management
Tagging: Tag models with performance metrics, training date, and dataset version

Continual / Online Learning Design

For online learning capability:

Incremental training: Process new queries in batches of 10K
Drift detection: Monitor query distribution drift using KL divergence
Replay buffer: Store recent training samples for periodic retraining
Online adaptation: Use small batches to adapt model weights for new query patterns

class OnlineLearningAdapter:
    def __init__(self, model, buffer_size=10000):
        self.model = model
        self.replay_buffer = []
        self.buffer_size = buffer_size
        
    def update_with_new_data(self, new_samples, lr=1e-6):
        """Update model with new samples using online learning"""
        # Add to replay buffer
        self.replay_buffer.extend(new_samples)
        if len(self.replay_buffer) > self.buffer_size:
            self.replay_buffer = self.replay_buffer[-self.buffer_size:]
            
        # Train on recent samples
        for sample in self.replay_buffer[-1000:]:  # Last 1000 samples
            self.model.train_step(sample, lr=lr)
            
    def detect_drift(self, new_query_distribution):
        """Detect distribution drift in query patterns"""
        # Simple KL divergence check
        kl_div = calculate_kl_divergence(
            self.current_distribution, 
            new_query_distribution
        )
        
        if kl_div > 0.1:  # Threshold for retraining trigger
            return True
        return False

Data Pipeline Monitoring and Validation

Data quality checks: Missing values, label distribution validation
Training data freshness: Daily updates with 24-hour lag for quality assurance
Cross-validation: 5-fold temporal cross-validation to ensure model generalization
Sample validation: Random sampling of training data for manual quality review

8. Monitoring & Observability

Data Drift Detection

Data drift detection is critical for maintaining model performance in a web search ranking system. We implement drift detection using Population Stability Index (PSI) for categorical features and Kolmogorov-Smirnov test for continuous features.

import numpy as np
from scipy import stats
from sklearn.metrics import pairwise_distances

class DriftDetector:
    def __init__(self, feature_names):
        self.feature_names = feature_names
        self.baseline_stats = {}
        
    def calculate_psi(self, current_data, baseline_data, bins=10):
        """Calculate PSI for categorical features"""
        psi_values = {}
        for feature in self.feature_names:
            if feature in baseline_data.columns:
                # Create bins from baseline data
                _, bin_edges = np.histogram(baseline_data[feature], bins=bins)
                
                # Count occurrences in both datasets
                baseline_counts, _ = np.histogram(baseline_data[feature], bins=bin_edges)
                current_counts, _ = np.histogram(current_data[feature], bins=bin_edges)
                
                # Avoid zero counts
                baseline_counts = np.where(baseline_counts == 0, 1e-6, baseline_counts)
                current_counts = np.where(current_counts == 0, 1e-6, current_counts)
                
                psi = np.sum((current_counts / len(current_data) - 
                             baseline_counts / len(baseline_data)) * 
                            np.log(current_counts / baseline_data))
                psi_values[feature] = psi
        return psi_values
    
    def detect_drift(self, current_data, baseline_data):
        """Detect drift in feature distributions"""
        drift_results = {}
        
        # Categorical features PSI
        categorical_features = [f for f in self.feature_names if current_data[f].dtype == 'object']
        drift_results['psi'] = self.calculate_psi(current_data, baseline_data)
        
        # Continuous features KS test
        continuous_features = [f for f in self.feature_names if f not in categorical_features]
        drift_results['ks'] = {}
        
        for feature in continuous_features:
            if feature in baseline_data.columns:
                ks_stat, p_value = stats.ks_2samp(
                    current_data[feature], baseline_data[feature])
                drift_results['ks'][feature] = {
                    'statistic': ks_stat,
                    'p_value': p_value
                }
        
        return drift_results

Prediction Drift and Label Drift Monitoring

Prediction drift monitors changes in model outputs over time, while label drift monitors shifts in the underlying distribution of target labels.

class PredictionDriftMonitor:
    def __init__(self, threshold=0.1):
        self.threshold = threshold
        self.reference_predictions = []
        
    def detect_prediction_drift(self, current_predictions):
        """Detect drift in model predictions"""
        if not self.reference_predictions:
            self.reference_predictions = current_predictions
            return False
            
        # Use Jensen-Shannon divergence for prediction distribution change
        js_div = self.jensen_shannon_divergence(
            self.reference_predictions, current_predictions)
        
        return js_div > self.threshold
    
    def jensen_shannon_divergence(self, p, q):
        """Calculate Jensen-Shannon divergence between two distributions"""
        p = np.array(p)
        q = np.array(q)
        
        # Normalize
        p = p / np.sum(p)
        q = q / np.sum(q)
        
        # Calculate midpoint
        m = (p + q) / 2
        
        # Calculate divergences
        divergence_p = stats.entropy(p, m)
        divergence_q = stats.entropy(q, m)
        
        return (divergence_p + divergence_q) / 2

class LabelDriftMonitor:
    def __init__(self, threshold=0.05):
        self.threshold = threshold
        self.label_distribution_history = []
        
    def detect_label_drift(self, current_labels):
        """Detect drift in label distribution"""
        if len(self.label_distribution_history) < 2:
            self.label_distribution_history.append(current_labels)
            return False
            
        # Compare recent label distributions
        recent_dist = self._get_label_distribution(current_labels)
        past_dist = self._get_label_distribution(
            self.label_distribution_history[-1])
            
        # Chi-square test for distribution change
        chi2_stat, p_value = stats.chisquare(
            recent_dist, past_dist)
            
        drift_detected = p_value < self.threshold
        self.label_distribution_history.append(current_labels)
        
        return drift_detected
    
    def _get_label_distribution(self, labels):
        """Get normalized label distribution"""
        unique, counts = np.unique(labels, return_counts=True)
        return counts / len(labels)

Model Performance Degradation Alerting

Model performance degradation is monitored through key metrics with established thresholds and escalation protocols.

class PerformanceMonitor:
    def __init__(self, metrics_config):
        self.metrics_config = metrics_config
        self.alert_thresholds = {
            'ndcg@10': {'warning': 0.85, 'critical': 0.80},
            'precision@10': {'warning': 0.70, 'critical': 0.65},
            'recall@10': {'warning': 0.60, 'critical': 0.55}
        }
        
    def check_performance_degradation(self, current_metrics):
        """Check if performance has degraded beyond thresholds"""
        alerts = []
        
        for metric_name, current_value in current_metrics.items():
            if metric_name in self.alert_thresholds:
                thresholds = self.alert_thresholds[metric_name]
                
                if current_value < thresholds['critical']:
                    alerts.append({
                        'type': 'CRITICAL',
                        'metric': metric_name,
                        'value': current_value,
                        'threshold': thresholds['critical']
                    })
                elif current_value < thresholds['warning']:
                    alerts.append({
                        'type': 'WARNING',
                        'metric': metric_name,
                        'value': current_value,
                        'threshold': thresholds['warning']
                    })
                    
        return alerts
    
    def setup_alerting(self, alerts):
        """Setup alerting system for performance degradation"""
        for alert in alerts:
            if alert['type'] == 'CRITICAL':
                # Send to pagerduty or Slack
                self.send_alert(
                    "CRITICAL: Performance degradation detected",
                    f"Metric {alert['metric']} dropped to {alert['value']}"
                )
            elif alert['type'] == 'WARNING':
                # Send to Slack or email
                self.send_alert(
                    "WARNING: Performance degradation detected",
                    f"Metric {alert['metric']} dropped to {alert['value']}"
                )
    
    def send_alert(self, subject, message):
        """Send alert to monitoring system"""
        # Implementation would integrate with Slack, PagerDuty, or email
        print(f"ALERT - {subject}: {message}")

Retraining Trigger Strategy

The system implements a multi-trigger retraining strategy combining scheduled, drift-based, and performance-based triggers.

class RetrainingTrigger:
    def __init__(self, drift_threshold=0.15, performance_threshold=0.05, 
                 time_window_days=30):
        self.drift_threshold = drift_threshold
        self.performance_threshold = performance_threshold
        self.time_window_days = time_window_days
        
    def should_retrain(self, drift_results, performance_metrics, 
                       time_since_last_retrain):
        """Determine if model should be retrained"""
        
        # Check drift triggers
        drift_triggers = []
        for feature, psi in drift_results.get('psi', {}).items():
            if psi > self.drift_threshold:
                drift_triggers.append(f"Feature {feature} PSI: {psi}")
        
        # Check performance triggers
        performance_triggers = []
        for metric, value in performance_metrics.items():
            if value < (1 - self.performance_threshold):
                performance_triggers.append(
                    f"Metric {metric} dropped to {value}"
                )
        
        # Check time-based trigger
        time_trigger = time_since_last_retrain > self.time_window_days
        
        # Decision logic
        if drift_triggers or performance_triggers or time_trigger:
            return {
                'should_retrain': True,
                'reasons': drift_triggers + performance_triggers + 
                          ['Time-based trigger'] if time_trigger else [],
                'severity': self._calculate_severity(
                    drift_triggers, performance_triggers)
            }
        
        return {'should_retrain': False, 'reasons': [], 'severity': 'LOW'}
    
    def _calculate_severity(self, drift_triggers, performance_triggers):
        """Calculate severity level for retraining trigger"""
        total_triggers = len(drift_triggers) + len(performance_triggers)
        if total_triggers >= 3:
            return 'HIGH'
        elif total_triggers >= 1:
            return 'MEDIUM'
        else:
            return 'LOW'

Logging Design

Comprehensive logging is implemented for prediction tracking, feature monitoring, and feedback collection.

import json
from datetime import datetime

class PredictionLogger:
    def __init__(self, log_file_path):
        self.log_file_path = log_file_path
        
    def log_prediction(self, query_id, user_id, items_ranked, 
                      model_scores, features, timestamp=None):
        """Log prediction data for monitoring and analysis"""
        if timestamp is None:
            timestamp = datetime.now().isoformat()
            
        log_entry = {
            'timestamp': timestamp,
            'query_id': query_id,
            'user_id': user_id,
            'items_ranked': items_ranked,
            'model_scores': model_scores,
            'features': features,
            'prediction_id': f"{query_id}_{timestamp}"
        }
        
        # Write to log file
        with open(self.log_file_path, 'a') as f:
            f.write(json.dumps(log_entry) + '\n')
            
    def log_feedback(self, query_id, clicked_items, 
                    relevance_labels, timestamp=None):
        """Log user feedback for model improvement"""
        if timestamp is None:
            timestamp = datetime.now().isoformat()
            
        feedback_entry = {
            'timestamp': timestamp,
            'query_id': query_id,
            'clicked_items': clicked_items,
            'relevance_labels': relevance_labels,
            'feedback_id': f"{query_id}_{timestamp}"
        }
        
        # Write to feedback log
        with open(f"{self.log_file_path}_feedback", 'a') as f:
            f.write(json.dumps(feedback_entry) + '\n')

class FeatureLogger:
    def __init__(self, feature_store):
        self.feature_store = feature_store
        
    def log_feature_updates(self, feature_updates):
        """Log feature updates for drift analysis"""
        # Implementation would store feature update logs
        pass

Feedback Loop for Continuous Improvement

The feedback loop enables continuous model improvement through user interaction data and retraining cycles.

class FeedbackLoop:
    def __init__(self, model, feature_store):
        self.model = model
        self.feature_store = feature_store
        self.feedback_collector = []
        
    def collect_feedback(self, query_id, clicked_items, 
                        relevance_labels, timestamp):
        """Collect and store user feedback"""
        feedback_data = {
            'query_id': query_id,
            'clicked_items': clicked_items,
            'relevance_labels': relevance_labels,
            'timestamp': timestamp
        }
        
        self.feedback_collector.append(feedback_data)
        
        # Store for retraining
        if len(self.feedback_collector) >= 1000:  # Batch size threshold
            self._process_feedback_batch()
            
    def _process_feedback_batch(self):
        """Process feedback batch for model retraining"""
        # Prepare training data from feedback
        training_data = self._prepare_training_data()
        
        # Retrain model with new data
        self.model.retrain(training_data)
        
        # Clear feedback buffer
        self.feedback_collector.clear()
        
    def _prepare_training_data(self):
        """Prepare training data from feedback"""
        # Implementation would convert feedback to training samples
        return {
            'features': [],  # Feature vectors from feedback
            'labels': []    # Relevance labels from feedback
        }

Key Monitoring Metrics

The system monitors the following key metrics:

Metric	Description	Alert Thresholds
NDCG@10	Normalized Discounted Cumulative Gain	Warning: 0.85, Critical: 0.80
Precision@10	Precision at top 10 results	Warning: 0.70, Critical: 0.65
Recall@10	Recall at top 10 results	Warning: 0.60, Critical: 0.55
PSI	Population Stability Index for features	Warning: 0.10, Critical: 0.15
Model Drift	Jensen-Shannon divergence in predictions	Warning: 0.05, Critical: 0.10

Alerting and Escalation Protocol

WARNING Level: Performance drops below warning thresholds - Slack notification
CRITICAL Level: Performance drops below critical thresholds or significant drift detected - PagerDuty alert + team notification
RETRAINING Required: Drift or performance triggers - Automatic retraining pipeline initiation

Observability Dashboard Components

The observability dashboard includes:

Real-time performance metrics with trend visualization
Drift detection alerts with feature-specific analysis
Retraining status and schedule tracking
User feedback distribution and relevance patterns
Model version tracking and performance comparison

This monitoring framework ensures robust system operation, early detection of performance degradation, and continuous model improvement through feedback-driven cycles.

9. Responsible AI & Safety

Bias Analysis

Training data audit: Conduct comprehensive audit of web search query logs and document collections to identify potential biases in representation (e.g., gender, geographic, cultural) that could lead to unfair ranking of results.
Slice-based fairness evaluation: Evaluate model performance across key user segments (by region, age group, device type) to detect disparities in ranking quality and ensure equal opportunity for diverse content.

Fairness Metrics and Constraints

Primary fairness metric: Demographic parity in top-K result distribution across user demographics (region, language, device type) to prevent systematic underrepresentation of content from certain groups.
Decision rationale: Demographic parity is preferred over equalized odds for web search ranking as it addresses the fundamental fairness concern of equitable result distribution rather than focusing on conditional prediction accuracy.

Explainability Requirements and Approach

SHAP-based explainability: Implement SHAP feature importance for top-K ranking decisions to provide transparency on which query or document features drive result ordering.
Attention visualization: For transformer-based ranking models, surface attention weights to show which document sections or query terms receive highest attention during ranking.
Stakeholder alignment: Business team requires explainability for compliance and trust; technical team needs model interpretability for debugging.

For LLM Systems: Safety Guardrails

Content filtering: Implement content safety filters to block harmful, misleading, or inappropriate search results (e.g., misinformation, hate speech, illegal content).
Output validation: Validate search result summaries and snippets for factual accuracy using external fact-checking APIs.
Rate limiting: Enforce rate limits on search queries to prevent abuse and ensure system stability.
Red teaming: Regular red-teaming exercises with diverse content samples to identify edge cases and safety gaps in search ranking.

Model Card Documentation Plan

Comprehensive model card: Document training data sources, preprocessing steps, model architecture, performance metrics, fairness analysis, and known limitations.
Public transparency: Publish model cards for major search ranking models to provide external stakeholders with visibility into system capabilities and constraints.

Incident Response Plan for Model Failures or Harmful Outputs

Failure detection: Monitor for sudden drops in ranking quality, increase in flagged content, or user complaints indicating model drift or bias.
Escalation protocol:
- P0: Immediate model rollback and fallback to rule-based ranking for all queries
- P1: Isolate affected user segments and apply targeted model fixes
- P2: Log incident for root cause analysis and model retraining
Communication plan: Notify internal stakeholders within 1 hour of failure detection, and external users if system-wide impact exceeds threshold.

Example Implementation: Content Safety Filter

import re

class ContentSafetyFilter:
    def __init__(self):
        self.harmful_patterns = [
            r"fake.*news",
            r"illegal.*activity",
            r"hate.*speech",
            r"misinformation"
        ]
        
    def is_safe(self, document_text):
        """Check if document text contains harmful content"""
        for pattern in self.harmful_patterns:
            if re.search(pattern, document_text.lower()):
                return False
        return True

# Usage in ranking pipeline
def filter_unsafe_results(results, safety_filter):
    safe_results = []
    for result in results:
        if safety_filter.is_safe(result['content']):
            safe_results.append(result)
    return safe_results

10. Scalability & Reliability

Bottleneck Analysis per Pipeline Stage

Training Pipeline:

Data ingestion: 100GB/day → 10TB/month, 500GB/s peak throughput required
Feature computation: 200M users × 100 features = 20B features per training run
Model training: 50M samples, 100GB model size → 200GB GPU memory requirement for training
Back-of-envelope: 100k QPS serving requires 5000+ model replicas with 20GB each = 10TB RAM

Serving Pipeline:

Model inference: 50ms p99 latency target → 20k QPS capacity with 10ms per request
Feature store: 50M users × 100 features = 20B features stored, 100GB memory footprint
Cache: 1M concurrent queries × 500 bytes = 500MB cache per instance

Scale-up Strategy

Horizontal Scaling:

Serving: Auto-scale to 5000+ model replicas across 100+ nodes
Feature computation: Distributed feature store with sharding across 500+ nodes
Training: 1000+ GPU workers with data parallelism across 5000+ training instances

Load Balancing:

Round-robin distribution of queries across model replicas
Adaptive routing based on model performance and latency metrics

Fault Tolerance

Model Serving Fallbacks:

Primary model failure → fallback to pre-trained model or rule-based baseline
Graceful degradation: serve top-K results from cached popular items when model unavailable
Health checks every 5 seconds with circuit breaker pattern

Stale Model Serving:

Model versioning with rollback capability
Shadow mode for new model versions during validation phase
99.9% uptime SLA with automatic failover to backup model instances

Cold Start Handling Strategy

New User Handling:

Default ranking based on popular items from last 30 days
Content-based filtering using user profile features (interests, demographics)
Fallback to category-based ranking for users with no feature data

New Item Handling:

Content-based ranking using item metadata (title, description, category)
Collaborative filtering with recent popular items from same category
Temporal bias: rank new items by recency and popularity

Multi-region Serving Strategy

Geographic Distribution:

Primary region (US West): 60% of traffic, 5000 model replicas
Secondary region (EU Central): 25% of traffic, 1500 model replicas
Backup region (Asia Pacific): 15% of traffic, 500 model replicas

Cross-region Failover:

Traffic distribution: 60/25/15 split across regions
Regional model synchronization every 30 minutes
Automatic failover to secondary region when primary region latency exceeds threshold

Latency Budget Allocation:

Stage	p50	p99	Budget
Feature lookup	2ms	5ms	10ms
Model inference	15ms	30ms	50ms
Response formatting	1ms	3ms	5ms
Total	18ms	38ms	65ms

Capacity Planning:

100k QPS peak → 200k QPS with 2x buffer for traffic spikes
5000 model replicas × 20GB = 100TB RAM requirement
5000 GPU instances × 20GB = 100TB GPU memory requirement
5000 feature store nodes × 100GB = 500TB storage requirement

Reliability Targets:

99.9% availability SLA (1.44 minutes downtime per day)
50ms p99 latency target for search ranking
100k QPS sustained throughput with 2x traffic spike capacity
Multi-region failover within 30 seconds of primary region failure

11. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering / policy alignment

Query classification strategy: Should we treat web search as a pure ranking problem or incorporate query intent classification (informational vs navigational vs transactional) that may require separate model paths?
Relevance vs commercial intent: How do we balance organic search relevance with paid search results and sponsored content in the ranking function?
Cross-domain consistency: Should we maintain consistent ranking behavior across search domains (web, images, videos) or optimize each domain independently?
Content freshness policy: How aggressively should we prioritize fresh content vs. established authoritative sources in search rankings?

Key ML risks

Risk	Impact	Likelihood	Mitigation
Data drift in query semantics	High - ranking quality degradation	Medium	Implement continuous drift detection, adaptive retraining windows
Query ambiguity misinterpretation	High - incorrect result ranking	Medium	Multi-lingual query understanding, fallback to explicit user feedback
Cold start for new domains	Medium - ranking instability	High	Implement domain adaptation strategies and seed models
Bias amplification from training data	High - regulatory risk, brand damage	Low-Medium	Regular bias audits, fairness constraints in training
Overfitting to training data	Medium - poor generalization	Medium	Regularization, cross-validation, out-of-domain testing

Trade-offs made with justification

Accuracy vs. latency: Prioritizing sub-100ms response time over marginal accuracy gains (e.g., 2-3% AUC improvement) by using model compression and pre-computed features
Freshness vs. cost: Accepting 1-2 day delay in feature updates to reduce infrastructure costs and training overhead, balancing with real-time signals for critical features like trending content
Model size vs. quality: Choosing model size that fits within 100MB serving constraints while maintaining >95% of baseline performance to enable efficient deployment across edge devices and mobile clients
Online vs. batch training: Using batch training for model updates to maintain consistency and reproducibility, while implementing online learning for real-time feedback adaptation

Alternatives considered and rejected

Transformer-based ranking models: Rejected due to high latency (500ms+ p99) and complexity for real-time serving, despite superior accuracy gains
Multi-task learning for query classification: Rejected due to increased model complexity and marginal gains in ranking performance, favoring simpler rank-only models
Real-time feature computation: Rejected due to high latency cost (50-100ms per feature) and infrastructure complexity, favoring pre-computed features with periodic updates
Zero-shot ranking models: Rejected due to poor performance on domain-specific queries and lack of fine-grained control over ranking behavior

Technical debt acknowledged

Feature dependency complexity: Current feature store design creates tight coupling between features and models, making future model evolution more difficult
Model versioning gaps: Lack of comprehensive model versioning and rollback capabilities in current serving infrastructure increases risk of production issues
Offline vs. online feature inconsistency: Known gaps between offline training features and online serving features that may cause performance degradation
Limited A/B testing infrastructure: Current system lacks robust A/B testing capabilities for multi-model experiments, limiting ability to validate model improvements in production
No cross-domain model sharing: Each search domain (web, images, videos) trains independent models without leveraging shared knowledge, resulting in redundant training and suboptimal cross-domain generalization

12. Phased Implementation Roadmap

Phase 1: MVP (Weeks 1-8)

Objective: Establish baseline ranking system with rule-based fallback and basic ML model

Deliverables:

Rule-based search ranking engine (TF-IDF + BM25)
Basic ML model using logistic regression with 100K training samples
Online serving infrastructure with 100 QPS capacity
A/B testing framework for baseline comparison

Dependencies:

Data pipeline ready with 100K samples
Feature store MVP (batch features only)
Basic serving infrastructure (TorchServe or custom)

Critical Path:

Data labeling pipeline → 2. Basic model training → 3. Online serving setup → 4. A/B testing infrastructure

Phase 2: Scale (Weeks 9-16)

Objective: Expand to real-time features, distributed training, and comprehensive evaluation

Deliverables:

Real-time feature computation (user behavior, item freshness)
Distributed training pipeline (TPU/GPU cluster)
Feature store with online and offline stores
Advanced model architecture (DNN with attention)

Dependencies:

Feature store with real-time updates
Distributed training infrastructure (Ray or Horovod)
A/B testing with proper randomization unit

Critical Path:

Feature store implementation → 2. Distributed training pipeline → 3. Model retraining infrastructure → 4. Advanced model deployment

Phase 3: Excellence (Weeks 17-24)

Objective: Implement continuous learning, advanced models, and multi-task architecture

Deliverables:

Online learning capability with concept drift detection
Multi-task model (ranking + click prediction + relevance scoring)
Real-time re-ranking with transformer-based architecture
Production monitoring and alerting system

Dependencies:

Continuous learning framework (Drift detection, retraining triggers)
Multi-task model training pipeline
Production monitoring stack with alerting

Critical Path:

Continuous learning implementation → 2. Multi-task model training → 3. Real-time re-ranking system → 4. Production monitoring stack