Ranking Metrics Analysis for AI Systems - NDCG, MRR, and Precision@K

Ranking metrics form the cornerstone of evaluating search, recommendation, and content retrieval systems in modern AI applications, with NDCG, MRR, and Precision@K serving as the primary evaluation measures for assessing relevance and system performance.

1. Problem Statement & ML Task Definition

Clarifying questions

1. What is the expected scale of the ranking system in terms of users/items and QPS?

   • This determines whether we need to optimize for low-latency serving or batch processing, and informs model size and feature store design.

2. What is the primary business metric we're optimizing for?

   • Is it CTR, revenue, engagement, or user retention? This will guide our choice of ranking metrics and evaluation strategy.

3. What is the latency budget for real-time ranking requests?

   • For example, 50ms p99 for user-facing ranking or 200ms for batch re-ranking. This impacts model architecture and serving stack.

4. Are we optimizing for online or batch ranking?

   • Online systems require low-latency inference, while batch systems can use more complex models with longer training cycles.

5. What is the expected data freshness requirement?

   • Is it real-time (seconds), near-real-time (minutes), or batch (daily)? This affects feature computation and update frequency.

6. What is the expected accuracy vs. speed trade-off?

   • Is it more critical to rank highly relevant items at the top (high NDCG) or to deliver results quickly (low latency)?

7. What is the typical size of the candidate set for ranking?

   • For example, 1000 items or 100k+ items. This impacts retrieval strategy and model complexity.

Business use case and success criteria

The system is designed to rank items (e.g., articles, products, videos) for users in real-time to maximize engagement and business KPIs such as click-through rate (CTR), revenue, or user retention. The ranking model is deployed in a real-time serving pipeline to provide personalized item rankings for each user request.

Success criteria:

• Offline: NDCG@K, MRR@K, Precision@K (for relevance)
• Online: CTR, revenue per user, engagement duration
• Long-term: Delayed conversion rate, retention rate

ML problem formulation

This is a learning-to-rank (LTR) task, specifically a relevance ranking problem. The goal is to learn a function that maps a user-item interaction context to a ranked list of items, optimizing for the relevance of top-ranked items.

Justification:

• The task involves ranking items in a list, not predicting a binary label or continuous value.
• The objective is to maximize the utility of top-k items, which aligns with ranking metrics like NDCG and MRR.
• The model needs to understand complex user-item interactions, which is well-suited for neural ranking models.

ML evaluation metrics

Offline Evaluation Metrics:

• NDCG@K (Normalized Discounted Cumulative Gain): Measures the ranking quality by discounting gains based on position, with higher weights for top positions.
• MRR@K (Mean Reciprocal Rank): Evaluates the rank of the first relevant item.
• Precision@K: Measures the proportion of relevant items in top-K predictions.

Online Evaluation Metrics:

• CTR (Click-Through Rate): Measures the ratio of clicks to impressions.
• Engagement: Time spent on content, scroll depth, etc.
• Revenue per user: Monetization impact of the ranking.

Scale and constraints

Back-of-envelope estimates:

• Training data size: 100M user-item interactions (e.g., clicks, views, purchases)
• Number of features: ~10,000 (user features, item features, context features)
• Embedding matrix size: 1M users × 128-dim embedding = ~500MB (user embeddings)
• Serving QPS: 10,000 requests/sec → 50ms p99 latency target

Latency breakdown:

• Feature lookup + embedding computation: 10ms
• Model inference (DNN): 15ms
• Scoring + re-ranking: 10ms
• Response serialization: 5ms

Non-ML baseline strategy

Rule-based fallback:

• For new users or items with no historical data, use a simple rule-based ranking (e.g., popularity, recency).
• For cold-start users, rank items by global popularity or trending.
• For unknown item features, fallback to a default ranking model trained on historical data.

Cold start handling:

• Use content-based features (e.g., item category, tags) for new items.
• For new users, use demographic or behavioral proxies from similar users.

def baseline_ranking(user_id, item_ids):
    # Fallback to popularity or recency for cold start
    if user_id not in user_popularity:
        return sorted(item_ids, key=lambda x: item_popularity[x], reverse=True)
    # Or use user profile features to rank items
    return sorted(item_ids, key=lambda x: calculate_relevance(user_id, x), reverse=True)

2. Data Strategy

Training Data Sources

• Explicit feedback: User ratings, clicks, likes, shares, and direct relevance labels from human annotators or internal feedback mechanisms.
• Implicit signals: Page views, time spent, scroll depth, click-through rate (CTR), and session duration to infer user interest.
• Third-party data: Content metadata, category tags, user demographics from partner APIs or public datasets for enriching features.

Labeling Pipeline Design

• Human annotation: For high-quality labeled data (e.g., relevance scores for search results or item rankings). Use Amazon Mechanical Turk or internal annotator teams with inter-annotator agreement thresholds (e.g., Cohen's Kappa > 0.8).
• Weak supervision: Apply labeling functions (e.g., content similarity, user behavior heuristics) to generate pseudo-labels at scale.
• Programmatic labeling: Use click-through rate thresholds, dwell time, or engagement signals to automatically label items as relevant or irrelevant.

Class Imbalance Handling Strategy

• Justification: In ranking tasks, most items are often irrelevant to a user's query or interest, leading to highly skewed distributions.
• Strategy: Use weighted loss functions (e.g., focal loss) or undersampling of negative examples during training to balance the impact of rare relevant items.
• Alternative: Apply balanced sampling in batches or use class-aware loss functions (e.g., weighted cross-entropy) to ensure model learns from both rare and common items.

Negative Sampling Approach

• Strategy: Use in-batch negative sampling or hard negative mining to improve model robustness.
• Implementation: For each user-item pair, sample 10–20 negative items from the same session or user pool that are not relevant but are semantically similar.
• Justification: Reduces training noise and improves ranking precision by focusing on relevant item vs. irrelevant item comparisons.

Data Freshness and Update Frequency Requirements

• Training data: Updated daily or weekly to reflect recent user behavior and content changes.
• Real-time features: Updated every few minutes for session-based signals (e.g., recent clicks, dwell time).
• Cold start items: Updated hourly or on content ingestion.
• Back-of-envelope estimates:
  • Training data size: ~100M user-item interactions
  • Features per item: ~500 (user, item, context)
  • Embedding matrix size (for 1M users × 50-dim): ~2GB
  • Serving QPS: 10,000 → 50ms p99 latency target

Privacy Compliance (GDPR, CCPA)

• Anonymization: Remove PII (user IDs, IP addresses) and apply k-anonymity or differential privacy to user features.
• Data retention: Retain training data for 6 months, with automatic purging of outdated user behavior.
• Consent handling: Include opt-out mechanisms in feature pipelines and ensure compliance with data subject rights.

Data Quality Validation and Monitoring

• Validation checks: Ensure no missing labels, consistent timestamp formats, and valid user/item IDs.
• Monitoring: Track data drift using PSI (Population Stability Index) or KL divergence between training and serving distributions.
• Alerts: Set up alerts for sudden drops in label quality or feature distribution shifts.

import pandas as pd
from sklearn.model_selection import train_test_split

# Example: Data validation and labeling pipeline snippet
def validate_and_label(data):
    # Drop rows with missing labels or IDs
    data = data.dropna(subset=['user_id', 'item_id', 'relevance_score'])
    
    # Apply labeling function for weak supervision (e.g., CTR > 0.1 = relevant)
    data['weak_label'] = (data['ctr'] > 0.1).astype(int)
    
    return data

# Example: Class imbalance handling
def weighted_loss(y_true, y_pred):
    # Weighted binary crossentropy for imbalanced data
    pos_weight = 1.0 / (y_true.sum() / len(y_true))  # Adjust based on class ratio
    return tf.keras.losses.binary_crossentropy(y_true, y_pred, pos_weight=pos_weight)

3. Feature Engineering

Feature Taxonomy

Features are categorized into four main types:

1. User Features: User demographics, behavior patterns, historical engagement, preferences (e.g., age, gender, session count, time spent)
2. Item Features: Content metadata, category, popularity, freshness (e.g., item title, description, category tags, publication date)
3. Context Features: Temporal signals, device information, geographic location (e.g., hour of day, device type, country)
4. Interaction Features: User-item interaction history, recency, frequency, temporal decay (e.g., last seen time, click count)

Embedding Design

Entity Embeddings:

• User ID embeddings (dimension: 128)
• Item ID embeddings (dimension: 128)
• Category embeddings (dimension: 64)

Text Embeddings:

• Item title and description embeddings using BERT-base (dimension: 768)
• User preference text (from history) using sentence-BERT (dimension: 768)

Graph Embeddings:

• User-item interaction graph embeddings using Node2Vec (dimension: 128)

Real-time vs Batch Feature Computation

Feature Type	Computation Mode	Reason
User session count	Real-time	High volatility, requires up-to-date engagement
Item popularity	Batch	Stable over time, computed daily
User preference embedding	Real-time	Needs to capture recent behavior
Temporal features	Real-time	Hourly/minute level changes critical

Feature Store Design

Online Store:

• Low-latency key-value store (e.g., Redis or DynamoDB)
• Designed for sub-100ms response times
• Stores pre-computed real-time features

Offline Store:

• Data warehouse (e.g., BigQuery or Snowflake)
• Used for model training, batch feature generation
• Supports SQL-based feature engineering and aggregation

Feature Freshness and TTL

Feature	Source	Update Frequency	Store Type
User session count	Real-time stream	Every 5 mins	Online
Item popularity	Batch job	Daily	Offline
User preference embedding	Real-time stream	Every 10 mins	Online
Hour of day	System clock	Per request	Online
User category preference	Batch job	Weekly	Offline

Cross Features and Feature Interaction Strategy

Cross features are generated to capture complex user-item relationships:

• User-Item Interaction Count: user_id × item_id interaction frequency
• Temporal Decay Features: time_since_last_interaction × item_popularity
• User-Item Category Match: user_category_preference × item_category

Cross features are generated using feature crosses in TensorFlow Feature Columns or PyTorch's embedding layers.

Feature Importance and Selection Methodology

Methodology:

• LIME-based local explanation for feature importance in model predictions
• SHAP values for global feature ranking (using LightGBM or XGBoost as surrogate models)
• Permutation importance from trained model

Selection Strategy:

• Select top 500 features based on SHAP importance scores
• Remove highly correlated features (VIF > 5)
• Use feature selection algorithms like SelectKBest or Recursive Feature Elimination

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif

# Sample feature matrix (n_samples x n_features)
X = np.random.rand(1000, 1000)
y = np.random.randint(0, 2, 1000)

# Select top K features using ANOVA F-test
selector = SelectKBest(score_func=f_classif, k=500)
X_selected = selector.fit_transform(X, y)

# Get selected feature indices
selected_features = selector.get_support(indices=True)

Feature Normalization and Scaling

• Continuous Features: Min-Max scaling to [0,1] range for neural networks
• Categorical Features: One-hot encoding or target encoding (with smoothing)
• Embedding Features: L2-normalized to unit vectors for cosine similarity

Handling Sparse Features

• High Cardinality: Use embedding layers with adaptive learning rates
• Sparse Feature Representation: Use sparse tensors in PyTorch or TensorFlow to reduce memory overhead

Cold Start Handling

• New User: Use demographic-based embeddings or default user embedding
• New Item: Use content-based embeddings (e.g., title + description) or default item embedding
• Fallback Strategy: Use popular items or trending content as fallback for unknown users/items

Feature Versioning and Rollout

• Versioned Features: Each feature has a version number (e.g., user_session_count_v2)
• Gradual Rollout: Feature flags or A/B testing to gradually introduce new features
• Backward Compatibility: Maintain backward compatibility in feature schema for older models

Feature Validation Pipeline

• Data Quality Checks: Missing value thresholds, distribution shifts
• Statistical Tests: Chi-square test for categorical features, t-test for continuous features
• Feature Drift Detection: Using statistical tests (e.g., Kolmogorov-Smirnov) to detect drift between training and serving distributions

Example Feature Engineering Pipeline (Python)

import pandas as pd
from sklearn.preprocessing import StandardScaler

def engineer_features(df):
    # User features
    df['user_session_count'] = df.groupby('user_id')['session_id'].transform('count')
    df['user_avg_time_spent'] = df.groupby('user_id')['time_spent'].transform('mean')
    
    # Item features
    df['item_popularity'] = df.groupby('item_id')['user_id'].transform('count')
    df['item_avg_rating'] = df.groupby('item_id')['rating'].transform('mean')
    
    # Context features
    df['hour_of_day'] = pd.to_datetime(df['timestamp']).dt.hour
    
    # Interaction features
    df['user_item_interaction_count'] = df.groupby(['user_id', 'item_id']).size()
    
    # Cross features
    df['user_item_popularity_interaction'] = df['user_session_count'] * df['item_popularity']
    
    # Normalize continuous features
    scaler = StandardScaler()
    continuous_features = ['user_session_count', 'user_avg_time_spent', 
                          'item_popularity', 'item_avg_rating']
    df[continuous_features] = scaler.fit_transform(df[continuous_features])
    
    return df

4. Model Architecture

Model Selection with Justification

For a ranking system focused on NDCG, MRR, and Precision@K metrics, we select a Two-Tower Neural Collaborative Filtering (NCF) model with attention mechanisms. This choice balances:

• Expressiveness: Deep neural networks can capture complex user-item interactions
• Scalability: Two-tower architecture enables efficient training and serving
• Interpretability: Attention mechanisms provide explainable ranking signals
• Metric Alignment: Directly optimized for rank-quality metrics

We reject:

• Logistic Regression: Insufficient expressiveness for complex ranking
• GBM: Less effective at capturing user-item interaction patterns than neural models
• Traditional Matrix Factorization: Limited in modeling complex feature interactions

Architecture Diagram

graph TD
    A[User Embedding] --> B[User Tower]
    C[Item Embedding] --> D[Item Tower]
    B --> E[Merged Representation]
    D --> E
    E --> F[Attention Layer]
    F --> G[Scoring Function]
    G --> H[Ranking Output]

Two-Tower / Multi-Task Design

The model employs a two-tower architecture:

1. User Tower: Processes user features (demographics, behavior history)
2. Item Tower: Processes item features (content, category, metadata)

We implement a multi-task learning head with shared user and item towers but separate task-specific layers for:

• Pointwise ranking score (for NDCG/MRR)
• Pairwise ranking loss (for rank order preservation)
• Click-through rate prediction (for business alignment)

Pre-trained Model Selection and Fine-tuning Strategy

For foundational capabilities, we utilize a pre-trained BERT model for text embeddings (item descriptions, titles) and fine-tune it on our domain-specific ranking data. This approach:

• Reduces training time from weeks to days
• Leverages general language understanding for better text representation
• Enables transfer learning across related domains

Fine-tuning strategy:

1. Stage 1: Freeze BERT layers, train only task-specific heads
2. Stage 2: Gradually unfreeze BERT layers, continue joint training

Model Complexity vs Latency Trade-off Analysis

Model	Metric	Latency (p99)	Size
Logistic Regression	NDCG@10	5ms	2KB
Two-Tower MLP (32-64-32)	NDCG@10	25ms	5MB
Two-Tower + Attention	NDCG@10	40ms	15MB
BERT + Two-Tower	NDCG@10	80ms	250MB

Offline Evaluation Baseline Framing

Baseline: Logistic regression with user and item embeddings + interaction features
Proposed: Two-tower model with attention + BERT text embeddings

Expected improvement:

• NDCG@10: 15-20% increase
• MRR: 10-15% improvement
• Precision@K: 25-30% boost

Core Model Code Snippet

import torch
import torch.nn as nn
import torch.nn.functional as F

class TwoTowerRanker(nn.Module):
    def __init__(self, user_dim=128, item_dim=128, hidden_dims=[64, 32], dropout_rate=0.2):
        super().__init__()
        self.user_tower = nn.Sequential(
            nn.Linear(user_dim, hidden_dims[0]),
            nn.ReLU(),
            nn.Dropout(dropout_rate),
            nn.Linear(hidden_dims[0], hidden_dims[1]),
            nn.ReLU()
        )
        self.item_tower = nn.Sequential(
            nn.Linear(item_dim, hidden_dims[0]),
            nn.ReLU(),
            nn.Dropout(dropout_rate),
            nn.Linear(hidden_dims[0], hidden_dims[1]),
            nn.ReLU()
        )
        self.attention = nn.MultiheadAttention(
            embed_dim=hidden_dims[1], 
            num_heads=4, 
            dropout=dropout_rate
        )
        self.ranker = nn.Sequential(
            nn.Linear(hidden_dims[1], 32),
            nn.ReLU(),
            nn.Linear(32, 1)
        )
        
    def forward(self, user_emb, item_emb):
        # Encode user and item representations
        user_repr = self.user_tower(user_emb)
        item_repr = self.item_tower(item_emb)
        
        # Apply attention mechanism for better interaction modeling
        user_repr = user_repr.unsqueeze(0)  # [1, batch_size, hidden_dim]
        item_repr = item_repr.unsqueeze(0)  # [1, batch_size, hidden_dim]
        attended, _ = self.attention(user_repr, item_repr, item_repr)
        
        # Final scoring
        scores = self.ranker(attended.squeeze(0))
        return torch.sigmoid(scores).squeeze(-1)
    
    def compute_ranking_loss(self, user_emb, item_emb, labels):
        """Compute pairwise ranking loss for training"""
        scores = self.forward(user_emb, item_emb)
        
        # Pairwise hinge loss (simplified for example)
        pos_scores = scores[labels == 1]
        neg_scores = scores[labels == 0]
        
        # Margin ranking loss
        loss = torch.mean(F.relu(1.0 - (pos_scores.unsqueeze(1) - neg_scores.unsqueeze(0))))
        return loss

# Example usage for training
model = TwoTowerRanker()
user_embeddings = torch.randn(32, 128)  # batch_size=32, embedding_dim=128
item_embeddings = torch.randn(32, 128)
labels = torch.randint(0, 2, (32,))  # Binary labels

loss = model.compute_ranking_loss(user_embeddings, item_embeddings, labels)
print(f"Ranking loss: {loss.item():.4f}")

This implementation provides:

• Modular architecture with clear separation of concerns
• Attention mechanism for better user-item interaction modeling
• Scalable design supporting batch processing
• Production-ready loss function for ranking optimization
• Concrete training interface with example usage

5. Serving & Inference Architecture

System Diagram

graph TD
    A[User Query] --> B[Query Processing]
    B --> C[Feature Store Lookup]
    C --> D[Embedding Cache]
    D --> E[Model Serving]
    E --> F[Ranking Results]
    F --> G[Response to Client]

    subgraph "Online Serving Layer"
        B
        C
        D
        E
        F
        G
    end

    subgraph "Model Serving"
        E --> H[TorchServe/ONNX Runtime]
        H --> I[Two-Tower Model]
    end

    subgraph "Feature Store"
        C --> J[Online Feature Store]
        D --> K[Embedding Cache]
    end

Online vs Batch Serving Decision

For a ranking system, online serving is required because:

• User queries are real-time and require immediate response
• Ranking must be personalized per user with up-to-date features
• Low latency (sub-100ms) is critical for user experience
• Batch serving would introduce unacceptable delays

Two-Stage Pipeline

1. Candidate Retrieval (High Recall)
   - Approximate Nearest Neighbor (ANN) search
   - BM25 or embedding-based retrieval

2. Scoring & Re-ranking (High Precision)
   - Two-tower model scoring
   - Final rank ordering with relevance scores

Serving Infrastructure

Primary choice: TorchServe + ONNX Runtime

Why:

• Low latency: Optimized for inference with sub-50ms p99 response times
• Model format support: Handles PyTorch and ONNX models natively
• Scalability: Built-in support for multi-GPU and distributed serving
• Production ready: Used in production at Meta, Amazon, and other FAANGs

Alternative consideration: Triton Inference Server

• Pros: Supports multiple frameworks, better for heterogeneous model serving
• Cons: Higher latency overhead, more complex setup

Latency Budget Breakdown

Stage	p50 Latency	p99 Latency	Budget
Query Parsing	1ms	2ms	5ms
Feature Lookup	5ms	10ms	15ms
Model Inference	20ms	35ms	50ms
Ranking & Response	2ms	5ms	10ms
Total	28ms	52ms	80ms

Model Compression Strategy

Strategy	Impact on Accuracy	Latency Improvement	Model Size Reduction
Quantization (INT8)	-1% to -2%	30-40%	50%
Pruning (10%)	-2% to -3%	25-35%	40-50%
Distillation (Student Model)	-5% to -8%	60-70%	70-80%

Selected approach: INT8 quantization + pruning (15%) for 35-40% latency improvement with minimal accuracy loss.

Caching Strategy

Cache Type	Purpose	Size Estimate
Prediction Cache	Reuse top-K results for frequent queries	1GB
Embedding Cache	User/item embeddings in memory	2GB
KV Cache (LLMs)	For LLM-based ranking	5-10GB

Auto-scaling and Capacity Planning

QPS Target: 10,000 requests/sec (peak)

Capacity calculation:

• Average response time: 50ms
• Target p99 latency: 100ms
• Required concurrent requests: 10,000 QPS × 0.1s = 1,000 concurrent requests
• Per-node capacity: 500 concurrent requests (assuming 2x redundancy)
• Required nodes: 2 nodes per region

Auto-scaling strategy:

• CPU utilization >80% → Scale up
• QPS > 8,000/sec → Add new nodes
• Response time p99 > 150ms → Alert and scale up

Core Model Serving Code Sample

import torch
from torch import nn
import numpy as np

class TwoTowerRanker(nn.Module):
    def __init__(self, user_emb_dim=128, item_emb_dim=128, hidden_dim=256):
        super().__init__()
        self.user_encoder = nn.Sequential(
            nn.Linear(user_emb_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim // 2)
        )
        self.item_encoder = nn.Sequential(
            nn.Linear(item_emb_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim // 2)
        )
        self.score_layer = nn.Linear(hidden_dim // 2, 1)
        
    def forward(self, user_emb, item_emb):
        # Encode user and item embeddings
        user_repr = self.user_encoder(user_emb)
        item_repr = self.item_encoder(item_emb)
        
        # Compute dot product score
        scores = torch.sum(user_repr * item_repr, dim=1)
        return scores
    
    def rank_items(self, user_emb, candidate_items):
        """Rank candidate items for a given user"""
        # Compute scores for all candidates
        scores = []
        for item_emb in candidate_items:
            score = self.forward(user_emb, item_emb)
            scores.append(score.item())
        
        # Sort by score (descending)
        ranked_indices = np.argsort(scores)[::-1]
        return ranked_indices

# Example usage for inference
def serve_ranking_request(user_embedding, candidate_items):
    model = TwoTowerRanker()
    # Load pre-trained weights
    model.load_state_dict(torch.load('ranker_weights.pth'))
    
    # Convert to tensor if needed
    user_tensor = torch.tensor(user_embedding, dtype=torch.float32)
    
    # Rank items
    ranked_indices = model.rank_items(user_tensor, candidate_items)
    
    # Return top-K results
    return ranked_indices[:10]  # Top-10 ranking

# Example scoring function for NDCG/MRR calculation
def calculate_ranking_metrics(predictions, relevant_items, k=10):
    """Calculate ranking metrics for evaluation"""
    # predictions: list of (item_id, score) tuples sorted by score descending
    pred_items = [item_id for item_id, _ in predictions[:k]]
    
    # Calculate MRR
    mrr = 0.0
    for i, item in enumerate(pred_items):
        if item in relevant_items:
            mrr = 1.0 / (i + 1)
            break
    
    # Calculate Precision@K
    precision_k = sum(1 for item in pred_items if item in relevant_items) / k
    
    # Calculate NDCG@K
    dcg = 0.0
    idcg = 0.0
    for i, item in enumerate(pred_items):
        if item in relevant_items:
            dcg += 1.0 / np.log2(i + 2)
        if i < len(relevant_items):
            idcg += 1.0 / np.log2(i + 2)
    
    ndcg = dcg / idcg if idcg > 0 else 0.0
    
    return {"MRR": mrr, "Precision@K": precision_k, "NDCG@K": ndcg}

6. Evaluation Framework

Offline Evaluation Methodology

For offline evaluation of ranking models, we use a temporal split approach to ensure that the model is evaluated on future data relative to training, preventing data leakage. This is critical for ranking systems where the temporal order of interactions matters.

We split our dataset into:

• Training: Last 6 months of data (2023-01-01 to 2023-06-30)
• Validation: Next 2 months (2023-07-01 to 2023-08-31)
• Test: Final 2 months (2023-09-01 to 2023-10-31)

We also apply time-based cross-validation with 3 splits for robustness. For each split, we train on the earlier portion and evaluate on the later portion.

Slice-Based Evaluation

We define critical slices for evaluation to detect potential model degradation or bias:

• User Demographics: Age groups (18–25, 26–35, 36–50), gender
• Device Types: Mobile, Desktop, Tablet
• Geographic Regions: US, EU, APAC, LATAM
• Engagement Levels: High, Medium, Low (based on past interaction frequency)
• Content Categories: News, Sports, Entertainment, Education

Each slice is evaluated using the same ranking metrics to ensure fairness and detect performance gaps.

A/B Test Design

We design a randomized controlled experiment with the following specifications:

• Randomization Unit: User ID (to avoid leakage between users)

• Sample Size Calculation:

  • Target: 5% lift in CTR (primary metric)
  • Power: 80%
  • Significance Level: 5%
  • Estimated baseline CTR: 2.5%
  • Required Sample Size: ~100,000 users per variant

• Duration: 2 weeks (to capture weekly seasonality and ensure statistical significance)

• Primary Metric: Click-Through Rate (CTR) — measured over 24-hour window post-serving

• Secondary Metrics: NDCG@10, MRR@10, Precision@5

Guardrail Metrics and Blocking Conditions

To ensure model safety and stability:

• Guardrail Metrics:

  • Mean Reciprocal Rank (MRR): Must not decrease by more than 5% from baseline
  • NDCG@10: Must not drop below 95% of baseline
  • Precision@5: Must not fall below 90% of baseline

• Blocking Conditions:

  • If any guardrail metric drops by more than 10%, the experiment is paused
  • If CTR drops by more than 3% for more than 2 consecutive days, the model is rolled back to previous version

Shadow Mode Rollout Plan

Before full A/B testing, we implement a shadow mode rollout:

• All traffic is routed to both the current model and new model in parallel
• Predictions are logged but not served to users
• Offline comparison of predictions between models using same test set
• Evaluate metrics like NDCG@10, MRR@10, and Precision@5 in shadow mode
• If shadow results show >5% improvement in key metrics, proceed to A/B test

Long-Term Metric Measurement

We define delayed conversion and engagement decay as long-term success indicators:

• Delayed Conversion: Track whether users who clicked on items from the top 5 results later convert (e.g., purchase, sign-up) within 7 days
• Engagement Decay: Monitor if engagement with items from top-k rankings drops significantly over time (e.g., 30-day decay in view count or click rate)

We also implement user retention tracking to ensure that the model does not lead to short-term gains at the expense of long-term user satisfaction.

import numpy as np

def calculate_ndcg(predictions, labels, k=10):
    """Calculate NDCG@K for a list of predictions and labels."""
    def dcg_at_k(relevance, k):
        return np.sum([(relevance[i] / np.log2(i + 2)) for i in range(min(k, len(relevance)))])
    
    def idcg_at_k(relevance, k):
        sorted_relevance = sorted(relevance, reverse=True)
        return dcg_at_k(sorted_relevance, k)
    
    ndcg = []
    for pred, label in zip(predictions, labels):
        dcg = dcg_at_k(pred, k)
        idcg = idcg_at_k(label, k)
        ndcg.append(dcg / idcg if idcg > 0 else 0.0)
    
    return np.mean(ndcg)

def calculate_mrr(predictions, labels):
    """Calculate MRR for a list of predictions and labels."""
    mrr = []
    for pred, label in zip(predictions, labels):
        # Assuming labels are binary (0 or 1) and we want to find the first relevant item
        for i, score in enumerate(pred):
            if label[i] == 1:
                mrr.append(1.0 / (i + 1))
                break
        else:
            mrr.append(0.0)
    return np.mean(mrr)

def calculate_precision_at_k(predictions, labels, k=5):
    """Calculate Precision@K for a list of predictions and labels."""
    precision_at_k = []
    for pred, label in zip(predictions, labels):
        relevant_items = sum(label[:k])
        retrieved_items = sum(pred[:k])
        if retrieved_items > 0:
            precision_at_k.append(relevant_items / retrieved_items)
        else:
            precision_at_k.append(0.0)
    return np.mean(precision_at_k)

# Example usage:
# predictions = [[1, 0, 1, 0, 1], [0, 1, 1, 0, 0]]
# labels = [[1, 0, 1, 0, 0], [0, 1, 1, 0, 0]]
# ndcg = calculate_ndcg(predictions, labels, k=5)
# mrr = calculate_mrr(predictions, labels)
# precision = calculate_precision_at_k(predictions, labels, k=5)

7. Training Pipeline

Training Data Pipeline

The training pipeline begins with data ingestion from multiple sources including user interaction logs, item metadata, and contextual signals. Data is processed through a multi-stage ETL pipeline that handles:

• Deduplication and data cleaning
• Feature extraction from raw logs (user ID, item ID, timestamps, actions)
• Negative sampling to balance the dataset
• Temporal ordering for time-based splits

import pandas as pd
from sklearn.model_selection import train_test_split

class TrainingDataPipeline:
    def __init__(self, data_source):
        self.data_source = data_source
    
    def load_and_preprocess(self):
        # Load raw interaction logs
        df = pd.read_csv(self.data_source)
        
        # Remove duplicates and filter for valid interactions
        df = df.drop_duplicates(subset=['user_id', 'item_id'])
        
        # Create negative samples (1:1 ratio)
        df_negative = self._create_negative_samples(df)
        
        # Combine positive and negative samples
        df_combined = pd.concat([df, df_negative], ignore_index=True)
        
        return df_combined
    
    def _create_negative_samples(self, df):
        # Sample negative items for each user
        negative_samples = []
        for user in df['user_id'].unique():
            user_interactions = df[df['user_id'] == user]['item_id']
            all_items = set(df['item_id'].unique())
            negative_items = list(all_items - set(user_interactions))
            
            # Sample 1 negative per positive interaction
            for _, row in df[df['user_id'] == user].iterrows():
                if len(negative_items) > 0:
                    sampled_item = np.random.choice(negative_items)
                    negative_samples.append({
                        'user_id': user,
                        'item_id': sampled_item,
                        'label': 0
                    })
        
        return pd.DataFrame(negative_samples)

Distributed Training Strategy

For large-scale training, we implement data parallelism using PyTorch's DistributedDataParallel (DDP) to distribute training across multiple GPUs. Model parallelism is used for very large models that exceed GPU memory capacity.

Key infrastructure considerations:

• 8x A100 GPUs per training node (40GB VRAM each)
• 32 nodes for full distributed training
• Training time: ~4 hours per epoch on 256M samples

Training Infrastructure

• GPU cluster: 32 nodes × 8 A100 GPUs each (40GB VRAM)
• Storage: NVMe SSDs for fast data loading (~500GB/s read bandwidth)
• Memory: 256GB RAM per node for data caching
• Estimated training time: ~4 hours per epoch on 256M samples

Hyperparameter Tuning Approach

We use Bayesian optimization for hyperparameter tuning with the following key parameters:

• Learning rate (0.001 to 0.1)
• Batch size (32 to 1024)
• Embedding dimension (64 to 512)
• Number of layers (2 to 8)
• Dropout rate (0.1 to 0.5)

Experiment Tracking and Reproducibility

We use MLflow for experiment tracking with:

• Versioned model artifacts (weights, configs, logs)
• Parameter logging and metric tracking
• Model registry with version control
• Reproducible training through seed setting

Model Versioning and Registry Strategy

Models are versioned using semantic versioning (v1.0.0, v1.1.0) with:

• Major version: Breaking changes in architecture or training data
• Minor version: Model improvements or new features
• Patch version: Bug fixes or minor adjustments

Continual / Online Learning Design

For online learning, we implement:

• Incremental model updates every 24 hours using recent data
• Online gradient descent with learning rate decay
• Sliding window approach to maintain model relevance
• Model drift detection to trigger retraining cycles

class OnlineLearningManager:
    def __init__(self, model, learning_rate=0.001):
        self.model = model
        self.learning_rate = learning_rate
        
    def update_model(self, new_data_batch):
        # Apply online gradient updates to model weights
        for batch in new_data_batch:
            loss = self.model.compute_loss(batch)
            self.model.backward(loss)
            self.model.step(self.learning_rate)
            
    def detect_drift(self, current_data):
        # Simple drift detection using feature distribution comparison
        drift_threshold = 0.1
        return self._calculate_drift(current_data) > drift_threshold

8. Monitoring & Observability

Data Drift Detection

Data drift detection is critical for maintaining model performance over time. We implement drift monitoring using:

• Population Stability Index (PSI) for categorical features to detect shifts in distribution
• Kullback-Leibler (KL) divergence for continuous features to quantify distributional changes
• Feature-wise drift thresholds: 0.1 PSI or 0.05 KL divergence triggers alerting

import numpy as np
from scipy.stats import ks_2samp

def psi(expected, actual, num_bins=10):
    """Calculate Population Stability Index"""
    expected = np.array(expected)
    actual = np.array(actual)
    
    # Bin the data
    bins = np.histogram_bin_edges(expected, bins=num_bins)
    exp_freq = np.histogram(expected, bins=bins)[0]
    act_freq = np.histogram(actual, bins=bins)[0]
    
    # Avoid zero division
    exp_freq = np.where(exp_freq == 0, 1e-6, exp_freq)
    act_freq = np.where(act_freq == 0, 1e-6, act_freq)
    
    psi_value = np.sum((act_freq - exp_freq) * np.log(act_freq / exp_freq))
    return psi_value

def kl_divergence(p, q):
    """Compute KL divergence between two distributions"""
    p = np.array(p)
    q = np.array(q)
    return np.sum(np.where((p != 0) & (q != 0), p * np.log(p / q), 0))

Prediction Drift and Label Drift Monitoring

We track both prediction drift (model output distribution shift) and label drift (actual outcomes changing over time):

• Prediction drift: Monitor distribution of predicted scores using KS test or histogram overlap
• Label drift: Track change in target distribution over time windows (e.g., daily, weekly)

Model Performance Degradation Alerting

We implement multi-tiered alerting:

• Primary metric degradation: 5% drop in NDCG@K or MRR over rolling 7-day window
• Secondary metric degradation: 3% drop in Precision@K or recall
• Alert escalation:
  • P1 (immediate): >5% drop triggers alerting and model investigation
  • P2 (automated rollback): >10% drop triggers model rollback to previous version

Retraining Trigger Strategy

We implement a hybrid retraining strategy:

• Scheduled retraining: Weekly model retraining with new data
• Drift-triggered retraining: When PSI > 0.1 or KL divergence > 0.05 for key features
• Performance-triggered retraining: When NDCG@K drops below threshold for 3 consecutive days

Logging Design

We maintain comprehensive logging for model observability:

class PredictionLogger:
    def __init__(self):
        self.prediction_logs = []
    
    def log_prediction(self, user_id, item_ids, scores, labels, features):
        log_entry = {
            "timestamp": time.time(),
            "user_id": user_id,
            "item_ids": item_ids,
            "scores": scores.tolist(),
            "labels": labels.tolist(),
            "features": features,
            "ndcg_k": self.compute_ndcg(scores, labels, k=10),
            "mrr": self.compute_mrr(scores, labels),
            "precision_k": self.compute_precision_at_k(scores, labels, k=5)
        }
        self.prediction_logs.append(log_entry)
    
    def compute_ndcg(self, scores, labels, k=10):
        # Implementation of NDCG calculation
        pass
    
    def compute_mrr(self, scores, labels):
        # Implementation of MRR calculation
        pass
    
    def compute_precision_at_k(self, scores, labels, k=5):
        # Implementation of Precision@K calculation
        pass

Feedback Loop for Continuous Improvement

We design a feedback loop that captures model performance and user behavior:

• User interaction logs: Capture clicks, views, time spent on items
• Relevance feedback: Collect explicit feedback (rating, thumbs up/down)
• A/B test results: Track performance differences between model versions
• Model retraining pipeline: Automated retraining when drift thresholds exceeded

Key Metrics and Thresholds

Metric	Alert Threshold	Escalation
NDCG@K	5% drop over 7 days	P1 alert
MRR	3% drop over 7 days	P1 alert
Precision@K	2% drop over 3 days	P2 alert
PSI	>0.1 for categorical features	P1 alert
KL divergence	>0.05 for continuous features	P1 alert

Observability Dashboard Components

• Real-time performance dashboard: NDCG, MRR, Precision@K over time
• Drift monitoring dashboard: PSI/KL values per feature
• Prediction distribution dashboard: Histograms of model scores and labels
• A/B test results: Performance comparison between versions
• Error rate dashboard: Prediction failures and model confidence levels

Retraining and Rollback Strategy

When drift thresholds are exceeded:

1. Immediate alert: Notify ML team via Slack/PagerDuty
2. Automated retraining: Trigger retraining pipeline with updated data
3. Shadow testing: Test new model in shadow mode for 24 hours
4. Gradual rollout: Gradually increase traffic to new model (10% → 50% → 100%)
5. Rollback capability: If performance degrades further, revert to previous model version

This monitoring framework ensures robust model maintenance and continuous improvement while providing clear signals for intervention when performance degrades.

9. Responsible AI & Safety

Bias Analysis

Bias in ranking systems can manifest through user or item demographics, leading to unfair treatment of underrepresented groups. We conduct a training data audit to identify imbalances in user demographics (age, gender, geography) and item categories. Slice-based fairness evaluation will be performed across key user segments (e.g., age groups, regions) to ensure equalized odds and demographic parity.

Fairness Metrics and Constraints

We implement fairness constraints using equalized odds as the primary metric, ensuring that true positive and false positive rates are similar across user slices. For example, if the system ranks items differently for users in different age groups, we ensure that the ranking quality (measured by NDCG/MRR) is consistent across slices.

Explainability Requirements and Approach

We use SHAP (SHapley Additive exPlanations) to explain individual predictions and model-level feature importance for ranking decisions. For real-time serving, we provide attention weights from transformer-based models as an interpretability layer. Stakeholders will receive explanations in the form of top contributing features (e.g., user age, item category) that drive a specific rank.

For LLM Systems: Safety Guardrails

If this system involves LLMs or generation, we implement:

• Content filtering to prevent harmful outputs (e.g., hate speech, misinformation)
• Output validation using external fact-checkers or moderation APIs
• Rate limiting to prevent abuse and ensure system stability
• Red teaming with human evaluators to identify edge cases and safety gaps

Model Card Documentation Plan

We maintain a model card that includes:

• Model architecture and training details
• Performance metrics (NDCG, MRR, Precision@K)
• Fairness and bias analysis
• Limitations and assumptions
• Data sources and preprocessing steps

Incident Response Plan for Model Failures or Harmful Outputs

In the event of model drift, bias escalation, or harmful output generation:

1. Immediate alerting via monitoring dashboards (e.g., NDCG drop > 5%)
2. Automatic rollback to the last stable model version
3. Human review of flagged outputs or user feedback
4. Post-mortem analysis to identify root cause and update training pipeline

We maintain a failure mode catalog that lists potential issues (e.g., cold start for new users, data drift in item popularity) and their mitigation strategies. This ensures that any failure scenario is handled with a pre-defined response process.

# Example of fairness-aware ranking logic in Python (conceptual)
import numpy as np

def calculate_fairness_metrics(ranked_items, user_slices):
    """
    Calculate fairness metrics like equalized odds across slices.
    """
    slice_metrics = {}
    for slice_name, users in user_slices.items():
        # Compute NDCG and MRR per slice
        ndcg_slice = compute_ndcg(ranked_items[users])
        mrr_slice = compute_mrr(ranked_items[users])
        slice_metrics[slice_name] = {
            "ndcg": ndcg_slice,
            "mrr": mrr_slice
        }
    return slice_metrics

def compute_ndcg(ranked_items, k=10):
    """
    Compute NDCG@K for a ranked list.
    """
    dcg = 0.0
    idcg = 0.0
    for i, item in enumerate(ranked_items[:k]):
        rel = item.relevance  # assume relevance score
        dcg += (2 ** rel - 1) / np.log2(i + 2)
        if i == 0:
            idcg = (2 ** rel - 1) / np.log2(i + 2)
    return dcg / idcg if idcg > 0 else 0.0

def compute_mrr(ranked_items):
    """
    Compute MRR for a ranked list.
    """
    for i, item in enumerate(ranked_items):
        if item.is_relevant:
            return 1.0 / (i + 1)
    return 0.0

10. Scalability & Reliability

Bottleneck Analysis per Pipeline Stage

Training Pipeline:

• Data ingestion: 100GB/day → 10TB/month, 500M samples → 200GB feature matrix
• Feature engineering: 10s per sample → 50M samples × 10s = 500M seconds = ~5.7 days on single GPU
• Model training: 200GB model → 150GB embedding matrix + 50GB dense layers
• Model serving: 100K QPS → 8.6M requests/day → 100GB/day storage for prediction logs

Serving Pipeline:

• Candidate retrieval (50ms p99) → 200ms total latency budget
• Scoring stage (50ms p99) → 200ms remaining for re-ranking + caching
• Embedding lookup (10ms p99) → 50K items × 128-dim embeddings = 256MB cache
• Prediction caching (20ms p99) → 10M cached items × 500 bytes = 5GB

Scale-up Strategy

Horizontal Scaling:

• Serving: 1000 nodes × 200 QPS = 200K QPS capacity
• Feature computation: 500 nodes × 100M samples/day = 50B samples/day
• Model training: 20 GPU nodes × 100GB model = 2TB distributed storage

Distributed Feature Computation:

• Use Ray or Spark for parallel feature extraction
• Partition by user ID hash to ensure consistent feature computation
• Cache computed features in Redis or Memcached for low-latency access

Fault Tolerance

Model Serving Fallbacks:

• Primary model failure → fallback to last known good version
• Cache miss → return default rank order or rule-based baseline
• Embedding lookup failure → use precomputed embedding fallback

Stale Model Serving:

• Implement model versioning with automatic rollback capability
• Use model registry to track versions and enable A/B testing
• Set up health checks every 5 minutes to detect model degradation

Cold Start Handling Strategy

New Users:

• Use content-based filtering with user profile features
• Apply collaborative filtering with similar users (kNN)
• Fall back to popular items ranking for truly new users

New Items:

• Use content-based features (category, tags, metadata)
• Apply item-to-item similarity with existing items
• Implement candidate generation from recent trending items

Multi-Region Serving Strategy

Deployment Architecture:

• Primary region (e.g., us-west-2) for real-time serving
• Secondary regions (us-east-1, eu-central-1) for failover and latency reduction
• Cross-region replication of feature store with 1-hour TTL

Traffic Management:

• Use service mesh (Istio) for traffic splitting and failover
• Implement region-aware routing with latency-based load balancing
• Set up geo-replicated model storage with automatic sync

Latency Budget Allocation:

Stage	p50	p99	Budget
Request routing	1ms	5ms	5ms
Feature lookup	2ms	10ms	10ms
Model inference	5ms	20ms	20ms
Response formatting	1ms	5ms	5ms
Total	9ms	40ms	40ms

Capacity Planning

QPS Projection:

• Current: 100K QPS
• Projected growth: 5x over 2 years → 500K QPS
• Peak hour: 1.5x base QPS → 150K QPS

Infrastructure Scaling:

• Auto-scaling groups with target CPU utilization at 70%
• GPU nodes provisioned in batches of 10 for cost efficiency
• Feature store with 500GB capacity per region

Reliability Targets:

• SLA: 99.9% availability (1.44 minutes downtime/day)
• MTBF: 30 days for primary services
• MTTR: <5 minutes for service recovery
• Error rate: <0.1% for user-facing requests

Performance Monitoring

Bottleneck Detection:

• Monitor request latency distribution across all stages
• Track feature lookup success rate and cache hit ratio
• Implement alerting for latency spikes >2x baseline

Resource Utilization:

• GPU memory usage (target <80% to avoid OOM)
• CPU utilization (target <70% for headroom)
• Network bandwidth usage (monitor for cross-region traffic)

Failure Recovery:

• Implement circuit breaker pattern for external dependencies
• Use retry logic with exponential backoff for transient failures
• Set up alerting for service degradation and model drift

11. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering / policy alignment

• Ranking vs. Relevance Trade-off: Should we prioritize top-K precision or overall ranking quality? This decision affects model training objectives and user experience trade-offs.
• User vs. Item Cold Start Strategy: How to handle new users/items with no historical interaction data? Should we fall back to popularity-based ranking or use content-based features?
• Cross-Device Consistency: Should the same model be used across mobile, web, and desktop platforms or should device-specific models be trained?
• Content Safety vs. Relevance: For content recommendation, how to balance relevance with safety (e.g., filtering harmful content) without significantly reducing user engagement?

Key ML risks

Risk	Impact	Likelihood	Mitigation
Data Drift in User Preferences	High - Model performance degradation	Medium - User behavior evolves over time	Implement real-time drift detection, set up automatic retraining triggers
Label Noise in Implicit Feedback	Medium - Misleading training signals	High - Inherent noise in click-through data	Use weak supervision techniques, implement label validation pipelines
Cold Start for New Users/Items	Medium - Reduced personalization quality	High - New user/item influx	Implement hybrid strategies with content-based features and popularity fallbacks
Model Overfitting to Training Data	High - Poor generalization	Medium - Complex model architectures	Use proper cross-validation, early stopping, and regularization techniques
Bias Amplification in Ranking	High - Unfair treatment of user segments	Medium - Historical data bias	Implement fairness-aware training, slice-based evaluation, and demographic parity constraints

Trade-offs made with justification

Trade-off	Justification
Accuracy vs. Latency	Prioritize sub-100ms serving latency over marginal accuracy gains for real-time ranking use cases
Freshness vs. Cost	Accept some staleness in batch features to reduce compute cost, while maintaining real-time feature updates for critical signals
Model Size vs. Quality	Optimize model size to fit within serving constraints while maintaining >95% of baseline performance
Online vs. Batch Training	Use batch training for stable model updates, with online learning for rapid adaptation to new user behavior
Precision@K vs. NDCG	Prioritize Precision@K as primary metric for business KPIs (e.g., CTR), while using NDCG for comprehensive ranking quality assessment

Alternatives considered and rejected

Alternative	Reason for Rejection
Transformer-based Models	Too computationally expensive for real-time serving, outweighing benefits in latency-sensitive applications
Multi-Task Learning	Complex to implement and maintain; simpler two-tower architecture provides sufficient performance
Ensemble of Models	Increased complexity and latency; single model with proper architecture achieves desired performance
Full Re-ranking Pipeline	Overkill for simple ranking tasks; candidate retrieval + scoring is sufficient for most use cases
Deep Reinforcement Learning	High complexity and training instability; supervised learning with proper ranking objectives is more reliable

Technical debt acknowledged

• Hardcoded Evaluation Metrics: Current implementation uses hardcoded metric definitions that should be made configurable for different ranking objectives.
• Limited Slice-Based Evaluation: Initial implementation only supports basic user segment slices; more granular slice evaluation is needed for comprehensive fairness analysis.
• Monolithic Model Architecture: The model architecture is not modular enough to support easy experimentation with different ranking objectives or embedding strategies.
• Limited Real-time Feature Support: Current design assumes batch feature computation; real-time feature updates are not fully supported in the initial implementation.
• No A/B Testing Infrastructure: The system lacks proper A/B testing infrastructure, which should be implemented before full production rollout to ensure safe model updates.

12. Phased Implementation Roadmap

Phase 1: MVP (Weeks 1-4)

Objective: Establish baseline ranking system with rule-based fallback and simple ML model.

Deliverables:

• Rule-based baseline ranking (e.g., popularity + recency)
• First ML model using logistic regression with basic features
• Online serving infrastructure (TorchServe or custom)
• Basic A/B testing infrastructure with 10% shadow mode
• Feature store with minimal feature set (user ID, item ID, basic metadata)

Dependencies:

• Data pipeline established
• Feature store infrastructure provisioned
• Basic model serving stack deployed

Phase 2: Scale (Weeks 5-8)

Objective: Expand to full feature set and distributed training.

Deliverables:

• Full feature store with real-time and batch features
• Distributed training pipeline (Ray or PyTorch DDP)
• Multi-task model architecture (ranking + relevance scoring)
• A/B testing infrastructure with proper randomization unit
• Shadow mode rollout to 50% of traffic

Dependencies:

• Phase 1 completed
• Feature store populated with all features
• Distributed training infrastructure available

Phase 3: Excellence (Weeks 9-12)

Objective: Advanced ranking models with continuous learning and real-time features.

Deliverables:

• Transformer-based ranking model (e.g., BERT-based)
• Real-time feature computation pipeline
• Continuous learning system with drift detection
• Advanced A/B testing with multiple metrics (NDCG, MRR, Precision@K)
• Production-grade monitoring and alerting

Dependencies:

• Phase 2 completed
• Model performance meets business thresholds
• Monitoring and observability systems in place

Critical Path Dependencies:

1. Data Pipeline (Week 1) → Phase 1
2. Feature Store (Week 2) → Phase 2
3. Model Training (Week 3) → Phase 2
4. Serving Infrastructure (Week 2) → Phase 1
5. A/B Testing (Week 3) → Phase 2

Risk Mitigation:

• Data Quality Risk: Implement data validation at each stage
• Latency Risk: Monitor QPS and latency in shadow mode before full rollout
• Model Drift Risk: Implement drift detection and retraining triggers
• Scale Risk: Design horizontal scaling capability from day one

Success Criteria:

• NDCG@10 improves by 15% over baseline
• MRR improves by 20% over rule-based system
• Precision@K meets 85% threshold for top-K recommendations
• 99th percentile latency under 200ms in production

Resource Allocation:

• Weeks 1-4: 2 engineers, 1 data scientist
• Weeks 5-8: 3 engineers, 2 data scientists
• Weeks 9-12: 4 engineers, 3 data scientists

Rollout Strategy:

1. Week 1-2: Internal shadow mode testing
2. Week 3-4: 10% production shadow mode
3. Week 5-6: 25% production shadow mode
4. Week 7-8: Full production rollout with A/B test
5. Week 9-12: Continuous monitoring and model updates

Technical Debt:

• Initial rule-based fallback system to be replaced with ML model
• Feature store schema evolution to be handled through versioned features
• Shadow mode infrastructure to be integrated with full A/B testing framework