Ad Ranking System Design: Real-Time ML Platform for Advertising Revenue Optimization

Modern digital advertising platforms require sophisticated real-time ranking systems to optimize ad placements and maximize revenue, combining user behavior signals with complex ML models for instantaneous decision making.

1. Problem Statement & ML Task Definition

Clarifying questions

1. What is the expected scale of our ad inventory and user base? (e.g., millions of ads, hundreds of millions of users)
2. What is the latency budget for ad ranking? (e.g., <50ms for real-time decisions)
3. Is this primarily an online or batch serving system? (e.g., real-time ad selection vs. pre-computed recommendations)
4. What is the primary business metric we're optimizing for? (e.g., CTR, revenue per thousand impressions, conversion rate)
5. What is the expected data freshness requirement? (e.g., real-time features vs. daily batch updates)
6. What is the acceptable trade-off between model accuracy and serving latency?
7. How will we handle cold start scenarios for new advertisers or users?

Business use case and success criteria

The ad ranking system is designed to maximize revenue per thousand impressions (RPM) by efficiently selecting and ordering relevant advertisements for each user session. The system must balance user experience with advertiser value, ensuring that high-quality, relevant ads are shown to users while maximizing ad platform monetization.

Success criteria include:

• Revenue per thousand impressions (RPM) improvement of 15% over baseline
• 20% increase in click-through rate (CTR) for relevant ads
• 95th percentile latency under 50ms for real-time ranking decisions
• 99.9% uptime with graceful degradation during failures

ML problem formulation

This is a learning-to-rank (LTR) problem, specifically a multi-class ranking task. The system must rank a set of candidate ads for each user session, optimizing for the expected revenue or engagement value.

Justification:

• The system needs to order ads by predicted relevance/value for each user session
• It's not a simple classification problem since we're ranking multiple items simultaneously
• The objective function is typically based on expected revenue or user engagement, not just binary click/no-click labels
• Multi-class ranking allows for proper handling of multiple ads with different values

ML evaluation metrics

Offline Evaluation:

• AUC-ROC for binary click prediction (when available)
• NDCG@K for ranked list quality (e.g., NDCG@5, NDCG@10)
• Precision@K (e.g., Precision@5 for top 5 ads)
• MAP@K (Mean Average Precision at K)

Online Evaluation:

• Click-through rate (CTR) - primary metric for user engagement
• Revenue per thousand impressions (RPM)
• Conversion rate (if applicable to ad types)
• User retention and session duration

Scale and constraints

Back-of-envelope estimates:

• Training data size: 100M+ user-ad interactions (50GB raw data)
• Number of features: 10,000+ (user features + item features + context features)
• Embedding matrix memory footprint: 10,000 users × 256 dimensions × 4 bytes = ~10MB (per user)
• Serving QPS math: 50ms latency budget → 20 QPS per model instance, 1000 instances for 20,000 QPS

Constraints:

• QPS target: 20,000 requests/sec (50ms p99 latency)
• Data freshness: Real-time features with 1-second TTL
• Model size: <50MB for efficient serving
• Training data volume: 100M+ samples, 50GB raw data

Non-ML baseline strategy

Rule-based fallback system:

1. Cold start handling: Default to top-performing ads based on historical performance
2. User segmentation: Use user demographics and interests to select relevant ad categories
3. Advertiser priority: Prioritize ads from high-paying advertisers or those with recent bidding activity
4. Content matching: Match ad content to page context or user browsing history
5. Frequency capping: Prevent overexposure to same ad across sessions

For new users, fallback to popular ads or category-based recommendations. For new advertisers, use default performance metrics from similar advertisers in the same category.

2. Data Strategy

Training Data Sources

The training data for an ad ranking system comes from multiple sources:

• Explicit feedback: Click-through rate (CTR) data, conversion events, and user feedback signals (e.g., "not interested" buttons)
• Implicit signals: View-through rate, dwell time, scroll behavior, and ad impression logs
• User behavior data: Historical click patterns, browsing history, demographic information, and device characteristics
• Ad content features: Ad text, image metadata, category tags, and targeting criteria
• Context signals: Time of day, geographic location, device type, and network conditions
• Third-party data: Partner analytics, audience segmentation data, and contextual content features

Labeling Pipeline Design

The labeling pipeline is designed to handle both human-annotated and automatically-generated labels:

• Human annotation: For high-value signals like conversion events, ad relevance scores, and quality assessments (using Amazon Mechanical Turk or internal annotators)
• Weak supervision: Using heuristics for CTR prediction (e.g., "if user clicked on similar ads, they likely would click this one")
• Programmatic labeling: For cold start cases or when human labels are not available, using behavioral heuristics like "user has clicked on 5+ ads in last hour" to label as positive

Class Imbalance Handling Strategy

Ad ranking systems typically suffer from severe class imbalance (most impressions don't get clicked). The strategy includes:

• Weighted sampling: Assign higher weights to positive samples during training
• Focal Loss: To focus training on hard-to-classify examples (especially rare clicks)
• Threshold tuning: Adjust classification thresholds to optimize for business metrics like revenue per thousand impressions (RPM)

Negative Sampling Approach

Negative sampling is critical for training stable models:

• Hard negative sampling: Sample ads that were previously clicked by similar users but not shown to current user
• Random negative sampling: Sample from the pool of non-clicked impressions (with probability proportional to exposure)
• Stratified sampling: Ensure representation of different user segments, ad categories, and time windows

Data Freshness and Update Frequency Requirements

• Real-time signals: User behavior, session data (updated every few seconds)
• Batch features: Historical user preferences, ad metadata (updated daily)
• Context features: Time-based signals, weather data (updated hourly)
• Label freshness: CTR labels updated every 15 minutes to capture recent user behavior

Privacy Compliance

• GDPR/CCPA compliance: All personal data is anonymized using hashing or tokenization
• Data retention policy: User behavior logs retained for 180 days, then purged
• Consent management: Explicit opt-in for behavioral targeting and ad personalization
• Differential privacy: For aggregate statistics, apply differential privacy mechanisms to prevent re-identification

Data Quality Validation and Monitoring

• Data validation checks: Ensure no nulls in key fields, valid timestamp ranges, and proper feature encoding
• Feature drift monitoring: Track distribution shifts in user features over time using PSI (Population Stability Index)
• Label consistency checks: Detect anomalies in click patterns or labeling errors through statistical outlier detection
• Data freshness monitoring: Alert on data pipelines that haven't updated within expected intervals

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder

class AdRankingDataPipeline:
    def __init__(self):
        self.label_encoder = LabelEncoder()
        
    def load_and_preprocess_data(self, raw_data_path):
        df = pd.read_csv(raw_data_path)
        
        # Handle missing values
        df.fillna({'user_age': 0, 'ad_category': 'unknown'}, inplace=True)
        
        # Encode categorical features
        df['user_id_encoded'] = self.label_encoder.fit_transform(df['user_id'])
        
        # Create time-based features
        df['hour_of_day'] = pd.to_datetime(df['timestamp']).dt.hour
        
        return df
    
    def sample_negatives(self, df, ratio=1.0):
        """Sample negative examples to balance dataset"""
        positive = df[df['clicked'] == 1]
        negative = df[df['clicked'] == 0]
        
        # Downsample negatives to match positive samples
        negative_sampled = negative.sample(n=len(positive) * ratio, random_state=42)
        
        return pd.concat([positive, negative_sampled])

3. Feature Engineering

Feature Taxonomy

Feature Type	Examples	Source	Purpose
User Features	Age, gender, location, device type, historical CTR, purchase history	User profile data, behavior logs	Personalization, targeting
Item Features	Category, price, brand, image features, text description	Product catalog, inventory systems	Relevance, match quality
Context Features	Time of day, seasonality, device type, network type	Real-time context APIs	Temporal relevance
Interaction Features	User-item click history, dwell time, conversion rate, recency	Event logs, session data	Behavioral patterns

Embedding Design

Entity Embeddings:

• User ID embeddings (128-dim): trained via click-through prediction
• Item ID embeddings (256-dim): trained via item popularity and relevance signals
• Category embeddings (64-dim): pre-trained from product taxonomy

Text Embeddings:

• Product title + description embeddings (512-dim): using BERT-base fine-tuned on ad relevance data
• User query embeddings (512-dim): for search-based ad matching

Graph Embeddings:

• User-item interaction graph embeddings (128-dim): using Node2Vec for session-based recommendations

Real-time vs Batch Feature Computation

Feature Type	Computation Mode	Reason
User session features	Real-time	High temporal sensitivity
Historical engagement	Batch	Stable, computed nightly
Item popularity scores	Batch	Computed once per day
Real-time context	Real-time	Required for immediate relevance

Feature Store Design

graph TD
    A[User Profile DB] --> B[Feature Store]
    C[Item Catalog] --> B
    D[Event Logs] --> B
    E[Batch Processing] --> B
    B --> F[Real-time Serving]
    B --> G[Training Pipeline]

Feature Freshness and TTL

Feature	Source	Update Frequency	Store Type
User demographics	Profile DB	Daily	Online
Item price	Catalog API	Hourly	Online
User recent clicks	Event Logs	Real-time	Online
Item popularity	Batch job	Daily	Offline
User session features	Session logs	Real-time	Online

Cross Features and Feature Interactions

Cross features are generated using hash-based feature crossing to handle combinatorial explosion:

def generate_cross_features(user_features, item_features):
    """Generate cross features for ad ranking"""
    cross_features = []
    
    # Hash-based feature crossing (avoid combinatorial explosion)
    for user_feat in user_features:
        for item_feat in item_features:
            # Simple hash-based cross feature
            cross_key = hash(f"{user_feat}_{item_feat}") % 10000
            cross_features.append(f"cross_{cross_key}")
    
    return cross_features

# Example usage
user_features = ['age_group', 'device_type']
item_features = ['category', 'price_tier']
crosses = generate_cross_features(user_features, item_features)

Feature Importance and Selection

Methodology:

• SHAP values for global feature importance analysis
• Permutation importance in training pipeline
• A/B testing to validate feature impact

Selection Strategy:

1. Remove features with SHAP importance < 0.01
2. Keep top 500 most important features for serving
3. Use feature groups in model to reduce dimensionality

from sklearn.inspection import permutation_importance
import numpy as np

def compute_feature_importance(model, X_test, y_test):
    """Compute permutation importance for feature selection"""
    baseline_score = model.score(X_test, y_test)
    
    importances = []
    for i in range(X_test.shape[1]):
        X_permuted = X_test.copy()
        X_permuted[:, i] = np.random.permutation(X_permuted[:, i])
        
        permuted_score = model.score(X_permuted, y_test)
        importance = baseline_score - permuted_score
        importances.append(importance)
    
    return np.array(importances)

4. Model Architecture

Model Selection with Justification

For an ad ranking system, I'll select a Two-Tower Deep Neural Network architecture with Multi-Task Learning capabilities. This choice balances the need for strong ranking performance with scalability and interpretability.

Justification:

• Two-Tower Architecture: Separates user and item representations, enabling efficient computation and better handling of sparse features
• Multi-Task Learning: Can jointly optimize for CTR, conversion rate, and revenue maximization
• Scalability: Can be distributed across multiple GPUs with efficient embedding lookups
• Interpretability: Clear separation of user and item embeddings makes debugging easier

Alternatives Considered:

• GBM (XGBoost/LightGBM): Good for structured features but struggles with high-dimensional sparse embeddings
• Transformer-based models: Excellent for sequence modeling but overkill for typical ad ranking with limited context
• Factorization Machines: Good for feature interactions but less expressive than DNNs

Architecture Diagram

graph TD
    A[User Features] --> B[User Tower]
    C[Item Features] --> D[Item Tower]
    B --> E[Merged Embedding]
    D --> E
    E --> F[MLP Layers]
    F --> G[CTR Score]
    F --> H[Conversion Score]
    F --> I[Revenue Score]

Two-Tower / Multi-Task Design

The system uses a two-tower architecture where:

1. User Tower: Processes user features (demographics, browsing history, preferences)
2. Item Tower: Processes item features (ad content, category, pricing)
3. Multi-Task Head: Predicts multiple business objectives simultaneously

Core Model Code Snippet

import torch
import torch.nn as nn
import torch.nn.functional as F

class TwoTowerRankingModel(nn.Module):
    def __init__(self, 
                 user_feature_dim=1000,
                 item_feature_dim=500,
                 embedding_dim=128,
                 hidden_dims=[256, 128, 64]):
        super().__init__()
        
        # User tower
        self.user_embedding = nn.Embedding(user_feature_dim, embedding_dim)
        self.user_mlp = nn.Sequential(
            nn.Linear(embedding_dim, hidden_dims[0]),
            nn.ReLU(),
            nn.Linear(hidden_dims[0], hidden_dims[1]),
            nn.ReLU()
        )
        
        # Item tower
        self.item_embedding = nn.Embedding(item_feature_dim, embedding_dim)
        self.item_mlp = nn.Sequential(
            nn.Linear(embedding_dim, hidden_dims[0]),
            nn.ReLU(),
            nn.Linear(hidden_dims[0], hidden_dims[1]),
            nn.ReLU()
        )
        
        # Multi-task head
        self.ctr_head = nn.Linear(hidden_dims[1], 1)
        self.conv_head = nn.Linear(hidden_dims[1], 1)
        self.revenue_head = nn.Linear(hidden_dims[1], 1)
        
    def forward(self, user_ids, item_ids):
        # User tower
        user_emb = self.user_embedding(user_ids).mean(dim=1)  # Average over sequence if needed
        user_repr = self.user_mlp(user_emb)
        
        # Item tower
        item_emb = self.item_embedding(item_ids).mean(dim=1)
        item_repr = self.item_mlp(item_emb)
        
        # Combine representations
        combined = torch.mul(user_repr, item_repr)  # Element-wise multiplication
        
        # Multi-task predictions
        ctr_score = torch.sigmoid(self.ctr_head(combined))
        conv_score = torch.sigmoid(self.conv_head(combined))
        revenue_score = torch.relu(self.revenue_head(combined))
        
        return {
            'ctr': ctr_score,
            'conversion': conv_score,
            'revenue': revenue_score
        }
    
    def compute_loss(self, user_ids, item_ids, ctr_labels, conv_labels, revenue_labels):
        predictions = self.forward(user_ids, item_ids)
        
        # Binary cross-entropy for CTR and conversion
        ctr_loss = F.binary_cross_entropy(predictions['ctr'].squeeze(), ctr_labels)
        conv_loss = F.binary_cross_entropy(predictions['conversion'].squeeze(), conv_labels)
        
        # Regression loss for revenue (MSE)
        revenue_loss = F.mse_loss(predictions['revenue'].squeeze(), revenue_labels)
        
        # Combined loss with task weights
        total_loss = 0.5 * ctr_loss + 0.3 * conv_loss + 0.2 * revenue_loss
        
        return total_loss, {
            'ctr_loss': ctr_loss.item(),
            'conv_loss': conv_loss.item(),
            'revenue_loss': revenue_loss.item()
        }

# Example usage for inference
def rank_ads(model, user_features, candidate_items):
    model.eval()
    with torch.no_grad():
        # Assuming user_features and candidate_items are preprocessed
        scores = model(user_features, candidate_items)
        # Return ranked items by CTR score
        return torch.argsort(scores['ctr'], descending=True)

Model Complexity vs Latency Trade-off Analysis

Model	Parameters	Inference Latency (ms)	Memory Usage	Notes
Logistic Regression	10K	0.5	1MB	Fast but limited expressiveness
Two-Tower DNN	2M	5-10	50MB	Good balance for production
Transformer (small)	5M	20-30	200MB	Better for sequence modeling
Large Transformer	50M	100+	1GB	Overkill for typical use cases

Offline Evaluation Baseline Framing

Baseline: Logistic Regression with engineered features and TF-IDF embeddings Expected Performance:

• AUC-ROC: ~0.75 (baseline)
• NDCG@10: ~0.35 (baseline)
• Precision@10: ~0.25 (baseline)

Proposed Model: Two-Tower DNN with multi-task learning

• Expected Improvement: +5-10% in AUC-ROC, +15-25% in NDCG@10

Model Training Strategy

The model uses a multi-task loss function that balances the three primary objectives:

1. CTR Prediction (binary classification) - 50% weight
2. Conversion Prediction (binary classification) - 30% weight
3. Revenue Prediction (regression) - 20% weight

This approach enables the model to learn representations that optimize for business impact rather than just click-through rate alone.

5. Serving & Inference Architecture

System Diagram

graph TD
    A[Ad Request] --> B[Load Balancer]
    B --> C[Ranking Service]
    C --> D[Feature Store]
    D --> E[Embedding Cache]
    C --> F[Model Server]
    F --> G[TorchServe/ONNX Runtime]
    G --> H[Scoring Results]
    H --> I[Ranking Output]
    I --> J[Ad Selection]
    J --> K[Ad Delivery]
    
    subgraph "Real-time Serving"
        C
        D
        F
        G
    end
    
    subgraph "Feature Store"
        D
        E
    end

Online vs Batch Serving Decision

Online serving chosen for:

• Real-time ad ranking with sub-100ms latency requirement
• User-specific personalization needs
• Immediate feedback loop for CTR optimization
• Interactive user experience requirements

Batch serving not suitable because:

• Ad requests require individualized ranking decisions
• User behavior changes rapidly and requires fresh feature computation
• Immediate revenue impact demands real-time response

Two-Stage Pipeline

class AdRankingPipeline:
    def __init__(self):
        self.candidate_retrieval = CandidateRetriever()
        self.scoring_model = ScoringModel()
        self.reranking_model = RerankingModel()
    
    def rank_ads(self, user_context, ad_candidates):
        # Stage 1: Candidate Retrieval
        candidates = self.candidate_retrieval.retrieve(user_context, ad_candidates)
        
        # Stage 2: Scoring and Re-ranking
        scores = self.scoring_model.score(user_context, candidates)
        ranked_ads = self.reranking_model.rerank(scores, user_context)
        
        return ranked_ads

class CandidateRetriever:
    def retrieve(self, user_context, ad_candidates):
        # Retrieve top-K candidates using user affinity scores
        # Could use approximate nearest neighbor search (FAISS/Annoy)
        return top_k_candidates

class ScoringModel:
    def score(self, user_context, candidates):
        # Fast scoring model for initial ranking
        # Returns raw scores per candidate
        return [score for candidate in candidates]

class RerankingModel:
    def rerank(self, scores, user_context):
        # Final re-ranking with more complex features
        # Could include contextual features, diversity constraints
        return sorted(scores, key=lambda x: x.score, reverse=True)

Serving Infrastructure

Model Server Choice: TorchServe with ONNX Runtime

• Justification:
  • Supports model versioning and A/B testing
  • Low latency inference (sub-50ms p99)
  • Good integration with MLflow model registry
  • Production-ready with built-in monitoring

Model Compression Strategy:

• Quantization: INT8 quantization reduces model size by 4x with <2% accuracy loss
• Pruning: Structured pruning removes 30% of parameters with minimal impact
• Distillation: Student model trained to mimic teacher model performance

Latency Budget Breakdown

Stage	p50 Latency	p99 Latency	Budget
Feature Retrieval	15ms	30ms	30ms
Model Inference	25ms	45ms	50ms
Scoring & Ranking	10ms	20ms	25ms
Total	50ms	95ms	105ms

Caching Strategy

class AdServingCache:
    def __init__(self):
        self.prediction_cache = LRUCache(maxsize=100000)
        self.embedding_cache = TTLCache(ttl=3600)  # 1 hour
        self.kv_cache = {}
    
    def get_cached_prediction(self, user_id, ad_ids):
        cache_key = f"{user_id}_{hash(tuple(ad_ids))}"
        if cache_key in self.prediction_cache:
            return self.prediction_cache[cache_key]
        return None
    
    def cache_prediction(self, user_id, ad_ids, predictions):
        cache_key = f"{user_id}_{hash(tuple(ad_ids))}"
        self.prediction_cache[cache_key] = predictions

class FeatureCache:
    def __init__(self):
        self.user_features = {}
        self.item_features = {}
    
    def get_user_features(self, user_id):
        if user_id in self.user_features:
            return self.user_features[user_id]
        # Fetch from feature store and cache
        return self.fetch_and_cache_user_features(user_id)

Auto-scaling and Capacity Planning

QPS Estimation:

• Peak QPS: 50,000 requests/sec
• Model inference time: ~50ms p99
• Required model servers: 50,000 * 0.05 = 2,500 concurrent inferences
• Conservative estimate: 3,000 servers with 2x redundancy

Auto-scaling Strategy:

• Horizontal scaling based on QPS and latency
• Target CPU utilization: 70%
• Scale up when p99 latency > 100ms
• Scale down when QPS < 1,000 and CPU < 30%

Capacity Planning:

• 50,000 QPS peak → 100 model servers (20 per region)
• 50ms p99 target → 10ms headroom for system overhead
• 20% buffer for traffic spikes and maintenance windows

6. Evaluation Framework

Offline Evaluation Methodology

We use temporal cross-validation to evaluate model performance, splitting data chronologically into train/validation/test sets (e.g., 2023Q1-2023Q3 for training, Q4 for validation, Q1 2024 for testing). This approach better simulates real-world deployment conditions where models trained on older data must generalize to newer user behavior and ad content. We also implement time-based holdout splits for each user segment to ensure temporal consistency in model evaluation.

Slice-Based Evaluation

We evaluate model performance across critical user segments:

• Demographic slices: Age groups (18-25, 26-35, 36-50), gender
• Geographic slices: Region, country, urban/rural classification
• Device slices: Mobile vs desktop, OS version
• Behavioral slices: New users vs returning users, high/low engagement users

Key slices include:

• High-value customers (top 10% revenue contributors)
• New user cohort (first 30 days of usage)
• Mobile-first users (90% mobile traffic)
• Regional segments with different ad performance patterns

A/B Test Design

Randomization Unit: Individual user sessions (not users, to avoid session-level correlation effects)

Sample Size Calculation:

• Power: 80%
• Significance level: 5%
• Expected lift: 5% CTR improvement
• Baseline CTR: 2.5%
• Estimated sample size: ~1.2 million sessions per variant

Test Duration: 30 days (to capture seasonal patterns and ensure statistical significance)

Primary Metric: Click-Through Rate (CTR) with 95% confidence interval. Secondary metrics include:

• Revenue per thousand impressions (RPM)
• User engagement duration
• Conversion rate (if applicable)

Guardrail Metrics and Blocking Conditions

Guardrail thresholds for launch:

• CTR Drop: >5% decrease from baseline
• Engagement Drop: >10% reduction in session duration
• Revenue Drop: >3% decline in RPM
• System Latency: p99 > 200ms (if not within budget)
• Error Rate: >0.1% failure rate in serving

If any guardrail metric exceeds thresholds, system automatically reverts to baseline model and alerts ML team.

Shadow Mode Rollout Plan

• Phase 1 (Week 1): 5% of traffic routed to new model in shadow mode
• Phase 2 (Week 2): 25% of traffic in shadow mode
• Phase 3 (Week 3): 50% of traffic in shadow mode
• Phase 4 (Week 4): Full production rollout with A/B testing

Shadow mode ensures model behavior is validated in real-time before full deployment.

Long-Term Metric Measurement

• Delayed Conversion: Track 7-day and 30-day conversion rates to ensure model doesn't sacrifice long-term user retention for short-term CTR gains
• Engagement Decay: Monitor session frequency and retention over 30-day period to prevent model from pushing aggressive ad content that drives user churn
• Advertiser Satisfaction: Track advertiser CPM and campaign performance over 90-day windows to ensure model aligns with business KPIs

# Example A/B test evaluation code snippet
import numpy as np
from scipy import stats

def ab_test_evaluation(control_ctr, test_ctr, control_sessions, test_sessions):
    """
    Evaluate A/B test results for CTR improvement
    """
    # Calculate confidence intervals
    control_ci = stats.norm.interval(
        0.95, 
        loc=control_ctr, 
        scale=np.sqrt(control_ctr * (1 - control_ctr) / control_sessions)
    )
    
    test_ci = stats.norm.interval(
        0.95, 
        loc=test_ctr, 
        scale=np.sqrt(test_ctr * (1 - test_ctr) / test_sessions)
    )
    
    # Calculate lift and confidence bounds
    lift = (test_ctr - control_ctr) / control_ctr * 100
    lift_lower_bound = (test_ci[0] - control_ci[1]) / control_ci[1] * 100
    lift_upper_bound = (test_ci[1] - control_ci[0]) / control_ci[0] * 100
    
    return {
        "lift_percent": lift,
        "lower_bound": lift_lower_bound,
        "upper_bound": lift_upper_bound,
        "statistically_significant": (
            lift_lower_bound > 0 and lift_upper_bound > 0
        )
    }

# Example usage:
results = ab_test_evaluation(
    control_ctr=0.025,
    test_ctr=0.0262, 
    control_sessions=100000,
    test_sessions=100000
)

7. Training Pipeline

Training Data Pipeline

The training pipeline processes raw ad interaction data through ETL workflows to generate training samples. The pipeline includes:

1. Data Ingestion: Raw logs from ad serving systems, user behavior tracking, and campaign data from marketing platforms
2. Preprocessing:
   • Feature extraction from user context, item metadata, and temporal signals
   • Label generation using explicit feedback (clicks) or implicit signals (time spent, conversions)
   • Data cleaning and deduplication
3. Shuffling and Batching:
   • Shuffled batches of training samples to prevent temporal bias
   • Batch size of 1024 for efficient GPU utilization

import torch
from torch.utils.data import Dataset, DataLoader

class AdRankingDataset(Dataset):
    def __init__(self, features, labels):
        self.features = features
        self.labels = labels
    
    def __len__(self):
        return len(self.labels)
    
    def __getitem__(self, idx):
        return {
            'user_features': self.features['user'][idx],
            'item_features': self.features['item'][idx],
            'context_features': self.features['context'][idx],
            'label': self.labels[idx]
        }

# Example data loading pipeline
def create_training_dataloader(features, labels, batch_size=1024):
    dataset = AdRankingDataset(features, labels)
    return DataLoader(dataset, batch_size=batch_size, shuffle=True)

Distributed Training Strategy

For large-scale training, we implement data parallelism with model parallelism for embedding layers:

• Data Parallelism: Split training data across multiple GPUs to enable efficient processing of large datasets
• Model Parallelism: Embedding tables are sharded across GPUs to handle large embedding matrices (e.g., 10M user IDs with 128-dim embeddings)
• Pipeline Parallelism: For transformer-based models, we pipeline computation stages to overlap communication and computation

Training Infrastructure

• Compute Cluster: 8x A100 GPUs (40GB VRAM each) for distributed training
• Storage: SSD-backed storage with 1PB capacity for training data and model checkpoints
• Training Time Estimate:
  • 50M samples with 10k features → ~24 hours on 8x A100
  • Model size: ~5GB (dense + embedding layers)
• Memory Usage:
  • Per GPU: ~30GB (embedding tables + activations)
  • Total: ~240GB for full training

Hyperparameter Tuning Approach

We use Bayesian optimization with the following key hyperparameters:

1. Learning Rate: [0.001, 0.01] (log scale)
2. Batch Size: [512, 2048] (discrete)
3. Embedding Dimension: [64, 256] (discrete)
4. Number of Layers: [2, 6] (discrete)
5. Dropout Rate: [0.1, 0.5] (continuous)

from skopt import gp_minimize
from skopt.space import Real, Integer

def objective(hyperparams):
    lr, batch_size, embed_dim, num_layers, dropout = hyperparams
    
    # Initialize model with given hyperparameters
    model = AdRankingModel(
        embedding_dim=embed_dim,
        num_layers=num_layers,
        dropout=dropout
    )
    
    # Train model and return validation AUC-ROC
    train_loss = train_model(model, lr, batch_size)
    val_auc = evaluate_model(model, validation_data)
    
    return -val_auc  # Minimize negative AUC

# Bayesian optimization search space
search_space = [
    Real(1e-4, 1e-2, prior='log-uniform', name='learning_rate'),
    Integer(256, 2048, name='batch_size'),
    Integer(64, 256, name='embedding_dim'),
    Integer(2, 6, name='num_layers'),
    Real(0.1, 0.5, name='dropout_rate')
]

# Run optimization
best_result = gp_minimize(objective, search_space, n_calls=50)

Experiment Tracking and Reproducibility

• MLflow: Track experiments, parameters, metrics, and model artifacts
• Version Control: Git LFS for large model checkpoints
• Configuration Management: YAML-based config files with parameter sweeps

Model Versioning and Registry Strategy

• Model Registry: MLflow model registry with version control (v1.0, v1.1, etc.)
• Tagging: Release tags (production, staging, experimental)
• Rollback Capability: Versioned model artifacts with automatic rollback on performance degradation

Continual / Online Learning Design

• Online Learning Strategy:
  • Incremental updates using recent user feedback (clicks, conversions)
  • Sliding window of last 30 days for online learning
• Replay Buffer: Store recent samples (1M samples) for periodic retraining
• Adaptive Learning Rate: Reduce learning rate by 50% when performance degrades
• Ensemble Updates: Maintain 3 most recent model versions for ensemble voting

class OnlineLearningManager:
    def __init__(self, model, replay_buffer_size=1000000):
        self.model = model
        self.replay_buffer = []
        self.replay_buffer_size = replay_buffer_size
        
    def update_model(self, new_samples, lr=0.001):
        # Add new samples to replay buffer
        self.replay_buffer.extend(new_samples)
        if len(self.replay_buffer) > self.replay_buffer_size:
            self.replay_buffer = self.replay_buffer[-self.replay_buffer_size:]
            
        # Train on recent samples
        for sample in self.replay_buffer[-1000:]:  # Last 1000 samples
            self.model.train_step(sample, lr)
            
    def adaptive_learning_rate(self, validation_auc):
        if validation_auc < 0.85:  # Threshold for degradation
            self.model.adjust_learning_rate(0.5)  # Reduce by half

Data Pipeline Monitoring

• Throughput: Track data ingestion rate (1M samples/hour target)
• Quality Metrics: Data completeness, label distribution balance
• Failure Handling: Retry logic for failed data ingestion with exponential backoff
• Checkpointing: Save training state every 1000 steps for fault tolerance

8. Monitoring & Observability

Data Drift Detection

Data drift detection is critical for maintaining model performance in ad ranking systems where user behavior and advertiser landscapes change continuously. We implement drift detection using:

• PSI (Population Stability Index) for categorical features with thresholds at 0.1 (warning) and 0.25 (critical)
• KL Divergence for continuous features with thresholds at 0.05 (warning) and 0.1 (critical)
• Feature-wise drift monitoring with 30-day rolling windows

import numpy as np
from scipy.stats import ks_2samp, entropy

class DriftDetector:
    def __init__(self, feature_names):
        self.feature_names = feature_names
        self.baseline_stats = {}
        
    def compute_psi(self, current_dist, baseline_dist, bins=10):
        """Compute Population Stability Index"""
        # Ensure same binning across distributions
        combined_bins = np.histogram(np.concatenate([current_dist, baseline_dist]), bins=bins)[1]
        current_hist, _ = np.histogram(current_dist, bins=combined_bins)
        baseline_hist, _ = np.histogram(baseline_dist, bins=combined_bins)
        
        # Avoid zero probability
        current_hist = np.clip(current_hist, 1e-6, None)
        baseline_hist = np.clip(baseline_hist, 1e-6, None)
        
        psi = np.sum((current_hist - baseline_hist) * np.log(current_hist / baseline_hist))
        return psi
    
    def detect_drift(self, current_features, baseline_features):
        drift_report = {}
        for feature in self.feature_names:
            if feature in current_features and feature in baseline_features:
                drift_report[feature] = {
                    'psi': self.compute_psi(current_features[feature], baseline_features[feature]),
                    'ks_test': ks_2samp(current_features[feature], baseline_features[feature]).pvalue
                }
        return drift_report

Prediction Drift and Label Drift Monitoring

Prediction drift monitors changes in model outputs, while label drift monitors actual performance degradation:

• Prediction drift: Track distribution shift in predicted CTRs using statistical tests
• Label drift: Monitor actual conversion rates vs predicted probabilities

Model Performance Degradation Alerting

We implement multi-layered alerting:

Metric	Threshold	Alert Level	Detection Window
AUC-ROC drop	5%	Warning	7-day rolling
NDCG@10 drop	3%	Critical	30-day rolling
CTR drift	10%	Warning	7-day rolling

Retraining Trigger Strategy

Retraining is triggered by:

1. Scheduled retraining (weekly or bi-weekly)
2. Drift-triggered when PSI > 0.25 or KL divergence > 0.1
3. Performance-triggered when AUC drops by 5% over 30 days

Logging Design

Prediction logs include:

class PredictionLogger:
    def __init__(self):
        self.log_file = "prediction_logs.json"
        
    def log_prediction(self, request_id, user_features, item_features, 
                      prediction, model_version, timestamp):
        log_entry = {
            "request_id": request_id,
            "user_features": user_features,
            "item_features": item_features,
            "prediction": prediction,
            "model_version": model_version,
            "timestamp": timestamp,
            "metadata": {
                "latency_ms": 150,  # Example latency
                "source": "realtime_serving"
            }
        }
        # Write to logging system (e.g., Kafka, BigQuery)
        return log_entry

Feedback Loop for Continuous Improvement

We implement a feedback collection system:

1. Click-through rate (CTR) feedback from user interactions
2. Conversion tracking for high-value ads
3. User feedback signals (thumbs up/down, engagement duration)
4. Advertiser performance data for bid optimization

class FeedbackCollector:
    def __init__(self):
        self.feedback_queue = []
        
    def collect_feedback(self, ad_id, user_id, event_type, timestamp):
        feedback = {
            "ad_id": ad_id,
            "user_id": user_id,
            "event_type": event_type,  # click, conversion, view
            "timestamp": timestamp,
            "value": self.calculate_event_value(event_type)
        }
        self.feedback_queue.append(feedback)
        
    def calculate_event_value(self, event_type):
        # Assign weights to different event types
        weights = {
            "click": 1.0,
            "conversion": 5.0,
            "view": 0.1
        }
        return weights.get(event_type, 0.0)

Key Metrics for Drift Detection

Feature Type	Drift Metric	Threshold	Alert Level
User demographics	PSI	0.25	Critical
Item categories	KL divergence	0.1	Warning
Temporal features	KS test	0.05	Warning
Context features	PSI	0.1	Warning

Anomaly Detection for Model Outputs

We use statistical anomaly detection on model predictions:

from sklearn.ensemble import IsolationForest

class AnomalyDetector:
    def __init__(self, contamination=0.1):
        self.model = IsolationForest(contamination=contamination)
        
    def detect_anomalies(self, predictions):
        # Normalize predictions to [0,1] range
        normalized = (predictions - np.min(predictions)) / (np.max(predictions) - np.min(predictions))
        anomalies = self.model.fit_predict(normalized.reshape(-1, 1))
        return anomalies

Alerting and Escalation

• Warning level: Slack alerts with drift details
• Critical level: PagerDuty escalation to ML team
• Automated retraining: Triggered for critical drifts
• Manual review: Required for warning-level drifts in high-volume segments

Data Quality Monitoring

• Missing feature rate: Alert if >5% of requests missing features
• Feature value ranges: Validate feature values are within expected bounds
• Duplicate request detection: Track and flag repeated ad impressions

Model Card Updates

Each drift or performance change triggers an update to the model card documenting:

• Drift details and root cause analysis
• Performance degradation metrics
• Retraining status and new model version

This monitoring framework ensures continuous model health, early drift detection, and automated retraining to maintain system performance in the dynamic ad ecosystem.

9. Responsible AI & Safety

Bias Analysis

• Training data audit: Conduct comprehensive audit of ad placement data to identify potential demographic, geographic, or categorical biases. Analyze distribution of ads by industry, gender, age group, and region to detect over/under-representation.
• Slice-based fairness evaluation: Evaluate model performance across user segments (age groups, gender, geographic regions) to detect disparate impact. Monitor for bias in click-through rates (CTR) and conversion rates by demographic slices.

Fairness Metrics and Constraints

• Primary fairness metric: Demographic parity (equalized exposure across user groups) with target tolerance of ±2% difference in ad exposure rates
• Secondary metric: Equalized odds (same probability of positive outcome across slices)
• Decision rationale: Demographic parity chosen over equalized odds as it's more aligned with business goal of equitable ad distribution while maintaining revenue targets

Explainability Requirements and Approach

• SHAP values: Implement SHAP-based explainability for key ad ranking decisions, particularly for high-value advertisers
• Attention visualization: For transformer-based models, provide attention weight visualizations to show which features influenced ranking decisions
• Stakeholder requirements: Advertisers need transparency in targeting logic; internal teams require model interpretability for debugging and compliance

For LLM Systems (if applicable)

• Content filtering: Implement content safety filters to prevent inappropriate ad content, including profanity detection and category-based restrictions
• Output validation: Validate that ranked ads comply with brand safety policies and regulatory requirements (e.g., health, finance, alcohol)
• Rate limiting: Enforce rate limits on ad serving to prevent spam or excessive targeting of specific user segments
• Red teaming: Regular adversarial testing to identify potential bias in ad targeting and content generation

Model Card Documentation Plan

• Content: Document model architecture, training data sources, performance metrics, fairness considerations, and deployment details
• Frequency: Update model cards quarterly or with major model releases
• Distribution: Make available to internal stakeholders, compliance teams, and regulatory bodies as needed

Incident Response Plan for Model Failures or Harmful Outputs

• Failure detection: Monitor for sudden drops in CTR, conversion rates, or user engagement that may indicate model drift or bias
• Escalation protocol:
  • Critical issues (≥5% CTR drop): Immediate model rollback and investigation
  • Moderate issues (2-5% CTR drop): Model retraining with bias correction
  • Minor issues (<2% CTR drop): Flag for manual review and potential model adjustment
• Communication: Establish communication channels with business stakeholders, legal team, and compliance for incident reporting and mitigation

# Example model card snippet - fairness metrics implementation
import pandas as pd
from sklearn.metrics import demographic_parity_score, equalized_odds_score

def calculate_fairness_metrics(y_true, y_pred, sensitive_features):
    """
    Calculate fairness metrics for ad ranking system
    """
    # Demographic parity - same exposure rate across groups
    dp_score = demographic_parity_score(y_true, y_pred, sensitive_features)
    
    # Equalized odds - same true positive rates across groups  
    eo_score = equalized_odds_score(y_true, y_pred, sensitive_features)
    
    return {
        "demographic_parity": dp_score,
        "equalized_odds": eo_score,
        "fairness_violation": abs(dp_score) > 0.02  # Threshold for concern
    }

# Example usage:
# fairness_metrics = calculate_fairness_metrics(
#     y_true=actual_clicks,
#     y_pred=predicted_clicks,
#     sensitive_features=user_demographics
# )

10. Scalability & Reliability

Bottleneck Analysis per Pipeline Stage

Training Pipeline:

• Data processing: 200GB daily training data → 10x feature engineering overhead → 2TB batch processing
• Model training: 50M samples × 1000 features → ~50GB model parameters → 2-4 hours training on 8xV100
• Model serialization: 50GB model → 200MB compressed with quantization

Serving Pipeline:

• Request QPS: 10,000 → 50ms p99 latency target → 500ms total pipeline budget
• Feature lookup: 100ms per request → 200ms for 2-stage retrieval + scoring
• Model inference: 5ms p99 → 10ms for 2-stage pipeline + 5ms for re-ranking

Back-of-Envelope Calculations:

Training data size: 200GB/day × 30 days = 6TB/month
Feature embedding matrix: 1M users × 128 dimensions × 4 bytes = 512MB
Model size: 50M samples × 1000 features × 4 bytes = 200GB (uncompressed)
Serving QPS: 10,000 requests/sec × 50ms = 500,000 concurrent predictions

Scale-up Strategy

Horizontal Scaling:

• Feature store: 1000 shards for user/item features
• Model serving: 500 model replicas across 10 regions
• Batch processing: 200 workers for data pipeline parallelization

Distributed Feature Computation:

• User features: 1000 shards, 200ms per shard lookup
• Item features: 500 shards, 100ms per shard lookup

Fault Tolerance

Model Serving Fallbacks:

class ModelServingFallback:
    def __init__(self, primary_model, fallback_model):
        self.primary = primary_model
        self.fallback = fallback_model
    
    def predict_with_fallback(self, inputs):
        try:
            return self.primary.predict(inputs)
        except Exception as e:
            logger.error(f"Primary model failed: {e}")
            return self.fallback.predict(inputs)

Stale Model Serving:

• Cache model version with TTL of 24 hours
• Fallback to last known good model if newer version fails

Cold Start Handling Strategy

New User Handling:

class ColdStartHandler:
    def __init__(self, popularity_baseline, fallback_model):
        self.popularity = popularity_baseline
        self.fallback = fallback_model
    
    def rank_for_new_user(self, items):
        # Use popularity + demographic features
        if not self.is_known_user():
            return self.popularity.rank(items)
        return self.fallback.rank(items)

New Item Handling:

• Use content-based features from item metadata
• Fall back to category-level popularity ranking
• Implement item embedding warm-up with pre-trained vectors

Multi-Region Serving Strategy

Global Load Balancing:

class MultiRegionRanker:
    def __init__(self, regions):
        self.regions = regions
        self.latency_monitor = LatencyMonitor()
    
    def rank_with_optimal_region(self, user_id, items):
        region = self.select_optimal_region(user_id)
        return self.regions[region].rank(items)
    
    def select_optimal_region(self, user_id):
        # Route to region with lowest latency for user
        return self.latency_monitor.get_lowest_latency_region(user_id)

Cross-Region Model Synchronization:

• Sync model checkpoints every 6 hours
• Use consistent hashing for region assignment
• Implement region-specific A/B testing with shared metrics dashboard

Regional Capacity Planning:

Region A: 40% of QPS, 2000 replicas
Region B: 30% of QPS, 1500 replicas  
Region C: 30% of QPS, 1500 replicas

Global Consistency:

• Use distributed consensus protocol (Raft) for model versioning
• Implement read-repair mechanism for stale feature lookups
• Maintain global feature statistics for cross-region consistency checks

11. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering / policy alignment

• Cross-platform ad policy alignment: Should the same ranking model be used across web, mobile, and TV platforms? Different platforms may have different user behaviors and ad formats.
• Advertiser budget allocation: How should the system handle budget constraints from advertisers? Should it prioritize high-budget advertisers or ensure fair distribution across all advertisers?
• User privacy vs. personalization trade-off: How much user behavioral data should be used for personalization? What level of tracking is acceptable under privacy regulations?

Key ML risks

Risk	Impact	Likelihood	Mitigation
Data drift in user behavior	High (reduced model performance)	Medium	Implement real-time drift detection, automatic retraining triggers
Negative feedback loops from low-performing ads	Medium (reduced CTR, ad quality degradation)	High	Implement diversity constraints in ranking, monitor feedback loops
Cold start problem for new advertisers/items	Medium (reduced ad quality)	High	Use content-based features, fallback to popular ads
Model bias toward high-paying advertisers	Medium (reduced fairness)	Medium	Implement fairness constraints, regular slice evaluation
Model overfitting to historical data	High (reduced generalization)	Medium	Use temporal splits, regularization, cross-validation

Trade-offs made with justification

• Accuracy vs. latency: Prioritize low-latency serving (sub-100ms) over marginal accuracy gains, as users expect near-instant ad display. This justifies model simplification and pre-computation over complex real-time features.
• Freshness vs. cost: Balance between real-time feature updates and infrastructure costs. Batch features updated hourly, while critical user behavior features are updated every 15 minutes to optimize cost vs. performance.
• Model size vs. quality: Use model compression (quantization, pruning) to maintain sub-100ms serving latency while preserving >95% of model quality for production use.
• Online vs. batch training: Prioritize batch training for stability and reproducibility over online learning to avoid training instability from real-time feedback loops.

Alternatives considered and rejected

• Transformer-based models: Rejected due to high latency requirements (500ms+ inference time) and complexity for real-time serving in a high-QPS environment.
• Full online learning: Rejected due to risk of training drift and instability in real-time feedback loops, preferring batch retraining with drift detection.
• Multi-armed bandit approach: Rejected for production use due to exploration-exploitation trade-off complexity and potential revenue loss during experimentation phases.

Technical debt acknowledged

• Feature store implementation: Current design assumes all features are available in the feature store, but some real-time features may require additional infrastructure.
• Model versioning: No robust model rollback capability implemented yet, which could be critical in case of production regressions.
• Cross-platform consistency: Lack of unified model versioning across platforms may lead to inconsistent user experiences.
• Explainability gaps: Limited explainability for complex models, which may impact trust and regulatory compliance in the future.

12. Phased Implementation Roadmap

Phase 1: MVP (Weeks 1-4)

Objective: Establish baseline ranking system with rule-based fallback and basic ML model

Deliverables:

• Rule-based baseline ranking system (CTR-based, historical performance)
• Simple logistic regression model with basic features
• Online serving infrastructure for batch inference
• Basic A/B testing framework

Key Tasks:

1. Implement rule-based ranking (e.g., bid price, historical CTR, ad quality score)
2. Train initial logistic regression model with 100k training samples
3. Deploy simple model serving endpoint (TorchServe or custom)
4. Set up basic feature store with 20 core features
5. Implement shadow mode A/B testing (5% traffic)

Dependencies:

• Data pipeline ready, model training infrastructure established

Phase 2: Scale (Weeks 5-8)

Objective: Expand to real-time features, distributed training, and robust evaluation

Deliverables:

• Real-time feature computation pipeline
• Distributed training infrastructure (data parallelism)
• Multi-task model architecture (click-through, conversion, revenue)
• Comprehensive A/B testing infrastructure
• Feature store with 50+ features

Key Tasks:

1. Build real-time feature computation pipeline (Kafka + Spark)
2. Implement distributed training with PyTorch DDP
3. Deploy multi-task model (CTR + conversion + revenue)
4. Scale feature store to handle 100M+ users
5. Implement full A/B testing with proper randomization unit (user ID)
6. Set up online evaluation dashboard

Dependencies:

• Feature store and training infrastructure ready, model serving stable

Phase 3: Excellence (Weeks 9-12)

Objective: Advanced models, continuous learning, and production-grade reliability

Deliverables:

• Transformer-based ranking model with attention mechanisms
• Online learning pipeline for continuous model updates
• Production-grade monitoring and alerting system
• Multi-region serving capability (US, EU, Asia)
• Advanced explainability and fairness monitoring

Key Tasks:

1. Implement transformer-based ranking model with 200M+ parameters
2. Deploy continuous learning pipeline (daily retraining, drift detection)
3. Set up comprehensive monitoring (data drift, prediction drift, model performance)
4. Implement multi-region serving with 99.9% SLA
5. Add fairness and explainability monitoring (SHAP, demographic parity)
6. Establish model card documentation and incident response procedures

Dependencies:

• All Phase 2 components stable, production monitoring in place

Critical Path Dependencies:

1. Data Pipeline - Must be complete before Phase 2
2. Feature Store - Critical for real-time serving in Phase 2
3. Model Serving - Must support A/B testing before Phase 2
4. Training Infrastructure - Required for distributed training in Phase 2
5. A/B Testing Framework - Essential for validating Phase 1 and 2 models

Risk Mitigation:

• Phase 1: Start with simple model to reduce complexity and allow faster iteration
• Phase 2: Implement shadow mode before full A/B test to validate performance
• Phase 3: Gradual rollout with multi-region support to ensure reliability

Success Metrics:

• Phase 1: 20% improvement in CTR vs baseline
• Phase 2: 30% improvement in conversion rate, 15% reduction in latency
• Phase 3: 40% improvement in revenue per impression, 95% model accuracy retention