AI-Driven Engagement Metrics System for Video Platforms

Modern video platforms require sophisticated AI systems to measure and optimize user engagement through precise watch time and session length tracking, enabling data-driven content and recommendation strategies.

1. Problem Statement & ML Task Definition

Clarifying questions

1. What is the target user population and content type? Are we optimizing for video content, articles, or mixed media?
2. What is the latency budget for real-time engagement predictions? Is this a batch scoring system or must we support sub-100ms response times?
3. What is the expected QPS for serving predictions? This impacts model size, caching strategy, and infrastructure choice.
4. What is the primary business metric we're optimizing for? Is it total watch time, session length, or engagement rate (e.g., % of content consumed)?
5. What is the training data size and feature count? This affects model complexity, embedding dimensions, and distributed training strategy.
6. How important is the trade-off between accuracy and speed? For example, can we accept a 5% drop in AUC for a 3x latency improvement?
7. What is the expected data freshness requirement? Do we need real-time feature updates or can we batch features every hour?

Business use case and success criteria

The system aims to predict user engagement metrics (watch time, session length) to optimize content delivery and personalization. The primary goal is to increase user retention by recommending content that maximizes engagement duration.

Success criteria:

• Offline: AUC-ROC > 0.85, NDCG@10 > 0.75 for ranking tasks, RMSE < 0.3 for regression
• Online: 15% improvement in average watch time, 10% increase in session length, 20% reduction in early exits (content skipped before 25%)
• Business: 5% increase in user retention rate over 3 months, 10% increase in content consumption time

ML problem formulation

This is a regression task for predicting continuous engagement metrics (watch time, session length) with the following justifications:

1. Continuous output: Engagement metrics are naturally continuous and can vary from seconds to hours
2. Monotonic relationship: Longer content consumption is generally preferred by users
3. Fine-grained optimization: Regression allows for precise prediction of engagement levels, enabling better content recommendations
4. Business alignment: Revenue and retention improvements correlate directly with longer engagement times

ML evaluation metrics

Offline Evaluation:

• RMSE (Root Mean Square Error): Measures prediction accuracy for continuous engagement time
• MAE (Mean Absolute Error): Robust to outliers in engagement duration predictions
• R² Score: Indicates proportion of variance explained by the model

Online Evaluation:

• CTR (Click-Through Rate): For content recommendation systems
• Engagement Duration: Average watch time or session length
• User Retention Rate: Percentage of users returning after content consumption
• Content Completion Rate: % of users who finish consumed content

Scale and constraints

Back-of-envelope estimates:

• Training data: 50M user-item interactions (100GB raw data)
• Features: 50K+ features (user demographics, content metadata, temporal signals)
• Embedding matrix size: 1M users × 128-dim embeddings = ~500MB
• Serving QPS: 10K QPS → 36M requests/hour (assuming 50ms p99 latency target)
• Model size: ~1GB for DNN model with 50M parameters

Latency budget breakdown:

• Feature computation: <20ms (batch + real-time)
• Model inference: <50ms (p99) for 10K QPS
• Total end-to-end: <100ms

Non-ML baseline strategy

Rule-based fallback:

• Cold start handling: Default to popular content or trending items for new users/items
• Content diversity: Ensure variety in recommendations even when model confidence is low
• Engagement threshold: If predicted engagement < 10 seconds, fall back to content with >50% completion rate

Fallback logic:

def get_engagement_prediction(user_id, content_id, model_output):
    if model_output is None or model_output < 10:
        # Fallback to popular content
        return get_popular_content(user_id)
    return model_output

2. Data Strategy

Training Data Sources

The training data will be sourced from multiple complementary streams:

1. Explicit Feedback: User ratings, likes/dislikes, watch completion rates, and explicit engagement signals (e.g., "I want to see more like this")
2. Implicit Signals:
   • Watch time duration and session length (primary target variables)
   • Scroll behavior, pause/resume patterns, seek behavior
   • Time spent on content pages before engagement
3. Third-Party Data:
   • Content metadata (genre, duration, release year, cast/crew)
   • User demographic and geographic data
   • Content popularity signals from external platforms

Labeling Pipeline Design

The labeling pipeline will use a hybrid approach:

1. Programmatic Labeling:

   • For watch time and session length, we'll use natural thresholds (e.g., >50% of content watched = "engaged")
   • For session length, we'll categorize into bins (short < 2 mins, medium 2-10 mins, long > 10 mins)
   • These labels will be generated from existing user behavior logs

2. Weak Supervision:

   • Use heuristics to label edge cases (e.g., users who watch 90% of content but don't finish = partially engaged)
   • Apply noisy label correction using Snorkel-style weak supervision

3. Human Annotation:

   • For edge case validation and model refinement
   • Sample 5% of sessions for manual labeling to validate model predictions

Class Imbalance Handling Strategy

The engagement distribution is highly skewed:

• 80% of users have short sessions (<5 mins)
• 15% have medium sessions (5-30 mins)
• 5% have long sessions (>30 mins)

Strategy:

• Use weighted loss functions (inverse frequency weighting)
• Apply stratified sampling during training to ensure representation of all engagement levels
• Implement Focal Loss for long-tail engagement classes

Negative Sampling Approach

For session length prediction, we'll use:

1. Time-based negative sampling:
   • Sample sessions with significantly different durations (e.g., 5 min vs 30 min)
2. Content-based sampling:
   • Sample from content categories with different engagement patterns (e.g., documentaries vs entertainment)
3. User-based sampling:
   • Sample from users with different engagement profiles (high vs low engagement)

Data Freshness and Update Frequency

Data Type	Source	Update Frequency	Retention
User behavior logs	Real-time stream	Every 5 mins	30 days
Content metadata	ETL pipeline	Daily	Permanent
User demographics	API sync	Weekly	Permanent
Engagement labels	Derived from logs	Every 15 mins	30 days

Privacy Compliance

All data processing will comply with:

• GDPR: Anonymization of PII, consent management, right to deletion
• CCPA: User opt-out capability, data access requests

Implementation:

• Apply differential privacy to user features (ε=0.1)
• Use k-anonymity for content categorization
• Implement data retention policies with automatic deletion after 30 days

Data Quality Validation and Monitoring

1. Data Validation Checks:

   • Session duration > 0 and < 24 hours
   • Watch time ≤ content duration
   • User IDs are valid and not duplicated

2. Monitoring Framework:

   • Daily data quality reports (missing values, outliers)
   • Real-time anomaly detection for engagement patterns
   • Feature drift monitoring using PSI and KL divergence

3. Data Drift Detection:

   • Weekly drift analysis for user demographics and content metadata
   • Monthly evaluation of session length distribution shifts

import pandas as pd
from scipy import stats
import numpy as np

def detect_drift_baseline(current_data, reference_data, feature):
    """
    Detect drift between current and reference data using KL divergence
    """
    # Normalize distributions for comparison
    current_hist, _ = np.histogram(current_data[feature], bins=50, density=True)
    reference_hist, _ = np.histogram(reference_data[feature], bins=50, density=True)
    
    # Calculate KL divergence
    kl_div = stats.entropy(current_hist, reference_hist)
    
    return kl_div > 0.1  # Threshold for drift detection

# Example usage for session length drift detection
def validate_session_length(session_df):
    """
    Validate session length data quality
    """
    # Check for invalid durations
    invalid_sessions = session_df[
        (session_df['session_length'] <= 0) | 
        (session_df['session_length'] > 24 * 60)  # More than 24 hours
    ]
    
    if not invalid_sessions.empty:
        print(f"Found {len(invalid_sessions)} invalid session durations")
        
    # Check for outliers using IQR method
    Q1 = session_df['session_length'].quantile(0.25)
    Q3 = session_df['session_length'].quantile(0.75)
    IQR = Q3 - Q1
    outliers = session_df[
        (session_df['session_length'] < Q1 - 1.5 * IQR) |
        (session_df['session_length'] > Q3 + 1.5 * IQR)
    ]
    
    return len(outliers) == 0

3. Feature Engineering

Feature Taxonomy

Features are organized into four main categories:

1. User Features: User demographics, behavior patterns, preferences, and engagement history
2. Item Features: Content metadata, category, popularity, and temporal properties
3. Context Features: Time of day, device type, geographic location, and session context
4. Interaction Features: User-item interaction signals, temporal decay, and sequence patterns

Embedding Design

Entity Embeddings:

• User ID embeddings: 128-dimensional, trained via click-through rate prediction
• Content ID embeddings: 256-dimensional, learned through content recommendation objectives

Text Embeddings:

• Title and description embeddings: 384-dimensional using BERT-base
• Category embeddings: 64-dimensional one-hot encoded with learned weights

Graph Embeddings:

• User-content interaction graph embeddings: 128-dimensional using Node2Vec
• Content similarity graph: 64-dimensional using GraphSAGE

Real-time vs Batch Feature Computation

Feature Type	Computation Mode	Reason
User session count	Batch	Computed daily, no real-time need
Content popularity	Batch	Updated hourly, stable signals
User preference scores	Real-time	Required for live ranking
Temporal decay factors	Real-time	Time-sensitive, computed per request

Feature Store Design

Online Store:

• Low-latency Redis-based store for real-time serving (p99 < 10ms)
• Supports key-value lookups with TTL management

Offline Store:

• Parquet-based data lake for training and batch processing
• Partitioned by date, optimized for query performance

Feature Freshness and TTL

Feature	Source	Update Frequency	Store Type
User session count	Analytics DB	Daily	Batch
Content engagement rate	ML Pipeline	Hourly	Online
User preference vector	Model Inference	Real-time	Online
Content category	Content DB	Daily	Batch
Time of day	System clock	Per request	Real-time

Cross Features and Feature Interactions

Cross features are engineered to capture complex user-content dynamics:

• User-Content Interaction: User preference × Content category
• Temporal Patterns: Time of day × User session count
• Behavioral Sequences: Previous content engagement × Current content popularity

Feature Importance and Selection

Feature importance is determined through:

1. SHAP values from XGBoost baseline model
2. Permutation importance from trained DNN models
3. Variance-based selection for high-cardinality features

High cardinality features (user IDs, content IDs) are handled through embedding lookup rather than direct encoding to reduce dimensionality and improve generalization.

import torch
import torch.nn as nn
import torch.nn.functional as F

class FeatureProcessor(nn.Module):
    def __init__(self, user_emb_dim=128, content_emb_dim=256):
        super().__init__()
        self.user_embedding = nn.Embedding(1000000, user_emb_dim)
        self.content_embedding = nn.Embedding(500000, content_emb_dim)
        self.category_embedding = nn.Embedding(100, 64)
        
    def forward(self, user_id, content_id, category_id):
        # Embedding lookups
        user_emb = self.user_embedding(user_id)
        content_emb = self.content_embedding(content_id)
        category_emb = self.category_embedding(category_id)
        
        # Feature combination
        combined_features = torch.cat([
            user_emb, 
            content_emb, 
            category_emb
        ], dim=-1)
        
        return combined_features

4. Model Architecture

Model Selection with Justification

For engagement prediction (watch time, session length), I select a two-tower neural network architecture with multi-task learning capabilities. This choice balances:

• Accuracy: Deep networks can capture complex user-item interactions
• Scalability: Two-tower design enables efficient training and serving
• Flexibility: Multi-task learning can optimize for both watch time and session length simultaneously

Comparison Matrix:

Model	Pros	Cons	Fit for Engagement
Logistic Regression	Fast, interpretable	Limited expressivity	❌
GBM (XGBoost/LightGBM)	Good baseline, feature importance	Slower inference, less scalable	⚠️
DNN (Two-Tower)	High accuracy, scalable, handles sparse features	More complex to train	✅
Transformer	Strong sequential modeling	High latency, complex training	⚠️

Architecture Diagram

graph TD
    A[User Features] --> B[User Tower]
    C[Item Features] --> D[Item Tower]
    B --> E[Merged Representation]
    D --> E
    E --> F[Multi-Task Head]
    F --> G[Watch Time Regressor]
    F --> H[Session Length Regressor]
    I[Context Features] --> J[Context Encoder]
    J --> F

Two-Tower / Multi-Task Design

The model uses a two-tower architecture where:

• User Tower: Processes user features (demographics, historical behavior, preferences)
• Item Tower: Processes item features (content metadata, category, popularity)
• Multi-Task Head: Predicts both watch time and session length

This design enables:

1. Shared representation learning for better generalization
2. Task-specific adaptation through separate output layers
3. Joint optimization of both engagement metrics

Pre-trained Model Selection and Fine-tuning Strategy

For content features, I propose using a pre-trained BERT model fine-tuned on:

• Video title embeddings
• Description embeddings
• Category embeddings

This approach leverages pre-trained language understanding while adapting to the specific engagement domain.

Model Complexity vs Latency Trade-off Analysis

Model	Parameters	Inference Time (ms)	Memory Usage (GB)	Accuracy (NDCG@10)
Two-Tower Baseline	5M	12	0.8	0.65
Two-Tower + BERT	25M	28	2.4	0.72
Transformer (Simplified)	15M	45	3.2	0.70

Offline Evaluation Baseline Framing

Baseline: Rule-based model using:

• Popularity-based ranking (most watched content)
• User preference heuristics (recently watched, liked items)

Expected Performance:

• NDCG@10: ~0.65 (baseline)
• RMSE for watch time: ~25 mins
• RMSE for session length: ~15 mins

Core Model Code Snippet

import torch
import torch.nn as nn
import torch.nn.functional as F

class TwoTowerEngagementModel(nn.Module):
    def __init__(self, user_dim=512, item_dim=512, context_dim=128, 
                 hidden_dims=[256, 128], dropout_rate=0.2):
        super().__init__()
        
        # User tower
        self.user_tower = nn.Sequential(
            nn.Linear(user_dim, 256),
            nn.ReLU(),
            nn.Dropout(dropout_rate),
            nn.Linear(256, 128),
            nn.ReLU()
        )
        
        # Item tower
        self.item_tower = nn.Sequential(
            nn.Linear(item_dim, 256),
            nn.ReLU(),
            nn.Dropout(dropout_rate),
            nn.Linear(256, 128),
            nn.ReLU()
        )
        
        # Context encoder
        self.context_encoder = nn.Sequential(
            nn.Linear(context_dim, 64),
            nn.ReLU()
        )
        
        # Multi-task head
        self.watch_time_head = nn.Sequential(
            nn.Linear(128 + 64, 64),
            nn.ReLU(),
            nn.Linear(64, 1)
        )
        
        self.session_length_head = nn.Sequential(
            nn.Linear(128 + 64, 64),
            nn.ReLU(),
            nn.Linear(64, 1)
        )
        
    def forward(self, user_features, item_features, context_features):
        # Encode user and item representations
        user_emb = self.user_tower(user_features)
        item_emb = self.item_tower(item_features)
        
        # Combine representations
        combined = torch.cat([user_emb, item_emb], dim=1)
        
        # Encode context
        context_emb = self.context_encoder(context_features)
        
        # Multi-task prediction
        combined_with_context = torch.cat([combined, context_emb], dim=1)
        
        watch_time_pred = self.watch_time_head(combined_with_context)
        session_length_pred = self.session_length_head(combined_with_context)
        
        return watch_time_pred, session_length_pred
    
    def compute_loss(self, watch_time_pred, session_length_pred, 
                     watch_time_true, session_length_true):
        # Watch time regression loss (MSE)
        watch_loss = F.mse_loss(watch_time_pred.squeeze(), watch_time_true.squeeze())
        
        # Session length regression loss (MSE)
        session_loss = F.mse_loss(session_length_pred.squeeze(), session_length_true.squeeze())
        
        # Combined loss (weighted average)
        total_loss = 0.7 * watch_loss + 0.3 * session_loss
        
        return total_loss, watch_loss, session_loss

# Example usage
model = TwoTowerEngagementModel()
user_feat = torch.randn(32, 512)  # batch_size x user_dim
item_feat = torch.randn(32, 512)  # batch_size x item_dim
context_feat = torch.randn(32, 128)  # batch_size x context_dim

watch_pred, session_pred = model(user_feat, item_feat, context_feat)
loss, watch_loss, session_loss = model.compute_loss(
    watch_pred, session_pred,
    torch.randn(32, 1), torch.randn(32, 1)
)

print(f"Total Loss: {loss.item():.4f}")

This implementation provides:

• Modular architecture for easy extension
• Multi-task learning capability
• Clear separation of concerns (user/item towers, context processing)
• Concrete loss computation for regression tasks
• Production-ready tensor operations using PyTorch

The model is designed to be served in production with efficient inference and supports both watch time and session length prediction as separate but related tasks.

5. Serving & Inference Architecture

System Diagram

graph TD
    A[User Request] --> B[Load Balancer]
    B --> C[Model Serving Gateway]
    C --> D{Online/Offline}
    D -->|Online| E[Real-time Feature Store]
    D -->|Offline| F[Batch Feature Store]
    E --> G[Embedding Cache]
    G --> H[Model Server]
    F --> I[Feature Processing]
    I --> H
    H --> J[Scoring Engine]
    J --> K[Ranking Service]
    K --> L[Response Formatter]
    L --> M[Client Response]
    
    subgraph "Serving Layer"
        C
        G
        H
        J
        K
        L
    end
    
    subgraph "Feature Layer"
        E
        F
        I
    end

Online vs Batch Serving Decision

For engagement metrics prediction, we require real-time serving due to:

• User session length and watch time are time-sensitive
• Immediate feedback loop needed for content recommendation
• Low latency requirement (< 100ms p99 for interactive experience)
• Personalized content needs to be served within user session context

Batch serving is appropriate for:

• Historical engagement analysis
• Weekly/monthly trend reporting
• Offline model retraining

Two-Stage Pipeline

1. Candidate Retrieval → 2. Scoring & Re-ranking

Stage 1: Candidate Retrieval

• Use approximate nearest neighbor (ANN) search for efficient item retrieval
• Leverages pre-computed item embeddings and user profiles
• Returns top-K candidates (e.g., 100 items) based on similarity

Stage 2: Scoring & Re-ranking

• Apply trained model to compute engagement scores for candidate items
• Re-rank candidates based on predicted watch time/session length
• Apply business rules (e.g., diversity, freshness constraints)

Serving Infrastructure

We use TorchServe as the model serving framework with these key features:

1. Model Versioning: Supports multiple model versions for A/B testing
2. Multi-Model Support: Can serve both ranking and classification models
3. GPU Acceleration: Leverages GPU for fast inference on large batches
4. REST API: Simple HTTP interface for integration with microservices

Justification: TorchServe provides good balance between performance and ease of deployment, supports model versioning natively, and integrates well with existing ML pipelines.

Latency Budget Breakdown

Stage	p50 Latency	p99 Latency	Budget
Feature Retrieval	2ms	10ms	15ms
Model Inference	5ms	25ms	30ms
Scoring & Ranking	3ms	15ms	20ms
Response Formatting	1ms	5ms	10ms
Total	11ms	55ms	75ms

Model Compression Strategy

We implement a multi-layer compression approach:

1. Quantization: INT8 quantization reduces model size by 75% with <2% accuracy loss
2. Pruning: Remove 30% of model weights with minimal impact on performance
3. Distillation: Use knowledge distillation to create smaller, faster student models

Trade-offs:

• Quantization: 25% model size reduction, ~1% accuracy degradation
• Pruning: 30% parameter reduction, ~2% accuracy loss
• Distillation: 50% size reduction, 1.5% accuracy degradation

Caching Strategy

Prediction Cache:

• Cache top-K recommendations for users with recent engagement
• TTL: 5 minutes for active users, 30 minutes for inactive users

Embedding Cache:

• Store user/item embeddings in memory for fast retrieval
• Memory footprint: ~2GB for 1M users × 128-dim embeddings

KV Cache for LLMs (if applicable):

• For content generation tasks, cache attention keys/values
• Reduce redundant computation in sequential content prediction

Auto-scaling and Capacity Planning

Horizontal Scaling Strategy:

• Auto-scale model servers based on QPS (queries per second)
• Target: 1000 QPS with 20% buffer for traffic spikes

Capacity Calculation:

QPS = 1000 (target) × 1.2 (buffer) = 1200 QPS
Per server capacity = 500 QPS (assuming 8 GPU servers)
Servers needed = ceil(1200 / 500) = 3 servers

Fault Tolerance:

• Fallback to rule-based model when primary model fails
• Stale model serving with graceful degradation
• Health checks every 30 seconds for all services

Cold Start Handling:

• Default engagement prediction for new users (e.g., average session length)
• Fallback to popular content ranking for new items
• Gradual transition to personalized predictions over 1000 sessions

6. Evaluation Framework

Offline Evaluation Methodology

For offline evaluation, we use a temporal split approach to ensure that our model is evaluated on future data relative to the training period, preventing data leakage. The dataset is split chronologically into training (last 6 months), validation (1 month), and test (1 month) sets. This approach is preferred over random splits to maintain temporal consistency in user behavior patterns.

Cross-validation is applied using time-based folds (e.g., 4-fold temporal cross-validation) to ensure robustness across different time periods and seasonal variations. For regression tasks predicting engagement duration, we compute RMSE and MAE as primary metrics, with R² to understand variance explained.

Slice-Based Evaluation

We define critical slices for engagement prediction evaluation:

• User segments: New vs. returning users, high-activity vs. low-activity users
• Content categories: Video type (educational, entertainment, news)
• Device types: Mobile vs. desktop, different screen sizes
• Geographic regions: US vs. Europe vs. Asia-Pacific

Each slice is evaluated with RMSE, MAE, and R² to detect performance degradation in specific user or content segments. For example, if model performance drops significantly for mobile users, it indicates potential device-specific feature gaps or data drift.

A/B Test Design

We implement a user-level randomized A/B test with the following parameters:

• Randomization unit: User ID
• Sample size calculation: Using power analysis with 80% power and 5% significance level, we estimate ~25k users per variant for detecting a 10% improvement in watch time
• Test duration: 3 weeks to capture seasonal variation and ensure statistical significance
• Primary metric: Average watch time per session (target: 10% improvement)

Secondary metrics include session count, content retention rate, and revenue per user. The test is designed to detect both direct engagement improvements and indirect business impact.

Guardrail Metrics and Blocking Conditions

To ensure system stability, we define guardrails:

• Engagement drop threshold: If average watch time decreases by more than 5% in the test group, the experiment is paused
• User satisfaction score: If average user satisfaction (from survey data) drops below 85%, the experiment is terminated
• System latency: If average inference time exceeds 200ms, the model is flagged for optimization

Shadow Mode Rollout Plan

Before full A/B testing, we implement shadow mode where the new model's predictions are logged but not served to users. Shadow mode runs for 2 weeks with 10% of traffic to validate prediction consistency and detect any unexpected behavior patterns. This ensures that model performance in real-world conditions is captured before full deployment.

Long-Term Metric Measurement

We monitor delayed conversion and engagement decay metrics:

• Delayed conversion: Measure if users who watched longer content are more likely to convert (e.g., purchase, subscribe) within 7 days of initial session
• Engagement decay: Track session length decay over time (e.g., 1-day, 7-day, 30-day retention of engagement levels)
• User lifetime value (LTV): Evaluate whether increased session length translates to higher LTV over 90-day periods

These metrics provide long-term signal validation and ensure that our model improves not just immediate engagement but also sustainable user retention and monetization.

7. Training Pipeline

Training Data Pipeline

The training data pipeline follows a multi-stage ETL process:

1. Data Collection: Raw engagement signals (watch time, session length) from user activity logs, with associated features (user demographics, content metadata, temporal context)
2. Preprocessing:
   • Feature engineering (user/item embeddings, temporal aggregations)
   • Data cleaning (removing outliers, handling missing values)
   • Label generation (watch time > threshold → engagement = 1, else 0 for binary classification)
3. Shuffling and Batching:
   • Data is shuffled using reservoir sampling to maintain distribution balance
   • Batches of 1024 samples for efficient GPU utilization

import torch
from torch.utils.data import Dataset, DataLoader

class EngagementDataset(Dataset):
    def __init__(self, features, labels):
        self.features = features
        self.labels = labels
    
    def __len__(self):
        return len(self.features)
    
    def __getitem__(self, idx):
        return {
            'user_features': torch.tensor(self.features[idx]['user'], dtype=torch.float32),
            'item_features': torch.tensor(self.features[idx]['item'], dtype=torch.float32),
            'context_features': torch.tensor(self.features[idx]['context'], dtype=torch.float32),
            'label': torch.tensor(self.labels[idx], dtype=torch.long)
        }

# Example data loading
train_dataset = EngagementDataset(train_features, train_labels)
train_loader = DataLoader(train_dataset, batch_size=1024, shuffle=True)

Distributed Training Strategy

For large-scale training on 1000+ GPUs:

• Data Parallelism: Each GPU processes a subset of the batch, gradients are synchronized via AllReduce
• Model Parallelism: For very large models (>1B parameters), model parallelism splits layers across GPUs
• Pipeline Parallelism: For transformer-based architectures, pipeline stages are distributed across devices

Training Infrastructure

• Compute Cluster: 500x A100 GPUs (40GB VRAM each) in a multi-rack setup
• Storage: 50PB SSD storage for training data, 200TB for model checkpoints
• Training Time: Expected ~3 days for full dataset training with 20 epochs
• Memory Usage: ~15GB per GPU for model + features

Hyperparameter Tuning Approach

• Search Strategy: Bayesian Optimization with Gaussian Process priors
• Key Hyperparameters:
  • Learning rate (1e-5 to 1e-2)
  • Batch size (256, 512, 1024)
  • Dropout rate (0.1 to 0.5)
  • Weight decay (1e-6 to 1e-3)
  • Number of attention heads (8, 12, 16)

Experiment Tracking and Reproducibility

• MLflow Integration:
  • Logging hyperparameters, metrics, model artifacts
  • Version control for model checkpoints and training configs
• Reproducibility:
  • Fixed random seeds (42) for all components
  • Docker containers with pinned dependencies
  • Configuration files stored in Git

Model Versioning and Registry Strategy

• Model Registry: MLflow model registry with versioned models (v1.0, v1.1, etc.)
• Versioning: Semantic versioning with major.minor.patch format
• Model Validation:
  • Offline evaluation on holdout set (AUC > 0.85)
  • Online A/B testing validation (statistical significance > 95%)

Continual / Online Learning Design

• Online Learning:
  • Incremental updates using Adam optimizer with learning rate decay
  • Data drift detection triggers retraining when PSI > 0.1
• Feedback Loop:
  • Real-time prediction logs collected every 5 minutes
  • Model retraining triggered when prediction drift exceeds threshold (AUC drop > 0.02)
• Data Freshness:
  • Daily model retraining with 30-day lookback window
  • Weekly full retraining for long-term trend adaptation

8. Monitoring & Observability

Data Drift Detection

Data drift detection is crucial for maintaining model performance over time. We implement drift detection using Population Stability Index (PSI) and Kullback-Leibler divergence for continuous features, and Chi-square tests for categorical features.

import numpy as np
from scipy import stats
import pandas as pd

def calculate_psi(expected, actual, bins=10):
    """Calculate PSI between expected and actual distributions"""
    # Ensure both arrays have same bins
    _, bin_edges = np.histogram(expected, bins=bins)
    
    # Count occurrences in each bin
    expected_counts, _ = np.histogram(expected, bins=bin_edges)
    actual_counts, _ = np.histogram(actual, bins=bin_edges)
    
    # Avoid division by zero
    expected_counts = np.where(expected_counts == 0, 1e-6, expected_counts)
    actual_counts = np.where(actual_counts == 0, 1e-6, actual_counts)
    
    psi = np.sum((actual_counts - expected_counts) * 
                 np.log(actual_counts / expected_counts))
    return psi

def detect_drift_features(current_features, reference_features, threshold=0.1):
    """Detect drift in feature distributions"""
    drift_results = {}
    
    for feature_name in current_features.columns:
        if feature_name in reference_features.columns:
            psi = calculate_psi(
                reference_features[feature_name].values,
                current_features[feature_name].values
            )
            
            drift_results[feature_name] = {
                'psi': psi,
                'drift_detected': psi > threshold
            }
    
    return drift_results

Prediction Drift and Label Drift Monitoring

We monitor prediction drift by comparing model outputs over time, and label drift through tracking distribution shifts in target variables.

Model Performance Degradation Alerting

We implement alerting thresholds for key metrics:

• AUC-ROC drop > 5% over 7-day window
• NDCG@K drop > 3% over 30-day window
• Mean absolute error increase > 10% over 7-day window

Retraining Trigger Strategy

Retraining is triggered by:

1. Scheduled retraining (every 30 days or when data volume exceeds threshold)
2. Drift-triggered (PSI > 0.15 for key features)
3. Performance-triggered (AUC drop > 5% or NDCG drop > 3%)

Logging Design

Prediction logs capture:

• Request ID, timestamp, user ID, item ID
• Input features (user, item, context)
• Model predictions and confidence scores
• Ground truth labels for offline evaluation

import json
from datetime import datetime

class PredictionLogger:
    def __init__(self, log_file_path):
        self.log_file_path = log_file_path
    
    def log_prediction(self, request_id, user_id, item_id, features, 
                      prediction, confidence, ground_truth=None):
        log_entry = {
            'request_id': request_id,
            'timestamp': datetime.now().isoformat(),
            'user_id': user_id,
            'item_id': item_id,
            'features': features,
            'prediction': prediction,
            'confidence': confidence,
            'ground_truth': ground_truth
        }
        
        with open(self.log_file_path, 'a') as f:
            f.write(json.dumps(log_entry) + '\n')

Feedback Loop for Continuous Improvement

We implement a feedback loop that:

1. Collects user engagement signals (watch time, session length) as ground truth
2. Triggers retraining when drift thresholds are exceeded
3. Updates feature importance and model weights based on new data patterns

class FeedbackLoop:
    def __init__(self, model, feature_store):
        self.model = model
        self.feature_store = feature_store
    
    def process_feedback(self, feedback_data):
        """Process user engagement feedback for model improvement"""
        # Update feature importance based on recent performance
        self.update_feature_importance(feedback_data)
        
        # Retrain model if drift thresholds exceeded
        if self.should_retrain(feedback_data):
            self.retrain_model(feedback_data)
    
    def should_retrain(self, feedback_data):
        """Determine if retraining is needed based on drift"""
        # Check for significant prediction drift
        drift_detected = self.detect_prediction_drift(feedback_data)
        
        # Check for significant label drift
        label_drift = self.detect_label_drift(feedback_data)
        
        return drift_detected or label_drift
    
    def detect_prediction_drift(self, feedback_data):
        """Detect drift in model predictions"""
        # Implementation for prediction drift detection
        return False  # Placeholder
    
    def detect_label_drift(self, feedback_data):
        """Detect drift in target variable distribution"""
        # Implementation for label drift detection
        return False  # Placeholder
    
    def update_feature_importance(self, feedback_data):
        """Update feature importance based on new feedback"""
        # Update feature importance scores
        pass
    
    def retrain_model(self, feedback_data):
        """Retrain model with new feedback data"""
        # Retraining logic
        pass

9. Responsible AI & Safety

Bias Analysis

• Training data audit: Conduct comprehensive analysis of user demographics, geographic distribution, and content categories to detect potential representation gaps or overrepresentation of certain groups.
• Slice-based fairness evaluation: Evaluate model performance across user segments (age groups, gender, region) and content categories to detect disparities in engagement predictions.
• Impact assessment: Analyze whether the model systematically under/over-predicts engagement for specific user groups or content types, potentially leading to unfair content distribution.

Fairness Metrics and Constraints

• Primary fairness metric: Demographic parity in predicted engagement scores across user segments (e.g., gender, age groups).
• Secondary metric: Equalized odds in engagement prediction between high and low engagement user subpopulations.
• Decision rationale: Demographic parity is prioritized over equalized odds to ensure equitable treatment across user segments rather than just maintaining statistical balance in predictions.

Explainability Requirements and Approach

• Stakeholder needs: Content creators require explainability to understand why certain content is recommended, while platform engineers need insights for system optimization.
• Approach:
  • Attention-based visualization for transformer models to highlight which content features drive engagement predictions.
  • SHAP values for linear and tree-based components to provide feature importance explanations.
  • Gradient-based attribution methods (e.g., Integrated Gradients) for DNN components to understand feature interactions.
• Implementation:

  # Example SHAP explanation for engagement prediction model
  import shap
  explainer = shap.Explainer(model, X_train)
  shap_values = explainer(X_test[:10])
  shap.summary_plot(shap_values, X_test[:10])

LLM Safety Guardrails

• Content filtering: Implement content safety filters to prevent recommendation of harmful or inappropriate content, including explicit material, misinformation, or hate speech.
• Output validation: Validate predicted engagement scores for outliers or implausible values that may indicate model misbehavior.
• Rate limiting: Apply rate limiting on engagement prediction requests to prevent abuse or system overload from malicious actors.
• Red teaming: Conduct adversarial testing with content safety experts to identify potential edge cases or safety gaps in the model's behavior.

Model Card Documentation Plan

• Content: Document model architecture, training data sources, performance metrics, fairness analysis, and known limitations.
• Distribution: Make model cards available to internal stakeholders (data scientists, ML engineers, product managers) and external partners as required.
• Versioning: Maintain versioned model cards that update with each model release to reflect changes in training data, performance, or safety measures.

Incident Response Plan for Model Failures

• Failure detection: Monitor for sudden drops in engagement prediction accuracy or unexpected patterns in recommendations.
• Escalation protocol:
  • Alert ML team and platform engineers when prediction drift exceeds threshold (e.g., >10% drop in AUC-ROC).
  • Activate shadow mode to route traffic through a stable baseline model while investigating root cause.
• Recovery steps:
  • Revert to previous stable model version if necessary.
  • Implement hotfixes or retraining if root cause is identified within acceptable time window (e.g., 24 hours).
• Communication: Notify relevant stakeholders (product, content, safety teams) when model failures are detected or resolved.

10. Scalability & Reliability

Bottleneck Analysis per Pipeline Stage

Training Pipeline:

• Data processing: 100GB/day training data → 500GB total, requires 2x parallel ETL with 100GB/s bandwidth
• Model training: 50M samples, 200K features → 100GB model size, 30min training on 8x A100s
• Feature engineering: 50M users × 100 features → 5TB feature store, 20ms query latency target

Serving Pipeline:

• Real-time scoring: 10K QPS → 50ms p99 latency target, requires 200 concurrent model instances
• Feature lookup: 50M users × 100 features → 5TB online store, 2ms lookup target
• Model inference: 10K QPS → 50ms latency, requires model compression to <2GB

Back-of-envelope calculations:

• Training data size: 50M samples × 200 features × 4 bytes = 40GB (compressed)
• Feature store size: 50M users × 100 features × 4 bytes = 20GB (compressed)
• Model size: 100M parameters × 4 bytes = 400MB (compressed)
• Serving QPS math: 10K QPS × 50ms = 500 seconds/sec → 500 concurrent requests

Scale-up Strategy

Horizontal Scaling:

• Serving: Deploy 50 model instances across 10 nodes with auto-scaling (2x redundancy)
• Feature store: Use distributed Redis cluster with 100GB capacity per node
• Training: Scale to 20x A100s for distributed training with data parallelism

Feature Computation Scaling:

• Batch feature computation: Use Spark with 50 executors for parallel processing
• Real-time features: Cache in Redis with TTL of 1 hour, refresh every 30 minutes

Fault Tolerance

Model Serving Fallbacks:

• Primary model: 100% traffic, fallback to rule-based engine (watch time > 30min → 1.0)
• Secondary model: 50% traffic, fallback to last known good version (30-day rolling window)
• Stale model handling: 5-minute stale threshold with warm-up period of 10 minutes

Stale Model Detection:

• Monitor prediction drift every 30 minutes
• Trigger model retraining if AUC drops below 0.75 or p99 latency exceeds 100ms
• Implement model versioning with rollback capability

Cold Start Handling Strategy

New User Handling:

• Default watch time: 15 minutes (based on historical distribution)
• Fallback to user demographic features (age, location) for personalized baseline
• Use content-based filtering with item popularity as fallback

New Item Handling:

• Default session length: 5 minutes (based on historical distribution)
• Fallback to item category features (genre, duration) for personalized baseline
• Use content-based filtering with item metadata as fallback

Fallback Logic Implementation:

def get_fallback_prediction(user_id, item_id):
    if user_id in new_users:
        return {"watch_time": 15.0, "session_length": 5.0}
    elif item_id in new_items:
        return {"watch_time": 10.0, "session_length": 3.0}
    else:
        return {"watch_time": 25.0, "session_length": 8.0}

Multi-region Serving Strategy

Regional Deployment:

• Primary region (US-East): 60% traffic, 100 model instances
• Secondary regions (EU-West, AP-Southeast): 20% each, 50 model instances per region
• Failover mechanism: Route traffic to nearest healthy region if primary region latency > 200ms

Cross-region Consistency:

• Feature store replication with 5-minute sync interval
• Model version synchronization using GitOps approach
• Global feature cache with 1-hour TTL and region-specific fallbacks

Traffic Management:

def route_request(request):
    region = get_user_region(request.user_id)
    if is_primary_region(region) and primary_region_healthy():
        return route_to_primary(request)
    else:
        return route_to_fallback_region(region, request)

Latency Budget Allocation:

Stage	p50 Latency	p99 Latency	Budget
Feature Lookup	2ms	5ms	10ms
Model Inference	30ms	50ms	60ms
Response Packaging	1ms	3ms	5ms
Total	33ms	58ms	75ms

Reliability Targets:

• 99.9% uptime for model serving
• <50ms p99 latency for real-time predictions
• <1% error rate in prediction responses
• <5% of traffic affected by regional outages

11. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering / policy alignment

• User vs item-level engagement modeling: Should we model engagement at the user-item level or session level? This affects feature design and model architecture.
• Cross-platform engagement consistency: How do we handle engagement across multiple devices/platforms? Do we need to normalize or aggregate engagement signals?
• Engagement vs. retention trade-off: Should we optimize for immediate watch time or long-term retention? This affects the choice of loss function and evaluation metrics.
• Content categorization for engagement: How do we define content categories (e.g., "educational", "entertainment") for engagement modeling? This impacts feature engineering and model segmentation.

Key ML risks

Risk	Impact	Likelihood	Mitigation
Data drift in engagement patterns	High (model performance degradation)	Medium	Implement continuous drift detection, retraining triggers, and model versioning
Cold start for new content	Medium (reduced recommendation quality)	High	Implement content-based features, collaborative filtering fallbacks, and hybrid models
Bias in engagement signal	High (unfair content promotion)	Medium	Audit training data for bias, implement fairness constraints, and use slice-based evaluation
Overfitting to short-term engagement	Medium (reduced long-term user satisfaction)	High	Use temporal splits in training, implement regularization, and monitor delayed metrics
Privacy compliance with user behavior data	High (regulatory penalties)	Medium	Implement differential privacy, anonymize features, and ensure data retention policies

Trade-offs made with justification

Trade-off	Justification
Accuracy vs. latency	Prioritize low-latency serving (50ms p99) over marginal accuracy gains to support real-time personalization and A/B testing
Freshness vs. cost	Balance real-time feature updates with infrastructure costs by using hybrid batch/real-time feature stores
Model size vs. quality	Optimize for model size to support multi-region serving and reduce latency, accepting small accuracy trade-offs
Online vs. batch training	Use online learning for real-time adaptation to user behavior changes, while maintaining batch training for stability
Personalization vs. diversity	Balance personalization with content diversity to prevent echo chamber effects and maintain user satisfaction

Alternatives considered and rejected

• Alternative 1: Simple content-based filtering: Rejected due to poor personalization capability and inability to capture user preferences dynamically
• Alternative 2: Collaborative filtering only: Rejected due to cold start problems and lack of content understanding
• Alternative 3: Pure session-based models: Rejected due to limited generalization and inability to capture long-term user preferences
• Alternative 4: Multi-task learning for multiple engagement metrics: Rejected due to complexity and potential negative transfer between tasks

Technical debt acknowledged

• Feature engineering debt: Initial feature set may not capture all engagement drivers, requiring future iterations for better signal extraction
• Model complexity debt: Early model versions may use overly simplified architectures that don't scale well to complex user behaviors
• Monitoring gap: Initial monitoring may not cover all edge cases and user segments, requiring ongoing refinement of alerting systems
• A/B testing infrastructure debt: Initial A/B test setup may not support all required experimental designs, requiring future enhancements to support multi-armed bandits and complex experiments

12. Phased Implementation Roadmap

Phase 1: MVP (Weeks 1-8)

Objective: Establish baseline engagement prediction system with rule-based fallback.

Deliverables:

• Rule-based baseline using historical session duration and watch time thresholds
• Basic feature extraction pipeline (user demographics, content metadata)
• Simple linear regression model trained on 10M historical sessions
• Online serving endpoint with 200ms p99 latency target

Dependencies:

• Data pipeline availability (Week 2)
• Feature store setup (Week 3)
• Model serving infrastructure (Week 4)

Critical Path:

1. Data ingestion and preprocessing (Week 1-2)
2. Feature engineering and storage (Week 2-3)
3. Model training and validation (Week 3-5)
4. Serving endpoint deployment (Week 5-6)
5. A/B test infrastructure setup (Week 6-7)
6. Production rollout (Week 7-8)

Phase 2: Scale (Weeks 9-16)

Objective: Expand to real-time features and distributed training.

Deliverables:

• Real-time feature computation pipeline (user behavior, content context)
• Distributed training infrastructure with data parallelism
• Multi-task model supporting both watch time and session length prediction
• Feature store with online/offline synchronization (15min TTL)
• A/B testing framework with statistical significance detection

Dependencies:

• Feature store scaling (Week 9)
• Distributed training cluster (Week 10)
• Real-time data pipeline (Week 11)
• Model versioning system (Week 12)

Critical Path:

1. Feature store expansion (Week 9-10)
2. Distributed training setup (Week 10-11)
3. Real-time feature pipeline (Week 11-12)
4. Multi-task model training (Week 12-13)
5. A/B testing infrastructure (Week 13-14)
6. Production deployment (Week 15-16)

Phase 3: Excellence (Weeks 17-24)

Objective: Advanced modeling and continuous learning capabilities.

Deliverables:

• Transformer-based model with attention over user interaction sequences
• Online learning system for continuous model updates
• Advanced drift detection and retraining triggers
• Multi-region serving with failover capability
• Comprehensive explainability dashboard for model decisions

Dependencies:

• Model compression and optimization (Week 17)
• Online learning framework (Week 18)
• Multi-region infrastructure (Week 19)
• Explainability tooling integration (Week 20)

Critical Path:

1. Model architecture design and implementation (Week 17-18)
2. Online learning system integration (Week 18-19)
3. Multi-region deployment (Week 19-20)
4. Drift detection and retraining pipeline (Week 20-21)
5. Explainability dashboard development (Week 21-22)
6. Production readiness and monitoring (Week 23-24)

Cross-Phase Dependencies:

• Feature store must support all three phases with increasing complexity
• Model versioning and registry required for all phases
• A/B testing infrastructure must be available from Phase 1
• Real-time data pipeline must scale with feature complexity

Risk Mitigation:

• Phase 1 fallback to rule-based system ensures minimal impact if ML fails
• Phase 2 includes robust monitoring and alerting for distributed systems
• Phase 3 incorporates continuous learning to prevent model drift
• All phases include comprehensive logging and observability for debugging