Twitter Facebook Feed Ranking System Design

Modern social media platforms require sophisticated ranking systems to personalize user feeds while balancing engagement, relevance, and platform goals through machine learning models and real-time processing architectures.

1. Problem Statement & ML Task Definition

Clarifying questions

1. What is the primary business objective for feed ranking? Is it engagement (likes, comments, shares), user retention, or revenue generation?
2. What is the expected QPS and latency budget? For example, can we tolerate 200ms p99 latency for real-time ranking?
3. What is the scale of users and content? For example, how many active users and posts are we dealing with?
4. What is the expected data freshness requirement? Should content be ranked based on real-time signals or can we use batched features?
5. What is the expected accuracy vs. speed trade-off? Should we prioritize top-K ranking precision or serving throughput?
6. What is the expected training data size and feature count? For example, how many user-item interactions do we have?
7. What is the success definition for the model? Is it a specific AUC-ROC threshold, NDCG improvement, or revenue lift?

Business use case and success criteria

The feed ranking system aims to improve user engagement by personalizing content for each user. The system must balance between maximizing user retention, increasing time spent on platform, and maintaining content diversity to avoid echo chambers.

Success criteria:

• Engagement rate: Increase in likes, comments, shares per user
• User retention: Improved day-7 and day-30 retention rates
• Time on platform: Average session duration increase
• Revenue impact: For monetized content, revenue per user increase

ML problem formulation

This is a ranking task with the following characteristics:

• Task type: Learning-to-rank (LTR) problem, specifically a listwise ranking approach where the entire feed is ranked as a unit
• Justification:
  • Users consume content in batches (feeds), not individually
  • The relative ordering of items matters more than absolute scores
  • We want to optimize for overall user satisfaction rather than individual item relevance

ML evaluation metrics

Offline metrics (training/validation):

• NDCG@K: Normalized Discounted Cumulative Gain for top-K ranking quality
• Precision@K: Precision at K items (e.g., 5, 10) for top-K content relevance
• AUC-ROC: For binary classification of relevant vs. irrelevant items (if using pairwise ranking)
• MAP@K: Mean Average Precision at K for multi-class ranking

Online metrics (production):

• CTR: Click-through rate on feed items
• Engagement rate: Likes, comments, shares per item
• Time spent on feed: Average time user spends viewing content
• User retention: Day-7 and day-30 retention rates

Scale and constraints

Back-of-envelope estimates:

• Training data size: 100B user-item interactions (e.g., 1M users × 100K items each)
• Number of features: 10,000–50,000 (user features + item features + context features)
• Embedding matrix memory footprint:
  • 1M users × 128-dim embeddings = ~500MB
  • 10M items × 128-dim embeddings = ~5GB
• Serving QPS math:
  • 10M daily active users, 500ms p99 latency = 20K QPS (assuming 1 request per user every 5 minutes)
  • For 100 items per feed, we need to rank 100 items per user in <500ms = ~5ms per item

Non-ML baseline strategy

Rule-based fallback:

• Cold start: For new users, use popular content or trending topics
• Content diversity: Ensure a mix of content types (text, images, videos)
• Time-based decay: Prioritize recent content with exponential decay
• User preference rules: If user has no interaction history, use default preferences based on demographics or region

# Example rule-based fallback logic (simplified)
def fallback_ranking(user_id, content_pool):
    if user_id in new_users:
        return trending_content[:10]  # Return top 10 trending items
    
    if user_id in inactive_users:
        return popular_content[:10]  # Return top 10 popular items
    
    return content_pool[:10]  # Default to first 10 items in pool

2. Data Strategy

Training Data Sources

The training data for a social media feed ranking system comes from multiple sources:

1. Explicit Feedback: Direct user actions like likes, retweets, comments, saves, and follows
2. Implicit Signals: View duration, scroll behavior, time spent on content, click-through rates (CTR), and engagement sequences
3. Third-Party Data: Content metadata, user demographics from public sources, trending topics, and content categorization from external APIs

Labeling Pipeline Design

Human Annotation: For high-value signals like content quality assessment, user intent classification, and sensitive content filtering. This is expensive but critical for training robust models.

Weak Supervision: Using heuristics and rules to generate labels at scale:

# Example weak supervision approach for content quality
def content_quality_label(content, user_history):
    if len(content) < 10:
        return "low_quality"
    elif user_engagement_rate(content) > 0.15:
        return "high_quality"
    else:
        return "medium_quality"

Programmatic Labeling: Using content classification models, spam detection systems, and automated moderation tools to label posts with confidence scores.

Class Imbalance Handling Strategy

Social media feeds naturally have class imbalance due to:

• Most posts receive little engagement
• Viral content is rare but highly impactful

Strategy: Use weighted sampling during training and apply Focal Loss for the ranking objective to reduce the impact of easy examples.

Negative Sampling Approach

Hard Negative Sampling: Sample posts that are similar to the user's preferences but received low engagement to train the model to distinguish between relevant and irrelevant content.

Diversity-aware Sampling: Ensure that negative samples span different categories to avoid overfitting to specific content types.

Data Freshness and Update Frequency Requirements

• Real-time signals: User actions (likes, follows) should be processed within 100ms
• Batch features: User profiles and item metadata updated hourly
• Content freshness: New posts should be available for ranking within 5 minutes

Privacy Compliance

• GDPR/CCPA: All user data is anonymized and pseudonymized
• Data Retention: 2-year retention policy with automatic deletion of expired data
• Consent Management: User consent tracking for data collection and processing

Data Quality Validation and Monitoring

• Data Validation Pipeline: Check for malformed content, missing fields, and inconsistent timestamps
• Monitoring Dashboard: Track data drift in user behavior patterns and content distribution
• Anomaly Detection: Flag unusual spikes in engagement or content types that may indicate data poisoning

# Data quality check example
def validate_user_action(action):
    required_fields = ['user_id', 'post_id', 'timestamp', 'action_type']
    for field in required_fields:
        if not action.get(field):
            raise ValueError(f"Missing required field: {field}")
    
    # Validate timestamp freshness (within last 30 days)
    if action['timestamp'] < datetime.now() - timedelta(days=30):
        raise ValueError("Action timestamp too old")
    
    return True

3. Feature Engineering

Feature Taxonomy

Features are categorized into four main types:

1. User Features: Demographics, engagement history, content preferences, social graph properties
2. Item Features: Content type, popularity metrics, author characteristics, temporal properties
3. Context Features: Time of day, device, location, session state
4. Interaction Features: User-item interaction history, recency, frequency, engagement signals

Embedding Design

Entity Embeddings:

• User embeddings (128-dim) trained via contrastive learning on user-item interactions
• Content/item embeddings (256-dim) from content representation models
• Topic/category embeddings (64-dim) from topic modeling

Text Embeddings:

• BERT-based sentence embeddings (768-dim) for post content
• Multi-modal embeddings combining text, image, and metadata features

Graph Embeddings:

• Node2Vec for social graph representation (128-dim)
• Heterogeneous graph embeddings for user-content relationships

Real-time vs Batch Feature Computation

Feature Type	Computation Mode	Justification
User engagement counts	Real-time	High frequency updates needed
Content popularity scores	Batch	Computed hourly, stable
Social graph features	Real-time	User relationships change frequently
Temporal features	Real-time	Time-sensitive, minute-level updates

Feature Store Design

Online Store:

• Low-latency KV store (Redis/FAISS) for real-time serving
• TTL of 1 hour for user features, 24 hours for content features

Offline Store:

• Data warehouse (Snowflake/BigQuery) for training and batch processing
• Partitioned by date for efficient querying

Feature Freshness and TTL

Feature	Source	Update Frequency	Store Type
User engagement count	Real-time stream	Every 5 min	Online
Content popularity	Batch job	Hourly	Offline
User topic preference	Real-time stream	Every 15 min	Online
Social graph centrality	Batch job	Daily	Offline
Content timestamp	Real-time stream	Every minute	Online

Cross Features and Feature Interactions

Cross features are generated using:

• User-content interaction buckets (user-topic, user-author)
• Temporal windows (last 1h, last 24h, last 7d)
• Content-type × user-engagement combinations

Cross-feature engineering uses:

def generate_cross_features(user_id, content_id, timestamp):
    # User-topic cross features
    user_topic = get_user_topic_vector(user_id)
    content_topic = get_content_topic_vector(content_id)
    
    # Temporal cross features
    time_window = timestamp.hour // 6  # 4 time buckets per day
    
    return {
        'user_topic_content_topic_sim': cosine_similarity(user_topic, content_topic),
        'time_window': time_window,
        'user_content_time_interaction': f"{user_id}_{content_id}_{time_window}"
    }

Feature Importance and Selection

Feature importance is evaluated using:

• SHAP values from gradient boosting models
• Permutation importance from trained models
• Statistical significance testing (ANOVA)

Feature selection strategy:

1. Remove features with < 0.01 SHAP importance
2. Apply variance threshold (remove features with < 0.05 variance)
3. Use L1 regularization in linear models for automatic feature selection

from sklearn.feature_selection import SelectKBest, f_classif
import numpy as np

class FeatureSelector:
    def __init__(self, k=1000):
        self.k = k
        self.selector = SelectKBest(score_func=f_classif, k=k)
        
    def select_features(self, X_train, y_train):
        # Select top k features based on ANOVA F-test
        X_selected = self.selector.fit_transform(X_train, y_train)
        
        # Get selected feature indices
        selected_indices = self.selector.get_support(indices=True)
        
        return X_selected, selected_indices
    
    def get_feature_importance(self, X_train, y_train):
        # Return feature importance scores for monitoring
        return self.selector.scores_

4. Model Architecture

Model Selection with Justification

For a Twitter/Facebook feed ranking system, I recommend a two-tower neural network architecture with a multi-task learning (MTL) framework. This approach offers the best balance of scalability, interpretability, and performance for content ranking systems.

Why not other architectures:

• Logistic Regression: Too simple to capture complex user-item interactions and temporal dynamics
• GBM (XGBoost/LightGBM): Good for structured features but struggles with high-dimensional sparse embeddings and temporal sequences
• DNN: Can capture complex interactions but lacks the scalability and efficiency of two-tower architectures for large-scale systems
• Transformer: Overkill for feed ranking where the primary goal is relevance rather than understanding context, and introduces significant latency overhead

Two-tower architecture advantages:

• Efficient for large-scale embedding lookups
• Natural fit for user-item interaction modeling
• Supports multi-task learning (relevance, engagement, safety)
• Can be easily extended to include temporal dynamics

Architecture Diagram

graph TD
    A[User Features] --> B[User Tower]
    C[Item Features] --> D[Item Tower]
    B --> E[Merge Layer]
    D --> E
    E --> F[Multi-Task Head]
    F --> G[Relevance Score]
    F --> H[Engagement Prediction]
    F --> I[Safety Score]

Two-Tower / Multi-Task Design

The model consists of:

1. User Tower: Processes user features (demographics, behavior history, preferences)
2. Item Tower: Processes content features (text, media, metadata)
3. Multi-Task Head: Predicts multiple objectives simultaneously

Core Model Code Snippet

import torch
import torch.nn as nn
import torch.nn.functional as F

class TwoTowerRanker(nn.Module):
    def __init__(self, user_emb_dim=128, item_emb_dim=128, 
                 hidden_dims=[256, 128], num_tasks=3):
        super().__init__()
        
        # User tower
        self.user_embedding = nn.Embedding(1000000, user_emb_dim)
        self.user_mlp = nn.Sequential(
            nn.Linear(user_emb_dim, hidden_dims[0]),
            nn.ReLU(),
            nn.Linear(hidden_dims[0], hidden_dims[1]),
            nn.ReLU()
        )
        
        # Item tower
        self.item_embedding = nn.Embedding(5000000, item_emb_dim)
        self.item_mlp = nn.Sequential(
            nn.Linear(item_emb_dim, hidden_dims[0]),
            nn.ReLU(),
            nn.Linear(hidden_dims[0], hidden_dims[1]),
            nn.ReLU()
        )
        
        # Multi-task prediction heads
        self.relevance_head = nn.Linear(hidden_dims[1], 1)
        self.engagement_head = nn.Linear(hidden_dims[1], 1)
        self.safety_head = nn.Linear(hidden_dims[1], 1)
        
    def forward(self, user_ids, item_ids):
        # User tower processing
        user_emb = self.user_embedding(user_ids)
        user_repr = self.user_mlp(user_emb)
        
        # Item tower processing
        item_emb = self.item_embedding(item_ids)
        item_repr = self.item_mlp(item_emb)
        
        # Interaction
        interaction = torch.mul(user_repr, item_repr)
        
        # Multi-task predictions
        relevance_score = self.relevance_head(interaction).sigmoid()
        engagement_pred = self.engagement_head(interaction).sigmoid()
        safety_score = self.safety_head(interaction).sigmoid()
        
        return {
            'relevance': relevance_score,
            'engagement': engagement_pred,
            'safety': safety_score
        }
    
    def compute_loss(self, user_ids, item_ids, relevance_labels, 
                     engagement_labels, safety_labels):
        predictions = self.forward(user_ids, item_ids)
        
        # Binary cross-entropy losses
        relevance_loss = F.binary_cross_entropy(
            predictions['relevance'].squeeze(), 
            relevance_labels.float()
        )
        
        engagement_loss = F.binary_cross_entropy(
            predictions['engagement'].squeeze(), 
            engagement_labels.float()
        )
        
        safety_loss = F.binary_cross_entropy(
            predictions['safety'].squeeze(), 
            safety_labels.float()
        )
        
        # Combined loss with task weights
        total_loss = (
            0.6 * relevance_loss + 
            0.3 * engagement_loss + 
            0.1 * safety_loss
        )
        
        return total_loss

# Example usage for inference
def rank_feed(model, user_id, candidate_items):
    """Rank a list of candidate items for a given user"""
    model.eval()
    with torch.no_grad():
        user_tensor = torch.tensor([user_id])
        item_tensors = torch.tensor(candidate_items)
        
        predictions = model(user_tensor, item_tensors)
        
        # Combine scores with task weights
        final_scores = (
            0.6 * predictions['relevance'].squeeze() +
            0.3 * predictions['engagement'].squeeze() +
            0.1 * (1 - predictions['safety'].squeeze())  # Inverse safety score
        )
        
        return torch.argsort(final_scores, descending=True)

Model Complexity vs Latency Trade-off Analysis

Model	Parameters	Inference Latency (ms)	Memory Footprint	Notes
Two-Tower Baseline	~50M	15-20 ms	2GB	Production ready
Two-Tower + Attention	~150M	30-40 ms	4GB	Better context modeling
Two-Tower + Temporal	~80M	25-35 ms	3GB	Handles temporal dynamics
Transformer (for comparison)	~200M+	100+ ms	8GB+	Too slow for real-time

Offline Evaluation Baseline Framing

Baseline Model: Logistic regression with user and item embeddings, trained on historical interactions

• Expected AUC-ROC: ~0.75 for relevance ranking
• Expected NDCG@10: ~0.35 for top-10 ranking
• Expected Precision@K: ~0.40 for K=5

Proposed Model: Two-tower neural network with multi-task learning

• Target AUC-ROC: ~0.85+ (25% improvement)
• Target NDCG@10: ~0.50+ (43% improvement)
• Target Precision@K: ~0.55+ (38% improvement)

Pre-trained Model Selection and Fine-tuning Strategy

For text content, we can leverage pre-trained BERT or RoBERTa embeddings:

# Example of fine-tuning strategy for text features
class TextEmbeddingAdapter(nn.Module):
    def __init__(self, pretrained_model_name='bert-base-uncased'):
        super().__init__()
        self.bert = AutoModel.from_pretrained(pretrained_model_name)
        self.text_projection = nn.Linear(768, 128)  # Reduce dimensionality
        
    def forward(self, text_input_ids, attention_mask):
        bert_output = self.bert(
            input_ids=text_input_ids,
            attention_mask=attention_mask
        )
        # Use [CLS] token embedding for classification
        cls_embedding = bert_output.last_hidden_state[:, 0, :]
        return self.text_projection(cls_embedding)

Model Versioning and Deployment Strategy

• Versioning: Semantic versioning (v1.0.0, v1.1.0) with model registry
• Deployment: Canary deployment with shadow mode testing
• Rollback: Automated rollback on performance degradation
• A/B Testing: Gradual rollout with traffic percentage control

The model architecture is designed for production deployment at scale, supporting real-time inference with sub-50ms latency while maintaining high-quality ranking performance through multi-task learning and proper embedding design.

5. Serving & Inference Architecture

System Diagram

graph TD
    A[User Request] --> B[Load Balancer]
    B --> C[Feed Ranking Service]
    C --> D{Candidate Retrieval}
    D --> E[Embedding Store]
    D --> F[Item Store]
    C --> G[Scoring Service]
    G --> H[Model Server]
    H --> I[TorchServe/ONNX Runtime]
    C --> J[Re-ranking Service]
    J --> K[Final Scorer]
    K --> L[Response Formatter]
    L --> M[Feed Response]
    M --> N[User Client]

    subgraph "Real-time Serving"
        C
        G
        J
        H
        K
        L
    end

    subgraph "Feature Stores"
        E
        F
    end

Online vs Batch Serving Decision

The feed ranking system requires real-time online serving due to:

• User engagement is time-sensitive (e.g., tweets disappear after 24 hours)
• Personalization must happen per-user with low latency (<200ms p99)
• User behavior changes dynamically (new tweets, follows, likes)
• Real-time feedback loop is critical for model improvement

Batch serving would introduce unacceptable delays and make the system unresponsive to user interactions.

Two-Stage Pipeline

Candidate Retrieval → Scoring → Re-ranking

1. Candidate Retrieval Stage:

   • Retrieve top-K candidates (e.g., 1000 tweets) using approximate nearest neighbor search
   • Use pre-computed embeddings from item and user stores
   • Apply coarse filtering (e.g., content type, time window)

2. Scoring Stage:

   • Score candidates using trained model with real-time features
   • Apply ranking-specific transformations (e.g., freshness boost, diversity penalty)

3. Re-ranking Stage:

   • Apply final adjustments (e.g., user preference drift, content safety filters)
   • Ensure diversity and fairness constraints

Serving Infrastructure

Model Server: TorchServe with ONNX Runtime backend for:

• Support for both PyTorch and ONNX models
• Built-in model versioning and A/B testing capability
• GPU-accelerated inference with automatic batching

Model Format: ONNX for cross-platform compatibility and optimized inference.

Latency Budget Breakdown

Stage	p50 Latency	p99 Latency	Budget
Candidate Retrieval	5ms	15ms	20ms
Scoring	10ms	30ms	40ms
Re-ranking	5ms	20ms	25ms
Response Formatting	2ms	8ms	10ms
Total	22ms	73ms	95ms

Model Compression Strategy

Quantization:

• INT8 quantization for model parameters and activations
• Expected 4x reduction in memory footprint with <2% accuracy loss

Pruning:

• Structured pruning to remove redundant connections
• Target: 30% reduction in model size with minimal impact on ranking quality

Distillation:

• Teacher-student model approach to reduce model size while maintaining performance
• Use pre-trained large model as teacher, smaller model as student

Caching Strategy

Prediction Cache:

• Cache top-K predictions for recently active users (100K users)
• TTL: 5 minutes to balance freshness and performance
• Cache hit rate target: >80%

Embedding Cache:

• Store user and item embeddings in LRU cache
• Size: 1GB for 1M embeddings (256-dim float32)
• Eviction policy: Least Recently Used

KV Cache for LLMs:

• Not applicable for typical feed ranking (no transformer-based generation)
• For future LLM integration: 100MB KV cache per user session

Auto-scaling and Capacity Planning

Horizontal Scaling:

• Auto-scale based on QPS with target: 1000 QPS per instance
• Target: 5000 concurrent users with 20ms p99 latency

Capacity Planning:

• Peak QPS: 15,000 requests/sec (estimated)
• Model server instances: 30-50 GPU instances (8x V100s)
• Feature store: 100GB memory for embeddings and features

Graceful Degradation:

• Fallback to rule-based ranking when model serving fails
• Cache warm-up strategy for new user sessions

# Sample model inference pipeline (Python)
import torch
from torch import nn
import numpy as np

class FeedRankingModel(nn.Module):
    def __init__(self, user_emb_dim=256, item_emb_dim=256, hidden_dim=128):
        super().__init__()
        self.user_embedding = nn.Embedding(1000000, user_emb_dim)
        self.item_embedding = nn.Embedding(10000000, item_emb_dim)
        self.mlp = nn.Sequential(
            nn.Linear(user_emb_dim + item_emb_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 1)
        )
    
    def forward(self, user_ids, item_ids):
        # Get embeddings
        user_emb = self.user_embedding(user_ids)
        item_emb = self.item_embedding(item_ids)
        
        # Concatenate and score
        features = torch.cat([user_emb, item_emb], dim=1)
        scores = self.mlp(features).squeeze()
        
        return torch.sigmoid(scores)

# Example usage for serving
def rank_feed_items(user_id, candidate_items):
    model = FeedRankingModel()
    # Load model weights from storage
    
    user_tensor = torch.tensor([user_id])
    scores = []
    
    for item_id in candidate_items:
        item_tensor = torch.tensor([item_id])
        score = model(user_tensor, item_tensor)
        scores.append(score.item())
    
    # Sort items by score (descending)
    ranked_items = [item for _, item in sorted(zip(scores, candidate_items), reverse=True)]
    
    return ranked_items[:10]  # Return top 10

6. Evaluation Framework

Offline Evaluation Methodology

For offline evaluation, we use temporal cross-validation to ensure our model generalizes to future data. We split training data chronologically, with the most recent 30 days reserved for validation and 30-60 days for testing. This approach addresses the temporal nature of user behavior and content freshness, preventing data leakage from future events influencing past model training. We also implement time-based holdout splits to validate model robustness across different temporal windows.

Slice-Based Evaluation

We define critical slices for comprehensive evaluation:

• User segments: New vs. active users, high-engagement vs. low-engagement users
• Content types: Text posts, images, videos, links, retweets
• Geographic regions: US vs. international, urban vs. rural users
• Device types: Mobile vs. desktop, different screen sizes
• Time of day: Morning vs. evening users

Each slice requires at least 1000 daily interactions for statistically significant evaluation. We monitor performance degradation across slices to detect potential bias or model drift.

A/B Test Design

We design a multi-armed bandit A/B test with the following specifications:

• Randomization unit: User ID (to prevent leakage between users)
• Sample size calculation: Using power analysis with 80% power and 5% significance level, requiring ~25,000 users per variant
• Test duration: 30 days for stable signal detection, with early stopping capability if confidence exceeds 95%
• Primary metric: Click-through rate (CTR) with 95% confidence intervals

Guardrail Metrics and Blocking Conditions

To ensure system stability, we implement guardrails:

• Minimum CTR threshold: Model must maintain at least 90% of baseline CTR
• Engagement drop threshold: If average session duration drops by more than 15%, automatic rollback
• Error rate cap: If inference errors exceed 0.1% of requests, system fallback to rule-based model

Shadow Mode Rollout Plan

Before full A/B testing, we run shadow mode for 7 days where:

1. All traffic is routed to both baseline and new model
2. New model predictions are logged but not served to users
3. Performance comparison is conducted on logged data
4. If new model shows >5% improvement in CTR, proceed to A/B test

Long-Term Metric Measurement

We track delayed conversion metrics for robust evaluation:

• 7-day retention: User engagement within 7 days of feed exposure
• Engagement decay: Session duration normalized to first day vs. 7-day average
• Revenue impact: Ad revenue per user over 30-day period
• User satisfaction: NPS scores collected via in-app surveys

# Example A/B test evaluation code snippet
import numpy as np
from scipy import stats

class ABTestEvaluator:
    def __init__(self, baseline_ctr, test_ctr, sample_size):
        self.baseline_ctr = baseline_ctr
        self.test_ctr = test_ctr
        self.sample_size = sample_size
    
    def calculate_p_value(self):
        # Two-sample proportion test for CTR comparison
        z_stat, p_value = stats.proportions_ztest(
            count=[self.baseline_ctr * self.sample_size, self.test_ctr * self.sample_size],
            nobs=[self.sample_size, self.sample_size]
        )
        return p_value
    
    def calculate_confidence_interval(self, confidence=0.95):
        alpha = 1 - confidence
        z_score = stats.norm.ppf(1 - alpha/2)
        
        # For binary data (CTR), use Wilson score interval
        p_hat = self.test_ctr
        n = self.sample_size
        
        denominator = 1 + z_score**2 / n
        center = (p_hat + z_score**2 / (2*n)) / denominator
        margin = z_score * np.sqrt((p_hat*(1-p_hat)/n + z_score**2/(4*n**2))) / denominator
        
        return (center - margin, center + margin)

This evaluation framework ensures both short-term performance and long-term user experience optimization while maintaining system reliability through guardrails and shadow testing.

7. Training Pipeline

Training Data Pipeline

The training data pipeline follows a multi-stage ETL process:

1. Data Collection: Collect user-item interactions (likes, retweets, comments, follows) and user/item metadata from various sources including:

   • Real-time interaction logs (Kafka streams)
   • Historical user behavior data (Hive/Parquet)
   • User profile information (user table)
   • Item content metadata (tweet/item table)

2. Preprocessing:

   • Normalize timestamps and handle time zones
   • Filter out spam or low-quality interactions
   • Apply temporal windows (e.g., last 30 days of data)
   • Create user/item IDs mapping to handle sparse representations

3. Shuffling and Batching:

   • Shuffle data in batches of 100K samples to prevent temporal correlation
   • Batch training data for efficient GPU utilization (batch size = 2048)
   • Apply stratified sampling to handle class imbalance

import tensorflow as tf
from tensorflow.data import Dataset

def create_training_dataset(data_paths, batch_size=2048):
    # Load from parquet files
    dataset = Dataset.from_tensor_slices(data_paths)
    
    # Shuffle and repeat for multiple epochs
    dataset = dataset.shuffle(buffer_size=100000)
    
    # Apply batching and prefetching for performance
    dataset = dataset.batch(batch_size)
    dataset = dataset.prefetch(tf.data.AUTOTUNE)
    
    return dataset

# Example usage
data_paths = ["s3://twitter-data/train_0.parquet", "s3://twitter-data/train_1.parquet"]
train_dataset = create_training_dataset(data_paths)

Distributed Training Strategy

For large-scale training, we implement data parallelism with model parallelism for the embedding layers:

1. Data Parallelism:

   • Split training data across multiple workers (e.g., 8 GPUs)
   • Each worker processes a subset of the training data
   • Gradient synchronization using AllReduce

2. Model Parallelism:

   • Embedding tables stored on separate GPUs
   • Forward pass with embedding lookup parallelization

3. Infrastructure:

   • 8x NVIDIA A100 GPUs (40GB VRAM each)
   • 256GB total memory capacity
   • 100 Gbps InfiniBand interconnect for synchronization

Training Infrastructure

• Cluster Sizing:

  • 8x A100 GPUs (32GB VRAM each) for model training
  • 16x CPU cores (Intel Xeon Platinum) for data preprocessing
  • 256GB RAM per node for large feature caching

• Training Time Estimation:

  • Training data: ~50M samples (20GB compressed)
  • Model size: ~5GB (embedding tables + dense layers)
  • Estimated training time: 4-6 hours on 8x A100 GPUs
  • Epochs required: 5-10 epochs for convergence

Hyperparameter Tuning Approach

We use Bayesian optimization with the following key hyperparameters:

1. Learning Rate: 1e-4 to 1e-2 (log scale)
2. Batch Size: 512 to 4096 (linear scale)
3. Embedding Dimension: 64 to 256 (linear scale)
4. Number of Dense Layers: 1-3 (discrete)
5. Dropout Rate: 0.1 to 0.5 (linear scale)

from skopt import gp_minimize
from skopt.space import Real, Integer

def objective(hyperparams):
    lr, batch_size, embedding_dim = hyperparams
    
    # Initialize model with given hyperparameters
    model = create_ranking_model(
        embedding_dim=embedding_dim,
        learning_rate=lr
    )
    
    # Train for one epoch and return validation loss
    train_loss = model.train_one_epoch(batch_size=batch_size)
    
    return train_loss

# Bayesian optimization search space
search_space = [
    Real(1e-4, 1e-2, prior='log-uniform', name='learning_rate'),
    Integer(512, 4096, name='batch_size'),
    Integer(64, 256, name='embedding_dim')
]

# Run optimization
best_params = gp_minimize(objective, search_space, n_calls=20)

Experiment Tracking and Reproducibility

We use MLflow for experiment tracking:

1. Experiment Registry:

   • Track model versions, hyperparameters, metrics
   • Store model artifacts (weights, config files)
   • Version control for training data and preprocessing steps

2. Reproducibility:

   • Save complete training configuration as YAML
   • Hash training data to ensure reproducibility
   • Use deterministic random seeds for all components

Model Versioning and Registry Strategy

• Version Control: Semantic versioning (v1.0.0, v1.1.0) based on performance improvements
• Registry: MLflow model registry with staging and production environments
• Rollback Capability: Automatic rollback to previous stable versions if performance degrades by >5%

Continual / Online Learning Design

For online learning, we implement:

1. Incremental Updates:

   • Update model weights with new samples every 24 hours
   • Use adaptive learning rate that decreases over time

2. Data Drift Detection:

   • Monitor feature distribution changes
   • Trigger retraining when drift exceeds threshold (PSI > 0.1)

3. Online Fine-tuning:

   • Use small batches (512 samples) for online updates
   • Apply learning rate decay to prevent catastrophic forgetting

class OnlineLearningEngine:
    def __init__(self, model, learning_rate=1e-4):
        self.model = model
        self.learning_rate = learning_rate
        
    def update_model(self, new_samples, epochs=1):
        """Update model with new samples"""
        for epoch in range(epochs):
            # Apply gradient updates to new samples
            self.model.train_step(new_samples, lr=self.learning_rate)
            
        # Decay learning rate for stability
        self.learning_rate *= 0.95
        
    def detect_drift(self, new_features):
        """Detect data drift using PSI"""
        psi = calculate_psi(self.model.previous_features, new_features)
        return psi > 0.1

8. Monitoring & Observability

Data Drift Detection

Data drift detection is critical for maintaining model performance in a dynamic social media environment. We implement drift detection using Population Stability Index (PSI) for categorical features and Kolmogorov-Smirnov (KS) test for continuous features.

import numpy as np
from scipy import stats
from sklearn.metrics import log_loss

class DriftDetector:
    def __init__(self, threshold=0.1):
        self.threshold = threshold
        
    def psi(self, old_dist, new_dist):
        """Calculate Population Stability Index"""
        # Ensure no zero probabilities
        old_dist = np.clip(old_dist, 1e-6, 1)
        new_dist = np.clip(new_dist, 1e-6, 1)
        
        psi_value = np.sum((new_dist - old_dist) * np.log(new_dist / old_dist))
        return psi_value
    
    def ks_test(self, old_data, new_data):
        """Kolmogorov-Smirnov test for continuous features"""
        ks_stat, p_value = stats.ks_2samp(old_data, new_data)
        return ks_stat, p_value

# Example usage for feature drift detection
drift_detector = DriftDetector()
user_age_drift = drift_detector.psi(
    old_dist=np.histogram(old_user_ages, bins=20)[0],
    new_dist=np.histogram(new_user_ages, bins=20)[0]
)

Prediction Drift and Label Drift Monitoring

Prediction drift monitors changes in model outputs, while label drift tracks shifts in ground truth distributions. These are monitored via:

• Prediction drift: Track distribution shift of model scores using KS test
• Label drift: Monitor change in engagement rate distribution over time

Model Performance Degradation Alerting

We implement alerting for model performance degradation with thresholds:

class PerformanceMonitor:
    def __init__(self):
        self.alert_thresholds = {
            'auc_drop': 0.05,  # 5% drop in AUC
            'ctr_drop': 0.10,  # 10% drop in CTR
            'log_loss_increase': 0.05  # 5% increase in log loss
        }
        
    def check_degradation(self, current_metrics, baseline_metrics):
        alerts = []
        for metric_name, threshold in self.alert_thresholds.items():
            if current_metrics[metric_name] < baseline_metrics[metric_name] * (1 - threshold):
                alerts.append({
                    'metric': metric_name,
                    'current': current_metrics[metric_name],
                    'baseline': baseline_metrics[metric_name],
                    'alert': True
                })
        return alerts

# Example degradation check
monitor = PerformanceMonitor()
alerts = monitor.check_degradation(
    current_metrics={'auc': 0.78, 'ctr': 0.025},
    baseline_metrics={'auc': 0.83, 'ctr': 0.028}
)

Retraining Trigger Strategy

We implement a hybrid retraining trigger system:

• Scheduled retraining: Every 2 weeks for model updates
• Drift-triggered: When PSI > 0.15 or KS p-value < 0.05 for key features
• Performance-triggered: When AUC drops by > 5% or CTR drops by > 10%

Logging Design

Prediction logs capture key information for model monitoring:

import json
from datetime import datetime

class PredictionLogger:
    def __init__(self, log_file="prediction_logs.json"):
        self.log_file = log_file
        
    def log_prediction(self, user_id, item_id, score, features, timestamp=None):
        log_entry = {
            "timestamp": timestamp or datetime.now().isoformat(),
            "user_id": user_id,
            "item_id": item_id,
            "score": score,
            "features": features,
            "prediction_metadata": {
                "model_version": "v2.1",
                "request_id": "req_12345"
            }
        }
        
        with open(self.log_file, "a") as f:
            f.write(json.dumps(log_entry) + "\n")

# Example logging
logger = PredictionLogger()
logger.log_prediction(
    user_id="user_123",
    item_id="post_456",
    score=0.87,
    features={"user_age": 28, "engagement_history": [1, 0, 1, 1]}
)

Feedback Loop for Continuous Improvement

We implement a feedback collection pipeline:

class FeedbackCollector:
    def __init__(self):
        self.feedback_storage = {}
        
    def collect_feedback(self, user_id, item_id, engagement_type, timestamp):
        """Collect user feedback on ranked items"""
        feedback_key = f"{user_id}_{item_id}"
        if feedback_key not in self.feedback_storage:
            self.feedback_storage[feedback_key] = []
            
        self.feedback_storage[feedback_key].append({
            "engagement_type": engagement_type,
            "timestamp": timestamp,
            "user_id": user_id,
            "item_id": item_id
        })
        
    def get_feedback_stats(self, time_window_hours=24):
        """Aggregate feedback over specified time window"""
        recent_feedback = []
        for key, feedback_list in self.feedback_storage.items():
            recent = [f for f in feedback_list 
                     if (datetime.now() - datetime.fromisoformat(f['timestamp'])).hours < time_window_hours]
            recent_feedback.extend(recent)
            
        engagement_stats = {}
        for fb in recent_feedback:
            etype = fb['engagement_type']
            engagement_stats[etype] = engagement_stats.get(etype, 0) + 1
            
        return engagement_stats

# Example feedback collection
feedback_collector = FeedbackCollector()
feedback_collector.collect_feedback(
    user_id="user_123",
    item_id="post_456",
    engagement_type="click",
    timestamp=datetime.now().isoformat()
)

Key Metrics for Monitoring

Metric	Source	Frequency	Alert Threshold
AUC-ROC	Offline evaluation	Daily	Drop > 5%
CTR	Online serving	Hourly	Drop > 10%
Log Loss	Online serving	Hourly	Increase > 5%
PSI	Feature drift	Daily	> 0.15
KS Test p-value	Feature drift	Daily	< 0.05

Model Card Integration

We maintain model cards that include:

• Performance metrics over time
• Data drift statistics
• Feature importance changes
• Retraining history and triggers

Incident Response Plan

When drift or degradation alerts are triggered:

1. Immediate: Pause model serving and fallback to baseline
2. Analysis: Investigate root cause (data drift vs model degradation)
3. Remediation: Retrain or re-deploy model with updated data
4. Verification: Run A/B test to validate improvements

This monitoring framework ensures robust model performance and enables proactive maintenance of the feed ranking system in a rapidly changing social media environment.

9. Responsible AI & Safety

Bias Analysis

Training data audit reveals potential demographic imbalances in content distribution, particularly affecting underrepresented communities. Slice-based fairness evaluation shows 15% lower engagement for minority user groups in initial model versions. Key bias metrics include:

• Demographic parity: 85% of users from all segments receive similar content exposure
• Equalized odds: 92% recall rate across user demographics for relevant content

Fairness Metrics and Constraints

Implement demographic parity constraints with 5% tolerance threshold to ensure equitable content distribution. Equalized odds constraint applied to minimize false positive rates across user segments. Fairness-aware loss function modification with fairness weight parameter (λ = 0.1) to balance accuracy and fairness trade-offs.

Explainability Requirements

SHAP value analysis for top 100 most influential features to explain content ranking decisions. Attention visualization for transformer-based models to show which user-content interactions drive scores. LIME-style local explanations for individual predictions with confidence intervals.

For LLM Systems: Safety Guardrails

Content filtering pipeline using multi-layered classifiers (toxicity, misinformation, spam) with 98% precision threshold. Output validation system that flags potentially harmful content for human review before serving. Rate limiting implemented at 100 requests per minute per user to prevent abuse.

Model Card Documentation Plan

Comprehensive model card including:

• Performance metrics by demographic slice
• Training data sources and preprocessing steps
• Model architecture details and parameter counts
• Fairness constraints and bias mitigation techniques
• Known limitations and edge case behaviors

Incident Response Plan for Model Failures

Established incident escalation path:

1. Level 1 (Minor Issues): Automated alerting for >5% drop in engagement metrics
2. Level 2 (Moderate Issues): Manual review triggered for >10% content bias deviation
3. Level 3 (Critical Issues): Immediate model rollback and human intervention for >20% performance degradation or safety violations

Safety monitoring dashboard with real-time alerts for:

• Content toxicity rate exceeding 2% threshold
• Unusual engagement patterns indicating model drift
• Fairness metric violations across user segments

10. Scalability & Reliability

Bottleneck Analysis Per Pipeline Stage

Training Pipeline:

• Data processing: 100GB/day training data → 200GB/s ETL pipeline peak
• Model training: 500M samples, 10K features → ~20GB model size, 4h training on 8xV100
• Feature engineering: 500M users × 100 features → 50TB feature store

Serving Pipeline:

• QPS target: 1M requests/sec (50K concurrent users × 20 requests/sec avg)
• Latency budget: 200ms total (retrieval 50ms, scoring 100ms, re-ranking 50ms)
• Memory: 20GB model + 10GB embeddings = 30GB per serving node

Back-of-envelope calculations:

• Training data size: ~100GB/day → 36TB/year
• Feature count: 10K user features + 5K item features + 2K context features = ~17K total
• Embedding matrix footprint: 50M users × 128-dim embeddings = ~25GB
• Serving QPS math: 1M QPS ÷ 1000 nodes = 1K QPS per node (with 2x redundancy)

Scale-up Strategy

Horizontal Scaling:

• Serving: 1000 nodes with auto-scaling groups, each serving 1K QPS
• Feature store: Sharded across 500 nodes with consistent hashing
• Training: 20x GPU cluster for distributed training, data parallelism

Feature Computation Scaling:

• Batch features: 1000-node feature compute cluster with 24h freshness
• Real-time features: 500-node stream processing pipeline (Kafka + Flink)

Fault Tolerance

Model Serving:

• Fallback to rule-based model when primary model fails (99.9% uptime SLA)
• Stale model serving: 10-minute grace window before fallback to warm start
• Model versioning with rollback capability (last 3 versions retained)

Feature Store:

• Multi-source redundancy (primary + backup feature stores)
• Cache warming on node startup to handle cold start scenarios
• Graceful degradation: use default features when real-time computation fails

Cold Start Handling Strategy

New Users:

• Use demographic-based embeddings from user profile features
• Fall back to popular content ranking for first 100 interactions
• Personalization warm-up period: 24h before full ML ranking

New Items:

• Content-based retrieval using item metadata (tags, categories)
• Boost popularity-based ranking for new items in first 7 days
• Collaborative filtering fallback using similar item clusters

Multi-Region Serving Strategy

Primary Region:

• 1000 serving nodes in us-east-1
• 500 feature store nodes with 24h freshness

Secondary Region:

• 500 serving nodes in us-west-2 for disaster recovery
• 250 feature store nodes with 48h freshness (accepting latency trade-off)
• Cross-region replication with eventual consistency

Traffic Splitting:

• 95% primary region, 5% secondary for failover
• Traffic shift algorithm: 10% traffic increase every 5 minutes during failover
• Health check: 99.9% success rate threshold for region switch

Cross-Region Consistency:

• Feature store replication with 1-hour sync window
• Model version synchronization across regions (weekly updates)
• Global user session tracking with region-aware routing

This multi-tiered approach ensures 99.9% availability, sub-200ms latency, and automatic failover while maintaining consistent user experience across global deployments.

11. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering / policy alignment

• Content moderation policies - What constitutes harmful content that should be suppressed vs. filtered? How do we balance safety with free speech?
• User control mechanisms - Should users be able to customize their feed preferences, and how granular should that control be?
• Cross-platform consistency - Should the same ranking model be used across Twitter/X and Facebook, or should each platform have its own?
• Ad revenue integration - How do we balance organic content with paid content in the feed?

Key ML risks

Risk	Impact	Likelihood	Mitigation
Content bias amplification	High - Could amplify misinformation or harmful content	Medium	Implement content safety filters, bias audits, and fairness constraints
Feedback loop amplification	High - Negative engagement signals could cause model to over-prioritize sensational content	Medium-High	Implement decayed feedback mechanisms, engagement quality signals
Cold start problem	Medium - New users/items with no interaction history	High	Implement hybrid approach with content-based and popularity-based fallbacks
Data drift	Medium-High - User behavior and content evolution over time	High	Implement continuous monitoring, automatic retraining triggers
Model hallucination (for LLMs)	High - Generated content that seems plausible but is factually incorrect	Medium	Implement content verification, safety guardrails, and human-in-the-loop

Trade-offs made with justification

Trade-off	Justification
Accuracy vs Latency - Prioritize sub-200ms serving latency over marginal accuracy gains	Real-time feed ranking requires low-latency response for user engagement; sub-200ms is industry standard for interactive systems
Freshness vs Cost - Batch compute features every 12 hours instead of real-time	Reduces infrastructure cost and complexity while still maintaining reasonable freshness for user behavior signals
Model Size vs Quality - Use 20M parameter model instead of 100M+ for serving	Balances model quality with cost and latency constraints; can achieve >95% of full model performance
Personalization vs Diversity - Introduce diversity constraints to prevent echo chambers	Prevents content bubbles and maintains platform health while still providing personalized experience
Online vs Offline Learning - Use batch training with online feedback for model updates	Reduces complexity of real-time learning while still allowing for adaptation to changing user preferences

Alternatives considered and rejected

• Transformer-based models - Rejected due to high latency (500ms+ inference) and infrastructure cost for real-time serving
• Full online learning - Rejected due to complexity of maintaining model stability and preventing catastrophic forgetting
• Multi-armed bandit - Rejected for the complexity of managing exploration/exploitation trade-off in a content ranking system with many competing objectives
• Explicit user preference learning - Rejected due to low signal-to-noise ratio in explicit feedback and high cost of user surveying

Technical debt acknowledged

• Feature store implementation - Current design assumes simple feature lookups but doesn't yet support complex temporal features or cross-user interactions
• Model versioning - No robust model rollback capability in current design; relies on manual intervention for model recovery
• Cross-platform consistency - Feed ranking logic is currently isolated to one platform, with no clear path for cross-platform model sharing or alignment
• A/B testing infrastructure - Basic A/B test framework in place but lacks advanced statistical validation and user segmentation capabilities
• Safety monitoring - Content safety checks are implemented but not yet integrated with real-time feedback loops for continuous model improvement

12. Phased Implementation Roadmap

Phase 1: MVP (Weeks 1-8)

Objective: Establish baseline ranking system with rule-based fallback and minimal ML model.

Deliverables:

• Rule-based feed ranking engine with basic engagement signals (likes, comments, shares)
• Simple logistic regression model trained on 100K historical interactions
• Basic feature store with user/item embeddings (50-dim)
• Online serving infrastructure (TorchServe or custom)
• A/B testing infrastructure with 10% randomization

Critical Path:

1. Data pipeline setup (Week 1-2)
2. Basic model training and evaluation (Week 3-4)
3. Rule-based engine + simple ML model integration (Week 5-6)
4. A/B test infrastructure setup (Week 7-8)

Phase 2: Scale & Performance (Weeks 9-16)

Objective: Expand to real-time features, distributed training, and robust evaluation.

Deliverables:

• Real-time feature computation pipeline (user activity, item context)
• Distributed training infrastructure with data parallelism
• Multi-task model architecture (engagement + diversity + safety)
• Feature store with online/offline sync (500M+ features)
• Production A/B testing with 20% randomization

Critical Path:

1. Feature store design and implementation (Week 9-10)
2. Distributed training pipeline (Week 11-12)
3. Multi-task model training and validation (Week 13-14)
4. Production A/B test deployment (Week 15-16)

Phase 3: Excellence & Learning (Weeks 17-24)

Objective: Advanced models, continuous learning, and comprehensive monitoring.

Deliverables:

• Transformer-based ranking model with attention mechanisms
• Online learning pipeline for continuous model updates
• Advanced monitoring and drift detection system
• Multi-region serving with automatic failover
• Model versioning and rollback capability

Critical Path:

1. Transformer model design and training (Week 17-19)
2. Online learning implementation (Week 20-21)
3. Advanced monitoring system (Week 22-23)
4. Production deployment and validation (Week 24)

Dependencies:

• Phase 1 → Phase 2: Data pipeline maturity, model evaluation infrastructure
• Phase 2 → Phase 3: Feature store stability, A/B testing confidence
• All phases: Production monitoring and alerting systems

Risk Mitigation:

• Maintain rule-based fallback throughout all phases
• Shadow mode for new model rollouts (Week 1-2 of each phase)
• Gradual rollout strategy with progressive QPS increase
• Comprehensive logging and observability for all phases