AI-Powered People Recommendation System Design

Modern people recommendation systems leverage diverse data signals and ML algorithms to surface relevant connections in social networks, requiring careful architecture design for scalability, latency, and personalization.

1. Problem Statement & ML Task Definition

Clarifying questions

1. What is the expected scale of users and potential recommendations?
   • Are we targeting a global audience with billions of users or a more localized user base?
2. What is the latency budget for real-time recommendations?
   • Is sub-100ms required for user-facing interactions, or can we tolerate longer response times?
3. Is this a batch or online system?
   • Should recommendations be generated in real-time on user actions, or can we precompute and cache?
4. What is the primary business metric for success?
   • Is it engagement (click-through rate, time spent), user retention, or revenue?
5. What is the expected data freshness requirement?
   • Should we update recommendations within minutes, hours, or days?
6. What is the expected accuracy vs. speed trade-off?
   • Should we prioritize recall or precision in the recommendation list?
7. What is the scope of the recommendation task?
   • Should we recommend friends, follow suggestions, or both?

Business use case and success criteria

The system aims to recommend relevant people (friends or follow suggestions) to users based on their interests, behaviors, and social context. The goal is to increase user engagement by helping users discover new connections that align with their preferences, thereby increasing the likelihood of interactions such as follows, messages, or friend requests.

Success criteria:

• Click-through rate (CTR) on recommended profiles
• Engagement rate (e.g., time spent viewing profiles or initiating interactions)
• Conversion rate (e.g., successful follow/friend request)
• User retention over time (e.g., 30-day retention rate)
• User satisfaction score (via surveys or feedback)

ML problem formulation

This is a ranking problem, where the system learns to order a list of potential users (candidates) for each target user based on their relevance and likelihood of engagement.

Justification:

• The system needs to rank users by relevance, not simply classify them as "relevant" or "not relevant"
• Users are presented with a ranked list, and the system's performance is evaluated based on how well it ranks high-value candidates at the top
• The task involves learning from implicit feedback (e.g., views, likes, follows) and explicit signals (e.g., user preferences, content tags)

ML evaluation metrics

Offline evaluation metrics:

• NDCG@K: Normalized Discounted Cumulative Gain, useful for ranking quality
• Precision@K: Proportion of relevant recommendations in top-K list
• Recall@K: Proportion of relevant items captured in top-K recommendations
• AUC-ROC: For binary classification tasks (e.g., whether a user will interact with a recommendation)

Online evaluation metrics:

• CTR: Click-through rate on recommended profiles
• Engagement time: Average time spent viewing a recommended profile
• Conversion rate: Proportion of recommendations that result in follow or message actions
• Revenue impact: If the system is tied to monetization (e.g., sponsored profiles)

Scale and constraints

Back-of-envelope estimates:

• Training data size: 100B user-item interactions (e.g., views, follows, likes)
• Number of features: ~10K user and item features (including embeddings)
• Embedding matrix memory footprint:
  • User embedding: 10K users × 256 dimensions × 4 bytes = ~10MB
  • Item embedding: 100M items × 256 dimensions × 4 bytes = ~100GB
• Serving QPS:
  • 10M QPS (peak) → 1000 concurrent requests with 10ms p99 latency = 100M requests/sec
  • Assuming 50 recommendations per user, 200K concurrent users = 10M QPS

Constraints:

• Latency budget: < 50ms for real-time serving (p99)
• Freshness: Recommendations should update within 1 hour of user behavior changes
• Scalability: Must support 100M+ users and 1B+ items

Non-ML baseline strategy

Rule-based fallback:

• Recommend users with similar interests (based on content tags or categories)
• Recommend users in the same geographic region or timezone
• Cold start: Recommend popular or trending users based on global engagement

Cold start handling:

• For new users: Recommend popular profiles or those with high engagement
• For new items (profiles): Use content-based filtering or collaborative filtering with a small set of seed users

Fallback logic:

• If the ML model fails to return results or is unavailable, fall back to a rule-based system that recommends users based on basic heuristics (e.g., most popular, most recent, or content-based matches)

2. Data Strategy

Training Data Sources

• Explicit feedback: User actions like "follow", "unfollow", "block"
• Implicit signals:
  • Engagement with user content (likes, comments, shares)
  • Time spent viewing profiles
  • Direct messaging frequency
• Third-party data:
  • Social graph connections (Facebook, LinkedIn)
  • Shared interests from content consumption
  • Location data for proximity matching

Labeling Pipeline Design

• Human annotation: For high-value edge cases and validation samples (5% of training data)
• Weak supervision:
  • Rule-based labeling from existing social graph
  • Content similarity thresholds (e.g., shared interests > 80%)
• Programmatic labeling:
  • Follow relationships from social platforms (e.g., Twitter API)
  • Mutual friend count > 3 → positive label
  • Time since first interaction < 24h → positive label

Class Imbalance Handling Strategy

Most users will not follow others, creating severe imbalance (1:100+ negative to positive ratio). Strategy:

• Undersampling: Randomly sample 1% of negatives for training
• Oversampling: Boost minority positive samples by 2x
• Focal Loss: Apply focal loss to reduce impact of easy negatives during training

Negative Sampling Approach

• Hard negative sampling: Sample users with similar profiles but no existing relationship (50% of negatives)
• Random negative sampling: Sample from pool of users with no interaction (50% of negatives)
• Negative weight adjustment: Apply inverse frequency weighting to reduce overrepresentation of popular users

Data Freshness and Update Frequency Requirements

• Real-time: User actions (follow/unfollow) must be processed within 5 minutes
• Batch: Profile updates, content consumption logs every 2 hours
• Daily: Social graph expansion and new user onboarding

Privacy Compliance

• GDPR/CCPA:
  • All personal data anonymized before training
  • User consent logs maintained for compliance tracking
• Data retention:
  • User interaction history retained for 90 days
  • Profile data retained indefinitely with opt-out capability

Data Quality Validation and Monitoring

• Validation checks:
  • User ID uniqueness and format validation (UUID or numeric)
  • Follow cycles detection (A follows B, B follows A, but A does not follow C)
  • Temporal consistency (follow actions in future timestamps)
• Monitoring dashboard:
  • Daily data ingestion volume and quality metrics
  • Anomaly detection for sudden spikes in follow rates or user activity

def validate_follow_data(follow_events):
    """
    Validate follow event data for quality and consistency.
    
    Args:
        follow_events: List of follow event dictionaries
    
    Returns:
        bool: True if valid, False otherwise
    """
    # Check for cycles in follow relationships
    follows = {}
    for event in follow_events:
        follower_id = event['follower_id']
        followee_id = event['followee_id']
        
        # Validate IDs are not same
        if follower_id == followee_id:
            return False
            
        # Check for cycles (simplified)
        if follower_id in follows and followee_id in follows[follower_id]:
            return False
            
        # Add to tracking
        if follower_id not in follows:
            follows[follower_id] = set()
        follows[follower_id].add(followee_id)
    
    return True

def sample_negative_users(user_pool, positive_samples, num_negatives=1000):
    """
    Sample negative users for training with proper distribution.
    
    Args:
        user_pool: Set of all users in system
        positive_samples: Set of users with existing relationships
        num_negatives: Number of negative samples to generate
    
    Returns:
        list: List of (user_id, followee_id) tuples for training
    """
    import random
    
    negatives = []
    
    # Hard negatives (similar profiles but no relationship)
    for _ in range(int(num_negatives * 0.5)):
        user = random.choice(list(user_pool - positive_samples))
        followee = random.choice(list(user_pool - {user} - positive_samples))
        negatives.append((user, followee))
    
    # Random negatives (random user pairs)
    for _ in range(int(num_negatives * 0.5)):
        user = random.choice(list(user_pool))
        followee = random.choice(list(user_pool - {user}))
        negatives.append((user, followee))
    
    return negatives

3. Feature Engineering

Feature Taxonomy

Features are categorized into four main types:

1. User Features: Demographics, behavioral signals, social graph properties
2. Item Features: Profile information, content metadata, engagement statistics
3. Context Features: Time-based signals, device information, session context
4. Interaction Features: Historical engagement patterns, similarity measures

Embedding Design

Entity Embeddings (user_id, item_id):

• Dimensionality: 128-d for efficient serving
• Training: Collaborative filtering with negative sampling
• Update frequency: Daily

Text Embeddings (user bio, item description):

• Dimensionality: 256-d using BERT-base or SentenceTransformer
• Training: Pre-trained with domain-specific fine-tuning
• Update frequency: Weekly

Graph Embeddings (social connections, content similarity):

• Dimensionality: 128-d using Node2Vec or GraphSAGE
• Training: Online graph processing pipeline
• Update frequency: Real-time for new connections

Real-time vs Batch Feature Computation

Feature Type	Real-time?	Reasoning
User recent activity	Yes	Immediate feedback on user behavior
Item popularity	No	Can be pre-computed and cached
User similarity scores	No	Batch-computed from historical data
Context features	Yes	Time-sensitive, device-specific

Feature Store Design

Online Store: Low-latency KV store (e.g., Redis) for real-time serving with TTL of 1 hour Offline Store: Data warehouse (e.g., Snowflake) for training and batch processing

Feature Freshness & TTL Table

Feature	Source	Update Frequency	Store Type
User profile	DB	Daily	Offline
Recent activity count	Logs	Real-time	Online
Item engagement stats	Analytics	Hourly	Offline
User similarity	Graph engine	Daily	Online
Time-based signals	System clock	Per request	Online

Cross Features & Feature Interactions

Cross features are engineered to capture complex user-item relationships:

1. User-Item Interaction Patterns:

   • User's engagement with similar items
   • Item popularity normalized by user's historical preferences

2. Temporal Cross Features:

   • User activity time window with item posting time difference
   • Recent engagement vs overall engagement ratio

3. Contextual Interactions:

   • User-device combinations (e.g., mobile vs desktop behavior)
   • Time-of-day buckets with content category

Feature Importance & Selection Methodology

Method: SHAP-based feature importance from trained models, combined with permutation importance from validation set.

Selection Criteria:

• Top 200 features by SHAP importance (after removing highly correlated features)
• Minimum information gain threshold (0.01) for feature inclusion
• A/B test validation of top 50 features before full deployment

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif

class FeatureSelector:
    def __init__(self, k=200):
        self.k = k
        self.selector = SelectKBest(score_func=f_classif, k=k)
    
    def select_features(self, X_train, y_train, feature_names):
        # Fit selector on training data
        X_selected = self.selector.fit_transform(X_train, y_train)
        
        # Get selected feature indices
        selected_indices = self.selector.get_support(indices=True)
        
        # Return selected features and their names
        selected_features = [feature_names[i] for i in selected_indices]
        
        return X_selected, selected_features
    
    def get_feature_importance(self, X_train, y_train):
        # Calculate permutation importance for top features
        from sklearn.inspection import permutation_importance
        
        baseline_score = self.selector.score(X_train, y_train)
        
        importance_scores = []
        for i in range(X_train.shape[1]):
            X_permuted = X_train.copy()
            X_permuted[:, i] = np.random.permutation(X_permuted[:, i])
            
            perm_score = self.selector.score(X_permuted, y_train)
            importance = baseline_score - perm_score
            importance_scores.append(importance)
            
        return np.array(importance_scores)

4. Model Architecture

Model Selection with Justification

For a people recommendation system, we'll select a Two-Tower Neural Network architecture with Multi-Task Learning capabilities. This choice balances:

• Scalability: Can handle large-scale user and item embeddings
• Expressiveness: Captures complex user-item interactions through deep learning
• Flexibility: Supports multiple recommendation objectives (follow, friend request, engagement)
• Interpretability: Tower structure allows for clear understanding of user/item representations

Comparison with alternatives:

• Logistic Regression: Too simple to capture complex user preferences and interactions
• GBM: Good for structured features but struggles with high-dimensional sparse embeddings
• DNN: More expressive than LR, but lacks the two-tower structure for efficient user/item representation learning
• Transformer: Overkill for this use case, introduces unnecessary complexity and latency

Architecture Diagram

graph TD
    A[User Features] --> B[User Tower]
    C[Item Features] --> D[Item Tower]
    B --> E[Merged Embeddings]
    D --> E
    E --> F[Multi-Task Head]
    F --> G[Follow Score]
    F --> H[Engagement Score]
    F --> I[Relevance Score]

Two-Tower / Multi-Task Design

The system uses a two-tower architecture where:

1. User Tower: Processes user features to generate user embeddings
2. Item Tower: Processes item (user) features to generate item embeddings
3. Multi-Task Head: Outputs multiple recommendation scores simultaneously

This design enables:

• Shared representation learning across tasks
• Efficient batch inference with parallel computation
• Easy extension to new recommendation objectives

Pre-trained Model Selection and Fine-tuning Strategy

For foundational capabilities, we'll leverage:

• BERT-based encoder for text features (user bios, interests)
• Graph Neural Networks for social graph features
• Cross-attention mechanisms to model user-item interaction patterns

Fine-tuning will be done in a multi-stage approach:

1. Pre-train on large-scale social graph data
2. Fine-tune on target platform's specific user behavior patterns
3. Continual adaptation to new user preferences

Model Complexity vs Latency Trade-off Analysis

Model	Parameters	Inference Latency (ms)	Memory Footprint	Use Case
Logistic Regression	10K	0.5	1MB	Baseline
Two-Tower DNN (Small)	1M	5	50MB	Production
Two-Tower DNN (Medium)	10M	15	200MB	High Accuracy
Transformer-based	50M	50	1GB	Research

Offline Evaluation Baseline Framing

Baseline model: Matrix Factorization + Content-based features Expected baseline performance:

• AUC-ROC: ~0.75
• Precision@10: ~0.25
• NDCG@10: ~0.35

Proposed model should exceed baseline by:

• AUC-ROC: +0.10
• Precision@10: +0.15
• NDCG@10: +0.20

Core Model Code Snippet

import torch
import torch.nn as nn
import torch.nn.functional as F

class TwoTowerModel(nn.Module):
    def __init__(self, user_feature_dim=1000, item_feature_dim=1000, 
                 embedding_dim=128, hidden_dims=[256, 128]):
        super().__init__()
        
        # User tower
        self.user_tower = nn.Sequential(
            nn.Linear(user_feature_dim, hidden_dims[0]),
            nn.ReLU(),
            nn.Linear(hidden_dims[0], hidden_dims[1]),
            nn.ReLU(),
            nn.Linear(hidden_dims[1], embedding_dim)
        )
        
        # Item tower
        self.item_tower = nn.Sequential(
            nn.Linear(item_feature_dim, hidden_dims[0]),
            nn.ReLU(),
            nn.Linear(hidden_dims[0], hidden_dims[1]),
            nn.ReLU(),
            nn.Linear(hidden_dims[1], embedding_dim)
        )
        
        # Multi-task heads
        self.follow_head = nn.Linear(embedding_dim, 1)
        self.engagement_head = nn.Linear(embedding_dim, 1)
        self.relevance_head = nn.Linear(embedding_dim, 1)
        
    def forward(self, user_features, item_features):
        # Generate embeddings
        user_emb = self.user_tower(user_features)
        item_emb = self.item_tower(item_features)
        
        # Compute similarity scores
        dot_product = torch.sum(user_emb * item_emb, dim=1, keepdim=True)
        cosine_sim = F.cosine_similarity(user_emb, item_emb, dim=1, eps=1e-8)
        
        # Multi-task outputs
        follow_score = self.follow_head(user_emb * item_emb).squeeze()
        engagement_score = self.engagement_head(user_emb * item_emb).squeeze()
        relevance_score = self.relevance_head(user_emb * item_emb).squeeze()
        
        return {
            'follow_score': follow_score,
            'engagement_score': engagement_score,
            'relevance_score': relevance_score,
            'similarity': cosine_sim
        }
    
    def compute_loss(self, user_features, item_features, follow_labels):
        outputs = self.forward(user_features, item_features)
        
        # Binary cross-entropy loss for follow prediction
        follow_loss = F.binary_cross_entropy_with_logits(
            outputs['follow_score'], 
            follow_labels.float()
        )
        
        # Additional losses for other tasks
        engagement_loss = F.mse_loss(outputs['engagement_score'], 
                                   torch.zeros_like(outputs['engagement_score']))
        relevance_loss = F.mse_loss(outputs['relevance_score'], 
                                  torch.zeros_like(outputs['relevance_score']))
        
        total_loss = follow_loss + 0.1 * engagement_loss + 0.1 * relevance_loss
        return total_loss

# Example usage for inference
def predict_follow_probability(model, user_features, item_features):
    model.eval()
    with torch.no_grad():
        outputs = model(user_user_features, item_features)
        follow_prob = torch.sigmoid(outputs['follow_score'])
        return follow_prob

# Example training step
def train_step(model, user_batch, item_batch, labels, optimizer):
    model.train()
    optimizer.zero_grad()
    
    loss = model.compute_loss(user_batch, item_batch, labels)
    loss.backward()
    optimizer.step()
    
    return loss.item()

This implementation provides:

• Modular architecture with clear separation between user and item towers
• Multi-task learning capability for various recommendation objectives
• Efficient inference pipeline with batch processing support
• Production-ready loss functions for training and evaluation
• Scalable embedding dimensions that can be adjusted based on scale requirements

5. Serving & Inference Architecture

System Diagram

graph TD
    A[User Request] --> B[Load Balancer]
    B --> C[API Gateway]
    C --> D[Ranking Service]
    D --> E[Feature Store]
    E --> F[Embedding Cache]
    F --> G[Model Serving]
    G --> H[Scoring Engine]
    H --> I[Re-ranking Service]
    I --> J[Response Formatter]
    J --> K[User Response]
    
    subgraph "Real-time Serving Layer"
        D
        G
        H
        I
        J
    end
    
    subgraph "Feature Layer"
        E
        F
    end
    
    subgraph "Model Layer"
        G
        H
    end

Online vs Batch Serving Decision

Online serving is required for real-time friend recommendations to ensure low latency (under 200ms p99) and immediate user feedback. Batch serving would introduce significant latency delays that impact user experience, particularly for social interactions where timeliness is crucial. The system must support high QPS (estimated 10K+ requests/sec) with sub-200ms response times for a scalable, interactive recommendation experience.

Two-Stage Pipeline

The system implements a two-stage pipeline:

1. Candidate Retrieval: Fast approximate retrieval using pre-computed embeddings and inverted indexes (e.g., FAISS) to reduce the search space from millions to thousands of candidates
2. Scoring & Re-ranking: High-fidelity scoring using full model computation and re-ranking with business logic (e.g., diversity, freshness)

Serving Infrastructure

Model Serving Platform: TorchServe with custom inference pipeline for:

• Model Format: PyTorch models serialized with TorchScript
• Infrastructure: GPU-accelerated instances (NVIDIA A10 or H100) with 8GB+ VRAM
• Scalability: Horizontal scaling via Kubernetes pods with auto-scaling based on QPS

Inference Engine: Custom inference engine built on PyTorch with:

• Batch processing (min 32, max 128 samples)
• TensorRT optimization for inference latency reduction
• Model versioning and rollback capability

Latency Budget Breakdown

Stage	p50 Latency	p99 Latency	Budget
Feature Fetch	5ms	10ms	15ms
Embedding Lookup	2ms	5ms	10ms
Scoring	15ms	30ms	45ms
Re-ranking	10ms	25ms	35ms
Response Formatting	2ms	5ms	10ms
Total	34ms	75ms	115ms

Model Compression Strategy

• Quantization: INT8 quantization for embedding lookups and dense layers (15-20% size reduction with <1% accuracy loss)
• Pruning: Structured pruning for attention layers (30% reduction in parameters with 2% accuracy drop)
• Distillation: Student model trained to mimic teacher model (50% reduction in inference time with 5% accuracy loss)

Caching Strategy

• Prediction Cache: Redis-based cache (10M entries) with TTL of 5 minutes for frequent user queries
• Embedding Cache: In-memory cache (2GB) for hot user/item embeddings with LRU eviction policy
• KV Cache: For LLM-based content generation (if applicable), caching attention keys/values to reduce recomputation

Auto-scaling and Capacity Planning

• Horizontal Scaling: Kubernetes HPA with target CPU utilization at 70%
• Vertical Scaling: GPU instances (A10 or H100) with 8GB+ VRAM
• Capacity Planning:
  • Expected QPS: 10K/sec (peak 25K/sec)
  • Model inference time: ~34ms p50, ~75ms p99
  • Required serving capacity: 128 GPU instances (assuming 50ms response time target)
  • Cold start handling: Pre-warmed model instances with warm-up routines

Model Serving Pipeline Code Example

import torch
from torch import nn
from typing import Dict, List, Tuple
import numpy as np

class FriendRecommendationServingEngine:
    def __init__(self, model_path: str):
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.model = torch.jit.load(model_path).to(self.device)
        self.model.eval()
        
    def batch_predict(self, user_embeddings: torch.Tensor, 
                     candidate_embeddings: torch.Tensor) -> torch.Tensor:
        """
        Batch scoring for friend recommendations
        """
        with torch.no_grad():
            # Compute dot product similarity scores
            scores = torch.matmul(user_embeddings, candidate_embeddings.T)
            
            # Apply sigmoid for probability-like scores
            probabilities = torch.sigmoid(scores)
            
            return probabilities
    
    def rank_candidates(self, user_id: str, 
                       candidate_ids: List[str], 
                       features: Dict) -> List[Tuple[str, float]]:
        """
        Main inference pipeline for friend recommendation
        """
        # Fetch user and candidate embeddings from cache/store
        user_embedding = self.fetch_user_embedding(user_id)
        candidate_embeddings = self.fetch_candidate_embeddings(candidate_ids)
        
        # Compute scores
        scores = self.batch_predict(user_embedding, candidate_embeddings)
        
        # Convert to ranked list with confidence
        results = [(candidate_ids[i], scores[i].item()) 
                  for i in range(len(candidate_ids))]
        
        # Sort by score descending
        results.sort(key=lambda x: x[1], reverse=True)
        
        return results
    
    def fetch_user_embedding(self, user_id: str) -> torch.Tensor:
        # Placeholder for actual embedding lookup logic
        return torch.randn(1, 128).to(self.device)
    
    def fetch_candidate_embeddings(self, candidate_ids: List[str]) -> torch.Tensor:
        # Placeholder for actual embedding lookup logic
        return torch.randn(len(candidate_ids), 128).to(self.device)

# Example usage
engine = FriendRecommendationServingEngine("friend_recommender.pt")
user_id = "user_123"
candidates = ["user_456", "user_789", "user_012"]
recommendations = engine.rank_candidates(user_id, candidates, {})

This serving architecture balances low latency with high scalability while supporting the real-time nature of social recommendations. The two-stage approach ensures that users receive relevant suggestions quickly, with the ability to scale to millions of concurrent users.

6. Evaluation Framework

Offline Evaluation Methodology

For offline evaluation, we use a temporal split approach to ensure that our model is evaluated on future data points that were not seen during training. This prevents data leakage and provides a realistic assessment of model performance in a production-like setting.

We split the dataset chronologically into:

• Training set: 80% of data (oldest timestamps)
• Validation set: 10% of data (middle timestamps)
• Test set: 10% of data (most recent timestamps)

Cross-validation is not suitable here due to temporal dependencies in user behavior and the need to maintain chronological order for realistic evaluation.

Slice-based Evaluation

We evaluate model performance across critical slices to identify potential biases or performance degradation:

Slice Type	Definition	Critical Slices
User Demographics	Age, gender, location	New users (0-30 days), high-activity users
Platform Segments	Mobile vs desktop, device type	iOS vs Android, tablet vs phone
Geographic Regions	Country, city size	US, EU, APAC, emerging markets
Content Categories	User interests, content type	Tech, sports, entertainment

A/B Test Design

We implement a robust A/B testing framework to measure the real-world impact of our recommendation system:

Randomization Unit: User ID (to prevent spill-over effects between test groups)

Sample Size Calculation:

• Target metric: Click-through rate (CTR) improvement of 5%
• Power: 80%
• Significance level: 5%
• Estimated baseline CTR: 2.5%
• Required sample size per group: ~14,000 users

Test Duration: 30 days (to capture seasonal trends and ensure statistical significance)

Primary Metric: CTR on recommended profiles Secondary Metrics:

• Time spent on profile pages (engagement)
• Follow rate after recommendation
• Revenue impact (if monetized)

Guardrail Metrics and Blocking Conditions

To ensure system safety before full deployment:

Metric	Threshold	Action
CTR Drop	< 10% from baseline	Immediate rollback
Engagement Drop	< 5% from baseline	Pause serving
System Latency	> 200ms p99	Alert and investigate
Model Confidence	< 0.7 for 5% of predictions	Flag for review

Shadow Mode Rollout Plan

Before full A/B testing, we implement shadow mode where:

1. 5% of traffic is routed to the new model in parallel with the existing system
2. All predictions are logged but not served to users
3. Performance metrics are collected and compared against baseline
4. Model confidence scores and prediction distributions are analyzed for consistency

Long-term Metric Measurement

To capture the true impact of recommendations:

Delayed Metric	Time Horizon	Expected Impact
Conversion Rate	7 days	Increased follow rate
Retention Rate	30 days	Improved user stickiness
Revenue Impact	90 days	Monetization from engaged users
Content Engagement	30 days	Higher profile view rates

We track these metrics using cohort analysis to understand the long-term behavioral impact of our recommendations. The system is designed to detect and alert on any significant drift in these delayed metrics, enabling proactive model retraining or adjustment.

# Example A/B test evaluation logic
import numpy as np
from scipy import stats

def ab_test_evaluation(control_ctr, test_ctr, control_size, test_size):
    """
    Evaluate A/B test results for CTR improvement
    """
    # Calculate confidence intervals
    control_lower, control_upper = stats.norm.interval(0.95, loc=control_ctr, scale=np.sqrt(control_ctr*(1-control_ctr)/control_size))
    test_lower, test_upper = stats.norm.interval(0.95, loc=test_ctr, scale=np.sqrt(test_ctr*(1-test_ctr)/test_size))
    
    # Calculate improvement
    improvement = (test_ctr - control_ctr) / control_ctr * 100
    
    # Statistical significance test
    z_score = (test_ctr - control_ctr) / np.sqrt(control_ctr*(1-control_ctr)/control_size + test_ctr*(1-test_ctr)/test_size)
    p_value = 2 * (1 - stats.norm.cdf(abs(z_score)))
    
    return {
        "improvement_percent": improvement,
        "confidence_interval": (control_lower, control_upper, test_lower, test_upper),
        "statistical_significance": p_value < 0.05,
        "p_value": p_value
    }

# Example usage:
# results = ab_test_evaluation(0.025, 0.026, 15000, 15000)

7. Training Pipeline

Training Data Pipeline

The training pipeline processes raw interaction data through ETL workflows to generate training samples. Raw data sources include user interactions (follows, unfollows, profile views), user profiles, item metadata, and contextual signals. The pipeline uses Apache Airflow for orchestration with the following stages:

1. Data Extraction: Pull data from Kafka streams and S3 buckets containing user-event logs, profile data, and metadata
2. Data Transformation: Normalize timestamps, enrich with user demographics, item categories, and temporal features
3. Sampling & Labeling: Generate positive samples from actual follows and negative samples through random or stratified sampling. For each user-item pair, create a binary label (1 if followed, 0 otherwise)
4. Feature Engineering: Compute user and item embeddings, temporal features, and interaction statistics
5. Shuffling & Batching: Shuffle training data to prevent temporal bias, batch into 1024 samples per batch for efficient GPU utilization

import torch
from torch.utils.data import Dataset, DataLoader
import pandas as pd

class RecommendationDataset(Dataset):
    def __init__(self, data_path, feature_columns):
        self.data = pd.read_csv(data_path)
        self.features = feature_columns
        
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, idx):
        row = self.data.iloc[idx]
        features = torch.tensor(row[self.features].values, dtype=torch.float32)
        label = torch.tensor(row['followed'], dtype=torch.long)
        return features, label

# Example usage
train_dataset = RecommendationDataset('training_data.csv', ['user_emb', 'item_emb', 'temporal_features'])
train_loader = DataLoader(train_dataset, batch_size=1024, shuffle=True)

Distributed Training Strategy

For large-scale training, we use data parallelism with PyTorch's DistributedDataParallel (DDP) to scale across multiple GPUs. With 100M training samples and 500 features, we can train on a cluster of 8x A100 GPUs with 40GB VRAM each, achieving ~200K samples/sec training throughput.

Training Infrastructure

• Cluster Size: 8x A100 GPUs (40GB VRAM each) with 256GB total memory
• Training Time: ~4 hours for full dataset (assuming 100M samples, 500 features)
• Memory Usage: ~2GB per GPU for model and gradients
• Storage: 500GB SSD storage for training data and model checkpoints

Hyperparameter Tuning Approach

We implement Bayesian optimization using Optuna to tune key hyperparameters:

• Learning rate (0.001 - 0.01)
• Batch size (512 - 2048)
• Embedding dimension (64 - 256)
• Dropout rate (0.1 - 0.5)
• Weight decay (1e-6 - 1e-3)

Experiment Tracking and Reproducibility

We use MLflow for experiment tracking with:

• Versioned model checkpoints stored in S3
• Hyperparameter logging and comparison dashboard
• Metrics tracking (AUC, Precision@K, Recall@K)
• Model lineage and metadata management

Model Versioning and Registry Strategy

Model versions are managed through MLflow model registry with:

• Semantic versioning (v1.0.0, v1.1.0, etc.)
• Automatic model validation against baseline performance
• Rollback capability for failed deployments
• Tagging strategy (production, staging, experimental)

Continual / Online Learning Design

We implement incremental learning with:

• Online fine-tuning using recent user interactions (last 30 days)
• Adaptive learning rate decay based on performance drift
• Periodic retraining every 7 days with fresh data
• Online learning window of 100K recent samples for model updates

8. Monitoring & Observability

Data Drift Detection

Data drift monitoring is critical for maintaining model performance in a people recommendation system. We'll implement drift detection at multiple levels:

• Feature Drift: Using Population Stability Index (PSI) and KL divergence to detect shifts in user/item feature distributions
• Label Drift: Monitoring distribution shifts in engagement signals (follows, interactions)
• Temporal Drift: Tracking feature and label distributions over time windows to detect seasonal or trend changes

import numpy as np
from scipy import stats

def psi(expected, actual, num_buckets=10):
    """Calculate Population Stability Index"""
    # Bin data into buckets
    bins = np.linspace(np.min([expected, actual]), np.max([expected, actual]), num_buckets + 1)
    expected_counts = np.histogram(expected, bins=bins)[0]
    actual_counts = np.histogram(actual, bins=bins)[0]
    
    # Avoid zero counts
    expected_counts = np.where(expected_counts == 0, 1e-6, expected_counts)
    actual_counts = np.where(actual_counts == 0, 1e-6, actual_counts)
    
    psi_value = np.sum((actual_counts - expected_counts) * np.log(actual_counts / expected_counts))
    return psi_value

def kl_divergence(p, q):
    """Calculate KL divergence between two distributions"""
    return np.sum(p * np.log(p / q))

Prediction Drift and Label Drift Monitoring

We'll monitor:

• Prediction drift: Changes in model confidence distributions over time
• Label drift: Shifts in actual engagement patterns (e.g., users becoming less likely to follow)
• Performance degradation: Decline in key metrics like CTR, engagement rate

Model Performance Degradation Alerting

We'll implement alerting thresholds for key metrics:

Metric	Threshold	Alert Level
CTR Drop	>5%	Warning
NDCG Drop	>10%	Critical
User Engagement Drop	>3%	Warning
Model Confidence Shift	>20%	Warning

Retraining Trigger Strategy

We'll implement a hybrid retraining strategy:

1. Scheduled retraining: Weekly for batch models, daily for online learning systems
2. Drift-triggered retraining: When PSI > 0.1 for key features or KL divergence > 0.05 for labels
3. Performance-triggered retraining: When metrics drop below thresholds

Logging Design

We'll implement comprehensive logging:

import json
from datetime import datetime

class PredictionLogger:
    def __init__(self):
        self.prediction_log = []
    
    def log_prediction(self, user_id, candidate_ids, scores, features, timestamp=None):
        if timestamp is None:
            timestamp = datetime.now().isoformat()
            
        log_entry = {
            "timestamp": timestamp,
            "user_id": user_id,
            "candidates": candidate_ids,
            "scores": scores.tolist() if hasattr(scores, 'tolist') else scores,
            "features": features,
            "model_version": "v1.2"
        }
        
        self.prediction_log.append(log_entry)
        # Write to monitoring system or file sink
        return log_entry

# Example usage for drift detection logging
class DriftMonitor:
    def __init__(self):
        self.feature_drift_history = {}
    
    def detect_and_log_drift(self, feature_name, current_distribution, reference_distribution):
        psi_score = psi(current_distribution, reference_distribution)
        kl_score = kl_divergence(current_distribution, reference_distribution)
        
        drift_record = {
            "feature": feature_name,
            "psi_score": psi_score,
            "kl_score": kl_score,
            "timestamp": datetime.now().isoformat(),
            "threshold_exceeded": psi_score > 0.1 or kl_score > 0.05
        }
        
        return drift_record

Feedback Loop for Continuous Improvement

We'll establish a feedback loop that captures:

• User interaction signals: Follows, unfollows, engagement with recommendations
• Feature feedback: User behavior changes that might indicate feature drift
• Model performance signals: A/B test results, user satisfaction metrics

class FeedbackProcessor:
    def __init__(self):
        self.feedback_queue = []
    
    def process_feedback(self, user_id, candidate_id, action, timestamp):
        """Process user feedback for model improvement"""
        feedback = {
            "user_id": user_id,
            "candidate_id": candidate_id,
            "action": action,  # follow/unfollow/ignore
            "timestamp": timestamp.isoformat(),
            "source": "user_interaction"
        }
        
        # Queue for retraining pipeline or online learning updates
        self.feedback_queue.append(feedback)
        
        # Update feature importance or reweight training samples if needed
        return feedback

Key Monitoring Components

1. Real-time Dashboards: For tracking model performance, drift metrics, and system health
2. Anomaly Detection: Using statistical methods to identify unusual patterns in prediction distributions
3. User Behavior Tracking: Monitoring changes in user engagement with recommendations over time
4. A/B Test Monitoring: Real-time tracking of A/B test results and statistical significance

Alerting Strategy

• Critical Alerts: Performance degradation >10%, drift >0.2 PSI, system failures
• Warning Alerts: Performance degradation 5-10%, drift 0.1-0.2 PSI, low confidence predictions
• Info Alerts: Scheduled retraining completion, new feature availability

Data Quality Monitoring

We'll monitor:

• Missing feature values: Track rate of missing features in real-time
• Feature value ranges: Ensure features stay within expected bounds
• Prediction consistency: Detect sudden shifts in prediction confidence distributions

This comprehensive monitoring framework ensures that the people recommendation system maintains high performance and can quickly detect and respond to changes in user behavior, data distribution, or model degradation.

9. Responsible AI & Safety

Bias Analysis

• Training data audit: Conduct comprehensive audit of user demographics, geographic distribution, and behavioral patterns to identify potential representation gaps. Analyze for gender, age group, and cultural biases in the training data.
• Slice-based fairness evaluation: Evaluate model performance across key user segments (age groups, regions, device types) to detect disparate impacts. Monitor for bias in recommendation frequency and diversity.

Fairness Metrics and Constraints

• Primary fairness metric: Demographic parity (equal recommendation rates across protected attributes)
• Secondary metrics: Equalized odds (same true positive rates across slices)
• Decision rationale: Demographic parity is preferred over equalized odds for recommendation systems to ensure equitable exposure rather than just matching individual outcomes

Explainability Requirements

• SHAP values: Implement SHAP-based feature importance for top-K recommendations to provide interpretable reasons for each recommendation
• Attention visualization: For transformer-based models, surface attention weights to show which user-item interactions were most influential
• Stakeholder requirements:
  • Product team: High-level recommendation rationale for user onboarding
  • Legal team: Audit trail of recommendation decisions for compliance

For LLM Systems (if applicable)

• Content filtering: Implement content safety filters to prevent recommending users with inappropriate profiles or content
• Output validation: Validate that recommendations meet platform guidelines and community standards
• Rate limiting: Enforce rate limits on recommendation generation to prevent abuse or spam
• Red teaming: Regular security testing with adversarial inputs to identify potential bias or safety issues

Model Card Documentation Plan

• Required elements:
  • Model architecture and training methodology
  • Performance metrics across different user segments
  • Known limitations and bias conditions
  • Data sources and preprocessing steps
  • Usage guidelines and intended audience

Incident Response Plan for Model Failures

• Failure detection: Monitor for sudden drops in recommendation quality, increase in negative feedback, or unexpected user behavior patterns
• Escalation protocol:
  • Alert level 1: Performance degradation below threshold triggers automated rollback to previous version
  • Alert level 2: System-wide bias detected triggers manual review and potential model retraining
• Communication plan:
  • Internal: Notify ML team, product team, and compliance team
  • External: Prepare communication plan for users if recommendations are significantly impacted

# Example SHAP-based explainability function for recommendation system
import shap
import numpy as np

def explain_recommendation(model, user_features, item_features, feature_names):
    """
    Generate SHAP explanations for a recommendation
    """
    # Create explainer object
    explainer = shap.Explainer(model, user_features)
    
    # Get SHAP values for the top recommendation
    shap_values = explainer(item_features)
    
    # Format explanations
    explanations = {}
    for i, feature in enumerate(feature_names):
        explanations[feature] = shap_values.values[0][i]
    
    return explanations

# Example fairness monitoring function
def check_fairness_by_slice(model, test_data, slices):
    """
    Monitor model performance across different user slices
    """
    fairness_results = {}
    
    for slice_name, slice_data in slices.items():
        # Get predictions for the slice
        slice_predictions = model.predict(slice_data['features'])
        
        # Calculate key metrics per slice
        slice_metrics = {
            'recommendation_rate': np.mean(slice_predictions),
            'diversity_score': calculate_diversity(slice_predictions),
            'bias_ratio': calculate_bias_ratio(slice_data['user_attributes'])
        }
        
        fairness_results[slice_name] = slice_metrics
    
    return fairness_results

10. Scalability & Reliability

Bottleneck Analysis per Pipeline Stage

Training Pipeline:

• Data processing: 100GB training dataset → 500M user-item interactions → 200M unique users, 100M unique items
• Feature engineering: 50M features → 2GB embedding matrix (128-dim) → 20GB total for all embeddings
• Model training: 50M samples, 100M features → 200GB parameter space (assuming 32-bit floats)
• Back-of-envelope: 50M samples × 100M features = 5 trillion operations → 200GB memory requirement for model state

Serving Pipeline:

• QPS target: 10,000 requests/sec
• Latency budget: 200ms p99 (50ms for retrieval, 100ms for scoring, 50ms for re-ranking)
• Per-request: 100 candidate items → 100 × 200ms = 20 seconds of compute per second → 200x overhead for batch processing
• Model size: 5GB (compressed) → 100GB memory for 20 concurrent models (for A/B testing)
• Embedding cache: 10M users × 128 dimensions = 50MB per user → 500GB total for full user base

Scale-up Strategy

Horizontal Scaling:

• Serving: 1000 GPU instances (each with 8x V100s) → 8000 GPUs total for model inference
• Feature computation: 500 distributed feature workers → 200GB memory per worker for embedding cache
• Training: 1000 GPU nodes in distributed training cluster → 5000 GPUs total for model training

Load Balancing:

• Use consistent hashing for user routing to prediction shards
• Implement weighted round-robin for model version distribution (A/B testing)
• Auto-scaling: 20% headroom for QPS spikes → 12,000 QPS capacity

Fault Tolerance

Model Serving Fallbacks:

• Primary model (MLP) → fallback to rule-based system (user similarity + shared interests)
• Cache warm-up: 50% of user base pre-loaded into cache → 50ms fallback for cold start
• Model version rollback: 3 versions stored in cache → automatic fallback on performance degradation

Stale Model Handling:

• Model freshness: 1-day TTL for embeddings → 24-hour retraining window
• Fallback to pre-trained embeddings: 10% of users → 1M users fallback to static embeddings
• Graceful degradation: 50% of users → 100ms latency increase with 20% accuracy drop

Cold Start Handling Strategy

New Users:

• Rule-based fallback: 100% of new users → 500M users/day → 500M × 100 candidates = 50B recommendations
• Content-based: 20 features per user → 1000M features → 5GB embedding matrix for content vectors
• Collaborative filtering: 100M users → 50GB embedding matrix for user embeddings

New Items:

• Content-based: 50M items → 100 features per item → 5GB embedding matrix for item embeddings
• Popularity-based: 100 most recent items → 200ms latency for popularity ranking
• Hybrid: 50% content + 50% popularity → 100ms latency for new item recommendations

Multi-Region Serving Strategy

Global Deployment:

• Primary region (US-West): 50% of traffic → 5,000 QPS
• Secondary region (EU): 30% of traffic → 3,000 QPS
• Backup region (Asia): 20% of traffic → 2,000 QPS

Cross-Region Failover:

• Region health monitoring: 5-minute heartbeat → 99.9% uptime SLA
• Traffic shifting: 100ms failover time → 50% of traffic shifted to backup region
• Data synchronization: 1-hour sync window → 50M users × 20 features = 1GB daily sync

Latency Budget Allocation:

Stage	p50	p99	Budget
User embedding lookup	1ms	5ms	5ms
Candidate retrieval	20ms	50ms	50ms
Scoring model inference	50ms	100ms	100ms
Re-ranking	20ms	50ms	50ms
Response serialization	1ms	5ms	5ms
Total	92ms	210ms	200ms

# Example: Model inference with fallback logic
class FallbackRecommendationEngine:
    def __init__(self):
        self.primary_model = self.load_primary_model()
        self.fallback_model = self.load_fallback_model()
        self.cache = {}
        
    def recommend_with_fallback(self, user_id, num_recommendations=10):
        # Check cache first
        if user_id in self.cache:
            return self.cache[user_id]
            
        try:
            # Primary model inference
            recommendations = self.primary_model.infer(user_id, num_recommendations)
            # Validate results
            if len(recommendations) >= num_recommendations:
                self.cache[user_id] = recommendations
                return recommendations
        except Exception as e:
            print(f"Primary model failed for user {user_id}: {e}")
            
        # Fallback to rule-based system
        recommendations = self.fallback_model.infer(user_id, num_recommendations)
        self.cache[user_id] = recommendations
        return recommendations

# Example: Multi-region failover logic
class MultiRegionEngine:
    def __init__(self):
        self.regions = {
            'us-west': {'endpoint': 'api.us-west.example.com', 'healthy': True},
            'eu-east': {'endpoint': 'api.eu-east.example.com', 'healthy': True},
            'asia-southeast': {'endpoint': 'api.asia-southeast.example.com', 'healthy': True}
        }
        
    def get_recommendations(self, user_id, region='us-west'):
        # Try primary region first
        try:
            return self.make_request(self.regions[region]['endpoint'], user_id)
        except Exception as e:
            # Failover to next healthy region
            for region_name, region_info in self.regions.items():
                if region_info['healthy'] and region_name != region:
                    try:
                        return self.make_request(region_info['endpoint'], user_id)
                    except Exception as e2:
                        continue
            # Last resort: fallback to local rule-based system
            return self.fallback_recommendations(user_id)

11. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering / policy alignment

• User consent and data access: What level of user data (e.g., contact list, social graph) can be accessed for recommendation generation? This directly impacts model performance and compliance requirements.
• Cross-platform recommendation: Should recommendations be consistent across platforms (web, mobile) or tailored to each platform's usage patterns?
• Content moderation and safety: How should the system handle users with potentially harmful or spammy behavior? Should these users be filtered out entirely or have their recommendations restricted?
• Privacy-preserving recommendation: What level of differential privacy or federated learning should be applied to protect user data?

Key ML risks

Risk	Impact	Likelihood	Mitigation
Data drift in user behavior	High (reduced model performance)	Medium	Implement continuous drift detection and automatic retraining triggers
Cold start for new users/items	Medium (low recommendation quality)	High	Implement hybrid approach with content-based and popularity-based fallbacks
Bias amplification	High (unfair recommendations)	Medium	Regular fairness audits and slice-based evaluation
Scalability bottlenecks	High (system failure under load)	Medium	Design with horizontal scaling in mind and implement proper caching
Prediction latency spikes	Medium (user experience degradation)	Low-Medium	Implement adaptive model compression and pre-warming strategies

Trade-offs made with justification

Trade-off	Justification
Accuracy vs. latency	Prioritize sub-100ms response time for real-time serving over marginal accuracy gains from more complex models
Freshness vs. cost	Accept 1-2 day data freshness for batch features to reduce infrastructure costs, while maintaining real-time feature updates
Model size vs. quality	Balance between model compression for serving efficiency and maintaining recommendation quality through pruning and quantization
Personalization vs. privacy	Optimize for user-centric recommendations while ensuring compliance with privacy regulations through anonymization and differential privacy
Online vs. batch training	Use online learning for real-time feedback incorporation, but maintain batch training for model stability and long-term trend capture

Alternatives considered and rejected

• Content-based filtering: Rejected due to limited ability to capture complex user preferences and social dynamics
• Collaborative filtering with matrix factorization: Rejected for scalability issues in large-scale systems and inability to handle cold start problems effectively
• Rule-based systems: Rejected for lack of personalization and inability to adapt to changing user behavior
• Graph neural networks: Rejected for high computational cost in real-time serving and complexity in model training and maintenance

Technical debt acknowledged

• Feature store implementation: Current design assumes a mature feature store, but actual implementation will require significant engineering effort to support real-time updates and complex feature transformations
• Model versioning complexity: Multi-task model architecture introduces complexity in tracking and managing different model versions for various recommendation goals
• A/B testing infrastructure: Need to build comprehensive A/B testing framework from scratch, which introduces risk of incorrect experiment design or measurement bias
• Cross-platform consistency: Maintaining consistent recommendation quality across platforms introduces ongoing maintenance overhead and potential for divergent user experiences
• Privacy compliance implementation: Full compliance with GDPR, CCPA, and other regulations requires continuous updates to data handling and user consent mechanisms that are not fully implemented in current design

12. Phased Implementation Roadmap

Phase 1: MVP (Weeks 1-4)

Objective: Establish baseline recommendation system with rule-based fallback and basic ML model.

Deliverables:

• Rule-based baseline (popularity + mutual friends)
• Basic collaborative filtering model (user-item matrix factorization)
• Online serving infrastructure with 100 QPS capacity
• Simple A/B testing framework

Dependencies:

• Data pipeline ready with 1M users, 10M interactions
• Feature store MVP (batch features only)
• Basic model serving stack (TorchServe or Triton)

Critical Path:

1. Rule-based system implementation (Week 1)
2. Basic model training pipeline (Week 2)
3. Online serving stack setup (Week 3)
4. A/B testing infrastructure (Week 4)

Phase 2: Scale (Weeks 5-8)

Objective: Add real-time features, distributed training, and robust evaluation.

Deliverables:

• Real-time feature computation (user activity, context)
• Distributed training pipeline (data parallelism)
• Feature store with online and offline stores
• Comprehensive A/B testing infrastructure

Dependencies:

• Feature store fully operational with 50M features
• Distributed training cluster (2x GPU nodes)
• MLflow experiment tracking setup

Critical Path:

1. Feature store integration (Week 5)
2. Distributed training pipeline (Week 6)
3. Real-time feature computation (Week 7)
4. A/B testing framework enhancement (Week 8)

Phase 3: Excellence (Weeks 9-12)

Objective: Advanced models, continuous learning, and production-grade monitoring.

Deliverables:

• Transformer-based model with attention mechanisms
• Online learning capability for continuous model updates
• Advanced monitoring and drift detection
• Multi-region serving capability

Dependencies:

• Production-grade monitoring stack (Prometheus + Grafana)
• Online learning infrastructure (streaming data pipeline)
• Multi-region deployment capability

Critical Path:

1. Transformer model implementation (Week 9)
2. Online learning pipeline (Week 10)
3. Multi-region deployment (Week 11)
4. Production monitoring setup (Week 12)

Cross-Phase Dependencies:

• Data pipeline must be stable across all phases
• Feature store evolution from batch-only to real-time
• Model versioning and registry must support all phases
• A/B testing infrastructure must scale with user base growth

Risk Mitigation:

• Phase 1 serves as fallback for later phases
• All phases include monitoring and alerting from day one
• Shadow mode rollout capability built into Phase 2
• Gradual scale-up with capacity planning for each phase

Success Metrics:

• Phase 1: 50% improvement over rule-based baseline
• Phase 2: 30% increase in user engagement (e.g., follow rate)
• Phase 3: 25% improvement in recommendation quality (NDCG@10)
• All phases: <50ms p99 latency in serving infrastructure