Negative Sampling Strategies in Machine Learning Systems

Negative sampling is a critical optimization technique in machine learning systems that efficiently handles imbalanced datasets and improves model training performance by strategically selecting negative examples for training.

1. Problem Statement & ML Task Definition

Clarifying questions

1. What is the target business use case? Is this for recommendation systems, ad targeting, content ranking, or search relevance?
2. What is the expected scale of data and serving QPS? This affects negative sampling strategy choice (e.g., global vs. local sampling).
3. What is the latency budget for model training and inference? This impacts whether we can afford expensive sampling methods like importance sampling or need simpler approaches.
4. Is the system primarily online (real-time) or batch-oriented? This affects whether we need to pre-compute negative samples or can compute them on-demand.
5. What is the expected label distribution? Is there significant class imbalance that requires specific negative sampling techniques?
6. What is the primary success metric? Is it AUC-ROC, NDCG, CTR, or revenue-based KPIs that will guide sampling strategy?
7. What is the frequency of data updates? This affects whether we can use static or dynamic negative sampling.

Business use case and success criteria

The system aims to build a recommendation engine for a content platform, where the goal is to rank relevant content items for each user based on their historical interactions. The system will recommend articles, videos, or other content to maximize user engagement and retention.

Success criteria:

• Offline: AUC-ROC > 0.92, NDCG@10 > 0.85 for top-10 recommendations
• Online: CTR (Click-Through Rate) > 2.5%, engagement time per session > 30 seconds
• Business: Revenue per user (RPU) increase by 15% over baseline

ML problem formulation

This is a ranking task with a binary classification objective, where the model learns to distinguish between relevant and irrelevant content items for each user. The task is framed as a pairwise ranking problem, where the model learns to rank positive (clicked/relevant) items higher than negative (non-clicked/irrelevant) items.

Justification:

• The system is fundamentally about ranking content for users, not pure classification
• Binary labels (clicked vs. not clicked) are available for training
• Pairwise ranking is effective for recommendation systems where relative ordering matters more than absolute scores

ML evaluation metrics

Offline:

• AUC-ROC (for overall model discrimination)
• NDCG@K (Normalized Discounted Cumulative Gain for top-K ranking quality)
• Precision@K (Precision at K items recommended)
• BLEU score (for content generation tasks, if applicable)

Online:

• CTR (Click-Through Rate) - primary business metric
• Engagement time per session
• Revenue per user (RPU)
• User retention rate

Scale and constraints

Back-of-envelope estimates:

• Training data size: 100M user-item interactions (50M positive, 50M negative)
• Number of features: 10K (user features + item features + interaction features)
• Embedding matrix memory footprint: 10K features × 128 dimensions × 4 bytes = ~5MB per embedding table
• Serving QPS: 10,000 requests/sec (assuming 50ms p99 latency target)
• Training data processing: ~100GB raw data, 50GB processed features

Latency budget breakdown:

• Model inference: <50ms p99 (target)
• Feature lookup: <10ms p99
• Negative sampling: <5ms p99 (for online serving)
• Model update frequency: Daily

Non-ML baseline strategy

Rule-based fallback:

• For new users (cold start): Recommend popular content based on global popularity scores
• For new items: Use content-based filtering (similar item tags, categories)
• For low-confidence predictions: Fall back to default ranking based on recent user activity

Cold start handling:

• New users: Use demographic-based recommendations or trending content
• New items: Content-based similarity with existing popular items

# Baseline recommendation logic (simplified)
def baseline_recommendations(user_id, item_pool, user_profile):
    if not user_profile.get('engagement_history'):
        # Cold start: return popular items
        return get_popular_items(item_pool, top_k=10)
    
    # Rule-based: content similarity with recent items
    recent_items = get_recently_interacted(user_id, k=5)
    return get_similar_items(recent_items, item_pool, top_k=10)

2. Data Strategy

Training data sources

• Explicit feedback: User ratings, clicks, likes, shares, and direct engagement signals (e.g., "Save for later", "Add to cart")
• Implicit signals: Session duration, scroll depth, time spent on item, view-through rates, and engagement patterns (e.g., "Viewed 3 items in a row")
• Third-party data: Demographic information, user behavior from partner platforms, contextual signals (weather, time of day), and external content features (e.g., topic tags, sentiment scores)
• Synthetic data: Generated via data augmentation or domain-specific simulators for rare event modeling (e.g., long-tail item interactions)
• Historical data: Past user-item interactions, session logs, and behavioral sequences (e.g., browsing history, cart abandonment)

Labeling pipeline design

• Human annotation: For high-value, rare events (e.g., explicit user preferences for new items) with quality control via inter-annotator agreement and gold standard validation
• Weak supervision: Using heuristics (e.g., "User clicked item X if they viewed Y items in the last 5 minutes") or labeling functions to generate pseudo-labels for large-scale training
• Programmatic labeling: Automating labels using business rules (e.g., "If user viewed item and didn't click within 30 seconds, label as negative") or ML-based pseudo-labeling for cold-start items
• Active learning: Selectively labeling samples with high uncertainty or high impact to maximize label efficiency

Class imbalance handling strategy with justification

• Oversampling: For rare positive labels (e.g., high-value item purchases) to ensure model learns patterns for these events
• Undersampling: For majority negative samples (e.g., general browsing) to prevent model bias toward frequent interactions
• Class weights: Applied in loss functions to penalize misclassification of rare events more heavily (e.g., weight positive samples 5x higher)
• Focal Loss: For highly imbalanced datasets (e.g., <1% purchase rate) to focus training on hard-to-classify examples
• Justification: Without balancing, models may become biased toward frequent interactions and fail to generalize for rare but valuable events

Negative sampling approach

• Uniform sampling: Randomly sample negative items from the item pool (simple but may not reflect user preferences)
• Popularity-based sampling: Sample negatives proportionally to item popularity (e.g., items with higher view counts are more likely to be sampled)
• Inverse frequency sampling: Sample negatives inversely proportional to item occurrence (e.g., rare items more likely to be selected)
• Context-aware negative sampling: Sample negatives based on user context (e.g., items from same category or similar to viewed items)
• Hard negative mining: Select hard negatives using model confidence scores (e.g., items with low predicted probability but high actual interaction)
• Justification: Context-aware and hard negative sampling improve model robustness and generalization, especially in recommendation systems where users have diverse tastes

Data freshness and update frequency requirements

• Real-time signals: User behavior updates every 5–10 minutes (e.g., click, view, scroll)
• Batch updates: Item features and user profiles updated daily (e.g., content metadata, user demographics)
• Historical data: 30-day rolling window for training (to capture recent trends and user drift)
• Freshness SLA: 95% of training data within 24 hours of collection (critical for trending content)
• Back-of-envelope estimates:
  • Training data size: ~100M user-item interactions (50GB raw)
  • Number of features: ~10,000 (user + item + context)
  • Embedding matrix memory footprint: ~10GB (50M items × 256 dimensions)
  • Serving QPS math: 100K QPS with 20ms latency budget → 5ms per stage (retrieval, scoring, reranking)

Privacy compliance (GDPR, CCPA)

• Data anonymization: Remove PII fields (user IDs, email addresses) and apply k-anonymity to user groups
• Data retention policies: Retain only 6 months of behavioral data, with automatic deletion of older records
• User consent management: Implement opt-out mechanisms for tracking and data usage (e.g., "Do Not Track" flag)
• Access control: Role-based access to sensitive datasets with audit logs and encryption in transit and at rest
• Data minimization: Collect only necessary features for model training (e.g., no full browsing history, just key actions)

Data quality validation and monitoring

• Schema validation: Ensure all fields conform to expected types (e.g., numeric for item ID, string for user agent)
• Distribution drift detection: Monitor feature distributions using KL divergence or PSI (Population Stability Index) for early drift signals
• Duplicate detection: Identify and remove duplicate user-item interactions (e.g., same click logged multiple times)
• Missing value handling: Flag and impute missing features (e.g., use default values or skip training samples)
• Outlier detection: Flag and analyze outliers in interaction counts, user session lengths, or item popularity to prevent model skew

import numpy as np
from collections import Counter

def sample_negative_items_uniform(user_item_pairs, item_pool_size, num_negatives=5):
    """Uniform negative sampling for recommendation systems."""
    negatives = []
    for user_id, item_id in user_item_pairs:
        sampled_items = np.random.choice(
            [i for i in range(item_pool_size) if i != item_id],
            size=num_negatives,
            replace=False
        )
        negatives.append(sampled_items)
    return negatives

def sample_negative_items_contextual(user_item_pairs, item_pool, user_contexts, num_negatives=5):
    """Context-aware negative sampling."""
    negatives = []
    for user_id, item_id in user_item_pairs:
        # Example: sample from same category or similar items
        context = user_contexts.get(user_id, {})
        category = context.get('category', 'default')
        similar_items = [i for i in item_pool if i != item_id and item_pool[i].get('category') == category]
        sampled_items = np.random.choice(
            similar_items,
            size=min(num_negatives, len(similar_items)),
            replace=False
        )
        negatives.append(sampled_items)
    return negatives

# Example usage:
user_item_pairs = [(1, 10), (2, 20)]
item_pool_size = 100
negatives_uniform = sample_negative_items_uniform(user_item_pairs, item_pool_size)
print("Uniform negatives:", negatives_uniform)

# Context-aware example (mock data)
user_contexts = {1: {'category': 'electronics'}, 2: {'category': 'books'}}
item_pool = {i: {'category': 'electronics' if i < 50 else 'books'} for i in range(100)}
negatives_contextual = sample_negative_items_contextual(user_item_pairs, item_pool, user_contexts)
print("Contextual negatives:", negatives_contextual)

3. Feature Engineering

Feature Taxonomy

Features are categorized into four main types:

• User Features: User demographics, behavior patterns, preferences (e.g., age, gender, session duration, click frequency)
• Item Features: Content metadata, popularity metrics, category tags (e.g., title, description, rating, view count)
• Context Features: Temporal signals, device information, geographic location (e.g., hour of day, country, device type)
• Interaction Features: User-item engagement signals, temporal decay factors (e.g., last interaction time, recency score)

Embedding Design

Entity Embeddings:

• User ID embeddings: 128-dim, trained via collaborative filtering
• Item ID embeddings: 128-dim, learned from user-item interaction matrix

Text Embeddings:

• Item titles and descriptions: 256-dim using BERT-base (fine-tuned on domain data)
• User profiles: 128-dim using TF-IDF weighted average of item titles

Graph Embeddings:

• User-item interaction graph: 64-dim using Node2Vec for capturing neighborhood patterns

Memory Footprint:

• User embedding matrix: ~1M users × 128 dimensions = ~500MB
• Item embedding matrix: ~10M items × 128 dimensions = ~5GB
• Text embeddings: ~5GB (BERT model weights + cache)

Real-time vs Batch Feature Computation

Feature Type	Computation Mode	Reason
User ID, Item ID	Batch	Static identifiers
Session features	Real-time	Dynamic, requires low latency
Recent interactions	Real-time	High temporal relevance
Historical statistics	Batch	Expensive to compute in real-time

Feature Store Design

Online Store:

• Low-latency KV store (e.g., Redis or DynamoDB)
• Designed for sub-100ms retrieval with TTL support

Offline Store:

• Data warehouse (e.g., BigQuery or Snowflake)
• Used for training and batch feature generation

Feature Freshness Table:

Feature	Source	Update Frequency	Store Type
User age	Profile API	Daily	Online
Item popularity	Analytics DB	Hourly	Offline
Recent clicks	Real-time stream	Milliseconds	Online
User category	Historical logs	Daily	Offline

Cross Features and Feature Interactions

Cross features are generated using:

• Product interactions: User-category × Item-category, User-device × Item-category
• Temporal decay: User-item interaction time × item popularity

Cross features are computed in real-time during inference using feature cross tables.

Feature Importance and Selection

• L1 regularization in logistic regression for automatic feature selection
• SHAP values for model interpretability and ranking features by importance
• Permutation importance from trained models to validate feature relevance

import numpy as np
from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LogisticRegression

# Example feature selection using L1 regularization
def select_features_with_l1(X_train, y_train, X_test):
    model = LogisticRegression(penalty='l1', solver='liblinear')
    model.fit(X_train, y_train)
    
    # Select features with non-zero coefficients
    selector = SelectFromModel(model, prefit=True)
    X_train_selected = selector.transform(X_train)
    X_test_selected = selector.transform(X_test)
    
    return X_train_selected, X_test_selected, selector

4. Model Architecture

Model selection with justification (logistic regression / GBM / DNN / Transformer — trade-offs)

For a recommendation system with negative sampling, we select a Two-Tower DNN architecture as the primary model:

• Logistic Regression: Simple but insufficient for capturing complex user-item interactions and context dependencies
• GBM (XGBoost/LightGBM): Good for structured features but struggles with high-dimensional sparse embeddings and sequence modeling
• Two-Tower DNN: Ideal for recommendation systems with user/item embeddings, supports real-time inference, and scales well to large embedding spaces
• Transformer: Overkill for this use case without sequence modeling needs, introduces higher latency and complexity

The Two-Tower architecture offers:

• Strong representation learning through embeddings
• Efficient real-time scoring with low latency (ideal for serving QPS of 10k+)
• Scalable to large embedding spaces (user/item features up to 10M dimensions)
• Natural fit for negative sampling training with contrastive loss

Architecture diagram (Mermaid)

graph TD
    A[User Input] --> B[User Tower]
    C[Item Input] --> D[Item Tower]
    
    B --> E[User Embedding]
    D --> F[Item Embedding]
    
    E --> G[Merge Layer]
    F --> G
    
    G --> H[Dot Product/MLP]
    H --> I[Scoring Function]
    
    I --> J[Loss Function]
    
    subgraph Model
        B
        D
        E
        F
        G
        H
        I
        J
    end
    
    K[Positive Item] --> D
    L[Negative Samples] --> D

Two-tower / multi-task / ensemble design if applicable

Two-Tower architecture with:

• User Tower: Processes user features (user ID, demographics, historical behavior)
• Item Tower: Processes item features (item ID, category, metadata)
• Scoring: Dot product between user and item embeddings for ranking
• Loss: Binary Cross Entropy with negative sampling

Pre-trained model selection and fine-tuning strategy (for LLM / foundation model tasks)

Not applicable for recommendation systems without natural language content. For content-rich recommendations, we would:

• Use pre-trained BERT embeddings for item descriptions
• Fine-tune with contrastive learning on user-item interactions
• Apply knowledge distillation from larger models to reduce serving latency

Model complexity vs latency trade-off analysis (table: model / metric / latency / size)

Model	Metric	P50 Latency (ms)	Size (MB)
Logistic Regression	AUC	1	0.5
XGBoost	AUC	5	2
Two-Tower DNN (Small)	AUC	10	5
Two-Tower DNN (Medium)	AUC	25	20
Two-Tower DNN (Large)	AUC	60	100

Offline evaluation baseline framing (expected baseline vs proposed model)

Baseline: Logistic regression with user and item IDs as features
Expected improvement: 5-10% in AUC-ROC vs baseline

Core model code snippet

import torch
import torch.nn as nn
import torch.nn.functional as F

class TwoTowerModel(nn.Module):
    def __init__(self, user_vocab_size, item_vocab_size, embedding_dim=128):
        super().__init__()
        self.user_embedding = nn.Embedding(user_vocab_size, embedding_dim)
        self.item_embedding = nn.Embedding(item_vocab_size, embedding_dim)
        self.user_tower = nn.Sequential(
            nn.Linear(embedding_dim, 256),
            nn.ReLU(),
            nn.Linear(256, 128)
        )
        self.item_tower = nn.Sequential(
            nn.Linear(embedding_dim, 256),
            nn.ReLU(),
            nn.Linear(256, 128)
        )
    
    def forward(self, user_ids, item_ids):
        # Get embeddings
        user_emb = self.user_embedding(user_ids)  # [B, E]
        item_emb = self.item_embedding(item_ids)  # [B, E]
        
        # Process through towers
        user_repr = self.user_tower(user_emb)  # [B, 128]
        item_repr = self.item_tower(item_emb)  # [B, 128]
        
        # Compute dot product score
        scores = torch.sum(user_repr * item_repr, dim=1)  # [B]
        return scores
    
    def compute_loss(self, user_ids, positive_items, negative_items):
        # Compute positive scores
        pos_scores = self.forward(user_ids, positive_items)  # [B]
        
        # Compute negative scores
        neg_scores = self.forward(user_ids, negative_items)  # [B, N]
        
        # Binary cross entropy loss with contrastive sampling
        pos_labels = torch.ones_like(pos_scores)
        neg_labels = torch.zeros_like(neg_scores).flatten()
        
        # Reshape pos_scores to match neg_scores shape for concatenation
        pos_scores_expanded = pos_scores.unsqueeze(1).expand(-1, neg_scores.shape[1]).flatten()
        
        # Combine all scores and labels
        all_scores = torch.cat([pos_scores_expanded, neg_scores.flatten()])
        all_labels = torch.cat([pos_labels, neg_labels])
        
        # Compute BCE loss
        loss = F.binary_cross_entropy_with_logits(all_scores, all_labels.float())
        return loss

# Example usage:
# model = TwoTowerModel(user_vocab_size=10000, item_vocab_size=50000)
# loss = model.compute_loss(user_ids, pos_items, neg_items)

This implementation includes:

• Two-tower architecture with embedding layers and MLP towers
• Contrastive loss computation for negative sampling training
• Efficient batch processing for real-time serving
• Standard PyTorch tensor operations for optimal performance

5. Serving & Inference Architecture

System Diagram

graph TD
    A[User Request] --> B[Load Balancer]
    B --> C[Query Router]
    C --> D[Feature Store]
    D --> E[Embedding Cache]
    E --> F[Model Serving]
    F --> G[Scoring Engine]
    G --> H[Ranking Service]
    H --> I[Response Formatter]
    I --> J[Client Response]

    subgraph "Online Serving Layer"
        C
        F
        G
        H
        I
    end

    subgraph "Feature Layer"
        D
        E
    end

Online vs Batch Serving Decision

For recommendation systems, we prioritize online serving due to:

• Real-time personalization requirements (user session context)
• Low latency SLA (50ms p99 for user-facing experiences)
• Dynamic content relevance needs
• Immediate feedback loop for model improvements

Batch serving is used for:

• Offline model evaluation and retraining
• Scheduled feature generation and updates
• Historical analysis and reporting

Two-Stage Pipeline

Stage 1: Candidate Retrieval → Stage 2: Scoring & Re-ranking

Retrieval Stage (p50: 10ms, p99: 30ms)

• Uses approximate nearest neighbor (ANN) search with FAISS or Annoy
• Embedding-based retrieval for user and item representations
• Filters based on user preferences, content categories, or temporal constraints

Scoring Stage (p50: 15ms, p99: 50ms)

• Full model scoring of top-K candidates from retrieval stage
• Two-tower architecture with user and item embeddings
• Attention-based scoring or dot-product similarity

Re-ranking Stage (p50: 20ms, p99: 60ms)

• Diversity and fairness constraints
• Business rules (e.g., promoting new items, avoiding content fatigue)
• Context-aware re-ranking using temporal or session features

Serving Infrastructure

We use TorchServe for model serving with the following configuration:

# Example TorchServe model configuration (simplified)
{
    "model_name": "recommendation_model",
    "model_url": "s3://ml-models/recommendation_model.mar",
    "batch_size": 32,
    "max_batch_delay": 100,
    "gpu": True,
    "engine": "PYTORCH",
    "model_server": "torchserve"
}

Justification:

• Support for PyTorch models with native ONNX export capability
• Built-in batching and dynamic batching support
• Easy integration with Kubernetes for container orchestration
• Native support for model versioning and A/B testing

Latency Budget Breakdown

Stage	p50	p99	Budget
Feature Retrieval	5ms	15ms	20ms
Embedding Lookup	3ms	10ms	15ms
Model Scoring	15ms	50ms	60ms
Re-ranking	20ms	60ms	75ms
Response Formatting	2ms	5ms	10ms
Total	45ms	140ms	175ms

Model Compression Strategy

We apply quantization-aware training (QAT) and pruning:

# Example model compression in PyTorch
import torch
from torch.quantization import quantize_dynamic

# Quantize model for inference
model.eval()
quantized_model = quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8
)

# Pruning for reducing model size
from torch.nn.utils import prune

prune.l1_unstructured(model.fc1, name="weight", amount=0.3)

Trade-offs:

• 25% reduction in model size with <5% accuracy drop
• Reduced memory bandwidth usage for embedding lookups
• Increased inference latency due to quantization overhead (~2ms)

Caching Strategy

# Example caching layer using Redis for prediction results
import redis
import pickle

class PredictionCache:
    def __init__(self, redis_client):
        self.redis = redis_client
    
    def get_cached_prediction(self, user_id, item_ids):
        cache_key = f"rec:{user_id}:{hash(str(item_ids))}"
        cached = self.redis.get(cache_key)
        if cached:
            return pickle.loads(cached)
        return None
    
    def cache_prediction(self, user_id, item_ids, predictions, ttl=3600):
        cache_key = f"rec:{user_id}:{hash(str(item_ids))}"
        self.redis.setex(cache_key, ttl, pickle.dumps(predictions))

Cache Strategy:

• Prediction Cache: 1-hour TTL for user-item pairs with high confidence
• Embedding Cache: 5-minute TTL for frequently accessed embeddings
• KV Cache: For LLM-based content generation (if applicable)

Auto-scaling and Capacity Planning

# Example auto-scaling logic using Kubernetes HPA
from kubernetes import client

def calculate_target_cpu_utilization(current_qps, target_qps):
    # Simple linear scaling based on QPS
    return min(80, max(20, (current_qps / target_qps) * 100))

# HPA configuration
hpa_config = {
    "apiVersion": "autoscaling/v2",
    "kind": "HorizontalPodAutoscaler",
    "metadata": {"name": "recommendation-hpa"},
    "spec": {
        "scaleTargetRef": {
            "apiVersion": "apps/v1",
            "kind": "Deployment",
            "name": "recommendation-service"
        },
        "minReplicas": 3,
        "maxReplicas": 20,
        "metrics": [
            {
                "type": "Resource",
                "resource": {
                    "name": "cpu",
                    "target": {
                        "type": "Utilization",
                        "averageUtilization": 60
                    }
                }
            }
        ]
    }
}

Capacity Planning:

• QPS Target: 10,000 requests/sec (peak)
• Model Memory: ~2GB per model instance
• Cache Memory: ~50GB for embedding cache (estimated)
• Expected Scaling Factor: 2x peak capacity for 99th percentile handling

Auto-scaling Strategy:

• CPU utilization target: 60%
• QPS-based scaling: 5% QPS increase triggers scaling up
• Graceful degradation for low QPS (scale down to 1 replica)

6. Evaluation Framework

Offline Evaluation Methodology

For offline evaluation, we use a temporal split approach to ensure that our model is evaluated on data from the future relative to training data, preventing data leakage. This is critical for ranking systems where temporal consistency matters.

We split our dataset chronologically into:

• Training: 2023-01-01 to 2023-09-30 (75%)
• Validation: 2023-10-01 to 2023-10-31 (5%)
• Test: 2023-11-01 to 2023-11-30 (5%)

We also implement time-aware cross-validation to ensure robustness across different time periods.

Slice-Based Evaluation

We evaluate model performance across critical user slices to detect potential bias or degradation:

Slice	Definition	Rationale
New vs. Existing Users	Users with < 5 interactions vs. > 5 interactions	Ensures cold-start handling
High vs. Low Engagement	Top 20% vs Bottom 20% of engagement	Detects model fairness
Geographic Regions	US, EU, APAC	Identifies regional drift
Device Types	Mobile vs. Desktop	Ensures device-agnostic performance

A/B Test Design

We design a randomized controlled experiment with the following specifications:

• Randomization Unit: User ID (to avoid spill-over effects)

• Sample Size Calculation:

  • Target: 5% relative lift in CTR (0.05)
  • Power: 80%
  • Significance Level: 0.05
  • Estimated baseline CTR: 2.5%
  • Required sample size per group: ~14,000 users

• Duration: 3 weeks (to capture weekly seasonality)

• Primary Metric: Click-through rate (CTR) with 95% confidence interval

• Secondary Metrics: Revenue per user, session duration

Guardrail Metrics and Blocking Conditions

To ensure safe deployment, we define guardrail metrics:

Metric	Threshold	Action
CTR Drop	< 95% of baseline	Block deployment
Revenue Drop	< 90% of baseline	Block deployment
Model Confidence	< 80% for > 5% of predictions	Alert and rollback
User Engagement Drop	< 90% of baseline	Alert and rollback

Shadow Mode Rollout Plan

Before full A/B test, we implement a shadow mode rollout:

1. Route 5% of traffic through the new model in shadow mode
2. Log predictions and compare with baseline behavior
3. Validate that shadow model outputs are consistent with expected behavior
4. Run statistical tests on logged predictions to ensure they're within acceptable variance

Long-Term Metric Measurement

We track long-term user engagement and conversion:

Metric	Definition	Monitoring Period
Delayed Conversion Rate	Users who convert within 30 days of interaction	30-day post-activation
Engagement Decay	Session frequency over time	90-day tracking
Retention Rate	Users returning after 7, 30 days	30-day retention window

# Example A/B test evaluation logic in Python

import numpy as np
from scipy import stats

def ab_test_evaluation(
    control_ctr: float,
    treatment_ctr: float,
    control_users: int,
    treatment_users: int,
    confidence_level: float = 0.95
) -> dict:
    """
    Evaluate A/B test results for CTR difference.
    
    Returns:
        Dictionary with statistical significance and confidence bounds
    """
    # Calculate standard errors
    se_control = np.sqrt(control_ctr * (1 - control_ctr) / control_users)
    se_treatment = np.sqrt(treatment_ctr * (1 - treatment_ctr) / treatment_users)
    
    # Standard error of difference
    se_diff = np.sqrt(se_control**2 + se_treatment**2)
    
    # Z-score for confidence level
    z_score = stats.norm.ppf((1 + confidence_level) / 2)
    
    # Confidence interval bounds
    margin_error = z_score * se_diff
    lower_bound = treatment_ctr - control_ctr - margin_error
    upper_bound = treatment_ctr - control_ctr + margin_error
    
    # Statistical significance test (two-tailed)
    z_stat = (treatment_ctr - control_ctr) / se_diff
    p_value = 2 * (1 - stats.norm.cdf(abs(z_stat)))
    
    return {
        "p_value": p_value,
        "confidence_interval": (lower_bound, upper_bound),
        "statistically_significant": p_value < 0.05,
        "lift_percentage": (treatment_ctr - control_ctr) / control_ctr * 100
    }

# Example usage:
result = ab_test_evaluation(
    control_ctr=0.025,
    treatment_ctr=0.026,
    control_users=15000,
    treatment_users=15000
)
print(result)

7. Training Pipeline

Training Data Pipeline

The training pipeline follows a multi-stage ETL process:

1. Data Ingestion: Raw data flows from Kafka streams and S3 buckets
2. Preprocessing: Feature extraction, normalization, and filtering
3. Shuffling: Distributed shuffling using Apache Spark for large-scale data
4. Batching: Batching with configurable batch sizes (1024-8192 samples)
5. Sampling: Negative sampling with configurable ratios

import torch
from torch.utils.data import Dataset, DataLoader
import numpy as np

class NegativeSamplingDataset(Dataset):
    def __init__(self, user_ids, item_ids, labels, negative_ratio=5):
        self.user_ids = torch.tensor(user_ids, dtype=torch.long)
        self.item_ids = torch.tensor(item_ids, dtype=torch.long)
        self.labels = torch.tensor(labels, dtype=torch.float32)
        self.negative_ratio = negative_ratio
        
    def __len__(self):
        return len(self.user_ids)
    
    def __getitem__(self, idx):
        # Get positive sample
        user_id = self.user_ids[idx]
        item_id = self.item_ids[idx]
        label = self.labels[idx]
        
        # Generate negative samples
        if label == 1.0:  # Positive sample, generate negatives
            negative_items = []
            for _ in range(self.negative_ratio):
                # Sample from item distribution (can be uniform or weighted by popularity)
                neg_item = np.random.randint(0, self.item_ids.max().item())
                negative_items.append(neg_item)
            
            # Return positive sample + negatives
            return {
                'user_id': user_id,
                'item_id': item_id,
                'label': label,
                'negative_items': torch.tensor(negative_items, dtype=torch.long)
            }
        else:
            return {
                'user_id': user_id,
                'item_id': item_id,
                'label': label
            }

# Training loop example
def train_model(model, dataloader, optimizer, criterion, num_epochs=10):
    model.train()
    for epoch in range(num_epochs):
        total_loss = 0
        for batch in dataloader:
            optimizer.zero_grad()
            
            # Forward pass - model should handle both positive and negative samples
            user_embeddings = model.user_encoder(batch['user_id'])
            item_embeddings = model.item_encoder(batch['item_id'])
            
            # Compute loss for positive samples
            pos_scores = torch.sum(user_embeddings * item_embeddings, dim=1)
            pos_loss = criterion(pos_scores, torch.ones_like(pos_scores))
            
            # Compute loss for negative samples (if present)
            if 'negative_items' in batch:
                neg_embeddings = model.item_encoder(batch['negative_items'])
                neg_scores = torch.bmm(
                    user_embeddings.unsqueeze(1), 
                    neg_embeddings.transpose(1, 2)
                ).squeeze(1)
                
                # Binary cross entropy loss for negatives
                neg_loss = criterion(neg_scores, torch.zeros_like(neg_scores))
                loss = pos_loss + neg_loss
            else:
                loss = pos_loss
                
            loss.backward()
            optimizer.step()
            total_loss += loss.item()
        
        print(f"Epoch {epoch+1}/{num_epochs}, Loss: {total_loss/len(dataloader):.4f}")

Distributed Training Strategy

• Data Parallelism: Using PyTorch's DistributedDataParallel for multi-GPU training
• Model Parallelism: For very large models, splitting across different GPUs
• Pipeline Parallelism: For transformer-based architectures with multiple stages

Training cluster configuration:

• 8x A100 GPUs (40GB each)
• 256GB RAM per node
• 100 Gbps InfiniBand interconnect

Training Infrastructure

• GPU Cluster: 32x A100 GPUs across 4 nodes (8 GPUs per node)
• Training Time: ~2 hours for full dataset with batch size 4096
• Memory Usage: ~12GB per GPU (with mixed precision training)
• Data Pipeline: Apache Spark + Ray for distributed data processing

Hyperparameter Tuning Approach

• Search Strategy: Bayesian Optimization with Optuna
• Key Parameters:
  • Learning rate (1e-4 to 1e-2)
  • Batch size (512, 1024, 2048)
  • Negative sampling ratio (3, 5, 10)
  • Embedding dimensions (64, 128, 256)
  • Number of epochs (5-20)

Experiment Tracking and Reproducibility

• MLflow for experiment tracking, versioning, and model registry
• Parameter logging: All training parameters, metrics, and artifacts
• Model versioning: Semantic versioning with tags (v1.0.0, v1.1.0)
• Reproducibility: Docker containers with pinned dependencies

Model Versioning and Registry Strategy

• Model Registry: MLflow model registry with stages (Staging, Production)
• Versioning: Semantic versioning (v1.0.0, v1.0.1, v1.1.0)
• Tagging: Release tags (stable, experimental) and metadata (precision, recall)
• Rollback capability: Automatic rollback to previous stable version

Continual / Online Learning Design

• Online Update Frequency: Every 2 hours for new user/item interactions
• Incremental Training: Using small batches from recent data
• Drift Detection: Statistical tests (Kolmogorov-Smirnov) on feature distributions
• Re-training Trigger: When drift exceeds 5% threshold or model performance drops by 2%
• Data Freshness: 30-minute window for real-time updates

8. Monitoring & Observability

Data Drift Detection

Data drift detection is critical for maintaining model performance over time. We implement drift detection using Population Stability Index (PSI) and Kullback-Leibler divergence for continuous features, and Chi-square tests for categorical features. Drift thresholds are set at 0.1 PSI for continuous features and 0.05 for categorical features, with alerts triggered when drift exceeds these thresholds.

import numpy as np
from scipy import stats

def calculate_psi(expected, actual, bins=10):
    """Calculate PSI between expected and actual distributions"""
    # Ensure both arrays have same binning
    _, bin_edges = np.histogram(expected, bins=bins)
    
    # Count occurrences in each bin
    expected_counts, _ = np.histogram(expected, bins=bin_edges)
    actual_counts, _ = np.histogram(actual, bins=bin_edges)
    
    # Avoid division by zero
    expected_counts = np.where(expected_counts == 0, 1e-6, expected_counts)
    actual_counts = np.where(actual_counts == 0, 1e-6, actual_counts)
    
    # Calculate PSI
    psi = np.sum((actual_counts - expected_counts) * 
                 np.log(actual_counts / expected_counts))
    return psi

def detect_drift(expected_data, actual_data, feature_type="continuous"):
    """Detect drift between expected and actual data distributions"""
    if feature_type == "continuous":
        psi = calculate_psi(expected_data, actual_data)
        return psi > 0.1
    elif feature_type == "categorical":
        chi2, p_value = stats.chisquare(actual_data, expected_data)
        return p_value < 0.05

Prediction Drift and Label Drift Monitoring

Prediction drift is monitored by tracking the distribution of model predictions over time. We use the Kolmogorov-Smirnov test to compare prediction distributions between time windows. Label drift is monitored by tracking the distribution of labels in training vs. current data.

Model Performance Degradation Alerting

Performance degradation is monitored using a sliding window approach with 30-day lookback windows. Alert thresholds are:

• AUC drop > 5% over 30-day window
• Precision@K drop > 10% over 30-day window
• NDCG drop > 5% over 30-day window

Retraining Trigger Strategy

Retraining is triggered by:

1. Scheduled retraining: Monthly or bi-weekly based on business cycles
2. Drift-triggered: When PSI > 0.15 for any feature or prediction drift detected
3. Performance-triggered: When model performance drops below threshold for 5 consecutive days

Logging Design

Prediction logs include:

• User ID, item ID, timestamp
• Input features (with metadata)
• Model predictions and confidence scores
• Ground truth labels (when available)
• Negative sampling strategy used

Feature logs capture:

• Feature values at prediction time
• Feature update timestamps
• Feature freshness indicators

Label collection pipeline:

class LabelCollectionPipeline:
    def __init__(self):
        self.label_buffer = []
        
    def collect_label(self, user_id, item_id, timestamp, label):
        """Collect and store labels for model retraining"""
        self.label_buffer.append({
            'user_id': user_id,
            'item_id': item_id,
            'timestamp': timestamp,
            'label': label,
            'source': 'user_interaction'
        })
        
    def flush_to_storage(self):
        """Flush collected labels to storage system"""
        # Implementation for storing labels in database or data lake
        pass

Feedback Loop for Continuous Improvement

Feedback loop includes:

1. Label validation: Cross-checking labels with business logic
2. User feedback integration: Incorporating explicit user feedback (e.g., "not relevant" signals)
3. Model retraining: Automated retraining when drift thresholds exceeded
4. Feature evolution: Periodic feature importance analysis to identify new signal sources

Drift Detection Dashboard

We implement a drift monitoring dashboard with:

• Real-time PSI scores for all features
• Prediction distribution comparison charts
• Performance degradation trend visualization
• Alert severity levels (low, medium, high)
• Automated drift detection alerts sent to ML team via Slack/Email

Key Metrics for Monitoring

Metric	Description	Threshold	Alert Level
PSI	Population Stability Index	> 0.15	High
NDCG	Normalized Discounted Cumulative Gain	< 0.95 (30-day avg)	Medium
AUC	Area Under ROC Curve	< 0.85 (30-day avg)	High
Precision@K	Precision at K items	< 0.75 (30-day avg)	Medium
Drift Count	Number of drifted features	> 5	High

Alert Escalation Matrix

Alert Type	Response Time	Escalation Path
Low drift	2 hours	ML Engineer
Medium drift	4 hours	ML Team Lead
High drift	1 hour	Engineering Manager, Product

Retraining Pipeline Integration

Drift detection integrates with the retraining pipeline to:

1. Flag datasets requiring retraining
2. Generate drift reports for model evaluation
3. Trigger automated retraining workflows when thresholds exceeded
4. Maintain drift history for analysis and model evolution

This comprehensive monitoring framework ensures that model performance is maintained, drift is detected proactively, and the system can adapt to changing data distributions in real-time.

9. Responsible AI & Safety

Bias Analysis and Fairness

Bias in negative sampling can amplify existing disparities in the dataset. For example, if we disproportionately sample negatives from underrepresented user groups or item categories, we risk reinforcing systemic biases in model predictions. We must audit training data for demographic distributions and ensure that negative sampling does not disproportionately exclude any user segment or item type.

We will implement fairness metrics such as demographic parity and equalized odds to detect bias in model predictions across slices (e.g., gender, age group, geography). These metrics will be monitored in both offline training and online serving environments.

Fairness Metrics and Constraints

We define fairness constraints as follows:

• Demographic Parity: The model should predict positive outcomes with equal probability across user demographics (e.g., gender, age groups)
• Equalized Odds: True positive and false positive rates should be equal across user slices

For our ranking model, we will implement fairness-aware loss functions that penalize deviations from these constraints during training. The specific constraint will depend on business priorities and regulatory requirements.

Explainability Requirements

We will implement attention-based explainability for our ranking model to understand which features contribute most to the negative sampling decisions. For instance, if a particular item category is over-sampled in negatives, we want to understand whether this stems from the model's learned preferences or data bias.

We will use SHAP values for feature importance analysis and LIME for local explanations of individual predictions. These tools will help stakeholders understand model behavior and identify potential bias sources.

Safety Guardrails for LLM Systems

For systems involving large language models, we implement the following safety measures:

• Content Filtering: Pre-filtering of inputs and outputs using keyword blacklists and toxicity classifiers
• Rate Limiting: Per-user rate limits to prevent abuse and ensure system stability
• Red Teaming: Regular adversarial testing with human evaluators to identify edge cases and harmful outputs
• Output Validation: Post-processing checks for hallucination, factual accuracy, and alignment with safety policies

Model Card Documentation Plan

We will maintain comprehensive model cards that include:

• Training data sources and preprocessing steps
• Model architecture and hyperparameters
• Performance metrics across different slices
• Known limitations and bias considerations
• Safety measures implemented
• Fairness constraints and monitoring

Incident Response Plan for Model Failures

In case of model failures or harmful outputs:

1. Immediate Isolation: Automatically disable affected model versions and route traffic to fallback systems
2. Root Cause Analysis: Investigate drift in training data, model degradation, or system configuration changes
3. Manual Review: Engage human reviewers for flagged outputs before re-enabling the model
4. Communication Protocol: Notify stakeholders and impacted users as appropriate

The system will include automated alerts for:

• Drop in model performance (e.g., AUC below threshold)
• Drift in prediction distributions
• Unusual patterns in negative sampling (e.g., disproportionate negative samples from specific user segments)

This approach ensures responsible AI deployment while maintaining system reliability and user trust.

10. Scalability & Reliability

Bottleneck Analysis per Pipeline Stage

Stage	Bottleneck Type	Back-of-Envelope Estimate	Mitigation Strategy
Data Ingestion	I/O throughput	10GB/s peak for 10M daily interactions	Distributed file systems (S3/HDFS), parallel readers
Feature Computation	CPU/memory	500GB feature matrix, 100M users	Distributed feature store with caching
Model Training	GPU memory	20GB per batch for 1M items	Data parallelism, gradient checkpointing
Model Serving	Latency	50ms p99 target	Two-stage pipeline, model compression
Negative Sampling	Compute	100x sampling ratio = 100M samples/sec	Pre-computed pools, streaming sampling

Scale-up Strategy

Horizontal Scaling of Serving:

• Deploy model replicas across multiple regions to reduce latency
• Use Kubernetes for auto-scaling based on QPS metrics
• Implement consistent hashing for feature store distribution

Distributed Feature Computation:

• Partition feature computation by user ID ranges
• Use Ray or Dask for distributed feature extraction
• Cache frequently accessed features in Redis/Memcached

Fault Tolerance

Model Serving Fallbacks:

• Implement graceful degradation to rule-based fallback when model fails
• Use circuit breaker pattern for external dependencies (feature store, embedding services)
• Maintain last-known-good model version in local cache

Stale Model Serving:

class FallbackModel:
    def __init__(self, primary_model, fallback_strategy):
        self.primary_model = primary_model
        self.fallback_strategy = fallback_strategy
        self.last_good_version = None
        
    def predict(self, inputs):
        try:
            return self.primary_model.predict(inputs)
        except Exception as e:
            # Log error and fallback
            logger.warning(f"Model failed, using fallback: {e}")
            return self.fallback_strategy(inputs)

Cold Start Handling Strategy

New User Cold Start:

• Use content-based filtering with demographic features
• Implement user embedding initialization from similar users
• Fall back to popular items ranking for first interactions

New Item Cold Start:

• Use item metadata embeddings (category, tags)
• Apply collaborative filtering with item popularity
• Implement content-based retrieval from item features

Multi-Region Serving Strategy

Global Load Balancing:

class MultiRegionServing:
    def __init__(self, regions):
        self.regions = regions
        self.latency_metrics = {}
        
    def get_optimal_region(self, user_location):
        # Select region with lowest latency for user
        min_latency = float('inf')
        best_region = None
        
        for region in self.regions:
            latency = self.get_region_latency(region, user_location)
            if latency < min_latency:
                min_latency = latency
                best_region = region
                
        return best_region

Cross-Region Model Synchronization:

• Implement model versioning with region-specific model IDs
• Use consistent hash for model distribution across regions
• Maintain model synchronization through periodic retraining and rollout

Regional Failover:

• Set up regional failover clusters with automatic traffic routing
• Implement health checks for each region's model serving endpoints
• Use weighted round-robin distribution with region-specific weights based on performance

Reliability Patterns

Circuit Breaker for Feature Store:

from circuitbreaker import circuit

@circuit(failure_threshold=5, recovery_timeout=60)
def get_user_features(user_id):
    return feature_store.get(user_id)

Retry Logic with Exponential Backoff:

import time
import random

def retry_with_backoff(func, max_retries=3):
    for attempt in range(max_retries):
        try:
            return func()
        except Exception as e:
            if attempt == max_retries - 1:
                raise e
            wait_time = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait_time)

Prediction Caching with TTL:

class PredictionCache:
    def __init__(self, ttl_seconds=3600):
        self.cache = {}
        self.ttl = ttl_seconds
        
    def get(self, key):
        if key in self.cache:
            value, timestamp = self.cache[key]
            if time.time() - timestamp < self.ttl:
                return value
            else:
                del self.cache[key]
        return None
        
    def set(self, key, value):
        self.cache[key] = (value, time.time())

Performance Monitoring

Latency Distribution Tracking:

• Monitor p50, p90, p99 latency across all serving stages
• Track QPS and error rates per region and model version
• Implement alerting for latency spikes > 2x baseline

Resource Utilization:

• Track GPU memory usage, CPU utilization, and I/O throughput
• Monitor feature store cache hit rates (target > 90%)
• Implement auto-scaling based on predicted QPS trends

Capacity Planning

QPS Projection:

• Current: 10,000 QPS peak
• Projected growth: 5x over 12 months (50,000 QPS)
• Required capacity: 60,000 QPS with 20% headroom

Infrastructure Requirements:

• Model serving: 600 GPU instances (100 per region)
• Feature store: 500GB RAM cache capacity
• Data pipeline: 100GB/s bandwidth for real-time processing

Failure Budget Allocation:

• 99.9% uptime target (1.44 min daily downtime)
• 50ms p99 latency budget across all stages
• 10% of traffic for shadow testing and A/B experiments

11. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering / policy alignment

• Sampling strategy alignment: Should negative sampling be optimized for model convergence or user experience? This requires alignment between ML and product teams on whether to prioritize faster training or better user engagement.
• Cold start handling: How should new users/items be handled during negative sampling? Should they fall back to popular items or use a hybrid strategy?
• Data freshness vs. sampling efficiency: Should we prioritize recent negative samples or maintain stable training distributions?

Key ML risks

Risk	Impact	Likelihood	Mitigation
Imbalanced sampling bias	Model learns to favor popular items over rare ones	High	Implement class-aware sampling with inverse frequency weighting
Training-serving skew	Model trained on sampled data performs poorly in production	Medium	Use same sampling distribution in training and serving
Cold start amplification	New users/items get disproportionately sampled, affecting model quality	Medium	Implement adaptive sampling that balances popularity and novelty
Temporal drift in negative samples	Old negative samples become irrelevant for newer user preferences	Medium	Implement time-aware sampling with decay factors
Overfitting to sampled negatives	Model learns spurious patterns from frequently sampled items	Low-Medium	Use diverse negative sampling with multiple strategies

Trade-offs made with justification

Trade-off	Justification
Accuracy vs. latency	Using simpler negative sampling (e.g., uniform) reduces training time but may hurt model accuracy. For real-time systems, we accept moderate accuracy degradation for sub-100ms response times.
Freshness vs. cost	Real-time negative sampling increases infrastructure costs but improves model relevance. We use a hybrid approach: recent 10% of samples are fresh, others are pre-computed to balance cost and performance.
Model size vs. quality	Larger embedding matrices increase model size but improve representation accuracy. We use embedding dimensionality reduction via PCA and quantization to maintain quality while reducing size.
Sampling diversity vs. training efficiency	Diverse sampling improves model robustness but increases training time and complexity. We use a multi-stage approach where we first sample broadly, then refine with more targeted strategies.

Alternatives considered and rejected

• Uniform sampling: Simple but leads to poor model performance due to overrepresentation of popular items. Rejected for accuracy trade-off.
• Fully random sampling: Unbiased but inefficient, requiring many samples to achieve good coverage. Rejected for latency and cost considerations.
• Fixed popularity-based sampling: Too deterministic, doesn't adapt to user behavior changes. Rejected for lack of dynamism.
• Online learning with live feedback: Complex to implement and maintain, but introduces instability in model training. Rejected for operational complexity.

Technical debt acknowledged

• Sampling distribution inconsistency: Current implementation uses different sampling strategies across training and serving, creating potential performance gaps. This will be addressed in Phase 2 with unified sampling logic.
• Limited negative sample diversity: The current approach doesn't adequately capture rare item interactions, which may lead to suboptimal model generalization over time. This is mitigated through periodic retraining with updated distributions.
• Hardcoded sampling parameters: Sampling weights and thresholds are currently hardcoded, making it difficult to adapt to changing data distributions without code changes. These will be made configurable in the feature store layer.

# Sample negative sampling strategy implementation
import numpy as np
from collections import defaultdict

class NegativeSampler:
    def __init__(self, item_popularity, sampling_strategy="inverse_frequency"):
        self.item_popularity = item_popularity
        self.sampling_strategy = sampling_strategy
        self.weights = self._compute_weights()
    
    def _compute_weights(self):
        if self.sampling_strategy == "inverse_frequency":
            # Inverse frequency weighting to reduce popularity bias
            return 1.0 / (np.array(list(self.item_popularity.values())) + 1e-8)
        elif self.sampling_strategy == "log_frequency":
            # Log frequency weighting
            return 1.0 / (np.log(np.array(list(self.item_popularity.values())) + 1) + 1e-8)
        else:
            return np.ones(len(self.item_popularity))
    
    def sample_negatives(self, user_id, num_samples=10):
        # Sample negatives based on computed weights
        sampled_items = np.random.choice(
            list(self.item_popularity.keys()),
            size=num_samples,
            p=self.weights / np.sum(self.weights),
            replace=False
        )
        return sampled_items.tolist()

12. Phased Implementation Roadmap

Phase 1: MVP (Weeks 1-4)

Objective: Establish baseline system with rule-based fallback and simple ML model

• Week 1-2:
  • Implement basic rule-based ranking system as fallback
  • Set up training data pipeline with labeled examples (50K samples)
  • Build minimal feature store with user/item metadata
• Week 3-4:
  • Train and deploy simple logistic regression model (baseline)
  • Implement basic online serving infrastructure with TorchServe
  • Setup logging and monitoring for prediction results

Phase 2: Scale (Weeks 5-8)

Objective: Add real-time features, distributed training, and A/B testing

• Week 5-6:
  • Implement real-time feature computation pipeline (user behavior, item popularity)
  • Build two-tower model architecture with embedding layers
  • Deploy distributed training pipeline using Ray or PyTorch Distributed
• Week 7-8:
  • Set up A/B testing infrastructure with proper randomization unit
  • Implement negative sampling with importance weighting (50K samples)
  • Launch shadow mode for model validation

Phase 3: Excellence (Weeks 9-12)

Objective: Advanced models, continuous learning, and optimization

• Week 9-10:
  • Deploy transformer-based model with attention mechanisms (50M+ parameters)
  • Implement dynamic negative sampling with adaptive weights
  • Add real-time feedback loop for model retraining
• Week 11-12:
  • Enable continuous learning with online gradient updates
  • Optimize serving latency with model compression (quantization, pruning)
  • Implement multi-region deployment and auto-scaling

Critical Path Dependencies:

1. Data Pipeline must be completed before model training can begin
2. Feature Store availability is critical for real-time serving
3. A/B Testing Infrastructure must be ready before production rollout
4. Model Compression requires model performance baseline for accuracy validation

Risk Mitigation:

• Cold Start: Rule-based fallback ensures system availability during model training
• Data Drift: Shadow mode enables early detection of performance degradation
• Latency Budget: Two-stage pipeline (candidate retrieval → scoring) provides flexibility for optimization