Large-Scale Logistic Regression Scoring System Design

This document presents a production-ready system architecture for large-scale logistic regression scoring, addressing key challenges in distributed training, real-time inference, and model versioning for enterprise ML platforms.

1. Problem Statement & ML Task Definition

Clarifying questions

1. What is the expected scale of training data (e.g., number of samples, features)? Are we dealing with billions of rows or millions?
2. What is the latency budget for scoring requests? Is it sub-100ms for real-time inference or can we tolerate several hundred milliseconds?
3. Is this a batch scoring use case or real-time online serving? How does that impact model architecture and feature computation?
4. What is the primary business objective — maximizing revenue, engagement, or accuracy? Are there specific trade-offs between these?
5. What is the expected QPS (queries per second) for serving this model in production? This affects infrastructure and caching strategy.
6. What is the desired accuracy target (e.g., AUC-ROC > 0.85) and how does it compare to baseline models?
7. How often will the model need retraining? Is this a static model or does it require continuous learning?

Business use case and success criteria

The system will power a large-scale recommendation engine that scores user-item pairs for relevance in real-time. The model is used to rank items for users based on their preferences, with the goal of maximizing user engagement (e.g., click-through rate, time spent on content) and business KPIs like revenue or retention. The system must handle millions of concurrent scoring requests with sub-100ms latency to support real-time personalization in a high-throughput web application.

ML problem formulation

This is a binary classification task where the model predicts whether a user will engage with an item (e.g., click, watch, purchase). The task is formulated as:

P(y=1 | x) = σ(w^T * x + b)

Where:

• y is the binary label (1 = engaged, 0 = not engaged)
• x is the feature vector (user, item, and context features)
• σ is the sigmoid function

Justification:

• Binary classification is appropriate for engagement prediction where we have clear positive/negative labels
• Logistic regression provides interpretable coefficients and fast inference, suitable for real-time serving at scale
• The task is well-suited to linear models with engineered features

ML evaluation metrics

Offline Metrics (Training/Validation):

• AUC-ROC: Primary metric for overall discriminative power
• Precision@K: For top-K recommendation quality (e.g., precision at 10)
• NDCG@K: Normalized Discounted Cumulative Gain for ranking quality

Online Metrics (Production):

• CTR (Click-Through Rate): Direct business impact metric
• Revenue per user: Monetization success indicator
• Engagement time: User retention and satisfaction metric

Scale and constraints

Back-of-envelope estimates:

• Training data size: 100M user-item interactions (50M positive, 50M negative)
• Number of features: 10K sparse features (user ID, item ID, content features, context features)
• Embedding matrix memory footprint: 10K features × 128 dimensions × 4 bytes = ~5MB per embedding
• Serving QPS: 10,000 requests/sec (500ms p99 latency target)
• Model size: 10K × 128 = ~1.3MB (dense weight matrix)
• Training time: 2 hours on 8xGPU cluster

Non-ML baseline strategy

Baseline rule-based strategies:

1. Cold start handling: For new users/items, use popularity-based ranking or content-based similarity
2. Simple fallback logic: If model fails to score, return top-K most popular items
3. Context-aware thresholds: Use historical engagement rates for different user segments as fallbacks

# Baseline fallback logic example
def baseline_ranking(user_context, item_pool):
    if not user_context or not item_pool:
        return popular_items()  # fallback to most popular items
    
    # Simple content-based matching
    return content_similarity_ranking(user_context, item_pool)

2. Data Strategy

Training data sources

• Explicit feedback: User ratings, clicks, purchases, and direct engagement signals (e.g., thumbs up/down, save actions)
• Implicit signals: Page views, time spent, scroll depth, session duration, and navigation patterns
• Third-party data: Demographic information from partners, contextual signals (weather, events), and external content features
• Synthetic data: Generated samples for rare classes or edge cases to improve model robustness

Labeling pipeline design

• Human annotation: For explicit feedback and high-value implicit signals, with quality control through inter-rater agreement (Cohen's Kappa > 0.8)
• Weak supervision: Using heuristics and rule-based labeling for large-scale initial data generation (e.g., click-through rate > 0.5 → positive label)
• Programmatic labeling: For implicit signals, applying business logic thresholds (e.g., > 30s engagement → positive signal)

Class imbalance handling strategy

• Oversampling: Using SMOTE (Synthetic Minority Oversampling Technique) for rare positive samples to ensure adequate representation in training
• Class weights: Applying inverse frequency weighting during model training to penalize misclassifications of minority classes more heavily
• Threshold tuning: Optimizing decision thresholds post-training to balance precision and recall for imbalanced slices

Negative sampling approach

• Stratified negative sampling: Maintaining target distribution across user-item pairs to prevent overrepresentation of popular items
• Hard negative mining: Including examples where model confidence is high but prediction is incorrect to improve robustness
• Negative ratio: Using 1:5 ratio (positive:negative) to balance model performance and training efficiency

Data freshness and update frequency requirements

• Real-time signals: Updated every 5 minutes for user behavior (e.g., session events, clicks)
• Batch features: Updated daily for static or slowly changing user/item attributes (e.g., demographics, item categories)
• Freshness SLA: 95% of training data must be within 24 hours of the latest event timestamp

Privacy compliance

• GDPR/CCPA compliance: Anonymization of PII (e.g., hashing user IDs, removing direct identifiers)
• Data retention policy: 24-month retention for training data with automated deletion of expired records
• Consent management: Integration with consent APIs to ensure only opted-in users' data is used for training

Data quality validation and monitoring

• Data validation rules: Check for missing values, out-of-range features, and invalid timestamp formats
• Feature drift monitoring: Track PSI (Population Stability Index) for key features to detect data distribution shifts
• Label consistency checks: Validate that labels align with business logic (e.g., purchase event must have a corresponding transaction)

# Example data validation pipeline snippet (Python)
import pandas as pd
from scipy import stats

def validate_training_data(df):
    # Check for missing values in key features
    assert not df['user_id'].isnull().any(), "Missing user IDs"
    
    # Validate timestamp consistency
    df['timestamp'] = pd.to_datetime(df['timestamp'])
    assert (df['timestamp'].max() - df['timestamp'].min()).days <= 30, "Data older than 30 days"

    # Check for label consistency
    assert df['label'].isin([0, 1]).all(), "Invalid labels - must be binary"
    
    # Validate feature drift using PSI
    baseline_features = ['user_age', 'item_category']
    for feature in baseline_features:
        psi_value = calculate_psi(df[feature], df[feature].sample(frac=0.1))
        assert psi_value < 0.1, f"Feature drift detected in {feature}"

def calculate_psi(actual, expected):
    # Simple PSI calculation for continuous features
    actual_freq = pd.cut(actual, bins=10, retbins=False)
    expected_freq = pd.cut(expected, bins=10, retbins=False)
    
    psi = sum((actual_freq - expected_freq) * (actual_freq / expected_freq))
    return psi

3. Feature Engineering

Feature Taxonomy

Features are categorized into four main types:

1. User Features: User demographics, behavioral history, engagement patterns
2. Item Features: Content metadata, popularity metrics, category information
3. Context Features: Time-based signals, device information, geographic location
4. Interaction Features: User-item interaction patterns, temporal decay signals

Embedding Design

Entity Embeddings:

• User ID embeddings: 128-dimensional, trained via click-through rate prediction
• Item ID embeddings: 128-dimensional, learned from user-item interaction matrix

Text Embeddings:

• Item title/content: 256-dimensional using pre-trained BERT embeddings
• User profile text: 128-dimensional using TF-IDF weighted average

Graph Embeddings:

• User-item bipartite graph: 64-dimensional using Node2Vec for neighbor-aware features

Real-time vs Batch Feature Computation

Feature Type	Computation Mode	Reasoning
User action counts	Real-time	High latency sensitivity, frequent updates
Item popularity scores	Batch	Stable over time, computed daily
User session features	Real-time	Session context matters for scoring
Historical engagement	Batch	Can be pre-computed, reduces real-time load

Feature Store Design

Online Store:

• Low-latency key-value store (e.g., Redis) for real-time feature serving
• TTL: 1 hour for user session features, 24 hours for static features

Offline Store:

• Data warehouse (e.g., Snowflake) for training data and batch feature computation
• Partitioned by date for efficient querying

graph TD
    A[Feature Store] --> B[Online Store (Redis)]
    A --> C[Offline Store (Snowflake)]
    B --> D[Real-time Serving]
    C --> E[Training Pipeline]

Feature Freshness and TTL

Feature	Source	Update Frequency	Store Type
User session count	Real-time stream	Every 5 mins	Online
Item popularity	Batch job	Daily	Offline
User device type	User profile	When updated	Online
Item category	Content DB	Weekly	Offline
User engagement decay	Real-time stream	Every 15 mins	Online

Cross Features and Feature Interactions

Cross features are engineered using:

• Multiplicative interactions: User age × Item category (256 total combinations)
• Binning strategies: Age groups × Purchase frequency buckets
• Temporal decay: User-item interaction count with time-based weight

Feature Importance and Selection Methodology

Methodology:

1. L1 regularization coefficient analysis: Features with zero coefficients are pruned
2. SHAP value computation: Top 500 features selected based on mean absolute SHAP values
3. Permutation importance: Evaluated during cross-validation

Selection Process:

• Start with 5000 raw features
• Apply L1 regularization to reduce to 2000 features
• Use SHAP to select top 500 features for final model
• Retain 1000 historical features for A/B testing

import numpy as np
from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LogisticRegression

class FeatureSelector:
    def __init__(self, max_features=500):
        self.max_features = max_features
        self.feature_names = None
        
    def select_features(self, X, y):
        # L1 regularization for feature selection
        model = LogisticRegression(penalty='l1', solver='liblinear')
        model.fit(X, y)
        
        # Select top features based on L1 regularization
        selector = SelectFromModel(model, max_features=self.max_features)
        X_selected = selector.fit_transform(X, y)
        
        # Get selected feature indices
        selected_indices = np.where(selector.get_support())[0]
        
        return X_selected, selected_indices

Feature Engineering Pipeline

class FeatureEngineeringPipeline:
    def __init__(self):
        self.feature_store = {}
        
    def compute_user_features(self, user_data):
        """Compute real-time user features"""
        features = {}
        
        # Real-time session features
        features['session_count'] = user_data.get('session_count', 0)
        features['avg_session_duration'] = user_data.get('avg_session_duration', 0)
        
        # Binned features
        features['user_age_group'] = self._bin_user_age(user_data.get('age', 0))
        features['user_engagement_level'] = self._bin_engagement(user_data.get('engagement_score', 0))
        
        return features
        
    def _bin_user_age(self, age):
        if age < 18:
            return 'under_18'
        elif age < 35:
            return '18_34'
        elif age < 50:
            return '35_49'
        else:
            return '50_plus'
            
    def _bin_engagement(self, score):
        if score < 0.3:
            return 'low'
        elif score < 0.7:
            return 'medium'
        else:
            return 'high'

Feature Normalization and Scaling

All numerical features are normalized using:

• Min-Max scaling: For bounded features (0-1 range)
• Z-score normalization: For normally distributed features
• Log transformation: For skewed features (e.g., item popularity)

from sklearn.preprocessing import StandardScaler, MinMaxScaler

class FeatureNormalizer:
    def __init__(self):
        self.scalers = {}
        
    def fit_transform(self, X, feature_types):
        X_normalized = X.copy()
        
        for col in X.columns:
            if feature_types[col] == 'numerical':
                if col in ['item_popularity', 'user_engagement']:
                    # Log transform for skewed features
                    X_normalized[col] = np.log1p(X_normalized[col])
                    self.scalers[col] = StandardScaler()
                else:
                    # Min-Max scaling for bounded features
                    self.scalers[col] = MinMaxScaler()
                    
                X_normalized[col] = self.scalers[col].fit_transform(
                    X_normalized[[col]]
                )
                
        return X_normalized

4. Model Architecture

Model Selection with Justification

For large-scale logistic regression scoring, we select sparse logistic regression as our core model architecture. This choice is driven by:

1. Interpretability: Critical for business stakeholders to understand feature importance and model decisions
2. Scalability: Linear scaling with features, suitable for high-dimensional sparse feature spaces
3. Training efficiency: Can be trained on large datasets with minimal memory overhead
4. Production stability: Well-understood, mature ecosystem with proven reliability at scale

We consider alternatives:

• GBM models: Better accuracy but higher latency and complexity for real-time scoring
• DNNs: Higher capacity but overkill for this use case and significantly more complex to deploy
• Transformer-based models: Not applicable for simple scoring tasks

Architecture Diagram

graph TD
    A[Feature Store] --> B[Feature Processing]
    B --> C[Logistic Regression Model]
    C --> D[Scoring Engine]
    D --> E[Real-time Response]
    
    subgraph "Training Pipeline"
        F[Feature Engineering] --> G[Model Training]
        G --> H[Model Versioning]
    end
    
    subgraph "Serving Pipeline"
        I[Online Inference] --> J[Scoring]
        J --> K[Response Generation]
    end

Core Model Implementation

import numpy as np
from scipy.sparse import csr_matrix
from sklearn.linear_model import LogisticRegression
import torch

class SparseLogisticRegression:
    def __init__(self, learning_rate=0.01, max_iter=1000, regularization=0.01):
        self.learning_rate = learning_rate
        self.max_iter = max_iter
        self.regularization = regularization
        self.weights = None
        self.bias = 0.0
        
    def fit(self, X, y):
        """
        Fit sparse logistic regression model using gradient descent
        X: sparse matrix of shape (n_samples, n_features)
        y: array of labels in {0, 1}
        """
        n_samples, n_features = X.shape
        
        # Initialize weights and bias
        self.weights = np.random.normal(0, 0.01, n_features)
        self.bias = 0.0
        
        # Training loop
        for iteration in range(self.max_iter):
            # Forward pass - compute predictions
            linear_pred = X.dot(self.weights) + self.bias
            predictions = self._sigmoid(linear_pred)
            
            # Compute gradients
            error = predictions - y
            
            # Weight gradient (with L2 regularization)
            weight_grad = X.T.dot(error) / n_samples + self.regularization * self.weights
            
            # Bias gradient
            bias_grad = np.mean(error)
            
            # Update parameters
            self.weights -= self.learning_rate * weight_grad
            self.bias -= self.learning_rate * bias_grad
            
    def predict_proba(self, X):
        """Return probability estimates for samples in X"""
        linear_pred = X.dot(self.weights) + self.bias
        return self._sigmoid(linear_pred)
    
    def predict(self, X):
        """Return class predictions for samples in X"""
        return (self.predict_proba(X) >= 0.5).astype(int)
    
    def _sigmoid(self, x):
        """Numerically stable sigmoid function"""
        return np.exp(-np.logaddexp(0, -x))

# Example usage for scoring
def score_user_item_pair(user_features, item_features, model):
    """
    Score a user-item pair using sparse logistic regression
    """
    # Combine features (assuming they're already in sparse format)
    combined_features = user_features.multiply(item_features)  # Example interaction
    
    # Get probability score
    score = model.predict_proba(combined_features)
    
    return score[0] if hasattr(score, '__len__') else score

Model Complexity vs Latency Trade-off

Model	Parameters	Memory Usage	Inference Time	Accuracy
Sparse LR (baseline)	O(d)	O(d)	< 1ms	Moderate
Sparse LR + Feature Hashing	O(d)	O(d)	< 1ms	Moderate
Sparse LR + Cross Features	O(d²)	O(d²)	1-2ms	High
DNN (2-layer)	O(d²)	O(d²)	5-10ms	Very High

Offline Evaluation Baseline

Baseline model performance:

• AUC-ROC: ~0.75 (rule-based + simple features)
• Precision@K: ~0.30 (top-K recommendations)
• NDCG: ~0.45 (ranked list quality)

Proposed model aims to achieve:

• AUC-ROC: ~0.85+
• Precision@K: ~0.45+
• NDCG: ~0.65+

Two-Tower / Multi-task Design

For multi-task scenarios (e.g., scoring + ranking + classification), we implement a shared-embedding architecture:

class MultiTaskLogisticRegression:
    def __init__(self, n_tasks=2):
        self.tasks = {}
        for i in range(n_tasks):
            self.tasks[f'task_{i}'] = SparseLogisticRegression()
    
    def fit_task(self, task_id, X, y):
        """Fit a specific task"""
        self.tasks[task_id].fit(X, y)
    
    def predict_task(self, task_id, X):
        """Predict for a specific task"""
        return self.tasks[task_id].predict_proba(X)

Pre-trained Model Selection and Fine-tuning Strategy

For foundation model use cases:

• Pre-trained embeddings: Use pre-computed user/item embeddings from collaborative filtering or content-based models
• Fine-tuning: Apply linear layer on top of pre-trained embeddings for task-specific adaptation
• Transfer learning: Use same embedding space across multiple related tasks

Offline Evaluation Baseline Framing

Baseline model uses:

• Simple feature engineering: Raw user/item IDs, categorical features
• Basic training: SGD with L2 regularization
• Standard metrics: AUC, Precision@K, NDCG

Proposed model improves upon baseline by:

• Adding feature interactions and cross features
• Using proper regularization techniques
• Implementing online learning capabilities

Model Compression Strategy

For production deployment:

1. Quantization: 8-bit weight quantization (reduces memory by ~75%)
2. Pruning: Remove weights below threshold (reduces model size by ~50%)
3. Model distillation: Train smaller student model with soft targets from teacher model

Serving Infrastructure Justification

• TorchServe: For serving PyTorch models with built-in model compression and multi-GPU support
• Custom inference engine: For maximum control over latency and memory usage in high-throughput scenarios

Performance Targeting

• p50 inference latency: < 1ms
• p99 inference latency: < 5ms
• QPS capacity: 100,000+ requests/sec
• Memory footprint: < 50MB per model instance

This architecture provides a robust, scalable foundation for large-scale logistic regression scoring while maintaining the interpretability and performance characteristics required for real-time production systems.

5. Serving & Inference Architecture

System Diagram

graph TD
    A[User Request] --> B[Load Balancer]
    B --> C[Feature Store Lookup]
    C --> D[Model Server]
    D --> E[Scoring Result]
    E --> F[Response to User]
    
    subgraph Online Serving
        C
        D
        E
    end
    
    subgraph Feature Store
        C
    end

Online vs Batch Serving Decision

Online serving is required for real-time scoring with sub-100ms latency SLA. Batch serving would introduce 500ms+ latency and is unsuitable for interactive user experiences. The system must support 10K QPS with <50ms p99 latency.

Two-Stage Pipeline

Candidate Retrieval → Scoring → Re-ranking

For large-scale scoring, we implement a two-stage pipeline:

1. Candidate Retrieval: Fast approximate matching using inverted index or locality-sensitive hashing to reduce candidate set from millions to thousands
2. Scoring: Full logistic regression model evaluation on reduced candidate set
3. Re-ranking: Optional second-stage reranking with more complex features or models

Serving Infrastructure

We use TorchServe as the serving infrastructure because:

• Native support for PyTorch models with optimized inference
• Built-in model versioning and A/B testing capabilities
• Easy integration with Kubernetes for auto-scaling
• Support for model compression (quantization, pruning)
• Production-grade monitoring and metrics collection

Latency Budget Breakdown

Stage	p50 Latency	p99 Latency	Budget
Feature Lookup	5ms	10ms	15ms
Model Inference	2ms	8ms	10ms
Response Packaging	1ms	3ms	5ms
Total	8ms	21ms	30ms

Model Compression Strategy

We apply quantization-aware training to reduce model size from 50MB to 12MB while maintaining >99% accuracy. For further optimization, we implement:

• Pruning: Remove 40% of weights with minimal accuracy loss
• Distillation: Train smaller student model to mimic teacher model behavior

Caching Strategy

We implement a multi-tier caching system:

1. Prediction Cache: LRU cache for recent user-item pairs (50K entries, 10GB max)
2. Embedding Cache: Hot item/user embeddings cached in memory (5GB)
3. KV Cache: For LLM-based features, we cache attention keys/values (2GB)

Auto-scaling and Capacity Planning

We use Kubernetes HPA with:

• Target: 70% CPU utilization
• Scale up: 2x per minute (max 10 replicas)
• Scale down: 5x per minute (min 2 replicas)

Back-of-envelope capacity calculation:

• Target QPS: 10,000
• Per-request latency: 20ms (p99)
• Required throughput: 10,000 * 0.02 = 200 QPS per replica
• With 5% overhead for warmup and system load: 200 * 1.05 = 210 QPS per replica
• For 10K QPS: 10,000 / 210 ≈ 50 replicas minimum

Model Serving Fallbacks

In case of model failure:

1. Rule-based fallback: Default score of 0.5 for all items
2. Stale model serving: Serve last known good model version for up to 5 minutes
3. Graceful degradation: Reduce feature set to core features when full model unavailable

Cold Start Handling

For new users/items:

1. Default score: Use global average probability (0.3) as fallback
2. Content-based matching: For new items, use item metadata to find similar items from existing pool
3. Hybrid approach: Combine rule-based scores with minimal feature set until sufficient training data is collected

6. Evaluation Framework

Offline Evaluation Methodology

For offline evaluation of logistic regression models at scale, we employ a temporal split strategy to ensure realistic performance assessment and prevent data leakage. The dataset is partitioned chronologically into training (last 6 months), validation (1 month), and test (1 month) sets. This approach preserves temporal order and simulates real-world deployment conditions where future data is not available during training.

Cross-validation is conducted using time-aware folds to maintain temporal consistency. For large-scale models, we use stratified sampling to ensure balanced representation across key user segments and item categories in each fold.

Slice-Based Evaluation

To detect performance degradation across different user populations, we define critical slices:

Slice Type	Definition	Rationale
User Demographics	Age groups (18-25, 26-35, 36-50), gender	Different engagement patterns
Geographic Region	Country/region codes	Regional content preferences
Device Type	Mobile, Desktop, Tablet	Platform-specific behavior
Session Frequency	Low (<5 sessions/month), Medium (5-20), High (>20)	Engagement level impact
Content Category	News, Entertainment, Education	Topic-specific relevance

We evaluate model performance on each slice using AUC-ROC, Precision@K, and NDCG to identify potential bias or degradation in specific user segments.

A/B Test Design

Randomization Unit: User ID with session-level randomization to prevent leakage across sessions.

Sample Size Calculation:

• Power: 80%
• Alpha: 0.05
• Baseline CTR: 2.5%
• Minimum Detectable Effect: 0.2% (0.8% absolute change)
• Estimated sample size: ~150,000 users per variant

Test Duration: 30-day continuous test to capture seasonal variation and ensure statistical significance.

Primary Metric: Click-Through Rate (CTR) with 95% confidence intervals and p-values.

Secondary Metrics:

• Revenue per user (RPU)
• Session duration
• Conversion rate (if applicable)

Guardrail Metrics and Blocking Conditions

We implement strict guardrails to prevent model rollouts that could harm user experience:

Metric	Threshold	Action
AUC Drop	< 0.02 from baseline	Block rollout
Precision@K Drop	< 0.05 from baseline	Block rollout
Recall@K Drop	< 0.03 from baseline	Block rollout
User Engagement Drop	< 0.02 from baseline	Block rollout
Negative Feedback Rate	> 5% of total interactions	Block rollout

Shadow Mode Rollout Plan

Before full A/B testing, we implement a shadow mode where model predictions are logged but not served to users:

1. Phase 1 (Week 1): Shadow mode with 5% of traffic
2. Phase 2 (Week 2): Shadow mode with 25% of traffic
3. Phase 3 (Week 3): Shadow mode with 50% of traffic
4. Phase 4 (Week 4): Shadow mode with 100% traffic

During shadow mode, we collect prediction logs and compare model outputs against the current baseline to validate performance consistency.

Long-Term Metric Measurement

We track delayed conversion and engagement decay metrics:

# Example long-term metric calculation in Python
from datetime import timedelta
import pandas as pd

def calculate_delayed_conversion_metrics(predictions_df, user_sessions_df):
    """
    Calculate delayed conversion metrics for long-term impact assessment
    """
    # Join predictions with user session data
    merged = pd.merge(predictions_df, user_sessions_df, on='user_id')
    
    # Calculate 7-day and 30-day conversion rates
    delayed_conversions = {}
    
    for days in [7, 30]:
        cutoff_date = pd.Timestamp.now() - timedelta(days=days)
        recent_users = merged[merged['session_date'] >= cutoff_date]
        
        # Calculate conversion rate for users who saw recommendations
        conversion_rate = recent_users['converted'].mean()
        delayed_conversions[f'conversion_{days}d'] = conversion_rate
        
    return delayed_conversions

# Example engagement decay tracking
def track_engagement_decay(predictions_df, user_engagement_df):
    """
    Track engagement decay over time to assess model longevity
    """
    # Calculate engagement scores at different time intervals
    engagement_decay = {}
    
    for days in [1, 7, 30]:
        decay_period = pd.Timestamp.now() - timedelta(days=days)
        
        # Filter for users with engagement data in decay window
        relevant_users = user_engagement_df[
            (user_engagement_df['last_interaction'] >= decay_period) &
            (user_engagement_df['first_interaction'] < decay_period)
        ]
        
        avg_engagement = relevant_users['engagement_score'].mean()
        engagement_decay[f'engagement_{days}d'] = avg_engagement
        
    return engagement_decay

We monitor these metrics for 90-day post-launch to ensure model robustness and avoid degradation over time. The system automatically alerts when any metric drops below 95% of baseline performance, triggering investigation and potential model rollback.

7. Training Pipeline

Training Data Pipeline

The training data pipeline processes user-item interactions, features, and labels through a multi-stage ETL process:

1. Data Ingestion: Raw data flows from Kafka topics (user events, item metadata) and S3 buckets (historical data)
2. Preprocessing:
   • Feature normalization and encoding
   • Missing value imputation using median/mean strategies
   • Outlier detection and filtering
3. Shuffling: Data is shuffled using reservoir sampling to ensure random distribution across training batches
4. Batching: Data is batched using PyTorch DataLoader with configurable batch sizes (default: 1024 samples per batch)

import torch
from torch.utils.data import Dataset, DataLoader
import pandas as pd

class LogisticRegressionDataset(Dataset):
    def __init__(self, features_df, labels_series):
        self.features = torch.tensor(features_df.values, dtype=torch.float32)
        self.labels = torch.tensor(labels_series.values, dtype=torch.float32)
    
    def __len__(self):
        return len(self.labels)
    
    def __getitem__(self, idx):
        return self.features[idx], self.labels[idx]

# Example usage for training data pipeline
def build_training_dataloader(features_df, labels_series, batch_size=1024):
    dataset = LogisticRegressionDataset(features_df, labels_series)
    return DataLoader(dataset, batch_size=batch_size, shuffle=True, num_workers=4)

Distributed Training Strategy

For large-scale training, we use data parallelism with PyTorch's DistributedDataParallel (DDP):

• Cluster Configuration: 8x GPU nodes (A100 80GB) with NVLink connectivity
• Data Parallelism: Each GPU processes a subset of the training data (batch size = 1024 × 8 = 8192)
• Gradient Synchronization: All-reduce operations every 100 steps to maintain model consistency
• Memory Optimization: Gradient accumulation with gradient clipping to prevent memory overflow

Training Infrastructure

• Hardware: 8x A100 GPUs (80GB each) with 256GB total VRAM
• Training Time: Estimated 4 hours for 10M samples with batch size 1024
• Memory Footprint: ~3GB per GPU (including gradients and optimizer state)
• Compute Budget: 8x A100 GPUs for 4 hours = ~256 GPU-hours per training run

Hyperparameter Tuning Approach

We implement Bayesian optimization using Optuna for efficient hyperparameter search:

Key hyperparameters optimized:

1. Learning rate (0.001, 0.01, 0.1)
2. L2 regularization strength (1e-6, 1e-4, 1e-2)
3. Batch size (512, 1024, 2048)
4. Number of epochs (5, 10, 20)

import optuna
from sklearn.linear_model import LogisticRegression

def objective(trial):
    # Suggest hyperparameters
    lr = trial.suggest_float('learning_rate', 1e-4, 1e-1, log=True)
    l2 = trial.suggest_float('l2_regularization', 1e-6, 1e-2, log=True)
    
    # Train model with suggested params
    model = LogisticRegression(
        C=1/l2,  # Inverse of regularization strength
        solver='lbfgs',
        max_iter=1000,
        random_state=42
    )
    
    # Return validation AUC-ROC score (example)
    return model.fit(X_train, y_train).score(X_val, y_val)

Experiment Tracking and Reproducibility

We use MLflow for experiment tracking:

1. Experiment Logging:
   • Model parameters (learning rate, regularization)
   • Training metrics (loss, AUC-ROC)
   • Model artifacts (weights, feature importance)
2. Versioning:
   • Model versioning with Git commit hash
   • Feature set version tracking
3. Reproducibility:
   • Fixed random seeds (42) across all components
   • Docker containers with pinned dependencies

Model Versioning and Registry Strategy

• Version Control: Semantic versioning (v1.0.0, v1.1.0) with Git tags
• Registry: MLflow model registry with staging/production environments
• Rollback Capability: Automatic rollback to previous version on performance degradation
• Model Validation: Pre-launch validation against baseline model (expected 2% AUC improvement)

Continual / Online Learning Design

For incremental updates, we implement online learning with:

1. Incremental Training:
   • Mini-batch updates every 24 hours using new data
   • Learning rate decay (10% reduction per cycle)
2. Model Drift Detection:
   • PSI (Population Stability Index) for feature drift
   • AUC-ROC drop threshold (5% degradation triggers retraining)
3. Data Freshness:
   • 7-day rolling window for training data
   • Daily model updates with new user/item interactions

8. Monitoring & Observability

Data Drift Detection

Data drift monitoring is critical for maintaining model performance in production. We implement drift detection using Population Stability Index (PSI) and Kullback-Leibler divergence for continuous features, and Chi-squared tests for categorical features.

import numpy as np
from scipy import stats
import pandas as pd

def calculate_psi(expected, actual, bins=10):
    """Calculate Population Stability Index between two distributions."""
    expected_freq = np.histogram(expected, bins=bins)[0]
    actual_freq = np.histogram(actual, bins=bins)[0]
    
    # Avoid zero division
    expected_freq = np.clip(expected_freq, 1e-6, None)
    actual_freq = np.clip(actual_freq, 1e-6, None)
    
    psi = np.sum((actual_freq - expected_freq) * np.log(actual_freq / expected_freq))
    return psi

def detect_drift(features_expected, features_actual):
    """Detect drift in feature distributions."""
    drift_results = {}
    
    for feature_name in features_expected.columns:
        if features_expected[feature_name].dtype in ['int64', 'float64']:
            psi = calculate_psi(features_expected[feature_name], 
                              features_actual[feature_name])
            drift_results[feature_name] = {
                'psi': psi,
                'threshold': 0.1,  # Threshold for drift detection
                'drifted': psi > 0.1
            }
        else:
            # For categorical features, use chi-squared test
            chi2, p_value, dof, expected = stats.chi2_contingency(
                pd.crosstab(features_expected[feature_name], 
                           features_actual[feature_name]))
            drift_results[feature_name] = {
                'chi2': chi2,
                'p_value': p_value,
                'threshold': 0.05,
                'drifted': p_value < 0.05
            }
    
    return drift_results

Prediction Drift and Label Drift Monitoring

Prediction drift monitors whether the distribution of model predictions changes over time, indicating potential model degradation. Label drift monitors for shifts in target distributions.

Model Performance Degradation Alerting

We implement alerting thresholds based on key metrics:

• AUC-ROC: Drop > 5% from baseline triggers alert
• Precision@K: Drop > 10% from baseline triggers alert
• NDCG: Drop > 5% from baseline triggers alert

class ModelPerformanceMonitor:
    def __init__(self, baseline_metrics):
        self.baseline_metrics = baseline_metrics
        self.alert_thresholds = {
            'auc_roc': 0.05,
            'precision_at_k': 0.10,
            'ndcg': 0.05
        }
    
    def check_degradation(self, current_metrics):
        alerts = []
        for metric_name, threshold in self.alert_thresholds.items():
            if metric_name in current_metrics:
                diff = self.baseline_metrics[metric_name] - current_metrics[metric_name]
                if diff > threshold:
                    alerts.append({
                        'metric': metric_name,
                        'baseline': self.baseline_metrics[metric_name],
                        'current': current_metrics[metric_name],
                        'alert': True,
                        'severity': 'high' if diff > 0.1 else 'medium'
                    })
        return alerts

Retraining Trigger Strategy

We implement a hybrid retraining trigger strategy:

1. Scheduled retraining: Every 30 days for model refresh
2. Drift-triggered: When PSI > 0.15 or prediction drift exceeds threshold
3. Performance-triggered: When AUC drops by > 5% or precision@K drops by > 10%

Logging Design

Prediction logs capture:

• Input features and model predictions
• Timestamps and user/item identifiers
• Model confidence scores
• Ground truth labels (when available)

import json
from datetime import datetime

class PredictionLogger:
    def __init__(self, log_file_path):
        self.log_file_path = log_file_path
    
    def log_prediction(self, user_id, item_id, features, prediction, 
                      confidence, ground_truth=None):
        log_entry = {
            'timestamp': datetime.now().isoformat(),
            'user_id': user_id,
            'item_id': item_id,
            'features': features,
            'prediction': prediction,
            'confidence': confidence,
            'ground_truth': ground_truth
        }
        
        with open(self.log_file_path, 'a') as f:
            f.write(json.dumps(log_entry) + '\n')

Feedback Loop for Continuous Improvement

We implement a feedback loop that captures user interactions and retraining signals:

1. User behavior feedback: Click-through rates, engagement metrics
2. Model performance feedback: A/B test results, user satisfaction scores
3. Data quality feedback: Drift detection, feature importance changes

The system automatically flags drift events and performance drops for human review, enabling continuous model improvement through retraining cycles.

9. Responsible AI & Safety

Bias Analysis and Fairness

The logistic regression model must undergo comprehensive bias analysis to ensure equitable treatment across user segments. We'll audit training data for demographic disparities and evaluate model performance across slices defined by age, gender, geographic region, and device type. Fairness metrics include demographic parity (equal positive prediction rates across groups) and equalized odds (equal true positive rates). For example, if we observe that the model's positive prediction rate differs significantly between male and female users (e.g., 0.3 vs 0.15), we'll implement fairness constraints during training or post-processing adjustments.

# Fairness constraint implementation example (post-processing)
def adjust_predictions_for_fairness(predictions, sensitive_attributes, target_rate=0.2):
    """
    Adjust predictions to maintain target rate across sensitive groups
    """
    adjusted = predictions.copy()
    for group in set(sensitive_attributes):
        mask = sensitive_attributes == group
        current_rate = adjusted[mask].mean()
        adjustment_factor = target_rate / (current_rate + 1e-8)
        adjusted[mask] = np.clip(adjusted[mask] * adjustment_factor, 0, 1)
    return adjusted

Explainability Requirements

We'll implement SHAP (SHapley Additive exPlanations) to provide model interpretability for key stakeholders. SHAP values will be computed for critical user segments to understand which features drive positive/negative predictions. For production use, we'll surface top 3-5 most influential features per prediction to business users and engineers.

Safety Guardrails for Model Outputs

For any logistic regression model used in production, we implement:

• Output probability clipping to prevent extreme confidence scores (e.g., clamp probabilities between 0.05 and 0.95)
• Rate limiting for high-volume prediction requests to prevent abuse
• Content filtering for sensitive topics (if used in content recommendation context)
• Threshold-based rejection of predictions with low confidence (< 0.3) for critical use cases

Model Card Documentation

We maintain a model card documenting:

• Training data sources and preprocessing steps
• Performance metrics across different user segments
• Known limitations and bias conditions
• Model parameters (learning rate, regularization strength)
• Expected usage constraints and failure modes

Incident Response Plan

In case of model drift or harmful outputs:

1. Immediate alerting when prediction drift exceeds 5% threshold
2. Automated rollback to previous stable model version
3. Human review queue for flagged predictions (e.g., low confidence + high impact)
4. Emergency retraining pipeline activation when drift exceeds 10% threshold

Privacy and Compliance

All model training and inference will comply with GDPR and CCPA requirements:

• Data anonymization before training (removing PII from features)
• Feature-level differential privacy for sensitive attributes
• Regular data retention audits to ensure compliance with data deletion requirements

Model Robustness Testing

We conduct adversarial testing by injecting perturbations to input features and measuring prediction stability. For example, small changes in user age or item category should not dramatically alter the model's output probability. This ensures robustness against data poisoning and adversarial attacks.

Continuous Monitoring of Model Behavior

We implement continuous monitoring for:

• Prediction distribution shifts (Kolmogorov-Smirnov test)
• Feature importance drift (comparing top features over time)
• User feedback correlation with model confidence scores

This approach ensures responsible deployment of logistic regression models at scale while maintaining robustness, fairness, and safety standards appropriate for FAANG-level systems.

10. Scalability & Reliability

Bottleneck Analysis per Pipeline Stage

Back-of-envelope calculations for a typical large-scale logistic regression system serving 100M users with 100K features:

• Training Stage:

  • Training data: 1B samples × 100K features = 100TB
  • Model size: 100K parameters × 8 bytes = 800KB
  • Training time: 100K features × 100M samples = ~10B operations → 2-4 hours on 32 GPUs
  • Memory: 100GB for feature matrix + 800KB model = ~100GB peak

• Serving Stage:

  • QPS: 50,000 requests/sec
  • Per-request latency: 1ms (p99) → 50ms total serving time
  • Model inference: 100K features × 8 bytes = 800KB model + 1ms per feature → 1ms total
  • Feature lookups: 100K features × 1ms = 100ms (worst case) → bottleneck for real-time features

Scale-up Strategy

Horizontal Scaling of Serving:

• Deploy model across 1000+ serving instances (each with 8GB RAM)
• Use consistent hashing to distribute user requests across instances
• Implement load balancing with health checks and auto-scaling

Distributed Feature Computation:

• Use feature store with 100+ compute nodes for real-time feature lookups
• Cache frequently accessed features in memory (Redis or Memcached)
• Implement feature versioning to handle schema evolution

Fault Tolerance

Model Serving Fallbacks:

• Implement graceful degradation: if primary model fails, fall back to a simpler model or rule-based system
• Use circuit breaker pattern for downstream services (e.g., feature store, embedding lookup)
• Maintain model versioning with rollback capability

Stale Model Serving on Failure:

• Keep last-known good model in memory for immediate fallback
• Implement model versioning with health monitoring (e.g., p99 latency > 50ms triggers fallback)
• Use model validation before switching to new versions

Cold Start Handling Strategy

New Users:

• Use demographic features (age, location, device) to make predictions using user clusters
• Implement content-based filtering with item popularity as fallback
• Use pre-trained embeddings from similar user groups

New Items:

• Use item metadata features (category, tags) to match with user preferences
• Implement popularity-based ranking for new items
• Use collaborative filtering with item clusters to estimate relevance

Multi-region Serving Strategy

For global scale:

• Deploy model in 3 regions (US, EU, Asia) with regional feature stores
• Use CDN for model distribution and caching of frequent predictions
• Implement cross-region failover with geo-distributed feature stores
• Use region-specific models trained on local user behavior for better relevance

# Example cold start handling logic in Python
class ColdStartHandler:
    def __init__(self):
        self.default_scores = {
            'new_user': 0.5,
            'new_item': 0.3
        }
    
    def handle_new_user(self, user_features):
        # Fallback to demographic-based scoring
        return self.default_scores['new_user']
    
    def handle_new_item(self, item_features):
        # Fallback to popularity-based scoring
        return self.default_scores['new_item']

Reliability Patterns

Graceful Degradation:

• When feature store is unavailable, use default values or cached features
• When model inference fails, return pre-computed scores with confidence intervals

Health Monitoring:

• Implement liveness and readiness probes for model endpoints
• Monitor feature store latency, model accuracy drift, and prediction consistency

Retry Logic:

• Implement exponential backoff for transient failures in feature lookups
• Use circuit breaker to prevent cascading failures in dependent services

Performance Optimization

Batching Strategy:

• Process 1000+ requests in batches for better GPU utilization
• Use async I/O for feature lookups to reduce latency

Memory Optimization:

• Use sparse matrix representations for features with high sparsity
• Implement feature pruning to remove low-importance features from serving

Resource Allocation:

• Allocate dedicated GPU resources for model inference
• Use CPU cores for feature preprocessing and batch operations

11. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering / policy alignment

• Model refresh cadence vs. data freshness: How frequently should models be retrained in response to data drift vs. business priorities?
• Latency budget allocation: How much of the total serving latency should be allocated to model inference vs. feature retrieval?
• Feature update frequency: Should real-time features be updated at every request or batched for efficiency?
• Fallback behavior for model failure: Should the system fall back to a rule-based model or return an error when ML inference fails?

Key ML risks

Risk	Impact	Likelihood	Mitigation
Data drift in user behavior	High - model performance degrades rapidly	Medium - continuous monitoring can detect drift	Implement feature drift detection, trigger retraining on drift
Label shift in training data	Medium - model trained on outdated labels	Low-Medium - periodic validation of label quality	Establish feedback loop for label validation
Cold start problem for new users/items	Medium - poor initial recommendations	Medium - proactive embedding initialization	Implement content-based fallback for new entities
Model overfitting to training data	Medium - reduced generalization	Medium - cross-validation and regularization	Use L2 regularization, early stopping, cross-validation
Feature unavailability during serving	High - complete system failure	Low-Medium - feature store reliability	Implement feature fallbacks and caching

Trade-offs made with justification

Trade-off	Justification
Accuracy vs. latency	Logistic regression provides fast inference (microseconds) at the cost of limited expressiveness. For large-scale scoring, prioritizing low latency over marginal accuracy gains is acceptable for most use cases
Freshness vs. cost	Real-time feature updates increase infrastructure costs and complexity. Batched updates reduce cost but may introduce staleness. For large-scale systems, batched features with TTLs provide a good balance
Model size vs. quality	Larger models can capture more complex patterns but increase serving cost and latency. Logistic regression with feature hashing provides a good balance between model size and performance
Online vs. batch training	Online learning enables faster adaptation to new patterns but introduces noise and complexity. Batch training provides more stable models but with delayed responsiveness
Feature engineering vs. model complexity	Complex feature engineering can outperform more complex models. For large-scale systems, simple yet effective features with good coverage are preferred over highly complex feature interactions

Alternatives considered and rejected

• GBM (XGBoost/LightGBM): While more accurate, GBMs have higher latency and complexity for large-scale serving. Rejected due to latency constraints in the serving layer.
• Deep neural networks: DNNs provide better accuracy but introduce significant complexity in training, serving, and maintenance. Rejected for simplicity and scalability goals.
• Ensemble methods: Ensemble of simple models (e.g., logistic regression + decision trees) was considered but rejected due to increased serving complexity and latency overhead.
• Online learning with SGD: While enabling real-time adaptation, introduces instability and complexity in model state management. Rejected for system stability and predictability.

Technical debt acknowledged

• Feature store dependency: The system assumes a robust feature store, which introduces a single point of failure and complexity in managing feature versioning.
• Model versioning gaps: Current model versioning strategy lacks fine-grained control over feature set versions, potentially leading to inconsistent model behavior.
• Limited explainability: Logistic regression is inherently interpretable but lacks the ability to provide detailed explanations for individual predictions in complex feature spaces.
• No model retraining automation: Manual or semi-automated retraining process introduces risk of model drift and delayed response to data changes.

12. Phased Implementation Roadmap

Phase 1: MVP (Weeks 1-4)

Objective: Establish baseline system with rule-based fallback and basic logistic regression model

Deliverables:

• Rule-based scoring engine with configurable thresholds
• Basic logistic regression model trained on historical data (1M samples)
• Simple online serving endpoint with 100 QPS capacity
• Basic feature store with 50 core features

Dependencies:

• Data pipeline ready (offline store)
• Feature engineering pipeline complete
• Basic model training infrastructure in place

Critical Path:

1. Rule-based fallback implementation (Week 1)
2. Model training pipeline setup (Week 2)
3. Basic serving infrastructure (Week 3)
4. Initial model deployment and testing (Week 4)

Phase 2: Scale (Weeks 5-8)

Objective: Scale to production-grade system with feature store and distributed training

Deliverables:

• Feature store with real-time and batch capabilities
• Distributed training infrastructure (data parallelism)
• A/B testing framework with 20% shadow mode
• Model versioning and registry system
• 10x QPS capacity (1K QPS)

Dependencies:

• Feature store operational
• Distributed training infrastructure ready
• Model monitoring system in place

Critical Path:

1. Feature store implementation (Week 5)
2. Distributed training pipeline (Week 6)
3. A/B testing infrastructure (Week 7)
4. Production deployment (Week 8)

Phase 3: Excellence (Weeks 9-12)

Objective: Advanced features and continuous learning capabilities

Deliverables:

• Real-time feature computation pipeline
• Online learning capability with concept drift detection
• Advanced model monitoring and retraining triggers
• Multi-region serving infrastructure
• 100x QPS capacity (10K QPS)

Dependencies:

• Production-scale serving infrastructure
• Advanced monitoring systems operational
• Continuous learning pipeline ready

Critical Path:

1. Real-time feature pipeline (Week 9)
2. Online learning implementation (Week 10)
3. Multi-region deployment (Week 11)
4. Performance optimization and final testing (Week 12)

Critical Path Dependencies:

• Phase 1 must complete before Phase 2 can begin
• Feature store must be operational before distributed training
• A/B testing infrastructure requires model versioning capability
• Multi-region deployment depends on stable serving infrastructure

Risk Mitigation:

• Maintain backward compatibility during phase transitions
• Implement graceful degradation for rule-based fallback
• Monitor QPS capacity with shadow mode during scale-up
• Validate model performance at each phase transition

Success Metrics:

• QPS capacity scaling from 100 → 1K → 10K
• Model accuracy improvement of 15% over baseline
• 99.9% uptime during production deployment
• 20% reduction in latency from baseline to Phase 3