Recommendation Engine Interface Design

This document presents a production-ready low-level design for a recommendation engine interface that handles real-time user interactions, personalization logic, and scalable microservices architecture for modern streaming platforms.

1. Problem Statement & Requirements

Clarifying questions

1. What is the scale of users and items?
   • Is this a system for millions of users with hundreds of thousands of items, or smaller-scale use case?
2. What consistency requirements exist for recommendations?
   • Should recommendations be real-time, near-real-time, or batch-generated with eventual consistency?
3. What are the key entities and their relationships?
   • What constitutes a "user", "item", "interaction"? How do they relate to each other?
4. What is the read/write ratio for recommendations?
   • Are we serving millions of recommendation requests per second or more batch-oriented?
5. What latency sensitivity is expected?
   • Is sub-100ms response time critical, or is 500ms acceptable for batch recommendations?
6. What authentication model and access control strategy is required?
   • Should the system support OAuth2, JWTs, or API keys for different consumer types?
7. Are there external integrations or data sources (e.g., analytics, ML models)?
   • Should the system integrate with external ML services or data pipelines?

Functional requirements

• Recommendation API: Serve personalized recommendations to users based on their interaction history.
• User Profile Management: Store and update user preferences, behavior, and item interaction data.
• Item Catalog: Maintain metadata for items (products, articles, videos) that can be recommended.
• Real-time Scoring: Support real-time scoring of items for a user based on recent interactions.
• Batch Recommendation Generation: Periodically generate and cache recommendations for all users or batches of users.
• Feedback Loop: Accept user feedback (e.g., clicks, likes, purchases) to improve future recommendations.
• A/B Testing Support: Allow for different recommendation algorithms or strategies to be tested in parallel.

Non-functional requirements

• Latency Targets:
  • p50: <100ms for real-time recommendations
  • p95: <300ms for real-time recommendations
  • p99: <500ms for real-time recommendations
• Throughput:
  • Handle at least 10,000 real-time recommendation requests per second
  • Support batch generation of recommendations for 1M users within 2 hours
• Availability:
  • 99.9% uptime for real-time recommendations (SLA: 5 minutes of downtime/month)
• Scalability:
  • Support horizontal scaling to handle 10M+ users and 100M+ items
• Data Freshness:
  • Real-time recommendations should reflect user interactions within 5 seconds
  • Batch recommendations should be generated every 24 hours

Constraints and assumptions

• Assumptions:
  • User interaction data (clicks, views, purchases) is available in a structured format.
  • The system will support both collaborative filtering and content-based recommendation strategies.
  • Recommendations will be generated using ML models or pre-computed similarity matrices.
  • The system supports both real-time and batch recommendation workflows.
• Constraints:
  • No external ML model training or retraining is part of this system.
  • The system assumes that item metadata and user behavior data are already ingested into the system.
  • Recommendations will be served via a REST API or gRPC endpoint.

Out of scope

• ML model training and deployment — This system does not include ML model training or retraining capabilities.
• User session management — Session tracking and stateful context are not part of this design.
• Recommendation personalization beyond user history — Advanced personalization logic (e.g., contextual or temporal preferences) is out of scope.
• Cross-system integration — Integration with external systems like analytics, A/B testing platforms, or user management services is not part of this design.
• Recommendation explainability — Providing explanations for why a recommendation was made is not included.

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for the recommendation engine interface due to:

• Strong typing and automatic code generation
• Efficient binary serialization (lower payload size than JSON)
• Built-in support for streaming and bidirectional communication
• First-class support for versioning and schema evolution
• Better performance characteristics for high-throughput scenarios

API Endpoints

Method	Path	Description	Auth
GetRecommendations	/recommendations/v1/user/{userId}	Fetch personalized recommendations for a user	JWT
GetRecommendationsByCategory	/recommendations/v1/category/{categoryId}	Fetch recommendations by category	JWT
GetRecommendationsByItem	/recommendations/v1/item/{itemId}	Fetch similar items to a given item	JWT
ReportInteraction	/recommendations/v1/interaction	Report user interaction with a recommendation	JWT
GetUserPreferences	/recommendations/v1/user/{userId}/preferences	Fetch user's preference profile	JWT
UpdateUserPreferences	/recommendations/v1/user/{userId}/preferences	Update user's preference profile	JWT

Request and Response Schemas

GetRecommendations Request:

message GetRecommendationsRequest {
  string user_id = 1;
  int32 limit = 2;
  string category = 3;
  repeated string excluded_item_ids = 4;
  map<string, string> context = 5;
}

GetRecommendations Response:

message GetRecommendationsResponse {
  repeated RecommendationItem items = 1;
  string request_id = 2;
  int64 timestamp = 3;
}

message RecommendationItem {
  string item_id = 1;
  string item_type = 2;
  double score = 3;
  string category = 4;
  map<string, string> metadata = 5;
}

ReportInteraction Request:

message ReportInteractionRequest {
  string user_id = 1;
  string item_id = 2;
  string interaction_type = 3; // "click", "view", "purchase"
  string session_id = 4;
  int64 timestamp = 5;
  map<string, string> context = 6;
}

HTTP Status Codes and Error Response

Status Code	Description
200	Success
400	Bad Request (malformed request, validation errors)
401	Unauthorized (missing or invalid auth token)
403	Forbidden (insufficient permissions)
429	Rate Limited
500	Internal Server Error

Standard error response:

{
  "error": {
    "code": "INVALID_ARGUMENT",
    "message": "User ID is required",
    "details": [
      {
        "field": "user_id",
        "reason": "required"
      }
    ]
  }
}

API Versioning Strategy

Versioned using path prefix: /recommendations/v1/...
Supports backward compatibility through versioned endpoints and gradual migration.

Pagination Design

Cursor-based pagination is used because:

• More stable than offset-based pagination when data changes
• Better performance for large datasets
• Consistent results even with concurrent modifications

Authentication and Authorization

• Authentication: JWT tokens with audience validation
• Authorization: RBAC model where users have roles (user, admin, service)
• Permissions:
  • recommendations:read for reading recommendations
  • recommendations:write for reporting interactions and updating preferences

Rate Limiting

Endpoint	Limit	Period	Enforcement
GetRecommendations	100 req/min	1 minute	In-memory + Redis
ReportInteraction	500 req/min	1 minute	Redis-based token bucket

Idempotency Design

Mutation endpoints support idempotency via:

• Idempotency-Key header in requests
• Store request signatures with TTL (5 minutes)
• Return same response for duplicate requests within window

// Example idempotency check in service layer
public class IdempotencyService {
    private final RedisTemplate<String, String> redisTemplate;
    
    public boolean isIdempotent(String key) {
        String existing = redisTemplate.opsForValue().get(key);
        if (existing != null) {
            return true;
        }
        redisTemplate.opsForValue().set(key, "processed", 5, TimeUnit.MINUTES);
        return false;
    }
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    USER {
        string id PK
        string email
        string name
        timestamp created_at
        timestamp updated_at
    }

    RECOMMENDATION {
        string id PK
        string user_id FK
        string item_id
        string item_type
        float score
        timestamp created_at
        timestamp updated_at
    }

    ITEM {
        string id PK
        string type
        string title
        string description
        jsonb metadata
        timestamp created_at
        timestamp updated_at
    }

    USER_ITEM_INTERACTION {
        string id PK
        string user_id FK
        string item_id FK
        string interaction_type
        int rating
        timestamp timestamp
    }

    RECOMMENDATION_CONTEXT {
        string id PK
        string user_id FK
        jsonb context_data
        timestamp created_at
    }

    USER ||--o{ RECOMMENDATION : "has"
    USER ||--o{ USER_ITEM_INTERACTION : "has"
    ITEM ||--o{ RECOMMENDATION : "recommended"
    ITEM ||--o{ USER_ITEM_INTERACTION : "has"
    USER ||--o{ RECOMMENDATION_CONTEXT : "has"

Table Schema Definitions

-- Users table
CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email VARCHAR(255) UNIQUE NOT NULL,
    name VARCHAR(255),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Items table (items that can be recommended)
CREATE TABLE items (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    type VARCHAR(50) NOT NULL,
    title VARCHAR(255),
    description TEXT,
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- User-item interactions table
CREATE TABLE user_item_interactions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
    interaction_type VARCHAR(50) NOT NULL, -- 'view', 'like', 'purchase', etc.
    rating INTEGER CHECK (rating >= 0 AND rating <= 5),
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Recommendations table
CREATE TABLE recommendations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
    item_type VARCHAR(50),
    score FLOAT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Recommendation context table for storing user preferences and context
CREATE TABLE recommendation_contexts (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    context_data JSONB DEFAULT '{}',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Index Design

Index Name	Fields	Type	Purpose
idx_users_email	email	Unique	Fast user lookup by email
idx_items_type	type	Standard	Filter items by type
idx_user_item_interactions_user_timestamp	user_id, timestamp	Composite	Fast retrieval of recent user interactions
idx_user_item_interactions_item_timestamp	item_id, timestamp	Composite	Fast retrieval of recent item interactions
idx_recommendations_user_score	user_id, score DESC	Composite	Fast retrieval of top recommendations
idx_recommendations_item_user	item_id, user_id	Composite	Fast lookup of recommendations for specific items
idx_user_item_interactions_rating	rating	Partial	Query interactions by rating range

-- Indexes for performance optimization
CREATE UNIQUE INDEX idx_users_email ON users(email);
CREATE INDEX idx_items_type ON items(type);
CREATE INDEX idx_user_item_interactions_user_timestamp ON user_item_interactions(user_id, timestamp DESC);
CREATE INDEX idx_user_item_interactions_item_timestamp ON user_item_interactions(item_id, timestamp DESC);
CREATE INDEX idx_recommendations_user_score ON recommendations(user_id, score DESC);
CREATE INDEX idx_recommendations_item_user ON recommendations(item_id, user_id);
CREATE INDEX idx_user_item_interactions_rating ON user_item_interactions(rating) WHERE rating IS NOT NULL;

Primary Key Strategy

UUID (Universally Unique Identifier) is selected as the primary key strategy for all tables because:

1. Distributed system compatibility: UUIDs can be generated without coordination across distributed systems
2. Scalability: No need for centralized ID generation, avoiding bottlenecks in high-throughput systems
3. Security: UUIDs don't expose sequential patterns that could be exploited for data enumeration attacks
4. No contention: Eliminates race conditions in concurrent writes to the same table

Data Partitioning / Sharding Strategy

Shard by User ID (Hash-based):

• Partition Key: user_id hash modulo N shards
• Shard Count: 32 shards (configurable)
• Rationale:
  • Even distribution of users across shards
  • Predictable sharding with minimal re-sharding needs
  • Supports horizontal scaling for high user volumes

-- Example sharding pattern (conceptual)
-- Partitioned tables would have the same schema but distributed across shards
-- Shard 0: user_id hash % 32 = 0
-- Shard 1: user_id hash % 32 = 1
-- ...

Soft Delete vs. Hard Delete

Soft Delete is chosen with a deleted_at timestamp field:

1. Auditability: Historical recommendations and interactions can be audited
2. Recovery capability: Data can be recovered if accidentally deleted
3. Analytics preservation: Historical data is preserved for ML model training and analysis
4. Compliance: Meets regulatory requirements for data retention

Audit Trail Design

All tables include created_at and updated_at timestamps with audit fields:

-- Audit trail fields added to relevant tables
ALTER TABLE users ADD COLUMN created_by VARCHAR(255);
ALTER TABLE users ADD COLUMN updated_by VARCHAR(255);
ALTER TABLE items ADD COLUMN created_by VARCHAR(255);
ALTER TABLE items ADD COLUMN updated_by VARCHAR(255);

Data Archival Strategy

30-day retention policy with automatic archival:

1. Active data: Last 30 days of user interactions and recommendations
2. Archival process: Daily batch job moves data older than 30 days to archive tables
3. Archive tables:
   • user_item_interactions_archive
   • recommendations_archive
4. Retention policy: 1 year for analytics and compliance purposes

-- Archive table structure (conceptual)
CREATE TABLE user_item_interactions_archive (
    id UUID PRIMARY KEY,
    user_id UUID,
    item_id UUID,
    interaction_type VARCHAR(50),
    rating INTEGER,
    timestamp TIMESTAMP,
    created_at TIMESTAMP,
    archived_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

4. Class Design & Service Layer

classDiagram
    class RecommendationEngine {
        +recommend(User, Context) List~Recommendation~
        +updateUserPreferences(User, Preferences) void
        +getRecommendationsForUser(UserId, RequestContext) List~Recommendation~
    }

    class RecommendationService {
        -recommendationRepository RecommendationRepository
        -userPreferenceService UserPreferenceService
        -cacheService CacheService
        -eventPublisher EventPublisher
        +getRecommendationsForUser(UserId, RequestContext) List~Recommendation~
        +updateUserPreferences(User, Preferences) void
    }

    class RecommendationRepository {
        +save(Recommendation) void
        +findByUserAndContext(UserId, Context) List~Recommendation~
        +findTopNByUser(UserId, int) List~Recommendation~
    }

    class UserPreferenceService {
        +getUserPreferences(UserId) Preferences
        +updateUserPreferences(UserId, Preferences) void
    }

    class CacheService {
        +getRecommendations(UserId, Context) List~Recommendation~
        +putRecommendations(UserId, Context, List~Recommendation~) void
    }

    class EventPublisher {
        +publish(RecommendationEvent) void
    }

    class Recommendation {
        +id UUID
        +userId UserId
        +contentId ContentId
        +score Double
        +context Context
        +timestamp Instant
    }

    class Preferences {
        +userId UserId
        +interests Set~String~
        +exclusions Set~String~
        +lastUpdated Instant
    }

    class RequestContext {
        +userId UserId
        +timestamp Instant
        +userAgent String
        +location Location
    }

    class RecommendationStrategy {
        +apply(RecommendationContext) List~Recommendation~
    }

    class ContentBasedStrategy {
        +apply(RecommendationContext) List~Recommendation~
    }

    class CollaborativeFilteringStrategy {
        +apply(RecommendationContext) List~Recommendation~
    }

    class HybridStrategy {
        +apply(RecommendationContext) List~Recommendation~
    }

    RecommendationEngine --> RecommendationService
    RecommendationService --> RecommendationRepository
    RecommendationService --> UserPreferenceService
    RecommendationService --> CacheService
    RecommendationService --> EventPublisher
    RecommendationStrategy <|-- ContentBasedStrategy
    RecommendationStrategy <|-- CollaborativeFilteringStrategy
    RecommendationStrategy <|-- HybridStrategy
    RecommendationService --> RecommendationStrategy

stateDiagram-v2
    [*] --> Idle
    Idle --> Processing : New request
    Processing --> Cached : Cache hit
    Processing --> Computing : Cache miss
    Computing --> Stored : Results computed
    Stored --> Idle : Request completed
    Cached --> Idle : Request completed

Service Class Structure and Responsibilities

The core service layer consists of:

1. RecommendationEngine: Main interface for external consumers
2. RecommendationService: Implementation of core recommendation logic with:
   • User preference management
   • Cache integration
   • Event publishing
3. RecommendationRepository: Data access for recommendations
4. UserPreferenceService: User preference persistence and retrieval
5. CacheService: Caching layer for performance optimization

Interface / Abstract Class Design

public interface RecommendationStrategy {
    List<Recommendation> apply(RecommendationContext context);
}

public abstract class AbstractRecommendationService {
    protected final RecommendationRepository repository;
    protected final UserPreferenceService userPreferenceService;
    protected final CacheService cacheService;
    protected final EventPublisher eventPublisher;

    public AbstractRecommendationService(
            RecommendationRepository repository,
            UserPreferenceService userPreferenceService,
            CacheService cacheService,
            EventPublisher eventPublisher) {
        this.repository = repository;
        this.userPreferenceService = userPreferenceService;
        this.cacheService = cacheService;
        this.eventPublisher = eventPublisher;
    }

    protected List<Recommendation> getCachedOrCompute(
            UserId userId, 
            RequestContext context,
            Supplier<List<Recommendation>> computation) {
        List<Recommendation> cached = cacheService.getRecommendations(userId, context);
        if (cached != null && !cached.isEmpty()) {
            return cached;
        }
        
        List<Recommendation> computed = computation.get();
        cacheService.putRecommendations(userId, context, computed);
        return computed;
    }
}

Business Logic Encapsulation Patterns

The engine uses the Strategy Pattern to support multiple recommendation algorithms:

public class HybridRecommendationStrategy implements RecommendationStrategy {
    private final List<RecommendationStrategy> strategies;
    private final double[] weights;

    public HybridRecommendationStrategy(List<RecommendationStrategy> strategies, double[] weights) {
        this.strategies = strategies;
        this.weights = weights;
    }

    @Override
    public List<Recommendation> apply(RecommendationContext context) {
        Map<String, Double> combinedScores = new HashMap<>();
        
        for (int i = 0; i < strategies.size(); i++) {
            List<Recommendation> recommendations = strategies.get(i).apply(context);
            double weight = weights[i];
            
            for (Recommendation rec : recommendations) {
                combinedScores.merge(
                    rec.getContentId().toString(),
                    rec.getScore() * weight,
                    Double::sum
                );
            }
        }

        return combinedScores.entrySet().stream()
            .map(entry -> new Recommendation(
                UUID.randomUUID(),
                context.getUserId(),
                ContentId.from(entry.getKey()),
                entry.getValue(),
                context.getContext(),
                Instant.now()
            ))
            .sorted(Comparator.comparing(Recommendation::getScore).reversed())
            .limit(context.getLimit())
            .collect(Collectors.toList());
    }
}

Critical-Path Method Implementation

public List<Recommendation> getRecommendationsForUser(
        UserId userId, 
        RequestContext context) {
    
    // Transaction boundary - all operations in single DB transaction
    return transactionTemplate.execute(status -> {
        try {
            // 1. Get or create user preferences
            Preferences preferences = userPreferenceService.getUserPreferences(userId);
            
            // 2. Check cache first (optimistic approach)
            List<Recommendation> cached = cacheService.getRecommendations(userId, context);
            if (cached != null && !cached.isEmpty()) {
                return cached;
            }
            
            // 3. Apply hybrid recommendation strategy
            RecommendationContext strategyContext = new RecommendationContext(
                userId, 
                preferences, 
                context.getContext(),
                context.getLimit()
            );
            
            List<Recommendation> recommendations = hybridStrategy.apply(strategyContext);
            
            // 4. Store in cache and database
            recommendations.forEach(rec -> {
                repository.save(rec);
                cacheService.putRecommendations(userId, context, recommendations);
            });
            
            // 5. Publish event for async processing
            eventPublisher.publish(new RecommendationEvent(
                userId, 
                recommendations, 
                context.getTimestamp()
            ));
            
            return recommendations;
        } catch (Exception e) {
            status.setRollbackOnly();
            throw new RecommendationServiceException("Failed to generate recommendations", e);
        }
    });
}

Transaction Boundary Design

Each recommendation request is handled within a single database transaction:

• User preferences are read/updated atomically
• Recommendations are persisted to database
• Cache is updated with consistent state
• Events are published in same transaction scope

Service Interface Contracts

public interface RecommendationEngine {
    List<Recommendation> recommend(User user, RequestContext context);
    
    void updateUserPreferences(User user, Preferences preferences);
    
    List<Recommendation> getRecommendationsForUser(
        UserId userId, 
        RequestContext context);
}

Saga / Orchestration Pattern

For complex recommendation workflows involving external systems:

public class RecommendationSaga {
    private final List<RecommendationStep> steps;
    
    public void execute(RecommendationContext context) {
        List<RecommendationStep> failedSteps = new ArrayList<>();
        
        for (RecommendationStep step : steps) {
            try {
                step.execute(context);
            } catch (Exception e) {
                failedSteps.add(step);
                // Handle compensation logic if needed
                break;
            }
        }
        
        if (!failedSteps.isEmpty()) {
            // Rollback or retry strategy
            handleFailure(failedSteps, context);
        }
    }
}

Domain Object / Value Object Design

public class Recommendation {
    private final UUID id;
    private final UserId userId;
    private final ContentId contentId;
    private final Double score;
    private final Context context;
    private final Instant timestamp;

    // Constructor, getters, equals, hashCode
}

public class Preferences {
    private final UserId userId;
    private final Set<String> interests;
    private final Set<String> exclusions;
    private final Instant lastUpdated;

    // Constructor, getters, equals, hashCode
}

public class RequestContext {
    private final UserId userId;
    private final Instant timestamp;
    private final String userAgent;
    private final Location location;

    // Constructor, getters, equals, hashCode
}

5. Concurrency & Thread Safety

Concurrency Model

The recommendation engine will operate in a multi-threaded environment using Java's thread pool executor model. Each request processing will be handled by a dedicated thread from the application's thread pool, ensuring that concurrent recommendation requests can be processed in parallel without blocking each other. The system will leverage Java's java.util.concurrent package for managing thread safety and coordination.

Race Condition Prevention Strategy

Race conditions will be prevented through:

1. Immutable data structures for recommendation models and user preferences
2. Thread-local storage for per-request state tracking (e.g., request IDs, user session data)
3. Synchronized access to shared mutable state using read-write locks for cache and preference updates
4. Atomic operations for counters and metrics tracking

Optimistic vs. Pessimistic Locking Selection

Optimistic locking will be used for user preference updates and recommendation cache modifications due to:

• Low write contention in the recommendation system (most operations are reads)
• Better performance under concurrent read-heavy workloads
• Reduced lock contention and improved throughput
• Acceptable conflict resolution via retry mechanisms

Pessimistic locking is avoided as it would introduce significant latency and reduce system throughput for a read-heavy recommendation engine.

Deadlock Prevention Strategy

Deadlock prevention will be implemented through:

1. Fixed ordering of resource acquisition - All locks are acquired in a consistent order (e.g., by user ID hash)
2. Timeout-based lock acquisition - Maximum 50ms timeout for any lock acquisition
3. Deadlock detection monitoring - Periodic checks using thread dump analysis for debugging
4. Avoid nested locks - Locks are held for minimal duration and released immediately after use

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency will be handled through:

1. Request deduplication using a distributed cache (Redis) with TTL of 30 seconds to store request fingerprints
2. Idempotency key strategy - Each client-provided idempotency key is used to identify and deduplicate requests
3. Concurrent request queuing - When duplicate requests arrive, they will wait for the first to complete and return the same result
4. Request fingerprinting - SHA-256 hash of request parameters to detect duplicates

public class RecommendationService {
    private final RedisTemplate<String, String> redisTemplate;
    private final ReadWriteLock cacheLock = new ReentrantReadWriteLock();
    
    public RecommendationResponse getRecommendations(
            String userId, 
            String idempotencyKey, 
            RecommendationRequest request) {
        
        // Idempotency check
        String cacheKey = "recommendation:" + idempotencyKey;
        String cachedResult = redisTemplate.opsForValue().get(cacheKey);
        
        if (cachedResult != null) {
            return deserialize(cachedResult);
        }
        
        // Process request with optimistic locking
        RecommendationResponse response = processRecommendation(userId, request);
        
        // Cache result with TTL
        redisTemplate.opsForValue().set(
            cacheKey, 
            serialize(response), 
            30, TimeUnit.SECONDS
        );
        
        return response;
    }
    
    private RecommendationResponse processRecommendation(
            String userId, 
            RecommendationRequest request) {
        
        // Use read-write lock for thread-safe cache access
        cacheLock.readLock().lock();
        try {
            // Read from cache or database
            return fetchRecommendations(userId, request);
        } finally {
            cacheLock.readLock().unlock();
        }
    }
}

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant U as User
    participant FE as Frontend
    participant API as RecommendationAPI
    participant Cache as RedisCache
    participant DB as RecommendationDB
    participant ML as MLService

    U->>FE: Request recommendations
    FE->>API: GET /recommendations?userId=123&limit=10
    API->>Cache: Check if recommendations exist
    Cache-->>API: HIT (cached recommendations)
    API-->>FE: Return cached recommendations
    FE-->>U: Display recommendations

    Note over API,Cache: Cache miss scenario
    API->>Cache: Check if recommendations exist
    Cache-->>API: MISS (no cached recommendations)
    API->>DB: Fetch user preferences
    DB-->>API: User preferences data
    API->>ML: Request recommendations for user
    ML-->>API: Generated recommendations
    API->>Cache: Store recommendations with TTL
    Cache-->>API: Cache stored
    API-->>FE: Return recommendations
    FE-->>U: Display recommendations

sequenceDiagram
    participant U as User
    participant FE as Frontend
    participant API as RecommendationAPI
    participant Cache as RedisCache
    participant DB as RecommendationDB
    participant ML as MLService

    U->>FE: Request recommendations
    FE->>API: GET /recommendations?userId=123&limit=10
    API->>Cache: Check if recommendations exist
    Cache-->>API: MISS (no cached recommendations)
    API->>DB: Fetch user preferences
    DB-->>API: User preferences data
    API->>ML: Request recommendations for user
    ML-->>API: Service unavailable (503)
    API->>Cache: Store empty recommendations with short TTL
    Cache-->>API: Cache stored
    API-->>FE: Return empty recommendations with fallback flag
    FE-->>U: Display "No recommendations available"

sequenceDiagram
    participant U as User
    participant FE as Frontend
    participant API as RecommendationAPI
    participant Cache as RedisCache
    participant DB as RecommendationDB
    participant ML as MLService
    participant EventQueue as Kafka

    U->>FE: Request recommendations
    FE->>API: GET /recommendations?userId=123&limit=10
    API->>Cache: Check if recommendations exist
    Cache-->>API: MISS (no cached recommendations)
    API->>DB: Fetch user preferences
    DB-->>API: User preferences data
    API->>ML: Request recommendations for user
    ML-->>API: Recommendations generated
    API->>Cache: Store recommendations with TTL
    Cache-->>API: Cache stored
    API->>EventQueue: Publish recommendation event
    EventQueue-->>API: Event published successfully
    API-->>FE: Return recommendations
    FE-->>U: Display recommendations

    Note over EventQueue,ML: Async event processing
    EventQueue->>ML: Process recommendation update event
    ML-->>EventQueue: Event processed successfully

7. Caching Design

Cache Strategy Selection

For the recommendation engine, we'll use a cache-aside pattern with write-through behavior for critical data and write-behind for less time-sensitive operations. This approach provides the best balance between consistency, performance, and complexity.

Cache Key Design

Cache keys will follow a structured namespace pattern:

{namespace}:{entityType}:{entityId}:{cacheType}:{version}

Example keys:

• recommendation:user:12345:personalized:v1
• recommendation:item:67890:popularity:v1
• recommendation:category:54321:trending:v1

TTL Strategy

Data Type	TTL	Reason
Personalized recommendations	5 minutes	User preferences change frequently, need fresh data
Item popularity scores	30 minutes	Stable metrics, infrequent updates
Category trending data	1 hour	Less volatile than user-specific data
User preference profiles	24 hours	Expensive to compute, stable over time
Cold start recommendations	1 hour	New users need temporary fallbacks

Cache Eviction Policy

We'll implement a hybrid LRU + TTL eviction strategy:

• Primary policy: LRU (Least Recently Used) for memory efficiency
• Secondary policy: TTL-based expiration to handle stale data
• Cache warming strategy: Pre-populate popular recommendations during off-peak hours

Cache Stampede Prevention

To prevent cache stampedes:

1. Probabilistic early expiry: Set cache TTL to 75% of actual expiration time, allowing for 25% overlap
2. Distributed locks: Use Redis distributed locks (Redlock algorithm) for cache warming operations
3. Cache warming: Proactively warm cache with pre-computed recommendations during low-traffic periods

Distributed Cache Architecture

We'll use Redis Cluster with:

• 3 master nodes for read/write operations
• 3 replica nodes for high availability and read scaling
• Automatic failover with Redis Sentinel monitoring
• Cross-zone replication for geo-distributed deployments

Cache Invalidation Strategy

We'll use event-driven invalidation for real-time consistency:

1. Write-through pattern: When user interactions change, update cache immediately

2. Event-driven invalidation:

   • User preference updates → invalidate user-specific recommendations
   • Item rating changes → invalidate item popularity scores and related recommendations
   • New item added → invalidate trending category data

3. Graceful degradation: When cache is unavailable, fall back to database reads with circuit breaker pattern

Implementation Example (Java)

@Component
public class RecommendationCacheService {
    private final RedisTemplate<String, Object> redisTemplate;
    private final ObjectMapper objectMapper;
    
    public RecommendationCacheService(RedisTemplate<String, Object> redisTemplate, 
                                     ObjectMapper objectMapper) {
        this.redisTemplate = redisTemplate;
        this.objectMapper = objectMapper;
    }
    
    public Recommendation getCachedRecommendations(String userId, String cacheKey) {
        String key = buildCacheKey("recommendation", "user", userId, "personalized");
        
        // Try to get from cache with optimistic locking
        String cached = (String) redisTemplate.opsForValue().get(key);
        
        if (cached != null) {
            try {
                return objectMapper.readValue(cached, Recommendation.class);
            } catch (Exception e) {
                // Log error and proceed to recomputation
                log.error("Failed to deserialize cached recommendation for user: {}", userId, e);
            }
        }
        
        return null;
    }
    
    public void cacheRecommendations(String userId, Recommendation recommendations) {
        String key = buildCacheKey("recommendation", "user", userId, "personalized");
        
        try {
            String json = objectMapper.writeValueAsString(recommendations);
            
            // Use SET with NX (not exists) and EX (expire) for atomic operation
            Boolean success = redisTemplate.opsForValue()
                .setIfAbsent(key, json, Duration.ofMinutes(5));
            
            if (success) {
                log.debug("Cached recommendations for user: {}", userId);
            }
        } catch (Exception e) {
            log.error("Failed to cache recommendations for user: {}", userId, e);
        }
    }
    
    private String buildCacheKey(String namespace, String entityType, 
                                String entityId, String cacheType) {
        return String.format("%s:%s:%s:%s:v1", namespace, entityType, entityId, cacheType);
    }
    
    public void invalidateUserRecommendations(String userId) {
        String key = buildCacheKey("recommendation", "user", userId, "personalized");
        
        // Use Redis pipeline for atomic operations
        redisTemplate.delete(key);
        
        // Optionally trigger cache warming via async queue
        triggerCacheWarmup(userId);
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and persistence concerns. Each entity has its own repository interface with standardized CRUD operations and specialized query methods.

public interface RecommendationRepository {
    Optional<Recommendation> findById(UUID id);
    List<Recommendation> findByUserId(UUID userId);
    List<Recommendation> findByUserIdAndType(UUID userId, String type);
    Page<Recommendation> findByUserIdWithPagination(UUID userId, Pageable pageable);
    Recommendation save(Recommendation recommendation);
    void deleteById(UUID id);
    long countByUserId(UUID userId);
}

public interface UserPreferenceRepository {
    Optional<UserPreference> findByUserId(UUID userId);
    List<UserPreference> findByUserIdIn(List<UUID> userIds);
    UserPreference save(UserPreference preference);
    void deleteByUserId(UUID userId);
}

ORM vs Raw Query Decisions

For the recommendation engine, we use a hybrid approach:

1. ORM for simple CRUD operations: Hibernate/JPA is used for basic entity persistence and retrieval to leverage built-in features like lazy loading, caching, and type safety.

2. Raw queries for complex analytics: For recommendation algorithms that require complex aggregations or joins, raw SQL is used to achieve optimal performance.

@Repository
public class RecommendationRepositoryImpl implements RecommendationRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    @Override
    public List<Recommendation> findByUserIdAndType(UUID userId, String type) {
        // Using JPA for simple queries
        return entityManager.createQuery(
            "SELECT r FROM Recommendation r WHERE r.userId = :userId AND r.type = :type",
            Recommendation.class)
            .setParameter("userId", userId)
            .setParameter("type", type)
            .getResultList();
    }
    
    @Override
    public Page<Recommendation> findByUserIdWithPagination(UUID userId, Pageable pageable) {
        // Raw query for complex pagination with analytics
        String sql = """
            SELECT r.*, 
                   COUNT(*) OVER() as total_count
            FROM recommendations r 
            WHERE r.user_id = :userId 
            ORDER BY r.created_at DESC
            LIMIT :limit OFFSET :offset
            """;
            
        Query query = entityManager.createNativeQuery(sql, "RecommendationWithCount");
        query.setParameter("userId", userId);
        query.setParameter("limit", pageable.getPageSize());
        query.setParameter("offset", pageable.getOffset());
        
        return new PageImpl<>(query.getResultList(), pageable, totalCount);
    }
}

Read vs Write Separation

The system implements CQRS (Command Query Responsibility Segregation) pattern to separate read and write operations:

// Write-side repositories for domain mutations
@Repository
public class RecommendationWriteRepository {
    @PersistenceContext
    private EntityManager entityManager;
    
    public void saveRecommendations(List<Recommendation> recommendations) {
        for (Recommendation rec : recommendations) {
            entityManager.persist(rec);
        }
    }
    
    public void updateRecommendations(List<Recommendation> recommendations) {
        for (Recommendation rec : recommendations) {
            entityManager.merge(rec);
        }
    }
}

// Read-side repositories optimized for querying
@Repository
public class RecommendationReadRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    // Optimized for read-heavy workloads with materialized views
    public List<RecommendationView> getTopRecommendations(UUID userId, int limit) {
        String sql = """
            SELECT * FROM recommendation_views 
            WHERE user_id = :userId 
            ORDER BY score DESC 
            LIMIT :limit
            """;
            
        return entityManager.createNativeQuery(sql, "RecommendationView")
            .setParameter("userId", userId)
            .setParameter("limit", limit)
            .getResultList();
    }
}

Connection Pool Configuration

Connection pooling is configured with optimal settings for high-concurrency recommendation engine:

# application.yml
spring:
  datasource:
    hikari:
      maximum-pool-size: 50
      minimum-idle: 10
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      validation-timeout: 5000
      leak-detection-threshold: 60000

N+1 Query Prevention Strategy

To prevent N+1 query problems, we implement eager loading and batch fetching:

@Entity
@Table(name = "recommendations")
@NamedEntityGraph(
    name = "Recommendation.withUserPreferences",
    attributeNodes = @NamedAttributeNode(value = "userPreferences", fetchMode = FetchType.EAGER)
)
public class Recommendation {
    @Id
    private UUID id;
    
    @ManyToOne(fetch = FetchType.LAZY)
    @JoinColumn(name = "user_id")
    private User user;
    
    @OneToMany(mappedBy = "recommendation", fetch = FetchType.LAZY, cascade = CascadeType.ALL)
    private List<UserPreference> userPreferences = new ArrayList<>();
    
    // Batch loading for related entities
    @ManyToMany(fetch = FetchType.LAZY)
    @JoinTable(
        name = "recommendation_tags",
        joinColumns = @JoinColumn(name = "recommendation_id"),
        inverseJoinColumns = @JoinColumn(name = "tag_id")
    )
    private Set<Tag> tags = new HashSet<>();
}

// Repository method with batch loading
public List<Recommendation> findRecommendationsWithPreferences(List<UUID> userIds) {
    String jpql = """
        SELECT r FROM Recommendation r 
        LEFT JOIN FETCH r.userPreferences up 
        WHERE r.userId IN :userIds
        """;
    
    return entityManager.createQuery(jpql, Recommendation.class)
        .setParameter("userIds", userIds)
        .getResultList();
}

Batch Operation Design

Batch operations are implemented for efficient bulk data processing:

@Repository
public class BatchRecommendationRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    public void batchSaveRecommendations(List<Recommendation> recommendations) {
        int batchSize = 1000;
        
        for (int i = 0; i < recommendations.size(); i += batchSize) {
            int end = Math.min(i + batchSize, recommendations.size());
            List<Recommendation> batch = recommendations.subList(i, end);
            
            // Use bulk insert for better performance
            String sql = """
                INSERT INTO recommendations (id, user_id, content_id, score, created_at) 
                VALUES (?, ?, ?, ?, ?)
                """;
                
            try (PreparedStatement stmt = entityManager.unwrap(Session.class)
                    .connection().prepareStatement(sql)) {
                
                for (Recommendation rec : batch) {
                    stmt.setObject(1, rec.getId());
                    stmt.setObject(2, rec.getUserId());
                    stmt.setString(3, rec.getContentId());
                    stmt.setDouble(4, rec.getScore());
                    stmt.setTimestamp(5, Timestamp.valueOf(rec.getCreatedAt()));
                    stmt.addBatch();
                }
                
                stmt.executeBatch();
            } catch (SQLException e) {
                throw new DataAccessException("Failed to batch save recommendations", e);
            }
        }
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The recommendation engine will use a hybrid synchronous/asynchronous approach. The core recommendation generation will be handled synchronously for real-time user requests, while long-running training jobs and batch updates will be processed asynchronously.

Key triggers for async processing:

• User interaction events (clicks, views, purchases) - trigger immediate re-ranking
• Periodic model retraining jobs - scheduled daily/weekly
• Batch user behavior aggregation - hourly updates
• Cold start recommendations for new users - async warm-up

Message schema design

{
  "version": "1.0",
  "event_type": "user_interaction",
  "timestamp": "2023-10-15T14:30:00Z",
  "user_id": "user_12345",
  "interaction_type": "view",
  "item_id": "item_67890",
  "metadata": {
    "session_id": "sess_abc123",
    "source": "mobile_app"
  }
}

Dead letter queue strategy and poison message handling

All recommendation engine events will be processed through a dead letter queue (DLQ) system. Messages that fail more than 3 times will be routed to DLQ with a 24-hour delay before retrying. Poison messages (messages causing system instability) will be logged with full context and escalated to on-call engineers.

At-least-once vs. exactly-once delivery

At-least-once delivery is selected for the recommendation engine because:

• Duplicate recommendations are acceptable and can be filtered client-side
• The system is designed to handle idempotent operations gracefully
• Performance overhead of exactly-once is not justified for this use case

Retry policy for consumers

public class RecommendationRetryPolicy {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double BACKOFF_MULTIPLIER = 2.0;
    
    public boolean shouldRetry(int attempt, Exception exception) {
        if (attempt >= MAX_RETRIES) return false;
        
        // Don't retry on validation errors or business logic errors
        if (exception instanceof ValidationException || 
            exception instanceof BusinessLogicException) {
            return false;
        }
        
        return true;
    }
    
    public long getDelay(int attempt) {
        return (long) (BASE_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempt));
    }
}

Outbox pattern for reliable event publishing

The outbox pattern is implemented to ensure recommendation events are reliably published:

@Component
public class RecommendationEventPublisher {
    private final EventOutboxRepository outboxRepository;
    private final KafkaTemplate<String, RecommendationEvent> kafkaTemplate;
    
    @Transactional
    public void publishRecommendationEvent(RecommendationEvent event) {
        // Store in outbox table first
        OutboxEvent outboxEvent = new OutboxEvent(
            UUID.randomUUID(),
            "recommendation",
            event,
            EventStatus.PENDING
        );
        
        outboxRepository.save(outboxEvent);
        
        // Then publish to Kafka (in same transaction)
        kafkaTemplate.send("recommendations", event);
    }
}

Sequence diagram for async flows

sequenceDiagram
    participant User
    participant API
    participant EventQueue
    participant RecommendationService
    participant ModelTrainer
    
    User->>API: Submit interaction event
    API->>EventQueue: Queue event (user_interaction)
    EventQueue->>RecommendationService: Process event
    RecommendationService->>RecommendationService: Update user profile
    RecommendationService->>EventQueue: Queue re-ranking event
    
    Note over RecommendationService,ModelTrainer: Batch processing
    EventQueue->>ModelTrainer: Trigger daily retraining
    ModelTrainer->>ModelTrainer: Train new model version
    ModelTrainer->>EventQueue: Publish model update event

10. Error Handling Design

Error Classification Hierarchy

The recommendation engine follows a structured error classification system:

Error
├── ValidationException (400)
│   ├── InvalidInputException
│   └── MissingRequiredFieldException
├── BusinessLogicException (400)
│   ├── UserNotFoundException
│   └── RecommendationLimitExceededException
├── InfrastructureException (500)
│   ├── DatabaseException
│   ├── CacheException
│   └── ExternalServiceException
└── SystemException (500)
    ├── InternalServerException
    └── TimeoutException

Error Propagation Strategy

• Validation errors are handled at the API layer and immediately returned to client
• Business logic errors are propagated up through service layers to API layer for proper HTTP status codes
• Infrastructure errors (DB, cache) are wrapped in appropriate service exceptions and logged for monitoring
• System errors (internal server issues) are caught at the service layer and converted to 500s with stack traces for debugging

Retry Strategy with Exponential Backoff and Jitter

public class RetryConfig {
    private static final int MAX_RETRIES = 3;
    private static final long INITIAL_DELAY_MS = 1000;
    private static final double BACKOFF_MULTIPLIER = 2.0;
    private static final double JITTER_RANGE = 0.1; // 10% jitter
    
    public static long calculateDelay(int attempt) {
        long baseDelay = (long) (INITIAL_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempt));
        long jitter = (long) (baseDelay * JITTER_RANGE * (Math.random() - 0.5) * 2);
        return baseDelay + jitter;
    }
}

Circuit Breaker Design

@Component
public class RecommendationCircuitBreaker {
    private final CircuitBreaker circuitBreaker = CircuitBreaker.ofDefaults("recommendation-service");
    
    public <T> CompletableFuture<T> withCircuitBreaker(Supplier<CompletableFuture<T>> supplier) {
        return circuitBreaker
            .apply(supplier)
            .onFailure(throwable -> {
                // Log failure for monitoring
                log.error("Recommendation service circuit breaker trip", throwable);
            });
    }
}

Fallback Behavior Design

@Component
public class RecommendationFallbackHandler {
    
    public List<Recommendation> fallbackRecommendations(
            String userId, 
            RecommendationContext context,
            Throwable throwable) {
        
        // Log fallback trigger for monitoring
        log.warn("Falling back to default recommendations for user: {}, cause: {}", 
                 userId, throwable.getMessage());
        
        // Return basic trending recommendations or empty list
        return getTrendingRecommendations();
    }
    
    private List<Recommendation> getTrendingRecommendations() {
        // Implementation to fetch trending items
        return Collections.emptyList();
    }
}

Error Logging Format and Alerting Thresholds

Structured logging follows this format:

{
  "timestamp": "2023-10-15T14:30:45.123Z",
  "trace_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "span_id": "f0e9d8c7-b6a5-4321-0fed-cba987654321",
  "user_id": "user_12345",
  "error_code": "RECOMMENDATION_SERVICE_TIMEOUT",
  "error_message": "External service timeout after 5000ms",
  "error_type": "InfrastructureException",
  "service_name": "recommendation-engine",
  "request_path": "/api/v1/recommendations/user/user_12345",
  "http_status": 504,
  "stack_trace": "..."
}

Alerting thresholds:

• Critical (5xx errors): > 10 errors per minute for 5 minutes → On-call alert
• Warning (4xx errors): > 100 validation errors per minute → Alert
• Info (rate limiting): > 500 rate limit hits per hour → Log alert

Retry Policy Configuration

public class RecommendationRetryPolicy {
    private final RetryConfig retryConfig = new RetryConfig();
    
    public <T> CompletionStage<T> executeWithRetry(Supplier<CompletionStage<T>> operation) {
        return CompletableFuture
            .supplyAsync(() -> {
                int attempts = 0;
                while (true) {
                    try {
                        return operation.get().toCompletableFuture().join();
                    } catch (Exception e) {
                        attempts++;
                        if (attempts >= RetryConfig.MAX_RETRIES) {
                            throw new RuntimeException("Max retries exceeded", e);
                        }
                        
                        // Exponential backoff with jitter
                        long delay = RetryConfig.calculateDelay(attempts);
                        try {
                            Thread.sleep(delay);
                        } catch (InterruptedException ie) {
                            Thread.currentThread().interrupt();
                            throw new RuntimeException("Interrupted during retry", ie);
                        }
                    }
                }
            });
    }
}

Error Response Format

Standardized JSON error response:

{
  "error": {
    "code": "RECOMMENDATION_SERVICE_TIMEOUT",
    "message": "External service timeout after 5000ms",
    "timestamp": "2023-10-15T14:30:45.123Z",
    "request_id": "req_abc123"
  }
}

Error Handling in Service Layer

@Service
public class RecommendationService {
    
    public List<Recommendation> getRecommendations(String userId, RecommendationContext context) {
        try {
            // Validate input
            validateUserId(userId);
            
            // Fetch from cache first
            List<Recommendation> cached = recommendationCache.get(userId);
            if (cached != null) {
                return cached;
            }
            
            // Fetch from database or external service
            List<Recommendation> recommendations = fetchRecommendationsFromSource(userId, context);
            
            // Cache results
            recommendationCache.put(userId, recommendations);
            
            return recommendations;
            
        } catch (UserNotFoundException e) {
            // Log and rethrow for proper HTTP 404
            log.error("User not found: {}", userId, e);
            throw new UserNotFoundException("User not found", e);
        } catch (DatabaseException e) {
            // Log and wrap for 500 error
            log.error("Database error fetching recommendations for user: {}", userId, e);
            throw new InfrastructureException("Failed to fetch recommendations", e);
        } catch (Exception e) {
            // Log and wrap unknown errors
            log.error("Unexpected error fetching recommendations for user: {}", userId, e);
            throw new SystemException("Internal server error", e);
        }
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using a combination of:

• Custom validation annotations (e.g., @Min(1), @Max(1000)) for numeric ranges
• Regex patterns for string format validation (e.g., email, UUID)
• Apache Commons Validator for complex validations (credit card, phone number)
• OWASP Java HTML Sanitizer for preventing XSS in user-generated content fields

public class RecommendationRequest {
    @Min(1)
    @Max(1000)
    private Integer userId;
    
    @Min(1)
    @Max(50)
    private Integer limit;
    
    @Pattern(regexp = "^(RECOMMEND|TRENDING|PERSONALIZED)$")
    private String recommendationType;
}

SQL Injection Prevention

• Prepared statements with parameterized queries for all database interactions
• ORM framework (Hibernate) with strict mapping to prevent dynamic query construction
• Whitelisting of column names and table names in dynamic queries

Authentication Middleware Design

• JWT-based authentication with 24-hour expiration and refresh token support
• OAuth2 integration for third-party authentication (Google, Facebook)
• Authentication filter that validates JWT tokens and extracts user context
• API key authentication for service-to-service communication

Authorization Enforcement

• Role-Based Access Control (RBAC) with predefined roles: USER, ADMIN, ANALYST
• Permission checking at service layer using @PreAuthorize annotations
• Resource-level authorization for user-specific recommendations (e.g., only view own recommendations)
• Fine-grained permissions for admin operations (e.g., RECOMMENDATION_ADMIN, DATA_ACCESS)

@PreAuthorize("hasRole('USER') or hasRole('ADMIN')")
public List<Recommendation> getUserRecommendations(@AuthenticationPrincipal User user, int limit) {
    // Only return recommendations for the authenticated user
    return recommendationService.getRecommendationsForUser(user.getId(), limit);
}

Sensitive Data Encryption

• At rest: AES-256 encryption using AWS KMS for storing user preferences and historical data
• In transit: TLS 1.3 with HSTS enforcement for all API endpoints
• PII masking: All PII fields (user email, phone) are masked in logs and audit trails

PII Handling and Masking Strategy

• Email addresses are stored as user***@domain.com in logs and audit trails
• Phone numbers are masked to +1 (XXX) XXX-XXXX format for non-privileged access
• Full PII is only accessible to users with DATA_ACCESS role and is logged with audit trail

Secrets Management

• AWS Secrets Manager for storing database credentials, API keys, and encryption keys
• Rotating secrets every 90 days with automated key rotation
• Environment-specific secret paths: /prod/recommendation-engine/ vs /staging/recommendation-engine/

Audit Logging Design

• Logged events: User access, recommendation generation, model training, data access
• Log format includes: timestamp, user_id, action_type, resource_id, ip_address, session_id
• Retention policy: 90 days for audit logs with daily backups to S3 Glacier
• Compliance: Logs are structured to support GDPR and CCPA compliance requirements

{
  "timestamp": "2023-10-15T14:30:45.123Z",
  "user_id": "user-123",
  "action_type": "RECOMMENDATION_GENERATED",
  "resource_id": "rec-456",
  "ip_address": "192.168.1.100",
  "session_id": "sess-789"
}

12. Testing Strategy

Unit Testing Scope and Coverage Targets

Target: 85% code coverage for business logic classes, with at least 90% coverage for critical-path methods like recommendation generation and user preference updates.

Scope:

• Service layer classes (e.g., RecommendationService, UserPreferenceService)
• Repository interfaces and their implementations
• Domain object validation logic
• Event handlers and processors
• Utility classes for feature extraction and similarity calculations

Coverage Tools: JaCoCo for code coverage analysis, Mockito for mocking dependencies in unit tests.

Integration Testing Strategy

Database Integration:

• Use Testcontainers with PostgreSQL to spin up isolated test databases for each test suite
• Initialize schema using Flyway migrations in test environment to ensure consistency with production
• Use embedded H2 for fast unit tests that don't require full database features

External Service Integration:

• Use WireMock or Mockito to simulate external APIs (e.g., user profile service, content catalog)
• Mock Kafka consumers and producers to test event-driven flows without full infrastructure

End-to-End Test Scenarios:

• User preference update → recommendation generation → cache invalidation
• Batch processing of user interactions → feature vector updates → model retraining trigger

Contract Testing

Framework: Pact (consumer-driven contract testing)

Scope:

• Define contracts between recommendation engine and user service (for user preferences)
• Define contracts with content catalog service (for content metadata)
• Define contracts with analytics service (for interaction event publishing)

Versioning Strategy:

• Use Pact version tags to align with service release cycles
• Implement contract verification as part of CI pipeline

Performance Testing Tool and Key Scenarios

Tool: k6 (JavaScript-based load testing framework)

Key Performance Scenarios:

• Concurrent user recommendation requests (1000 concurrent users)
• Batch processing of interaction events (10,000 events/sec)
• Cold start performance for new user recommendations
• Recommendation freshness under high load (latency SLA: p95 < 200ms)

Metrics Tracked:

• Response time distribution (p50/p95/p99)
• Throughput (requests/sec)
• Error rate (5xx, 4xx)
• Memory and CPU utilization during load

Test Data Management Strategy

Fixtures:

• Use factory pattern to generate realistic user profiles, content items, and interaction events
• Define base fixtures for common test scenarios (new user, returning user, high-engagement user)
• Include sample feature vectors and similarity matrices for performance testing

Factories:

public class UserFactory {
    public static User createUserWithPreferences() {
        return User.builder()
            .id(UUID.randomUUID())
            .preferences(List.of("tech", "science"))
            .interactionHistory(createInteractionHistory())
            .build();
    }
}

Seeding Strategy:

• Use database seeding scripts for initial content catalog (e.g., 10,000 content items)
• Pre-populate user preferences and interaction data for performance testing
• Implement test data cleanup strategy to ensure isolation between test runs

Data Isolation:

• Use database schema per test suite or dedicated test database
• Implement transactional test isolation where possible to avoid cross-contamination

13. Observability Design

Structured Logging Schema

The logging system will use structured JSON format to ensure consistency and enable efficient querying. Each log entry will include the following required fields:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO",
  "service": "recommendation-engine",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user_12345",
  "request_id": "req_98765",
  "entity_id": "rec_12345",
  "operation": "getRecommendations",
  "duration_ms": 125,
  "error_code": "INTERNAL_ERROR",
  "error_message": "Database timeout",
  "stack_trace": "...",
  "method": "GET",
  "path": "/api/v1/recommendations",
  "ip_address": "192.168.1.100",
  "user_agent": "Mozilla/5.0...",
  "request_payload": {},
  "response_payload": {},
  "recommendations_count": 5,
  "recommendation_type": "collaborative_filtering",
  "latency_percentile": "p95"
}

PII masking rules:

• User IDs will be masked in logs when not strictly necessary for debugging
• IP addresses will be masked to the first 3 octets (e.g., 192.168.1.*)
• User agent strings will be truncated to first 100 characters

Distributed Tracing Instrumentation

OpenTelemetry will be used for distributed tracing with custom spans:

Key Operations to Trace:

1. recommendation_engine.get_recommendations - Main recommendation calculation
2. recommendation_engine.fetch_user_profile - User data retrieval
3. recommendation_engine.calculate_similarity - Similarity computation
4. recommendation_engine.fetch_item_data - Item metadata fetching

Span Attributes to Capture:

// Example span attributes for recommendation calculation
span.setAttribute("recommendation_type", "collaborative_filtering");
span.setAttribute("user_id", userId);
span.setAttribute("item_count", items.size());
span.setAttribute("algorithm_version", "v2.1");
span.setAttribute("cache_hit", false);
span.setAttribute("recommendations_returned", 5);

Metrics Design

RED Method Metrics (Rate, Errors, Duration):

Metric Name	Type	Labels	Description
recommendations_requests_total	Counter	status, type, algorithm	Total recommendation requests
recommendations_errors_total	Counter	error_type, algorithm	Recommendation processing errors
recommendations_duration_seconds	Histogram	type, algorithm	Response time distribution
recommendations_cache_hits_total	Counter	type, algorithm	Cache hit statistics
recommendations_items_processed_total	Counter	source, algorithm	Items processed in batch
recommendations_user_profile_fetch_duration_seconds	Histogram	source	User profile fetch latency
recommendations_similarity_calculation_duration_seconds	Histogram	algorithm	Similarity calculation latency

Additional Service-Level Metrics:

• recommendations_cache_miss_ratio - Gauge
• recommendations_queue_length - Gauge
• recommendations_processing_rate - Gauge
• recommendations_memory_usage_bytes - Gauge

Health Check Endpoint Design

// Health check endpoints with different levels of checks
@RestController
@RequestMapping("/health")
public class HealthController {
    
    @GetMapping("/live")
    public ResponseEntity<Map<String, Object>> live() {
        Map<String, Object> response = new HashMap<>();
        response.put("status", "healthy");
        response.put("timestamp", Instant.now());
        return ResponseEntity.ok(response);
    }
    
    @GetMapping("/ready")
    public ResponseEntity<Map<String, Object>> ready() {
        Map<String, Object> response = new HashMap<>();
        response.put("status", "ready");
        response.put("timestamp", Instant.now());
        response.put("database", databaseHealthCheck());
        response.put("cache", cacheHealthCheck());
        response.put("recommendation_engine", engineHealthCheck());
        return ResponseEntity.ok(response);
    }
    
    @GetMapping("/startup")
    public ResponseEntity<Map<String, Object>> startup() {
        Map<String, Object> response = new HashMap<>();
        response.put("status", "startup_complete");
        response.put("timestamp", Instant.now());
        response.put("dependencies_loaded", true);
        response.put("cache_ready", cacheReady());
        return ResponseEntity.ok(response);
    }
}

Alerting Thresholds

Critical Alerts (on-call escalation):

• recommendations_errors_total rate > 10/minute for 5 minutes → Critical
• recommendations_duration_seconds p95 > 500ms for 10 minutes → Critical
• recommendations_cache_hit_ratio < 70% for 15 minutes → Critical
• recommendations_processing_rate drops below 50% of baseline for 10 minutes → Critical

Warning Alerts (team notification):

• recommendations_duration_seconds p95 > 200ms for 5 minutes → Warning
• recommendations_errors_total rate > 2/minute for 3 minutes → Warning
• recommendations_cache_hit_ratio < 85% for 10 minutes → Warning

Alerting Configuration Example:

# Prometheus alert rules example
groups:
- name: recommendation-engine-alerts
  rules:
  - alert: HighRecommendationErrorRate
    expr: rate(recommendations_errors_total[5m]) > 10
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High recommendation error rate detected"
      description: "Recommendation engine is returning errors at {{ $value }} per minute"

  - alert: SlowRecommendationLatency
    expr: histogram_quantile(0.95, recommendations_duration_seconds) > 500
    for: 10m
    labels:
      severity: critical
    annotations:
      summary: "High recommendation latency"
      description: "95th percentile recommendation duration is {{ $value }}ms"

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• Recommendation freshness vs. performance trade-off: Should recommendations be computed in real-time or pre-computed with periodic batch jobs? Real-time provides better personalization but increases latency and computational cost. Batch processing offers better performance but introduces staleness.
• Personalization scope: Should recommendations be user-centric, content-centric, or hybrid? User-centric approaches require extensive user behavior tracking, while content-centric approaches focus on item similarity but may miss personal preferences.
• Cross-domain recommendation strategy: Should recommendations span across different content categories (e.g., movies + books) or be limited to same-domain items? Cross-domain recommendations require more complex feature engineering and may dilute relevance.

Known trade-offs and justifications

Trade-off	Option 1	Option 2	Justification
Real-time vs. Batch Processing	Real-time	Batch	Batch processing reduces computational cost and latency for users but introduces delay in reflecting recent user behavior. Real-time processing provides better personalization but increases infrastructure costs and complexity.
Cold Start Handling	User-based	Item-based	User-based cold start requires more user data to build accurate recommendations, while item-based approaches can leverage content features but may miss user preferences.
Scalability vs. Accuracy	Simple collaborative filtering	Matrix factorization	Simple approaches scale easily but may have lower accuracy. Complex algorithms provide better accuracy but require more computational resources and are harder to scale.
Consistency vs. Availability	Strong consistency	Eventual consistency	Strong consistency ensures users always see the same recommendations but may impact availability during high load. Eventual consistency improves availability but introduces potential inconsistency.

Risks and mitigation strategies

• Data quality degradation: Risk of poor recommendations due to noisy or outdated user behavior data. Mitigation: Implement data validation pipelines, anomaly detection, and regular data quality audits.
• Cold start problem: Risk of poor recommendations for new users or items with insufficient interaction data. Mitigation: Implement hybrid approaches combining content-based and collaborative filtering, use default recommendations for new users, and maintain popularity-based fallbacks.
• Scalability bottlenecks: Risk of recommendation system becoming a performance bottleneck as user base grows. Mitigation: Implement proper sharding strategies, use caching layers, and design for horizontal scaling with microservices architecture.
• Bias amplification: Risk of reinforcing existing biases in recommendations (e.g., popularity bias, gender bias). Mitigation: Implement bias detection mechanisms, diversity-aware recommendation algorithms, and regular fairness audits.
• Privacy compliance: Risk of violating user privacy regulations (GDPR, CCPA) when collecting and processing user data. Mitigation: Implement proper data anonymization, provide user controls for data sharing, and maintain audit trails of data usage.

Technical debt acknowledged

• Feature drift: Risk of recommendation algorithms becoming outdated as user behavior patterns evolve without regular retraining and model updates.
• Tight coupling between components: Risk of tight coupling between recommendation engine and other services, making future modifications difficult.
• Limited A/B testing capabilities: Risk of not having sufficient infrastructure to support robust A/B testing of different recommendation algorithms and strategies.
• Inadequate monitoring: Risk of insufficient observability into recommendation quality, user engagement with recommendations, and system performance metrics.
• Limited support for edge cases: Risk of not properly handling edge cases such as users with no interaction history or items with no user interactions, leading to suboptimal recommendations or system errors.