News Feed System Design - Scalable Content Aggregation Platform

A scalable news feed system that aggregates content from multiple sources, applies user preferences and ranking algorithms, and delivers personalized content to users with high availability and low latency requirements.

1. Problem Statement & Requirements

Clarifying questions

1. What is the scale of the system? What's the expected number of users, posts, and concurrent reads/writes?
2. What consistency requirements are there? Should the feed be strongly consistent, eventually consistent, or does it depend on content type?
3. What are the key entities and their relationships? Are users followed by other users, or do they follow topics/categories?
4. What's the read/write ratio? Is it predominantly reads (e.g., 90% feed reads) or more balanced?
5. What latency sensitivity is expected? Should feed generation happen in <100ms for real-time experience or can it be up to a few seconds?
6. What authentication model is in place? Is OAuth2, JWT, or API keys used for user identity?
7. Are there external integrations or data sources? Should the feed include content from third-party APIs, social media, or RSS feeds?

Functional requirements

• Users can follow other users or topics
• Users can view a personalized news feed based on their followed content sources
• Feed should include posts, articles, or other content types from followed sources
• Users can like, comment, or share posts in their feed
• Feed should support pagination and filtering (e.g., by date, content type)
• Users can mark posts as read or unread
• Feed should support real-time updates for new content from followed sources

Non-functional requirements

• Latency targets:
  • p50: <200ms for feed fetch
  • p95: <500ms for feed fetch
  • p99: <1s for feed fetch
• Throughput: Support 100K concurrent users fetching feeds with 500K posts per day
• Availability: 99.9% uptime (99.99% for core feed API)
• Scalability: Horizontal scaling to support 1M+ users

Constraints and assumptions

• Assumption: Users have a limited number of followed sources (e.g., <1000)
• Assumption: Feed content is primarily text-based with media attachments (images, videos)
• Constraint: Feed generation must be fast enough to support real-time user experience
• Assumption: Content is mostly immutable once published (no frequent edits)
• Constraint: No real-time feed updates for users who are offline or inactive

Out of scope

• Content moderation and spam filtering
• Feed personalization algorithms (ML-based ranking)
• Push notifications for feed updates
• Content creation and publishing workflows
• Feed analytics or engagement metrics tracking (e.g., view counts, like rates)
• Multi-language support for content
• Feed search functionality (search by keywords or tags)

2. API Design

API Style Selection

REST is selected for this design due to its simplicity, wide adoption, and ease of integration with existing frontend frameworks. REST provides clear resource-based URLs that make the API self-documenting and easy to understand for developers.

Endpoint Definitions

Method	Path	Description	Auth
GET	/api/v1/news-feed	Fetch user's personalized news feed	JWT
POST	/api/v1/news-feed/items	Create a new news item	JWT + Admin
PUT	/api/v1/news-feed/items/{id}	Update a news item	JWT + Admin
DELETE	/api/v1/news-feed/items/{id}	Delete a news item	JWT + Admin
GET	/api/v1/news-feed/items/{id}	Fetch a specific news item	JWT
POST	/api/v1/news-feed/subscriptions	Subscribe to a news source	JWT
DELETE	/api/v1/news-feed/subscriptions/{id}	Unsubscribe from a news source	JWT
GET	/api/v1/news-feed/categories	List all available categories	JWT
GET	/api/v1/news-feed/categories/{id}/items	Get news items by category	JWT

Request and Response Schemas

News Item Schema

{
  "id": "uuid",
  "title": "string",
  "content": "string",
  "author": "string",
  "category": "string",
  "publishedAt": "datetime",
  "source": "string",
  "imageUrl": "string",
  "isFeatured": "boolean"
}

News Feed Response

{
  "items": [
    {
      "id": "uuid",
      "title": "string",
      "content": "string",
      "author": "string",
      "category": "string",
      "publishedAt": "datetime",
      "source": "string",
      "imageUrl": "string",
      "isFeatured": "boolean"
    }
  ],
  "cursor": "string",
  "hasMore": "boolean"
}

Error Response

{
  "error": {
    "code": "string",
    "message": "string",
    "details": "object"
  }
}

HTTP Status Codes

• 200 OK - Successful GET, PUT, PATCH
• 201 Created - Successful POST
• 204 No Content - Successful DELETE
• 400 Bad Request - Invalid request parameters or body
• 401 Unauthorized - Missing or invalid authentication
• 403 Forbidden - Insufficient permissions
• 404 Not Found - Resource not found
• 429 Too Many Requests - Rate limit exceeded
• 500 Internal Server Error - Unexpected server error

API Versioning Strategy

Version 1 is implemented using path-based versioning (/api/v1/) to maintain clean separation between different API versions. This approach allows for backward compatibility and enables smooth migration of clients to newer versions.

Pagination Design

Cursor-based pagination is used with a cursor parameter for efficient handling of large datasets. This approach provides better performance and consistency compared to offset-based pagination, especially when data is constantly changing.

Authentication and Authorization

• JWT tokens are used for user authentication
• Admin role is required for content management operations
• User-specific subscriptions are enforced through JWT claims

Rate Limiting Strategy

• 100 requests per minute for anonymous users
• 1000 requests per minute for authenticated users
• Rate limiting enforced via Redis with sliding window algorithm

Idempotency Design

For mutation endpoints, idempotency keys are supported via X-Idempotency-Key header. The system stores the result of the first request and returns the same response for subsequent identical requests within a 5-minute window.

Example Request/Response

GET /api/v1/news-feed?limit=20&cursor=abc123 HTTP/1.1
Authorization: Bearer <jwt_token>
X-Idempotency-Key: unique_key_123

HTTP/1.1 200 OK
Content-Type: application/json

{
  "items": [
    {
      "id": "123e4567-e89b-12d3-a456-426614174000",
      "title": "Breaking News: Tech Innovation",
      "content": "Detailed content here...",
      "author": "John Doe",
      "category": "Technology",
      "publishedAt": "2023-10-15T14:30:00Z",
      "source": "TechNews",
      "imageUrl": "https://example.com/image.jpg",
      "isFeatured": true
    }
  ],
  "cursor": "def456",
  "hasMore": true
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    USER {
        string id PK
        string username
        string email
        timestamp created_at
        timestamp updated_at
        boolean is_active
    }

    ARTICLE {
        string id PK
        string title
        string content
        string author_id FK
        timestamp published_at
        timestamp created_at
        timestamp updated_at
        boolean is_published
    }

    USER_ARTICLE_FEED {
        string user_id FK
        string article_id FK
        timestamp added_at
        integer priority
    }

    USER_FOLLOW {
        string follower_id FK
        string following_id FK
        timestamp created_at
    }

    USER_ARTICLE_READ {
        string user_id FK
        string article_id FK
        timestamp read_at
    }

    USER {
        string id PK
        string username
        string email
        timestamp created_at
        timestamp updated_at
        boolean is_active
    }

    ARTICLE {
        string id PK
        string title
        string content
        string author_id FK
        timestamp published_at
        timestamp created_at
        timestamp updated_at
        boolean is_published
    }

    USER_ARTICLE_FEED {
        string user_id FK
        string article_id FK
        timestamp added_at
        integer priority
    }

    USER_FOLLOW {
        string follower_id FK
        string following_id FK
        timestamp created_at
    }

    USER_ARTICLE_READ {
        string user_id FK
        string article_id FK
        timestamp read_at
    }

    USER ||--o{ USER_ARTICLE_FEED : "has"
    USER ||--o{ USER_FOLLOW : "follows"
    USER ||--o{ USER_ARTICLE_READ : "reads"
    ARTICLE ||--o{ USER_ARTICLE_FEED : "in"
    ARTICLE ||--o{ USER_ARTICLE_READ : "read_by"
    USER ||--o{ USER_FOLLOW : "is_follower"
    USER ||--o{ USER_FOLLOW : "is_following"

Table Schema Definitions

Users Table

CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    username VARCHAR(50) UNIQUE NOT NULL,
    email VARCHAR(255) UNIQUE NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    is_active BOOLEAN DEFAULT TRUE,
    
    INDEX idx_users_email (email),
    INDEX idx_users_username (username),
    INDEX idx_users_created_at (created_at)
);

Articles Table

CREATE TABLE articles (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    title VARCHAR(255) NOT NULL,
    content TEXT NOT NULL,
    author_id UUID REFERENCES users(id),
    published_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    is_published BOOLEAN DEFAULT FALSE,
    
    INDEX idx_articles_author (author_id),
    INDEX idx_articles_published_at (published_at),
    INDEX idx_articles_created_at (created_at),
    INDEX idx_articles_is_published (is_published)
);

User Article Feed Table

CREATE TABLE user_article_feed (
    user_id UUID REFERENCES users(id),
    article_id UUID REFERENCES articles(id),
    added_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    priority INTEGER DEFAULT 0,
    
    PRIMARY KEY (user_id, article_id),
    INDEX idx_user_article_feed_user_added (user_id, added_at),
    INDEX idx_user_article_feed_article_priority (article_id, priority),
    INDEX idx_user_article_feed_user_priority (user_id, priority)
);

User Follow Table

CREATE TABLE user_follow (
    follower_id UUID REFERENCES users(id),
    following_id UUID REFERENCES users(id),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    PRIMARY KEY (follower_id, following_id),
    INDEX idx_user_follow_follower (follower_id),
    INDEX idx_user_follow_following (following_id)
);

User Article Read Table

CREATE TABLE user_article_read (
    user_id UUID REFERENCES users(id),
    article_id UUID REFERENCES articles(id),
    read_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    PRIMARY KEY (user_id, article_id),
    INDEX idx_user_article_read_user (user_id),
    INDEX idx_user_article_read_article (article_id)
);

Primary Key Strategy

UUIDs chosen over auto-increment IDs for the following reasons:

1. Scalability: UUIDs can be generated client-side without database round-trips
2. Global Uniqueness: No conflicts when merging data from distributed systems
3. Security: Harder to predict next ID for malicious actors
4. Sharding: Easier to distribute across multiple database shards

Index Design Justification

1. users.email: For fast email-based lookups (login, verification)
2. users.username: For username-based authentication and profile lookups
3. articles.author_id: For fetching articles by author
4. articles.published_at: For chronological article queries and feed generation
5. user_article_feed.user_id + added_at: For efficient feed pagination by time
6. user_article_feed.article_id + priority: For fast article lookup with priority consideration
7. user_article_feed.user_id + priority: For feed prioritization queries
8. user_follow.follower_id: For fetching user's followings
9. user_follow.following_id: For fetching followers of a user
10. user_article_read.user_id: For reading article history by user

Data Partitioning Strategy

Horizontal Sharding by User ID (UUID):

• Partition key: First 4 bytes of UUID (or first 2 characters of string representation)
• Sharding strategy: Consistent hashing with 64 shards for initial scale
• Each shard contains users, their feeds, and read history
• Articles are replicated across all shards for fast content lookup

Soft Delete vs. Hard Delete

Soft delete chosen with the following considerations:

1. Auditability: Historical feed state can be reconstructed for analytics
2. Recovery capability: Articles can be restored if accidentally deleted
3. User experience: Feed items can be re-added without content loss
4. Compliance: GDPR and other regulations may require data retention

Audit Trail Design

All write operations will be logged to an audit table:

CREATE TABLE audit_log (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    operation_type VARCHAR(20) NOT NULL, -- CREATE, UPDATE, DELETE
    table_name VARCHAR(50) NOT NULL,
    record_id UUID NOT NULL,
    user_id UUID REFERENCES users(id),
    changes JSONB,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_audit_log_table_record (table_name, record_id),
    INDEX idx_audit_log_user_time (user_id, created_at)
);

Data Archival Strategy

1. Articles:

   • Keep published articles for 5 years
   • Archive articles older than 5 years to cold storage (S3 or similar)
   • Maintain metadata in database for quick lookup

2. Feed Records:

   • Keep feed history for 30 days
   • Archive older entries to separate archive tables or external storage

3. Read History:

   • Keep read history for 90 days
   • Archive to data warehouse for analytics

-- Example archive table structure for articles older than 5 years
CREATE TABLE archived_articles (
    id UUID PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    content TEXT NOT NULL,
    author_id UUID,
    published_at TIMESTAMP,
    archived_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

4. Class Design & Service Layer

classDiagram
    class NewsFeedService {
        +createNewsFeed(userId: String, feedConfig: FeedConfig): NewsFeed
        +getNewsFeed(userId: String, cursor: String, limit: int): NewsFeedPage
        +addArticleToFeed(userId: String, articleId: String): void
        +removeArticleFromFeed(userId: String, articleId: String): void
        +updateFeedPreferences(userId: String, preferences: FeedPreferences): void
    }
    
    class ArticleService {
        +createArticle(article: Article): Article
        +getArticleById(articleId: String): Article
        +updateArticle(article: Article): void
        +deleteArticle(articleId: String): void
    }
    
    class FeedConfig {
        userId: String
        maxArticles: int
        contentTypes: List~String~
        categories: List~String~
    }
    
    class NewsFeed {
        userId: String
        articles: List~Article~
        createdAt: Instant
        updatedAt: Instant
    }
    
    class Article {
        id: String
        title: String
        content: String
        authorId: String
        category: String
        publishedAt: Instant
        tags: List~String~
    }
    
    class FeedPreferences {
        userId: String
        preferredCategories: List~String~
        excludedSources: List~String~
        maxArticlesPerHour: int
    }
    
    class FeedEvent {
        eventType: String
        userId: String
        articleId: String
        timestamp: Instant
    }
    
    class FeedEventPublisher {
        +publishFeedEvent(event: FeedEvent): void
    }
    
    class ArticleRepository {
        +save(article: Article): Article
        +findById(articleId: String): Optional~Article~
        +findAllByIds(articleIds: List~String~): List~Article~
    }
    
    class NewsFeedRepository {
        +save(feed: NewsFeed): NewsFeed
        +findById(userId: String): Optional~NewsFeed~
        +findByUserId(userId: String): NewsFeed
    }
    
    class FeedConfigRepository {
        +save(config: FeedConfig): FeedConfig
        +findByUserId(userId: String): Optional~FeedConfig~
    }
    
    NewsFeedService --> NewsFeedRepository : manages
    NewsFeedService --> ArticleRepository : queries
    NewsFeedService --> FeedConfigRepository : manages
    NewsFeedService --> ArticleService : delegates
    NewsFeedService --> FeedEventPublisher : publishes
    
    ArticleService --> ArticleRepository : persists
    ArticleService --> FeedEventPublisher : publishes
    
    FeedEventPublisher --> EventPublisher : delegates
    
    NewsFeedService --> FeedPreferences : uses
    ArticleService --> FeedPreferences : uses
    
    class EventPublisher {
        +publish(event: Event): void
    }
    
    EventPublisher <|-- FeedEventPublisher : implements
    
    class FeedStrategy {
        +buildFeed(feedConfig: FeedConfig, articleIds: List~String~): NewsFeed
    }
    
    class ContentBasedFeedStrategy {
        +buildFeed(feedConfig: FeedConfig, articleIds: List~String~): NewsFeed
    }
    
    class TimeBasedFeedStrategy {
        +buildFeed(feedConfig: FeedConfig, articleIds: List~String~): NewsFeed
    }
    
    FeedStrategy <|-- ContentBasedFeedStrategy : implements
    FeedStrategy <|-- TimeBasedFeedStrategy : implements
    
    NewsFeedService --> FeedStrategy : uses

stateDiagram-v2
    [*] --> Idle
    Idle --> Building : addArticle()
    Building --> Ready : buildComplete()
    Ready --> Updating : updatePreferences()
    Updating --> Ready : updateComplete()
    Ready --> Deleting : removeArticle()
    Deleting --> Ready : deleteComplete()

Service Class Structure and Responsibilities

The News Feed system is designed with a clean service layer that encapsulates business logic and orchestrates interactions between domain entities.

NewsFeedService: Main orchestration service responsible for:

• Creating and managing user news feeds
• Building and updating feed content based on preferences
• Managing feed configuration and preferences
• Publishing feed-related events for external systems

ArticleService: Manages article lifecycle including creation, retrieval, and deletion.

FeedEventPublisher: Handles event publishing for feed-related operations to support audit trail and external integrations.

Interface / Abstract Class Design

public interface FeedStrategy {
    NewsFeed buildFeed(FeedConfig config, List<String> articleIds);
}

public abstract class AbstractFeedService {
    protected final FeedConfigRepository feedConfigRepository;
    protected final NewsFeedRepository newsFeedRepository;
    protected final ArticleRepository articleRepository;
    protected final FeedEventPublisher eventPublisher;
    protected final FeedStrategy feedStrategy;

    public AbstractFeedService(FeedConfigRepository feedConfigRepository,
                              NewsFeedRepository newsFeedRepository,
                              ArticleRepository articleRepository,
                              FeedEventPublisher eventPublisher,
                              FeedStrategy feedStrategy) {
        this.feedConfigRepository = feedConfigRepository;
        this.newsFeedRepository = newsFeedRepository;
        this.articleRepository = articleRepository;
        this.eventPublisher = eventPublisher;
        this.feedStrategy = feedStrategy;
    }
}

Business Logic Encapsulation Patterns

Strategy Pattern: The system uses the Strategy pattern for building different types of feeds based on content and time preferences.

public class ContentBasedFeedStrategy implements FeedStrategy {
    private final Set<String> preferredCategories;
    
    public ContentBasedFeedStrategy(FeedPreferences preferences) {
        this.preferredCategories = new HashSet<>(preferences.getPreferredCategories());
    }
    
    @Override
    public NewsFeed buildFeed(FeedConfig config, List<String> articleIds) {
        // Filter articles by preferred categories
        return new NewsFeed(
            articleIds.stream()
                .filter(id -> isPreferredCategory(id))
                .limit(config.getMaxArticles())
                .collect(Collectors.toList())
        );
    }
    
    private boolean isPreferredCategory(String articleId) {
        // Implementation to check if article belongs to preferred category
        return true;
    }
}

Critical-Path Method Implementation

public NewsFeedPage getNewsFeed(String userId, String cursor, int limit) {
    // Transaction boundary: All operations within single transaction
    return transactionTemplate.execute(status -> {
        try {
            // Read feed config with pessimistic locking to prevent race conditions
            FeedConfig feedConfig = feedConfigRepository.findByUserId(userId)
                .orElseThrow(() -> new ResourceNotFoundException("Feed config not found for user: " + userId));
            
            // Read current feed with optimistic locking
            NewsFeed currentFeed = newsFeedRepository.findByUserId(userId)
                .orElseGet(() -> createNewFeed(userId, feedConfig));
            
            // Validate and refresh feed if needed
            NewsFeed updatedFeed = refreshFeedIfNeeded(currentFeed, feedConfig);
            
            // Build page with cursor-based pagination
            List<Article> articles = fetchArticlesForFeed(updatedFeed, limit);
            
            // Update feed timestamp
            updatedFeed.setUpdatedAt(Instant.now());
            newsFeedRepository.save(updatedFeed);
            
            // Publish feed access event
            eventPublisher.publish(new FeedEvent(
                "FEED_ACCESS", userId, null, Instant.now()));
            
            return new NewsFeedPage(articles, buildNextCursor(articles));
        } catch (Exception e) {
            status.setRollbackOnly();
            throw new InternalServerErrorException("Failed to build news feed", e);
        }
    });
}

Transaction Boundary Design

All operations within a single user's feed management are wrapped in a single database transaction to ensure consistency. This includes:

• Reading and updating feed configuration
• Building and persisting the updated feed
• Publishing events for audit trail

Service Interface Contracts

public interface NewsFeedService {
    NewsFeedPage getNewsFeed(String userId, String cursor, int limit);
    NewsFeed createNewsFeed(String userId, FeedConfig feedConfig);
    void addArticleToFeed(String userId, String articleId);
    void removeArticleFromFeed(String userId, String articleId);
    void updateFeedPreferences(String userId, FeedPreferences preferences);
}

Saga / Orchestration Pattern

For distributed transactions involving article creation and feed updates, we implement an orchestration pattern where the NewsFeedService coordinates between local database operations and external event publishing. The service ensures that all operations either complete successfully or are rolled back.

Domain Object / Value Object Design

public class NewsFeed {
    private String userId;
    private List<Article> articles;
    private Instant createdAt;
    private Instant updatedAt;
    
    // Constructor, getters, setters
}

public class FeedConfig {
    private String userId;
    private int maxArticles;
    private List<String> contentTypes;
    private List<String> categories;
    
    // Constructor, getters, setters
}

public class FeedPreferences {
    private String userId;
    private List<String> preferredCategories;
    private List<String> excludedSources;
    private int maxArticlesPerHour;
    
    // Constructor, getters, setters
}

5. Concurrency & Thread Safety

Concurrency Model

The news feed system will utilize a thread pool model with Java's ExecutorService for handling concurrent requests. Each incoming request (e.g., feed fetch, article read) will be processed by a worker thread from the pool. For high-throughput scenarios, we'll leverage CompletableFuture for asynchronous processing and non-blocking I/O operations where applicable.

Race Condition Prevention Strategy

Race conditions in the news feed system primarily occur during:

1. Feed updates: Concurrent writes to user's feed when multiple articles are published simultaneously
2. Article read tracking: Concurrent updates to article read counts or user read status
3. Feed reordering: Concurrent re-sorting of feed items based on freshness or popularity

We implement race condition prevention through:

• Database-level atomic operations (e.g., UPDATE ... SET count = count + 1 WHERE id = ?)
• Optimistic locking using version fields for feed item updates
• Database transactions for multi-step operations like article publishing and feed insertion

Optimistic vs. Pessimistic Locking Selection

We use optimistic locking for most operations because:

• News feed updates are frequent but not highly contended
• The system prioritizes throughput over strict consistency
• Most operations are read-heavy with infrequent writes

Pessimistic locking is used for:

• Feed reordering operations where strict ordering is required
• Critical article publishing workflows that must not be interleaved

Deadlock Prevention Strategy

Deadlock prevention is implemented through:

1. Consistent lock ordering: All operations acquire locks in a consistent order (e.g., by user ID, then article ID)
2. Timeout-based acquisition: Locks are acquired with a 500ms timeout to prevent indefinite blocking
3. Hierarchical lock management: Using a lock manager that tracks all active locks and prevents cycles

Idempotency Implementation for Concurrent Duplicate Requests

For operations like article publishing or feed refresh, we implement idempotency using:

1. Idempotency keys in request headers (e.g., X-Idempotency-Key)
2. Database deduplication: Store request metadata with unique keys and skip processing if already handled
3. Cache-based deduplication: For read-heavy operations, cache recent requests with TTL

public class IdempotencyService {
    private final RedisTemplate<String, String> redisTemplate;
    
    public boolean isDuplicate(String key) {
        String existing = redisTemplate.opsForValue().get(key);
        if (existing != null) {
            return true;
        }
        redisTemplate.opsForValue().set(key, "processed", 30, TimeUnit.MINUTES);
        return false;
    }
}

Thread Safety Patterns

1. Immutable Objects for Feed Items

public final class NewsArticle {
    private final String id;
    private final String title;
    private final String content;
    private final LocalDateTime publishedAt;
    private final String authorId;
    
    // Constructor and getters...
}

2. Thread-Safe Collections for Feed Caching

public class UserFeedCache {
    private final Map<String, List<NewsArticle>> userFeeds = 
        new ConcurrentHashMap<>();
    
    public void updateFeed(String userId, List<NewsArticle> articles) {
        userFeeds.put(userId, new CopyOnWriteArrayList<>(articles));
    }
    
    public List<NewsArticle> getFeed(String userId) {
        return userFeeds.getOrDefault(userId, Collections.emptyList());
    }
}

3. Synchronized Access to Shared State

public class FeedUpdateService {
    private final Object feedLock = new Object();
    
    public void updateFeedWithArticle(String userId, NewsArticle article) {
        synchronized (feedLock) {
            // Critical section - update feed and notify subscribers
            List<NewsArticle> currentFeed = getFeed(userId);
            currentFeed.add(0, article); // Add to top
            persistFeed(userId, currentFeed);
        }
    }
}

Concurrency Control for Feed Refresh Operations

When refreshing a user's feed, we implement a concurrent read with exclusive write pattern:

1. Multiple readers can concurrently read from the feed cache
2. When a refresh is needed, we acquire a write lock to prevent concurrent writes
3. Feed refreshes are batched and processed in the background to avoid blocking user requests

public class ConcurrentFeedService {
    private final ReadWriteLock feedLock = new ReentrantReadWriteLock();
    private final Lock readLock = feedLock.readLock();
    private final Lock writeLock = feedLock.writeLock();
    
    public List<NewsArticle> getUserFeed(String userId) {
        readLock.lock();
        try {
            return feedCache.get(userId);
        } finally {
            readLock.unlock();
        }
    }
    
    public void refreshUserFeed(String userId) {
        writeLock.lock();
        try {
            // Refresh logic here
            List<NewsArticle> newFeed = buildFreshFeed(userId);
            feedCache.put(userId, newFeed);
        } finally {
            writeLock.unlock();
        }
    }
}

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant U as User
    participant FE as Frontend
    participant API as NewsFeedAPI
    participant Cache as RedisCache
    participant DB as Database
    participant Queue as MessageQueue
    participant Worker as FeedWorker

    U->>FE: Request /feed?page=1&limit=20
    FE->>API: GET /feed?page=1&limit=20
    API->>Cache: Check if feed exists in cache
    alt Cache hit
        Cache-->>API: Return cached feed
        API-->>FE: Return cached feed (200)
    else Cache miss
        API->>DB: Query user's followed topics and recent articles
        DB-->>API: Return articles with metadata
        API->>Cache: Cache feed for 5 minutes
        Cache-->>API: Store feed with TTL
        API-->>FE: Return articles (200)
    end

    U->>FE: Request /feed?topic=tech&page=1&limit=20
    FE->>API: GET /feed?topic=tech&page=1&limit=20
    API->>Cache: Check if topic feed exists in cache
    alt Cache hit
        Cache-->>API: Return cached topic feed
        API-->>FE: Return cached topic feed (200)
    else Cache miss
        API->>DB: Query articles for topic 'tech'
        DB-->>API: Return articles with metadata
        API->>Cache: Cache topic feed for 10 minutes
        Cache-->>API: Store topic feed with TTL
        API-->>FE: Return articles (200)
    end

sequenceDiagram
    participant U as User
    participant FE as Frontend
    participant API as NewsFeedAPI
    participant Cache as RedisCache
    participant DB as Database
    participant Queue as MessageQueue
    participant Worker as FeedWorker

    U->>FE: Post article /articles
    FE->>API: POST /articles
    API->>DB: Insert new article with metadata
    DB-->>API: Return article ID and timestamp
    API->>Cache: Invalidate user's feed cache
    Cache-->>API: Invalidate cached feeds for user
    API->>Queue: Publish ArticleCreated event
    Queue-->>Worker: Event published to queue
    Worker->>DB: Update article feed index for all followers
    DB-->>Worker: Return success status
    Worker->>Queue: Publish FeedUpdated event
    Queue-->>API: Event published to queue
    API->>FE: Return created article (201)

sequenceDiagram
    participant U as User
    participant FE as Frontend
    participant API as NewsFeedAPI
    participant Cache as RedisCache
    participant DB as Database
    participant Queue as MessageQueue
    participant Worker as FeedWorker

    U->>FE: Follow topic /topics/tech/follow
    FE->>API: POST /topics/tech/follow
    API->>DB: Insert follow relationship
    DB-->>API: Return success status
    API->>Queue: Publish TopicFollowed event
    Queue-->>Worker: Event published to queue
    Worker->>DB: Update user's feed index with new topic articles
    DB-->>Worker: Return success status
    Worker->>Queue: Publish FeedUpdated event
    Queue-->>API: Event published to queue
    API->>Cache: Invalidate user's feed cache
    Cache-->>API: Invalidate cached feeds for user
    API->>FE: Return success (200)

sequenceDiagram
    participant U as User
    participant FE as Frontend
    participant API as NewsFeedAPI
    participant Cache as RedisCache
    participant DB as Database
    participant Queue as MessageQueue
    participant Worker as FeedWorker

    U->>FE: Request /feed?page=1&limit=20
    FE->>API: GET /feed?page=1&limit=20
    API->>Cache: Check if feed exists in cache
    alt Cache miss and DB timeout
        API->>DB: Query user's followed topics and recent articles
        DB-->>API: Timeout error (504)
        API->>FE: Return timeout error (504)
    else Cache miss and DB success
        API->>DB: Query user's followed topics and recent articles
        DB-->>API: Return articles with metadata
        API->>Cache: Cache feed for 5 minutes
        Cache-->>API: Store feed with TTL
        API-->>FE: Return articles (200)
    end

    U->>FE: Request /feed?topic=tech&page=1&limit=20
    FE->>API: GET /feed?topic=tech&page=1&limit=20
    API->>Cache: Check if topic feed exists in cache
    alt Cache miss and DB timeout
        API->>DB: Query articles for topic 'tech'
        DB-->>API: Timeout error (504)
        API->>FE: Return timeout error (504)
    else Cache miss and DB success
        API->>DB: Query articles for topic 'tech'
        DB-->>API: Return articles with metadata
        API->>Cache: Cache topic feed for 10 minutes
        Cache-->>API: Store topic feed with TTL
        API-->>FE: Return articles (200)
    end

7. Caching Design

Cache Strategy Selection

For the news feed system, we'll use a cache-aside pattern with write-through behavior for critical data and cache-aside for read-heavy content. This approach provides optimal control over cache lifecycle and ensures data consistency.

Cache Key Design

Cache keys will follow a structured namespace pattern:

{namespace}:{entity_type}:{entity_id}:{version}:{context}

Example keys:

• feed:user:12345:latest:v1:timeline
• feed:article:67890:published:v1:content
• feed:user:12345:latest:v1:following

TTL Strategy

Data Type	TTL	Reason
User feed timeline	30 seconds	Recent content changes require quick propagation
Article content	1 hour	Content is relatively stable, reduces database load
User preferences	1 day	Preferences change infrequently
User following list	5 minutes	Fast propagation of new follows
Feed statistics	10 minutes	Aggregated metrics can tolerate some staleness

Cache Eviction Policy

We'll implement a hybrid LRU + TTL eviction strategy:

• Primary: LRU-based eviction for memory pressure
• Secondary: TTL-based cleanup for stale content
• Cache size: 10GB (configurable per environment)

Cache Stampede Prevention

To prevent cache stampedes:

1. Probabilistic early expiry: Set cache TTL to 75% of configured TTL
2. Distributed locks: Use Redis Redlock for cache warming operations
3. Cache warming: Pre-warm popular feeds during low-traffic periods

Distributed Cache Architecture

We'll use Redis Cluster with:

• 3 master nodes, 3 replica nodes per shard
• Auto-failover configuration
• Memory persistence using RDB snapshots
• Redis Sentinel for monitoring and failover

Cache Invalidation Strategy

We'll implement event-driven invalidation:

1. Write operations: Invalidate related cache entries on data changes
2. User follows/unfollows: Invalidate user feed cache and following list
3. Article publishing: Invalidate relevant article content caches
4. Batch invalidation: Use Redis pipeline for bulk cache clearing

public class CacheInvalidator {
    private final RedisTemplate<String, Object> redisTemplate;
    
    public void invalidateUserFeed(String userId) {
        String key = "feed:user:" + userId + ":latest:*";
        Set<String> keys = redisTemplate.keys(key);
        if (keys != null && !keys.isEmpty()) {
            redisTemplate.delete(keys);
        }
    }
    
    public void invalidateArticleContent(String articleId) {
        String key = "feed:article:" + articleId + ":published:*";
        redisTemplate.delete(key);
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern provides a clean abstraction between the domain layer and data access logic, promoting loose coupling and testability. For the news feed system, we implement repositories for core entities like User, FeedItem, and UserSubscription.

public interface UserRepository {
    Optional<User> findById(UUID userId);
    List<User> findByIds(List<UUID> userIds);
    void save(User user);
    void delete(UUID userId);
}

public interface FeedItemRepository {
    Optional<FeedItem> findById(UUID itemId);
    List<FeedItem> findByUserIds(List<UUID> userIds, int limit);
    void save(FeedItem item);
    void delete(UUID itemId);
}

public interface UserSubscriptionRepository {
    List<UUID> findSubscribedUserIds(UUID userId);
    void save(UserSubscription subscription);
    void delete(UserSubscription subscription);
}

ORM vs Raw Query Decisions

For performance-critical operations like feed generation, we use raw SQL queries with prepared statements to avoid ORM overhead. For simpler CRUD operations, we leverage JPA/Hibernate for type safety and maintainability.

@Repository
public class FeedItemRepositoryImpl implements FeedItemRepository {
    
    @PersistenceContext
    private EntityManager entityManager;
    
    @Override
    public List<FeedItem> findByUserIds(List<UUID> userIds, int limit) {
        String sql = """
            SELECT f FROM FeedItem f 
            WHERE f.authorId IN :userIds 
            AND f.createdAt >= :cutoffDate
            ORDER BY f.createdAt DESC
            """;
            
        return entityManager.createQuery(sql, FeedItem.class)
                .setParameter("userIds", userIds)
                .setParameter("cutoffDate", calculateCutoffDate())
                .setMaxResults(limit)
                .getResultList();
    }
    
    private LocalDateTime calculateCutoffDate() {
        return LocalDateTime.now().minusDays(30); // 30-day feed window
    }
}

Read vs Write Separation

We implement CQRS (Command Query Responsibility Segregation) for the news feed to separate read and write operations. The write side handles content creation, while the read side optimizes for feed retrieval.

// Write-side (Command)
public class FeedItemCommandService {
    private final FeedItemRepository feedItemRepository;
    
    public void createFeedItem(FeedItem item) {
        feedItemRepository.save(item);
        // Publish to event bus for async read model update
        eventPublisher.publish(new FeedItemCreatedEvent(item));
    }
}

// Read-side (Query)
public class FeedReadService {
    private final FeedItemReadRepository feedItemReadRepository;
    
    public List<FeedItem> getUserFeed(UUID userId, int limit) {
        return feedItemReadRepository.findUserFeed(userId, limit);
    }
}

Connection Pool Configuration

We configure HikariCP with optimal settings for high-throughput feed operations:

spring:
  datasource:
    hikari:
      maximum-pool-size: 50
      minimum-idle: 10
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      leak-detection-threshold: 60000

N+1 Query Prevention Strategy

We implement eager loading and batch fetching to prevent N+1 queries:

@Query("SELECT f FROM FeedItem f JOIN FETCH f.author WHERE f.authorId IN :userIds")
List<FeedItem> findByUserIdsWithAuthor(@Param("userIds") List<UUID> userIds);

// Batch fetch for user subscriptions
@Query("SELECT s FROM UserSubscription s WHERE s.userId = :userId")
List<UserSubscription> findSubscriptionsByUserId(@Param("userId") UUID userId);

Batch Operation Design

For bulk operations like feed item creation or subscription updates, we implement batch processing:

@Repository
public class BatchFeedItemRepository {
    
    public void batchCreate(List<FeedItem> items) {
        String sql = "INSERT INTO feed_items (id, author_id, content, created_at) VALUES (?, ?, ?, ?)";
        
        try (Connection conn = dataSource.getConnection();
             PreparedStatement stmt = conn.prepareStatement(sql)) {
            
            for (FeedItem item : items) {
                stmt.setObject(1, item.getId());
                stmt.setObject(2, item.getAuthorId());
                stmt.setString(3, item.getContent());
                stmt.setTimestamp(4, Timestamp.valueOf(item.getCreatedAt()));
                stmt.addBatch();
            }
            
            stmt.executeBatch();
        } catch (SQLException e) {
            throw new DataAccessException("Failed to batch create feed items", e);
        }
    }
}

Read-Through Cache Integration

We implement read-through caching in repositories to reduce database load:

@Repository
public class CachedFeedItemRepository implements FeedItemRepository {
    
    private final FeedItemRepository delegate;
    private final CacheManager cacheManager;
    
    @Override
    public Optional<FeedItem> findById(UUID itemId) {
        String cacheKey = "feed_item:" + itemId.toString();
        
        return Optional.ofNullable(
            cacheManager.getCache("feedItems")
                .get(cacheKey, FeedItem.class)
        );
    }
    
    @Override
    public List<FeedItem> findByUserIds(List<UUID> userIds, int limit) {
        // Cache key includes user IDs and limit for freshness
        String cacheKey = "feed:" + userIds.stream().map(UUID::toString).sorted().collect(Collectors.joining(":")) + ":" + limit;
        
        return cacheManager.getCache("userFeeds")
            .get(cacheKey, new TypeReference<List<FeedItem>>() {});
    }
}

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The news feed system uses a hybrid synchronous/asynchronous approach where:

• Synchronous flows: User actions like following/unfollowing, posting articles, and reading feed items are handled synchronously to ensure immediate consistency for user interactions.
• Asynchronous flows: Content distribution, recommendation updates, and analytics processing are handled asynchronously to decouple system components and improve scalability.

Key asynchronous triggers:

1. Article published → Trigger recommendation engine update and feed distribution
2. User follows/unfollows → Update user's feed content and preferences
3. User interacts with article (likes, comments) → Update engagement metrics and trigger content recommendations
4. Content moderation → Update article visibility and notify relevant users

Message schema design with sample payload

{
  "version": "1.0",
  "event_type": "article_published",
  "timestamp": "2023-10-27T15:30:00Z",
  "payload": {
    "article_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "author_id": "u987654321",
    "category": "technology",
    "tags": ["ai", "machine-learning"],
    "content": "New breakthrough in neural networks...",
    "published_at": "2023-10-27T15:30:00Z"
  },
  "source": "content-service",
  "correlation_id": "c1d2e3f4-5678-90ab-cdef-1234567890ab"
}

Dead letter queue strategy and poison message handling

Poison messages are handled with a dead-letter queue (DLQ) strategy:

1. Retry logic: Messages are retried up to 3 times with exponential backoff (5s, 25s, 125s)
2. DLQ routing: Messages that fail after max retries are moved to a DLQ topic
3. Manual inspection: Failed messages in DLQ are logged with full context and require manual intervention
4. Monitoring: DLQ queue size and error rates are monitored with alerts

At-least-once vs. exactly-once delivery

At-least-once delivery is selected for this system because:

• The news feed service is idempotent in nature (reprocessing same article won't cause inconsistency)
• User actions are designed to be idempotent (e.g., multiple likes don't increase count)
• Simpler implementation and better performance characteristics
• Acceptable trade-off for better scalability

Retry policy for consumers

Retry policy configuration:

public class RetryPolicy {
    private static final int MAX_RETRIES = 3;
    private static final long INITIAL_DELAY_MS = 5000;
    private static final double BACKOFF_MULTIPLIER = 5.0;
    private static final long MAX_DELAY_MS = 300000; // 5 minutes
    
    public static long calculateDelay(int attempt) {
        if (attempt <= 0) return 0;
        long delay = (long) (INITIAL_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempt - 1));
        return Math.min(delay, MAX_DELAY_MS);
    }
}

Non-retryable error classification:

• Validation errors: Invalid article format, missing required fields
• Authorization errors: User not authorized to publish content
• Business logic errors: Article already published, user blocked

Outbox pattern for reliable event publishing

Outbox pattern is implemented to ensure event delivery reliability:

public class EventPublisher {
    private final EventRepository eventRepository;
    private final KafkaTemplate<String, String> kafkaTemplate;
    
    public void publishEvent(Event event) {
        // 1. Persist event to outbox table within same transaction
        EventOutbox outboxEvent = new EventOutbox(event);
        eventRepository.save(outboxEvent);
        
        // 2. Publish to Kafka in separate transaction
        kafkaTemplate.send("news-feed-events", event.toJson());
    }
}

Sequence diagram for async flows

sequenceDiagram
    participant ContentService
    participant EventPublisher
    participant Kafka
    participant FeedService
    participant RecommendationEngine
    
    ContentService->>EventPublisher: Publish article_published event
    EventPublisher->>EventPublisher: Save to outbox table (tx)
    EventPublisher->>Kafka: Publish to events topic
    Kafka->>FeedService: Deliver article_published event
    FeedService->>FeedService: Update user feeds (async)
    Kafka->>RecommendationEngine: Deliver article_published event
    RecommendationEngine->>RecommendationEngine: Update recommendations (async)

The outbox pattern ensures that events are published reliably even if the database transaction fails, providing eventual consistency for the news feed system.

10. Error Handling Design

Error Classification Hierarchy

The system implements a hierarchical error classification to enable proper handling and routing:

Error
├── DomainError (business logic violations)
│   ├── ValidationException
│   ├── BusinessLogicException
│   └── ResourceNotFoundException
├── InfrastructureError (system-level issues)
│   ├── DatabaseException
│   ├── ExternalServiceException
│   └── CacheException
└── SystemError (internal system failures)
    ├── InternalServerException
    └── ConfigurationException

Error Propagation Strategy

• Domain errors are caught and converted to appropriate HTTP status codes (400, 404, 409) at the API boundary
• Infrastructure errors are wrapped in system errors and logged with full stack trace for debugging
• System errors bubble up to the global error handler and are converted to 500 Internal Server Errors

Retry Strategy with Exponential Backoff and Jitter

For external service calls and transient infrastructure issues:

public class RetryableOperation {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;

    public <T> T executeWithRetry(Supplier<T> operation) {
        int attempts = 0;
        Exception lastException = null;

        while (attempts < MAX_RETRIES) {
            try {
                return operation.get();
            } catch (Exception e) {
                lastException = e;
                if (isRetryable(e) && attempts < MAX_RETRIES - 1) {
                    long delay = calculateBackoffDelay(attempts);
                    try {
                        Thread.sleep(delay);
                    } catch (InterruptedException ie) {
                        Thread.currentThread().interrupt();
                        throw new RuntimeException("Retry interrupted", ie);
                    }
                } else {
                    break;
                }
            }
            attempts++;
        }

        throw new RuntimeException("Operation failed after " + MAX_RETRIES + " attempts", lastException);
    }

    private boolean isRetryable(Exception e) {
        return e instanceof ExternalServiceException || 
               e instanceof DatabaseException ||
               e instanceof CacheException;
    }

    private long calculateBackoffDelay(int attempt) {
        long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempt));
        long jitter = (long) (delay * JITTER_FACTOR * (Math.random() - 0.5) * 2);
        return delay + jitter;
    }
}

Circuit Breaker Design

For external service dependencies with configurable thresholds:

public class CircuitBreaker {
    private final int failureThreshold;
    private final long timeoutMs;
    private final int successThreshold;

    private volatile CircuitState state = CircuitState.CLOSED;
    private volatile long lastFailureTime = 0;
    private volatile int failureCount = 0;
    private volatile int successCount = 0;

    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }

    public <T> T execute(Supplier<T> operation) {
        switch (state) {
            case CLOSED:
                return executeClosed(operation);
            case OPEN:
                return executeOpen(operation);
            case HALF_OPEN:
                return executeHalfOpen(operation);
            default:
                throw new IllegalStateException("Unknown circuit state");
        }
    }

    private <T> T executeClosed(Supplier<T> operation) {
        try {
            T result = operation.get();
            onSuccess();
            return result;
        } catch (Exception e) {
            onFailure(e);
            throw e;
        }
    }

    private <T> T executeOpen(Supplier<T> operation) {
        if (System.currentTimeMillis() - lastFailureTime > timeoutMs) {
            state = CircuitState.HALF_OPEN;
            return executeHalfOpen(operation);
        }
        throw new ExternalServiceException("Circuit breaker is OPEN");
    }

    private <T> T executeHalfOpen(Supplier<T> operation) {
        try {
            T result = operation.get();
            successCount++;
            if (successCount >= successThreshold) {
                state = CircuitState.CLOSED;
                successCount = 0;
            }
            return result;
        } catch (Exception e) {
            state = CircuitState.OPEN;
            lastFailureTime = System.currentTimeMillis();
            throw e;
        }
    }

    private void onSuccess() {
        successCount++;
        failureCount = 0;
    }

    private void onFailure(Exception e) {
        failureCount++;
        lastFailureTime = System.currentTimeMillis();
        if (failureCount >= failureThreshold) {
            state = CircuitState.OPEN;
        }
    }
}

Fallback Behavior Design

Each external dependency has a fallback implementation:

public class NewsFeedService {
    private final ExternalRecommendationService recommendationService;
    private final FallbackRecommendationService fallbackRecommendationService;

    public List<NewsItem> getRecommendedFeeds(String userId, int limit) {
        try {
            return recommendationService.getRecommendations(userId, limit);
        } catch (ExternalServiceException e) {
            // Fallback to pre-computed recommendations or default content
            return fallbackRecommendationService.getFallbackRecommendations(userId, limit);
        }
    }
}

Error Logging Format and Alerting Thresholds

Structured logging with consistent format:

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);

    public void logError(String operation, Exception error, Map<String, Object> context) {
        Map<String, Object> logContext = new HashMap<>();
        logContext.put("operation", operation);
        logContext.put("errorType", error.getClass().getSimpleName());
        logContext.put("errorMessage", error.getMessage());
        logContext.put("timestamp", Instant.now().toString());
        logContext.putAll(context);

        logger.error("Error in operation: {}", operation, error);
    }

    public void logCriticalError(String operation, Exception error) {
        // Send to monitoring system for alerting
        sendAlert(operation, error);
    }

    private void sendAlert(String operation, Exception error) {
        // Implementation for sending to alerting systems like PagerDuty or Slack
    }
}

Alerting Thresholds

Metric	Warning Level	Critical Level	Alerting Channel
External API failures	>5% error rate	>10% error rate	Slack/Email
Database connection timeouts	>50ms avg latency	>200ms avg latency	PagerDuty
Cache miss rate	>5% cache misses	>10% cache misses	Email
Service latency (p99)	>500ms	>1000ms	Slack

Error Response Format

Standardized JSON error response:

{
  "error": {
    "code": "NEWS_FEED_NOT_FOUND",
    "message": "News feed not found for user",
    "timestamp": "2023-10-15T14:30:00Z",
    "requestId": "req-1234567890"
  }
}

Error Handling in API Layer

@RestController
@RequestMapping("/api/news")
public class NewsFeedController {
    private final NewsFeedService newsFeedService;

    @GetMapping("/feed")
    public ResponseEntity<List<NewsItem>> getUserFeed(
            @RequestParam String userId,
            @RequestParam(defaultValue = "10") int limit) {
        try {
            List<NewsItem> feed = newsFeedService.getRecommendedFeeds(userId, limit);
            return ResponseEntity.ok(feed);
        } catch (ResourceNotFoundException e) {
            return ResponseEntity.notFound().build();
        } catch (ValidationException e) {
            return ResponseEntity.badRequest().body(Collections.emptyList());
        } catch (ExternalServiceException e) {
            // Log and return 503 for temporary failures
            logger.error("External service failure in feed generation", e);
            return ResponseEntity.status(503).body(Collections.emptyList());
        } catch (Exception e) {
            logger.error("Unexpected error in feed generation", e);
            return ResponseEntity.status(500).body(Collections.emptyList());
        }
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using a combination of:

• Custom validation annotations (e.g., @NotBlank, @Size) for basic constraints
• Apache Commons Validator for complex patterns (email, URL, phone number)
• OWASP Java HTML Sanitizer for user-generated content to prevent XSS
• Custom sanitization logic for specific fields like titles and descriptions

public class NewsFeedItemRequest {
    @NotBlank(message = "Title is required")
    @Size(max = 255, message = "Title must be less than 255 characters")
    private String title;
    
    @NotBlank(message = "Content is required")
    @Size(max = 10000, message = "Content must be less than 10000 characters")
    private String content;
    
    @ValidUrl(message = "Invalid URL format")
    private String imageUrl;
}

SQL Injection Prevention

• All database queries use parameterized prepared statements
• No dynamic query building with user input
• ORM framework (Hibernate) is used exclusively for data access

Authentication Middleware Design

• JWT-based authentication with 24-hour expiration
• Authentication filter intercepts all requests and validates JWT tokens
• Token refresh mechanism with 7-day refresh token validity
• Role-based access control via JWT claims

@Component
public class JwtAuthenticationFilter extends OncePerRequestFilter {
    @Override
    protected void doFilterInternal(HttpServletRequest request, 
                                   HttpServletResponse response, 
                                   FilterChain filterChain) throws ServletException, IOException {
        String token = extractToken(request);
        if (token != null && jwtService.validateToken(token)) {
            Authentication auth = jwtService.getAuthentication(token);
            SecurityContextHolder.getContext().setAuthentication(auth);
        }
        filterChain.doFilter(request, response);
    }
}

Authorization Enforcement

• RBAC (Role-Based Access Control) implemented at service layer
• Permission checking before any data access or modification operations
• User roles: USER, MODERATOR, ADMIN
• Resource-level permissions for content moderation actions

@PreAuthorize("hasRole('MODERATOR') or @newsFeedAuthorizationService.canEditContent(principal, #itemId)")
public void updateNewsItem(String itemId, NewsFeedUpdateRequest request) {
    // Implementation
}

Sensitive Data Encryption

• At rest: AES-256 encryption using AWS KMS for content storage
• In transit: TLS 1.3 with HSTS header enforcement
• PII encryption: PII fields (user emails, IPs) are encrypted using AES-256

PII Handling and Masking Strategy

• PII fields: User emails, IP addresses, device IDs are encrypted at rest
• Masking in logs: IPs shown as 192.168.***.***
• Audit logs: PII fields are redacted in logs but stored encrypted

Secrets Management

• AWS Secrets Manager for storing:
  • Database credentials
  • JWT signing keys
  • External API keys (Twitter, RSS feeds)
• Rotating secrets every 90 days with automatic key rotation
• Environment-specific secret paths to isolate dev/staging/prod secrets

Audit Logging Design

• Events logged: User actions (create, update, delete), system events (feed refresh)
• Log format:

{
  "timestamp": "2023-10-15T14:30:00Z",
  "user_id": "user_12345",
  "action": "CREATE_NEWS_ITEM",
  "resource_id": "item_67890",
  "ip_address": "192.168.***.***",
  "user_agent": "Mozilla/5.0...",
  "request_id": "req_abc123"
}

• Retention policy: 90 days for audit logs, 1 year for system events
• Compliance: Logs stored in encrypted format with access control

Security Headers Implementation

@Component
public class SecurityHeadersFilter implements Filter {
    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) {
        HttpServletResponse httpResponse = (HttpServletResponse) response;
        httpResponse.setHeader("X-Content-Type-Options", "nosniff");
        httpResponse.setHeader("X-Frame-Options", "DENY");
        httpResponse.setHeader("X-XSS-Protection", "1; mode=block");
        httpResponse.setHeader("Strict-Transport-Security", "max-age=31536000; includeSubDomains");
        chain.doFilter(request, response);
    }
}

Security Testing Considerations

• Static code analysis using SonarQube for security vulnerabilities
• Dynamic application security testing (DAST) with OWASP ZAP
• Penetration testing conducted quarterly by third-party vendors
• Security scanning integrated into CI/CD pipeline for every commit

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

• Service Layer: 85% code coverage, focusing on business logic and edge cases
• Repository Layer: 90% coverage with mock database interactions
• Domain Objects: 100% coverage for value objects and immutable entities
• API Layer: 75% coverage for request/response validation and error handling paths
• Security Layer: 100% coverage for authentication, authorization, and input validation logic

Integration Testing Strategy

• Database Integration: Test with embedded H2 or PostgreSQL in test containers for real database interactions
• External Service Mocks: Use WireMock or Mockito to simulate third-party API responses (e.g., user profile service, content recommendation engine)
• Event System: Use embedded Kafka or RabbitMQ for testing event-driven flows and consumer logic
• Caching Layer: Test with Redis in-memory container to validate cache behavior and invalidation logic
• Authentication Integration: Use mock OAuth2 server or JWT provider to test token validation and role-based access control

Contract Testing

• API Contracts: Use Pact for consumer-driven contract testing between frontend and backend services
• Event Contracts: Define Avro schemas with versioning strategy and validate message format consistency across producers and consumers
• External API Contracts: Implement contract tests with WireMock to ensure compatibility with content delivery networks and analytics services

Performance Testing Tool and Key Scenarios

• Load Testing: Use k6 or Gatling to simulate concurrent users (1000+ simultaneous requests) and measure response times
• Stress Testing: Simulate burst traffic (5000 concurrent users) to identify breaking points and memory leaks
• Key Scenarios:
  • News feed generation under high concurrent load (p95 < 200ms)
  • Content fetch and aggregation latency (p99 < 500ms)
  • Cache hit ratio validation (target: >90%)
  • Database query performance (index usage verification)

Test Data Management Strategy

• Fixtures: Define reusable test data sets for users, content categories, and feed preferences in JSON/YAML format
• Factories: Implement factory classes for generating realistic test data (e.g., UserFactory, ArticleFactory) with configurable attributes
• Seeding: Use database migration scripts or test data builders to populate test environments with consistent baseline data
• Data Isolation: Use database transaction rollback or separate test databases to ensure test isolation and prevent cross-test contamination

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema with the following required fields:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO|WARN|ERROR",
  "service": "news-feed-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user-12345",
  "request_id": "req-98765",
  "operation": "fetchNewsFeed",
  "entity_id": "feed-12345",
  "message": "News feed fetched successfully",
  "duration_ms": 123,
  "error_code": "FEED_NOT_FOUND",
  "error_message": "No feed found for user",
  "stack_trace": "..."
}

PII masking rules:

• User IDs are logged as user-***
• Email addresses are masked as ***@***.com
• Full names are truncated to first letter: J***n Doe

Distributed Tracing Instrumentation

Using OpenTelemetry with custom spans:

Key Operations to Trace:

1. fetchNewsFeed - Main API endpoint
2. buildFeedContent - Content aggregation logic
3. fetchUserPreferences - User preference loading
4. filterArticlesByTopic - Content filtering
5. publishFeedEvent - Event publishing

Span Attributes:

// Example span attributes for fetchNewsFeed operation
span.setAttribute("user.id", "user-12345");
span.setAttribute("feed.size", 20);
span.setAttribute("source.type", "realtime");
span.setAttribute("cache.hit", false);

Metrics Design

RED Method Metrics:

Metric Name	Type	Labels	Description
news_feed_requests_total	Counter	operation, status, user_type	Total requests processed
news_feed_errors_total	Counter	operation, error_type	Error counts by operation
news_feed_duration_seconds	Histogram	operation, user_type	Request processing time
news_feed_cache_hit_ratio	Gauge	cache_type	Cache hit rate percentage
news_feed_content_count	Histogram	content_type, source	Distribution of content types

Example Histogram Bucket Configuration:

// For duration histogram (50ms, 100ms, 250ms, 500ms, 1s, 2s, 5s)
Histogram.Timer durationTimer = Histogram.timer("news_feed_duration_seconds")
    .withLabels("operation", "fetchNewsFeed")
    .build();

Health Check Endpoint Design

GET /health
GET /ready
GET /live
GET /startup

Response Format:

{
  "status": "healthy|degraded|unhealthy",
  "checks": {
    "database": {
      "status": "healthy",
      "latency_ms": 12
    },
    "cache": {
      "status": "healthy",
      "latency_ms": 5
    },
    "message_queue": {
      "status": "healthy",
      "latency_ms": 8
    }
  },
  "timestamp": "2023-10-27T14:30:45.123Z"
}

Health Check Logic:

• /live - Basic service availability (memory, CPU)
• /ready - Service ready for traffic (DB, cache connections)
• /health - Full health including external dependencies
• /startup - Service startup validation (config, migrations)

Alerting Thresholds

Critical Alerts:

• news_feed_errors_total > 100 errors/minute → Alert
• news_feed_duration_seconds p99 > 5s → Alert
• news_feed_cache_hit_ratio < 60% → Alert
• Database connection pool utilization > 95% → Alert

Warning Alerts:

• news_feed_errors_total > 10 errors/minute → Warning
• news_feed_duration_seconds p99 > 2s → Warning
• Cache hit ratio < 70% → Warning

On-call Escalation:

1. First alert → Slack notification to dev team
2. Second alert within 5 minutes → On-call engineer
3. Third alert within 10 minutes → Manager escalation

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• Content freshness vs. performance trade-off: How much delay is acceptable between content creation and appearing in user feeds? This directly impacts cache invalidation strategy and real-time processing requirements.
• Personalization intensity: Should the feed be highly personalized (ML-driven) or broadly relevant? This affects the complexity of recommendation algorithms and the data requirements for user preference modeling.
• Feed refresh frequency: How often should feeds be recomputed? Real-time updates vs. batch processing creates different architectural and scalability considerations.
• Content type prioritization: How to balance between different content types (articles, videos, images) in the feed? This impacts ranking algorithms and UI design.

Known trade-offs and justifications

• Consistency vs. availability: Using eventual consistency for feed updates provides better availability but may cause temporary inconsistencies between users and their feeds. This is acceptable for news feeds where slight delays are tolerable.
• Read performance vs. write complexity: Optimizing for fast feed reads by pre-computing and caching feeds adds complexity to the write path (content creation, user follow/unfollow operations) but dramatically improves read performance for users.
• Storage vs. compute trade-off: Storing pre-computed feeds per user increases storage requirements but reduces CPU usage during feed retrieval, which is crucial for handling millions of concurrent users.
• Cache invalidation vs. cache freshness: Using time-based TTLs for feed caches simplifies implementation but may serve stale content longer than necessary, requiring careful balance of TTL values.

Risks and mitigation strategies

• Scalability risk: Feed generation for millions of users with billions of content items could overwhelm the system. Mitigation includes implementing sharding strategies, content partitioning by topic/region, and using distributed processing frameworks like Apache Spark or Kafka Streams.
• Cold start problem: New users with no followings or preferences will receive generic content. Mitigation includes implementing default content strategies, onboarding flows that capture user interests, and fallback algorithms for new users.
• Content drift: Feed content may drift away from user interests over time due to poor ranking algorithms or lack of feedback mechanisms. Mitigation includes implementing A/B testing for ranking algorithms, user feedback collection, and continuous model retraining.
• Cache stampede: Simultaneous feed refreshes for popular content could overwhelm backend services. Mitigation includes implementing probabilistic early cache expiry, distributed locks for feed generation, and staggered refresh schedules.

Technical debt acknowledged

• Feed recomputation overhead: Current design assumes feed recomputation on every content creation or user action, which could become a bottleneck at scale. Future improvements should consider incremental recomputation and background processing.
• Limited content diversity: The current model may not adequately represent content diversity, potentially leading to echo chamber effects. This requires ongoing monitoring and algorithmic adjustments.
• No content quality filtering: The system does not yet implement content quality or spam detection mechanisms, which could lead to poor user experience if low-quality content appears in feeds.
• Limited real-time personalization: The current design favors batch processing for feed generation, which may not support truly real-time personalization needs. This is acceptable for most news use cases but could be a limitation for highly interactive content platforms.
• No cross-platform feed synchronization: Feed content and preferences are not synchronized across different platforms (mobile, web), which could lead to inconsistent user experiences. This requires future integration with a centralized user preference store.