API Rate Limiter Design and Implementation

This document presents a production-ready rate limiting system design that handles API request throttling with multiple algorithm implementations and distributed consistency considerations.

1. Problem Statement & Requirements

Clarifying questions

1. What scale and throughput requirements are expected for the rate limiter?

   • Is it designed for a small microservice with 1000 requests/sec or a high-volume platform like a public API gateway serving millions of requests?

2. What consistency model is required for rate limits?

   • Should rate limits be strictly enforced in real-time (strong consistency) or can some tolerance exist (eventual consistency)?

3. What types of entities are being rate-limited?

   • Is it per-user, per-API key, per-IP address, or a combination? What are the primary entities and their relationships?

4. What is the expected read/write ratio for rate limit data?

   • Is it primarily checking limits (reads) or updating them (writes)? This affects caching and concurrency strategy.

5. What latency sensitivity is expected?

   • Should rate limit checks be sub-10ms to avoid impacting user experience?

6. What authentication model is in place?

   • Is the system using JWTs, API keys, OAuth2, or other mechanisms? How does rate limiting integrate with auth?

7. Are there any external integrations or dependencies for rate limit enforcement?

   • Does the system need to integrate with external services like Redis, Kafka, or a centralized auth service?

Functional requirements

• Rate limit enforcement per user/api key/IP address

  • Enforce limits based on configurable rules (e.g., 1000 requests/hour per user).

• Dynamic configuration of rate limits

  • Allow updating rate limit policies without downtime.

• Rate limit status reporting

  • Return current usage and remaining quota in response headers or body.

• Quota reset handling

  • Automatically reset quotas at defined intervals (e.g., hourly, daily).

• Support for burst limits

  • Allow short bursts of traffic within a defined window.

Non-functional requirements

• Latency targets

  • p50: <5ms, p95: <10ms, p99: <20ms for rate limit checks.

• Throughput

  • Support 100,000+ concurrent rate limit checks per second.

• Availability

  • 99.9% uptime with graceful degradation during outages.

• Scalability

  • Support horizontal scaling across multiple instances.

Constraints and assumptions

• Assumption: Rate limits are enforced at the service gateway or API layer, not in each microservice.
• Constraint: Rate limit data must be shared across all instances of the service in a distributed environment.
• Assumption: The system will use Redis or similar for rate limit state storage.
• Constraint: Rate limits are enforced in real-time with minimal impact on request processing.

Out of scope

• Rate limit policy creation UI

  • The system does not include a dashboard or admin interface for managing policies.

• Rate limit policy inheritance

  • Policies do not support inheritance or hierarchical configuration.

• Rate limit tiering

  • Different rate limits per user role or subscription level are not supported.

• Rate limit analytics dashboard

  • Metrics and reporting for rate limit usage are not part of this design.

• Rate limit blocking or throttling with delay

  • The system only returns HTTP 429 responses and does not delay or queue requests.

• Rate limit enforcement for internal services

  • Internal microservice-to-microservice communication is not considered.

2. API Design

API Style Selection

For rate limiting, I'll use REST API style because:

• It's the most widely adopted and understood pattern
• Provides clear HTTP semantics for rate limit headers and status codes
• Compatible with existing infrastructure and monitoring tools
• Supports standard HTTP methods (GET, POST, etc.) that are naturally rate-limited

Endpoint Definitions

Method	Path	Description	Auth
GET	/api/v1/rate-limits	Retrieve current rate limit configuration for user	JWT
POST	/api/v1/rate-limits	Create new rate limit configuration	JWT
PUT	/api/v1/rate-limits/{id}	Update existing rate limit configuration	JWT
DELETE	/api/v1/rate-limits/{id}	Delete rate limit configuration	JWT
GET	/api/v1/rate-limits/{id}/stats	Get usage statistics for a specific rate limit	JWT
POST	/api/v1/rate-limits/check	Check if request would exceed rate limits	None
GET	/api/v1/rate-limits/global	Get global rate limit configuration	JWT
PUT	/api/v1/rate-limits/global	Update global rate limit configuration	JWT

Request and Response Schemas

RateLimitRequest

{
  "id": "string",
  "userId": "string",
  "resource": "string",
  "limit": 100,
  "windowSeconds": 3600,
  "actions": ["GET", "POST"],
  "createdAt": "2023-01-01T00:00:00Z",
  "updatedAt": "2023-01-01T00:00:00Z"
}

RateLimitResponse

{
  "id": "string",
  "userId": "string",
  "resource": "string",
  "limit": 100,
  "windowSeconds": 3600,
  "actions": ["GET", "POST"],
  "createdAt": "2023-01-01T00:00:00Z",
  "updatedAt": "2023-01-01T00:00:00Z"
}

RateLimitCheckRequest

{
  "userId": "string",
  "resource": "string",
  "action": "GET"
}

RateLimitCheckResponse

{
  "allowed": true,
  "remaining": 99,
  "resetTime": "2023-01-01T00:00:00Z",
  "limit": 100
}

GlobalRateLimitConfig

{
  "limit": 1000,
  "windowSeconds": 3600,
  "actions": ["ALL"],
  "createdAt": "2023-01-01T00:00:00Z",
  "updatedAt": "2023-01-01T00:00:00Z"
}

HTTP Status Codes and Error Response Format

Standard HTTP Status Codes:

• 200 OK - Successful GET, PUT, DELETE
• 201 Created - Successful POST
• 400 Bad Request - Invalid request body or parameters
• 401 Unauthorized - Missing or invalid authentication
• 403 Forbidden - Insufficient permissions
• 429 Too Many Requests - Rate limit exceeded (with Retry-After header)
• 500 Internal Server Error - Server-side errors

Standardized Error Response Format:

{
  "error": {
    "code": "RATE_LIMIT_EXCEEDED",
    "message": "Rate limit exceeded for resource: api_call",
    "details": {
      "resource": "api_call",
      "limit": 100,
      "remaining": 0,
      "resetTime": "2023-01-01T00:00:00Z"
    },
    "timestamp": "2023-01-01T00:00:00Z",
    "requestId": "uuid"
  }
}

API Versioning Strategy

Version v1 will be used with the following approach:

• URL-based versioning (/api/v1/...)
• Semantic versioning with backward compatibility guarantee
• Version negotiation through Accept header (future extensibility)
• Migration path documented in changelog

Pagination Design

Cursor-based pagination is preferred over offset-based because:

• More efficient for large datasets
• Consistent results even when data changes during pagination
• Better performance with large offsets

For /api/v1/rate-limits:

GET /api/v1/rate-limits?limit=20&cursor=abc123

Authentication and Authorization

Authentication:

• JWT tokens for API authentication
• OAuth2 integration with authorization server (future extensibility)
• API key support for service-to-service authentication

Authorization:

• JWT claims validation (userId, roles)
• RBAC model where users can only access their own rate limits
• Admin role for global configuration access

Rate Limiting Per Endpoint

Endpoint	Limit	Window	Description
/api/v1/rate-limits/check	1000	60 seconds	Check endpoint (high volume)
/api/v1/rate-limits	100	60 seconds	CRUD operations
/api/v1/rate-limits/global	50	60 seconds	Global config
All other endpoints	1000	3600 seconds	Default per user

Idempotency Design

Idempotency Key Strategy:

• X-Idempotency-Key header for POST/PUT operations
• Key format: {userId}:{resource}:{timestamp}:{random}
• Store in Redis with TTL of 5 minutes
• If key exists and operation is same, return cached result
• Prevents duplicate processing in case of retries

Example Request Flow with Rate Limit Headers

POST /api/v1/rate-limits HTTP/1.1
Host: api.example.com
Authorization: Bearer <jwt>
Content-Type: application/json
X-Idempotency-Key: user123:api_call:1672531200:abc123

{
  "userId": "user123",
  "resource": "api_call",
  "limit": 100,
  "windowSeconds": 3600
}

HTTP/1.1 201 Created
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 99
X-RateLimit-Reset: 1672534800
Content-Type: application/json

{
  "id": "rl_abc123",
  "userId": "user123",
  "resource": "api_call",
  "limit": 100,
  "windowSeconds": 3600
}

When rate limit is exceeded:

HTTP/1.1 429 Too Many Requests
Retry-After: 3600
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1672534800
Content-Type: application/json

{
  "error": {
    "code": "RATE_LIMIT_EXCEEDED",
    "message": "Rate limit exceeded for resource: api_call"
  }
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    RATE_LIMIT_CONFIG {
        string id PK
        string resource_type
        string resource_id
        integer max_requests
        integer window_seconds
        timestamp created_at
        timestamp updated_at
        boolean is_active
    }

    RATE_LIMIT_RECORD {
        string id PK
        string config_id FK
        string user_id
        string client_ip
        integer request_count
        timestamp window_start
        timestamp window_end
        timestamp last_accessed
    }

    RATE_LIMIT_EVENT {
        string id PK
        string user_id
        string client_ip
        string resource_type
        string resource_id
        timestamp event_timestamp
        integer requests_made
        boolean is_blocked
        string reason
    }

    RATE_LIMIT_CONFIG ||--o{ RATE_LIMIT_RECORD : "has"
    RATE_LIMIT_CONFIG ||--o{ RATE_LIMIT_EVENT : "generates"

Table / Collection Schema Definitions

RateLimitConfig

CREATE TABLE rate_limit_configs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    resource_type VARCHAR(50) NOT NULL,
    resource_id VARCHAR(255),
    max_requests INTEGER NOT NULL CHECK (max_requests > 0),
    window_seconds INTEGER NOT NULL CHECK (window_seconds > 0),
    is_active BOOLEAN DEFAULT TRUE,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    
    -- Composite index for efficient lookups by resource type and ID
    INDEX idx_resource_type_id (resource_type, resource_id),
    -- Index for active configs only
    INDEX idx_active_resource (is_active, resource_type, resource_id)
);

RateLimitRecord

CREATE TABLE rate_limit_records (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    config_id UUID NOT NULL REFERENCES rate_limit_configs(id) ON DELETE CASCADE,
    user_id VARCHAR(255),
    client_ip INET NOT NULL,
    request_count INTEGER DEFAULT 0,
    window_start TIMESTAMP WITH TIME ZONE NOT NULL,
    window_end TIMESTAMP WITH TIME ZONE NOT NULL,
    last_accessed TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    
    -- Composite index for efficient window-based queries
    INDEX idx_config_window (config_id, window_start),
    INDEX idx_user_window (user_id, window_start),
    INDEX idx_ip_window (client_ip, window_start),
    INDEX idx_config_user_window (config_id, user_id, window_start),
    INDEX idx_last_accessed (last_accessed)
);

RateLimitEvent

CREATE TABLE rate_limit_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id VARCHAR(255),
    client_ip INET NOT NULL,
    resource_type VARCHAR(50) NOT NULL,
    resource_id VARCHAR(255),
    event_timestamp TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    requests_made INTEGER NOT NULL,
    is_blocked BOOLEAN DEFAULT FALSE,
    reason VARCHAR(255),
    
    -- Indexes for efficient querying by user, IP, and timestamp
    INDEX idx_user_timestamp (user_id, event_timestamp),
    INDEX idx_ip_timestamp (client_ip, event_timestamp),
    INDEX idx_resource_type_id (resource_type, resource_id),
    INDEX idx_event_timestamp (event_timestamp)
);

Primary Key Strategy

UUIDs are used as primary keys for all tables because:

1. Scalability: UUIDs allow distributed systems to generate IDs without coordination
2. Security: Prevents enumeration attacks by making IDs unpredictable
3. Partitioning: UUIDs can be used for sharding strategies when needed
4. No single point of failure: Unlike auto-increment, UUID generation doesn't require a centralized counter

Index Design Justification

1. RateLimitConfig:

   • idx_resource_type_id: Enables fast lookup of configs by resource type and ID combination
   • idx_active_resource: Efficient filtering for active rate limits only

2. RateLimitRecord:

   • idx_config_window: Fast window-based queries for rate limit checking
   • idx_user_window: Efficient user-specific rate limiting lookups
   • idx_ip_window: Fast IP-based rate limiting lookups
   • idx_config_user_window: Combined lookup for config + user rate limiting
   • idx_last_accessed: Efficient cleanup of stale records

3. RateLimitEvent:

   • idx_user_timestamp: Efficient user activity monitoring
   • idx_ip_timestamp: Fast IP-based monitoring
   • idx_resource_type_id: Resource-level event filtering
   • idx_event_timestamp: Time-based event queries

Data Partitioning / Sharding Strategy

• Shard by resource type and ID: Rate limit configs are partitioned by resource_type and resource_id to support per-resource rate limiting
• Time-based sharding: For RateLimitRecord, we can shard by window_start timestamp to support time-range queries
• Geographic sharding: For client IP-based rate limiting, we can shard by IP subnet ranges for better distribution

Soft Delete vs. Hard Delete Decision

Soft delete is used because:

1. Audit trail: Need to maintain historical rate limit events for compliance and debugging
2. Recovery capability: Configs might need to be restored after accidental deletion
3. Analytics: Historical data is valuable for rate limit tuning and anomaly detection

Audit Trail Design

All changes to rate_limit_configs are audited with:

• created_at, updated_at timestamps
• is_active flag changes are tracked in the system logs

Data Archival Strategy and Retention Policy

1. RateLimitEvent:

   • Retention: 90 days for compliance and analytics
   • Archival: Events older than 90 days are moved to cold storage (e.g., S3)
   • Purge: Events older than 1 year are permanently deleted

2. RateLimitRecord:

   • Retention: 30 days (window duration + some buffer)
   • Archival: Records that are older than 30 days are purged
   • Purge: Records older than 90 days are permanently deleted

3. RateLimitConfig:

   • Retention: Indefinite (configs are typically long-lived)
   • Archival: No archival needed as configs are not frequently accessed
   • Purge: Only soft-deleted records are purged after 6 months

4. Class Design & Service Layer

UML Class Diagram

classDiagram
    class RateLimiterService {
        -rateLimitRepository
        -cache
        -config
        +isAllowed(String userId, String endpoint, RateLimitConfig config)
        +updateRateLimitConfig(RateLimitConfig config)
        +getRateLimitStats(String userId, String endpoint)
    }

    class RateLimitConfig {
        -userId
        -endpoint
        -maxRequests
        -windowInSeconds
        -type
        +validate()
    }

    class RateLimitBucket {
        -userId
        -endpoint
        -tokens
        -lastRefillTime
        -maxTokens
        -refillRate
        +consume(int tokens)
        +refill()
        +getRemainingTokens()
    }

    class RateLimitRepository {
        +save(RateLimitBucket bucket)
        +findByUserIdAndEndpoint(String userId, String endpoint)
        +update(RateLimitBucket bucket)
    }

    class RateLimitStats {
        -userId
        -endpoint
        -requestsInWindow
        -windowStart
        -windowEnd
        -tokensRemaining
    }

    class TokenBucket {
        -capacity
        -tokens
        -refillRate
        -lastRefillTime
        +consume(int tokens)
        +refill()
    }

    class RateLimitException {
        -errorCode
        -message
        +getErrorCode()
        +getMessage()
    }

    RateLimiterService --> RateLimitRepository
    RateLimiterService --> TokenBucket
    RateLimiterService --> RateLimitConfig
    RateLimiterService --> RateLimitStats
    TokenBucket --> RateLimitBucket
    RateLimitBucket --> RateLimitConfig
    RateLimitBucket --> RateLimitStats

State Machine Diagram for Rate Limit Bucket

stateDiagram-v2
    [*] --> Active
    Active --> Blocked : Exceeds limit
    Active --> Active : Within limits
    Blocked --> Active : Window resets

Service Class Structure and Responsibilities

The RateLimiterService is the core component responsible for rate limiting logic. It handles:

• Checking if a request is allowed based on configured limits
• Managing rate limit configuration updates
• Tracking and reporting rate limit statistics
• Interacting with the data store for bucket persistence

Interface / Abstract Class Design

public interface RateLimitRepository {
    RateLimitBucket save(RateLimitBucket bucket);
    Optional<RateLimitBucket> findByUserIdAndEndpoint(String userId, String endpoint);
    RateLimitBucket update(RateLimitBucket bucket);
}

public interface RateLimitConfigValidator {
    void validate(RateLimitConfig config) throws RateLimitException;
}

Business Logic Encapsulation Patterns

Strategy Pattern Implementation: We use the Strategy pattern for different rate limit algorithms (token bucket, fixed window, sliding window).

public interface RateLimitStrategy {
    boolean isAllowed(RateLimitBucket bucket, int tokensRequired);
}

public class TokenBucketStrategy implements RateLimitStrategy {
    @Override
    public boolean isAllowed(RateLimitBucket bucket, int tokensRequired) {
        return bucket.consume(tokensRequired);
    }
}

public class FixedWindowStrategy implements RateLimitStrategy {
    @Override
    public boolean isAllowed(RateLimitBucket bucket, int tokensRequired) {
        // Implementation for fixed window strategy
        return true;
    }
}

Critical-Path Method Implementation

public boolean isAllowed(String userId, String endpoint, RateLimitConfig config) {
    // Transaction boundary: All operations within this method should be atomic
    try {
        // Acquire distributed lock to prevent race conditions
        String lockKey = "rate_limit_lock:" + userId + ":" + endpoint;
        boolean acquired = distributedLockManager.acquire(lockKey, 5000); // 5s timeout
        if (!acquired) {
            throw new RateLimitException("Rate limit check in progress");
        }

        // Fetch or create bucket
        Optional<RateLimitBucket> existingBucket = rateLimitRepository.findByUserIdAndEndpoint(userId, endpoint);
        RateLimitBucket bucket = existingBucket.orElseGet(() -> {
            RateLimitBucket newBucket = new RateLimitBucket(userId, endpoint, config);
            return rateLimitRepository.save(newBucket);
        });

        // Refill bucket if needed
        bucket.refill();

        // Check if request is allowed
        boolean allowed = rateLimitStrategy.isAllowed(bucket, 1);

        // Update bucket in DB
        rateLimitRepository.update(bucket);

        // Publish event for monitoring and analytics
        if (!allowed) {
            eventPublisher.publish(new RateLimitExceededEvent(userId, endpoint));
        }

        return allowed;
    } catch (Exception e) {
        // Log error and rethrow
        logger.error("Rate limit check failed for user: {}, endpoint: {}", userId, endpoint, e);
        throw new RateLimitException("Rate limit check failed");
    } finally {
        // Release distributed lock
        distributedLockManager.release(lockKey);
    }
}

Transaction Boundary Design

A single rate limit check operation constitutes a transaction. This includes:

1. Acquiring the distributed lock
2. Fetching or creating the rate limit bucket
3. Refilling tokens if needed
4. Consuming tokens if allowed
5. Persisting the updated bucket state
6. Releasing the distributed lock

Service Interface Contracts

public interface RateLimiterService {
    boolean isAllowed(String userId, String endpoint, RateLimitConfig config);
    void updateRateLimitConfig(RateLimitConfig config);
    RateLimitStats getRateLimitStats(String userId, String endpoint);
}

Saga / Orchestration Pattern

For distributed rate limiting across microservices, we would implement a saga pattern where:

• Each service maintains its own local rate limit buckets
• A global coordinator manages cross-service rate limits using a shared database or distributed cache

Domain Object / Value Object Design

public class RateLimitConfig {
    private String userId;
    private String endpoint;
    private int maxRequests;
    private int windowInSeconds;
    private RateLimitType type;
    
    // Getters and setters
}

public class RateLimitBucket {
    private String userId;
    private String endpoint;
    private int tokens;
    private long lastRefillTime;
    private int maxTokens;
    private int refillRate;
    
    // Methods: consume(), refill(), getRemainingTokens()
}

public enum RateLimitType {
    TOKEN_BUCKET,
    FIXED_WINDOW,
    SLIDING_WINDOW
}

5. Concurrency & Thread Safety

Concurrency Model

The rate limiter service operates in a multi-threaded environment with concurrent request processing. Each incoming API request is handled by a separate thread from the application's thread pool (e.g., using Netty or Spring's async executor). The core rate limiting logic must be thread-safe to handle simultaneous requests from multiple clients.

Race Condition Prevention Strategy

Race conditions can occur when multiple threads attempt to update the same rate limit bucket concurrently. To prevent this:

• Atomic operations for counter updates
• Lock-free data structures where possible (e.g., AtomicLong, ConcurrentHashMap)
• Synchronized blocks or methods for critical sections requiring exclusive access
• Use of compare-and-swap (CAS) operations to ensure atomic updates

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected for rate limiting because:

• Rate limiting typically involves read-heavy operations with infrequent writes
• The cost of handling conflicts (retries) is lower than maintaining exclusive locks
• It allows better throughput under concurrent load
• Conflicts are rare since tokens are usually consumed in batches or at low frequency

Implementation uses AtomicLong for token counters and ReentrantReadWriteLock for managing access to shared rate limit configurations.

Deadlock Prevention Strategy

Deadlocks are prevented by:

• Using consistent lock ordering across all rate limit operations
• Implementing timeouts on lock acquisition (e.g., 50ms)
• Avoiding nested locks where possible
• Using StampedLock for better performance in read-heavy scenarios

Idempotency Implementation for Concurrent Duplicate Requests

For rate-limited requests that may arrive concurrently:

• Each request includes a unique idempotencyKey in the header
• The system maintains an in-memory cache (ConcurrentHashMap) of recently processed keys with TTL (e.g., 10 seconds)
• If a duplicate key is detected, the system returns the cached result immediately without reprocessing
• This ensures that even if multiple threads process the same request, only one actual rate limit check occurs

public class RateLimiter {
    private final Map<String, Long> idempotencyCache = new ConcurrentHashMap<>();
    private final ReadWriteLock lock = new ReentrantReadWriteLock();
    
    public boolean isRateLimited(String clientId, String apiKey) {
        // Check idempotency cache first
        Long lastRequestTime = idempotencyCache.get(apiKey);
        if (lastRequestTime != null && System.currentTimeMillis() - lastRequestTime < 10000) {
            return false; // Already processed within 10 seconds
        }
        
        // Proceed with rate limiting logic using read lock
        lock.readLock().lock();
        try {
            // Implementation details...
            return false; // Placeholder
        } finally {
            lock.readLock().unlock();
        }
    }
}

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant Client
    participant APIGateway
    participant RateLimiter
    participant Redis
    participant Service

    Client->>APIGateway: Request to /api/users/123
    APIGateway->>RateLimiter: Check rate limit for user ID 123
    RateLimiter->>Redis: Get current request count for key user:123
    Redis-->>RateLimiter: Current count: 49
    RateLimiter->>Redis: Increment request count for key user:123
    Redis-->>RateLimiter: New count: 50
    alt Rate limit not exceeded
        RateLimiter->>Service: Forward request to service
        Service-->>RateLimiter: Response
        RateLimiter-->>APIGateway: Forward response
        APIGateway-->>Client: Return 200 OK
    else Rate limit exceeded
        RateLimiter-->>APIGateway: Return 429 Too Many Requests
        APIGateway-->>Client: Return 429 Too Many Requests
    end

sequenceDiagram
    participant Client
    participant APIGateway
    participant RateLimiter
    participant Redis

    Client->>APIGateway: Request to /api/users/123
    APIGateway->>RateLimiter: Check rate limit for user ID 123
    RateLimiter->>Redis: Get current request count for key user:123
    Redis-->>RateLimiter: Current count: 50 (exceeds limit)
    RateLimiter-->>APIGateway: Return 429 Too Many Requests
    APIGateway-->>Client: Return 429 Too Many Requests

sequenceDiagram
    participant Client
    participant APIGateway
    participant RateLimiter
    participant Redis
    participant EventPublisher

    Client->>APIGateway: Request to /api/users/123
    APIGateway->>RateLimiter: Check rate limit for user ID 123
    RateLimiter->>Redis: Get current request count for key user:123
    Redis-->>RateLimiter: Current count: 49
    RateLimiter->>Redis: Increment request count for key user:123
    Redis-->>RateLimiter: New count: 50
    alt Rate limit not exceeded
        RateLimiter->>EventPublisher: Publish rate limit event for user 123
        EventPublisher-->>RateLimiter: Event published successfully
        RateLimiter->>Service: Forward request to service
        Service-->>RateLimiter: Response
        RateLimiter-->>APIGateway: Forward response
        APIGateway-->>Client: Return 200 OK
    else Rate limit exceeded
        RateLimiter->>EventPublisher: Publish rate limit exceeded event for user 123
        EventPublisher-->>RateLimiter: Event published successfully
        RateLimiter-->>APIGateway: Return 429 Too Many Requests
        APIGateway-->>Client: Return 429 Too Many Requests
    end

7. Caching Design

Cache Strategy Selection

For rate limiting, we'll use cache-aside pattern with token bucket algorithm stored in Redis. This approach provides the best balance between performance and consistency for high-throughput systems.

Cache Key Design

Cache keys will follow this namespace structure:

rate_limit:{resource_type}:{resource_id}:{time_window}

Example keys:

• rate_limit:api_key:abc123:1m
• rate_limit:user:12345:1h
• rate_limit:ip:192.168.1.1:5m

TTL Configuration

Data Type	TTL	Reason
Token bucket state	1 hour	Bucket resets daily, but we want to keep recent usage
Rate limit configuration	1 day	Configuration changes are infrequent
Quota tracking	30 days	For analytics and historical tracking

Cache Eviction Policy

We'll use TTL-based eviction with a secondary LRU-like strategy for memory-constrained environments. Redis will handle automatic eviction based on TTL.

Cache Stampede Prevention

To prevent cache stampedes:

1. Probabilistic early expiry: Set TTL to 90% of actual expiration time
2. Distributed locks: Use Redis SETNX with TTL for atomic bucket updates
3. Cache warming: Pre-populate popular rate limit buckets during low-traffic periods

Distributed Cache Architecture

We'll use Redis Cluster with:

• 3 master nodes for high availability
• 3 replica nodes per master for read scaling
• Automatic failover with Redis Sentinel
• Sharding based on cache key prefix to distribute load

Cache Invalidation Strategy

We'll implement event-driven invalidation:

1. When rate limit configuration changes, invalidate all related buckets
2. When user activity occurs (e.g., API call), update token bucket state
3. When quotas reset, expire buckets with TTLs

Cache Implementation Example (Java)

@Component
public class RateLimitCache {
    private final RedisTemplate<String, String> redisTemplate;
    private final ObjectMapper objectMapper;
    
    public RateLimitResult checkRateLimit(String resourceType, String resourceId, 
                                         RateLimitConfig config) {
        String cacheKey = buildCacheKey(resourceType, resourceId, config.getTimeWindow());
        
        // Try to get from cache first
        String cachedBucket = redisTemplate.opsForValue().get(cacheKey);
        
        if (cachedBucket != null) {
            try {
                TokenBucket bucket = objectMapper.readValue(cachedBucket, TokenBucket.class);
                return new RateLimitResult(true, bucket);
            } catch (Exception e) {
                // Fall through to recalculate
            }
        }
        
        // Recalculate bucket state (this will be a distributed lock)
        return calculateAndCacheBucket(resourceType, resourceId, config, cacheKey);
    }
    
    private RateLimitResult calculateAndCacheBucket(String resourceType, String resourceId,
                                                   RateLimitConfig config, String cacheKey) {
        // Use Redis distributed lock to prevent race conditions
        String lockKey = "rate_limit_lock:" + cacheKey;
        String lockValue = UUID.randomUUID().toString();
        
        try {
            if (redisTemplate.opsForValue().setIfAbsent(lockKey, lockValue, 
                                                      Duration.ofSeconds(1), 
                                                      TimeUnit.SECONDS)) {
                // Critical section - calculate bucket state
                TokenBucket bucket = calculateTokenBucket(resourceType, resourceId, config);
                
                // Cache with TTL slightly longer than window to prevent stampede
                String bucketJson = objectMapper.writeValueAsString(bucket);
                redisTemplate.opsForValue().set(cacheKey, bucketJson, 
                                               config.getTimeWindow() * 2, TimeUnit.SECONDS);
                
                return new RateLimitResult(false, bucket);
            } else {
                // Another thread is calculating, wait and retry
                Thread.sleep(50);
                return checkRateLimit(resourceType, resourceId, config);
            }
        } catch (Exception e) {
            throw new RateLimitingException("Failed to calculate rate limit", e);
        } finally {
            // Release lock
            String script = "if redis.call('GET', KEYS[1]) == ARGV[1] then " +
                           "return redis.call('DEL', KEYS[1]) else return 0 end";
            redisTemplate.execute(new DefaultRedisScript<>(script, Long.class), 
                                Collections.singletonList(lockKey), lockValue);
        }
    }
    
    private String buildCacheKey(String resourceType, String resourceId, int timeWindow) {
        return String.format("rate_limit:%s:%s:%dm", resourceType, resourceId, timeWindow);
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and database operations. Each entity has its own repository interface and implementation.

public interface RateLimitRepository {
    RateLimitEntity findByKey(String key);
    void save(RateLimitEntity rateLimit);
    void update(RateLimitEntity rateLimit);
    void delete(String key);
}

@Repository
public class RateLimitRepositoryImpl implements RateLimitRepository {
    private final JdbcTemplate jdbcTemplate;
    private final ObjectMapper objectMapper;

    public RateLimitRepositoryImpl(JdbcTemplate jdbcTemplate, ObjectMapper objectMapper) {
        this.jdbcTemplate = jdbcTemplate;
        this.objectMapper = objectMapper;
    }

    @Override
    public RateLimitEntity findByKey(String key) {
        String sql = "SELECT * FROM rate_limits WHERE key = ?";
        try {
            return jdbcTemplate.queryForObject(sql, new Object[]{key}, new RateLimitRowMapper());
        } catch (EmptyResultDataAccessException e) {
            return null;
        }
    }

    @Override
    public void save(RateLimitEntity rateLimit) {
        String sql = "INSERT INTO rate_limits (key, limit_config, current_count, reset_time, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?)";
        jdbcTemplate.update(sql, 
            rateLimit.getKey(),
            objectMapper.writeValueAsString(rateLimit.getLimitConfig()),
            rateLimit.getCurrentCount(),
            rateLimit.getResetTime(),
            rateLimit.getCreatedAt(),
            rateLimit.getUpdatedAt()
        );
    }

    @Override
    public void update(RateLimitEntity rateLimit) {
        String sql = "UPDATE rate_limits SET limit_config = ?, current_count = ?, reset_time = ?, updated_at = ? WHERE key = ?";
        jdbcTemplate.update(sql,
            objectMapper.writeValueAsString(rateLimit.getLimitConfig()),
            rateLimit.getCurrentCount(),
            rateLimit.getResetTime(),
            rateLimit.getUpdatedAt(),
            rateLimit.getKey()
        );
    }

    @Override
    public void delete(String key) {
        String sql = "DELETE FROM rate_limits WHERE key = ?";
        jdbcTemplate.update(sql, key);
    }
}

ORM vs. Raw Query Decisions

For rate limiting, raw SQL queries are preferred over ORM for the following reasons:

1. Performance: Raw queries provide better performance for simple CRUD operations
2. Control: Direct control over query execution and optimization
3. Simplicity: Avoids ORM complexity for straightforward data access patterns

Read vs. Write Separation

The rate limiter uses a single database for both reads and writes since:

1. Low write volume: Rate limit updates are infrequent (per request)
2. Consistency requirements: Real-time rate limit checking requires strong consistency
3. Simplicity: Single database reduces operational complexity

Connection Pool Configuration

spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      validation-timeout: 5000

N+1 Query Prevention Strategy

Since rate limit data is accessed per request, we implement:

1. Batch operations: Fetch multiple rate limits in single queries when needed
2. Caching layer: Reduce database hits through caching
3. Pre-fetching: Load rate limits in batches during bulk operations

Batch Operation Design

public class BatchRateLimitUpdate {
    public void updateMultiple(List<RateLimitEntity> rateLimits) {
        String sql = "UPDATE rate_limits SET current_count = ?, reset_time = ?, updated_at = ? WHERE key = ?";
        
        jdbcTemplate.batchUpdate(sql, new BatchPreparedStatementSetter() {
            @Override
            public void setValues(PreparedStatement ps, int i) throws SQLException {
                RateLimitEntity rateLimit = rateLimits.get(i);
                ps.setInt(1, rateLimit.getCurrentCount());
                ps.setTimestamp(2, new Timestamp(rateLimit.getResetTime().getTime()));
                ps.setTimestamp(3, new Timestamp(System.currentTimeMillis()));
                ps.setString(4, rateLimit.getKey());
            }
            
            @Override
            public int getBatchSize() {
                return rateLimits.size();
            }
        });
    }
}

Data Access Layer Interface

public interface RateLimitDAO {
    Optional<RateLimitEntity> getRateLimit(String key);
    void saveRateLimit(RateLimitEntity rateLimit);
    void updateRateLimit(RateLimitEntity rateLimit);
    void deleteRateLimit(String key);
    List<RateLimitEntity> getRateLimitsByKeys(List<String> keys);
    void batchUpdateRateLimits(List<RateLimitEntity> rateLimits);
}

Transaction Management

@Service
@Transactional
public class RateLimitService {
    private final RateLimitDAO rateLimitDAO;
    
    public boolean consumeRateLimit(String key, int maxRequests, long windowSeconds) {
        RateLimitEntity rateLimit = rateLimitDAO.getRateLimit(key)
            .orElseGet(() -> createNewRateLimit(key, maxRequests, windowSeconds));
            
        if (rateLimit.getCurrentCount() >= maxRequests) {
            return false;
        }
        
        rateLimit.incrementCurrentCount();
        rateLimitDAO.updateRateLimit(rateLimit);
        return true;
    }
    
    private RateLimitEntity createNewRateLimit(String key, int maxRequests, long windowSeconds) {
        Date now = new Date();
        Date resetTime = new Date(now.getTime() + (windowSeconds * 1000));
        
        RateLimitEntity rateLimit = new RateLimitEntity();
        rateLimit.setKey(key);
        rateLimit.setLimitConfig(new RateLimitConfig(maxRequests, windowSeconds));
        rateLimit.setCurrentCount(1);
        rateLimit.setResetTime(resetTime);
        rateLimit.setCreatedAt(now);
        rateLimit.setUpdatedAt(now);
        
        rateLimitDAO.saveRateLimit(rateLimit);
        return rateLimit;
    }
}

Data Access Layer Configuration

@Configuration
@EnableTransactionManagement
public class RateLimitDataConfig {
    
    @Bean
    public JdbcTemplate jdbcTemplate(DataSource dataSource) {
        return new JdbcTemplate(dataSource);
    }
    
    @Bean
    public PlatformTransactionManager transactionManager(DataSource dataSource) {
        return new DataSourceTransactionManager(dataSource);
    }
}

Rate Limit Entity Mapping

public class RateLimitEntity {
    private String key;
    private RateLimitConfig limitConfig;
    private int currentCount;
    private Date resetTime;
    private Date createdAt;
    private Date updatedAt;
    
    // Getters and setters
}

public class RateLimitRowMapper implements RowMapper<RateLimitEntity> {
    private final ObjectMapper objectMapper;
    
    public RateLimitRowMapper(ObjectMapper objectMapper) {
        this.objectMapper = objectMapper;
    }
    
    @Override
    public RateLimitEntity mapRow(ResultSet rs, int rowNum) throws SQLException {
        RateLimitEntity entity = new RateLimitEntity();
        entity.setKey(rs.getString("key"));
        entity.setLimitConfig(objectMapper.readValue(rs.getString("limit_config"), RateLimitConfig.class));
        entity.setCurrentCount(rs.getInt("current_count"));
        entity.setResetTime(rs.getTimestamp("reset_time"));
        entity.setCreatedAt(rs.getTimestamp("created_at"));
        entity.setUpdatedAt(rs.getTimestamp("updated_at"));
        return entity;
    }
}

Performance Considerations

1. Query Optimization: All queries use indexed columns for fast lookup
2. Connection Pooling: Efficient resource utilization with proper sizing
3. Batch Operations: Reduce database round trips for bulk operations
4. Caching Integration: Reduce database load through caching layer
5. Indexing Strategy: Proper indexing on key column for fast lookups

Error Handling in Data Access Layer

public class RateLimitDAOImpl implements RateLimitDAO {
    private final JdbcTemplate jdbcTemplate;
    
    @Override
    public Optional<RateLimitEntity> getRateLimit(String key) {
        try {
            return Optional.ofNullable(jdbcTemplate.queryForObject(
                "SELECT * FROM rate_limits WHERE key = ?", 
                new Object[]{key}, 
                new RateLimitRowMapper()
            ));
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        } catch (DataAccessException e) {
            throw new RateLimitStorageException("Failed to retrieve rate limit for key: " + key, e);
        }
    }
}

The repository layer is designed for simplicity and performance while maintaining proper error handling and transaction boundaries. It provides a clean abstraction over the underlying data store with appropriate batching and connection management for high-throughput rate limiting operations.

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature — what triggers async vs. what stays synchronous, and why

Rate limiting is primarily a real-time system requirement that must respond to each API request within strict latency bounds. Therefore, the core rate limiting logic should remain synchronous and in-process to ensure low-latency response times. However, asynchronous processing is used for:

1. Rate limit bucket replenishment - The token bucket algorithm requires periodic updates to the rate limit state, which can be handled asynchronously to avoid blocking the main request path.
2. Analytics and reporting - Aggregating rate limit violations, quota usage statistics, and user behavior analytics can be done asynchronously to avoid impacting API response times.
3. Alerting and notifications - When a user approaches or exceeds their rate limits, asynchronous alerts can be sent to administrators or users via email/SMS without blocking the request.

Message schema design with a sample payload (Avro / Protobuf / JSON Schema — with versioning strategy)

{
  "type": "object",
  "title": "RateLimitEvent",
  "version": "1.0",
  "properties": {
    "eventId": {
      "type": "string",
      "description": "Unique identifier for the event"
    },
    "userId": {
      "type": "string",
      "description": "User identifier"
    },
    "apiEndpoint": {
      "type": "string",
      "description": "API endpoint that was accessed"
    },
    "requestTimestamp": {
      "type": "string",
      "format": "date-time",
      "description": "Timestamp when the request was made"
    },
    "rateLimitBucketKey": {
      "type": "string",
      "description": "Key identifying the rate limit bucket"
    },
    "requestCount": {
      "type": "integer",
      "description": "Number of requests made in the current window"
    },
    "limit": {
      "type": "integer",
      "description": "Rate limit for the bucket"
    },
    "remainingTokens": {
      "type": "integer",
      "description": "Remaining tokens in the bucket"
    },
    "violationType": {
      "type": "string",
      "enum": ["EXCEEDED", "NEAR_LIMIT", "NORMAL"],
      "description": "Type of rate limit violation"
    }
  },
  "required": ["eventId", "userId", "apiEndpoint", "requestTimestamp", "rateLimitBucketKey"]
}

Dead letter queue strategy and poison message handling

Poison messages in rate limiting context typically occur when:

1. Invalid or malformed rate limit event payloads
2. Consistently failing to process events due to system issues

For dead letter queue strategy:

• Messages that fail to be processed after 3 retries are moved to a DLQ
• The DLQ is monitored by an alerting system to notify operations teams
• Failed messages are logged with full context for debugging and potential manual reprocessing

At-least-once vs. exactly-once delivery — decision with justification for this feature

At-least-once delivery is chosen because:

1. Rate limit violations are idempotent - If a rate limit event is processed twice, it doesn't change the outcome for the user
2. Simpler implementation - Avoids complex state management and coordination required for exactly-once semantics
3. Performance considerations - Reduces the overhead of maintaining transactional state for event processing

Retry policy for consumers (max attempts, backoff formula, non-retryable error classification)

public class RateLimitEventConsumer {
    private static final int MAX_RETRIES = 3;
    private static final long INITIAL_BACKOFF_MS = 1000L;
    private static final double BACKOFF_MULTIPLIER = 2.0;
    private static final double JITTER_RANGE = 0.1;

    public void processRateLimitEvent(RateLimitEvent event) {
        int attempts = 0;
        while (attempts < MAX_RETRIES) {
            try {
                // Process the event
                processEvent(event);
                return;
            } catch (NonRetryableException e) {
                // Log and do not retry for non-retryable errors
                log.error("Non-retryable error processing rate limit event: {}", event, e);
                break;
            } catch (Exception e) {
                attempts++;
                if (attempts >= MAX_RETRIES) {
                    moveMessageToDLQ(event, e);
                    break;
                }
                
                // Calculate exponential backoff with jitter
                long delay = (long) (INITIAL_BACKOFF_MS * Math.pow(BACKOFF_MULTIPLIER, attempts - 1));
                delay = (long) (delay * (1 + JITTER_RANGE * (Math.random() - 0.5)));
                
                try {
                    Thread.sleep(delay);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new RuntimeException("Interrupted during retry delay", ie);
                }
            }
        }
    }
}

Outbox pattern for reliable event publishing — include or explicitly justify omission

Omission justified because:

1. Rate limit events are not critical for data consistency - They're primarily used for analytics and alerts, not core business logic
2. Simpler architecture - Avoids the complexity of maintaining an outbox table and managing transactional consistency
3. Eventual consistency is acceptable - The system can tolerate some lost events in favor of better performance and simpler implementation

Sequence diagram for all async flows (Mermaid)

sequenceDiagram
    participant Client
    participant APIGateway
    participant RateLimiter
    participant AsyncProcessor
    participant EventPublisher
    participant DLQ

    Client->>APIGateway: API Request
    APIGateway->>RateLimiter: Check Rate Limit
    RateLimiter-->>APIGateway: Allow/Reject
    APIGateway-->>Client: Response
    
    Note over RateLimiter,AsyncProcessor: Async Bucket Replenishment
    loop Every 1s
        RateLimiter->>AsyncProcessor: Update Bucket State
        AsyncProcessor->>EventPublisher: Publish RateLimitEvent
        EventPublisher->>AsyncProcessor: Acknowledge
    end
    
    Note over RateLimiter,AsyncProcessor: Async Analytics
    loop Every 5m
        RateLimiter->>AsyncProcessor: Aggregate Usage Stats
        AsyncProcessor->>EventPublisher: Publish AnalyticsEvent
    end
    
    Note over EventPublisher,DLQ: Error Handling
    EventPublisher->>AsyncProcessor: Retry Failed Events
    AsyncProcessor->>DLQ: Move to DLQ after max retries

10. Error Handling Design

Error Classification Hierarchy

The rate limiter system follows a three-tier error classification:

1. Validation Errors - Invalid input parameters (e.g., invalid API key, malformed rate limit configuration)
2. Infrastructure Errors - Cache or database connectivity issues (e.g., Redis timeout, DB connection pool exhausted)
3. Business Logic Errors - Rate limit exceeded or access denied scenarios

public enum ErrorType {
    VALIDATION_ERROR,
    INFRASTRUCTURE_ERROR,
    BUSINESS_LOGIC_ERROR
}

Error Propagation Strategy

Errors are handled at the service boundary and propagated upward through well-defined exceptions. Validation errors are immediately thrown to prevent downstream processing, while infrastructure errors may trigger retry mechanisms or fallback behavior.

Retry Strategy

Retry logic is implemented with exponential backoff and jitter to prevent thundering herd:

public class RetryConfig {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 100;
    private static final double JITTER_FACTOR = 0.2;
    
    public static long calculateDelay(int attempt) {
        long delay = (long) (BASE_DELAY_MS * Math.pow(2, attempt));
        long jitter = (long) (delay * JITTER_FACTOR * (Math.random() - 0.5));
        return Math.min(delay + jitter, 5000); // Cap at 5 seconds
    }
}

Circuit Breaker Design

For external dependencies like Redis or database, a circuit breaker pattern prevents cascading failures:

public class CircuitBreaker {
    private final int failureThreshold = 5;
    private final long timeout = 60000; // 1 minute
    private final int resetTimeout = 30000; // 30 seconds
    
    private int failureCount = 0;
    private long lastFailureTime = 0;
    private CircuitState state = CircuitState.CLOSED;
    
    public boolean canExecute() {
        if (state == CircuitState.OPEN) {
            if (System.currentTimeMillis() - lastFailureTime > resetTimeout) {
                state = CircuitState.HALF_OPEN;
            } else {
                return false;
            }
        }
        
        return state != CircuitState.OPEN;
    }
    
    public void recordSuccess() {
        if (state == CircuitState.HALF_OPEN) {
            state = CircuitState.CLOSED;
        }
        failureCount = 0;
    }
    
    public void recordFailure() {
        if (state == CircuitState.CLOSED) {
            failureCount++;
            lastFailureTime = System.currentTimeMillis();
            
            if (failureCount >= failureThreshold) {
                state = CircuitState.OPEN;
            }
        } else if (state == CircuitState.HALF_OPEN) {
            state = CircuitState.OPEN;
        }
    }
    
    enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }
}

Fallback Behavior Design

When rate limiting infrastructure fails, the system can fall back to a default behavior:

public class RateLimitFallback {
    public boolean allowRequest(String apiKey, String endpoint) {
        // Default behavior - allow all requests with generous limits
        return true;
    }
}

Error Logging Format

Structured logging includes trace context and error metadata:

{
  "timestamp": "2023-10-15T14:30:45.123Z",
  "trace_id": "abc123def456",
  "span_id": "xyz789uvw012",
  "user_id": "user-12345",
  "error_code": "RATE_LIMIT_EXCEEDED",
  "error_message": "API rate limit exceeded for key: api-key-123",
  "request_path": "/api/v1/users",
  "request_method": "POST",
  "error_type": "BUSINESS_LOGIC_ERROR",
  "retry_after": 30
}

Alerting Thresholds

Critical alerts are configured for:

• Rate limit exceeded: > 1000 occurrences per minute (warn) / > 5000 occurrences per minute (critical)
• Infrastructure failures: > 10 consecutive Redis connection timeouts (critical)
• Circuit breaker open: > 50% of requests failing (warn) / > 80% of requests failing (critical)

When thresholds are breached, alerts are sent to on-call engineers via Slack and email with detailed context including the affected API keys and endpoints.

11. Security Design

Input Validation and Sanitization Strategy

All API inputs must undergo strict validation using a combination of:

• Schema validation using JSON Schema or Protocol Buffers for structured data
• Regex pattern matching for specific formats (e.g., email, phone number)
• Length and range constraints to prevent buffer overflow attacks
• Type checking to ensure correct data types

For Java implementation, we use Jakarta Bean Validation (JSR-380) with Hibernate Validator:

public class RateLimitRequest {
    @NotNull
    @Size(min = 1, max = 64)
    private String apiKey;
    
    @NotNull
    @Min(1)
    @Max(10000)
    private Integer requestsPerWindow;
    
    @NotNull
    @Min(1000)
    @Max(3600000)
    private Long windowMs;
}

SQL Injection Prevention

All database interactions use parameterized queries with prepared statements. No dynamic query construction or string concatenation is allowed:

String sql = "SELECT * FROM rate_limits WHERE api_key = ?";
PreparedStatement stmt = connection.prepareStatement(sql);
stmt.setString(1, apiKey);
ResultSet rs = stmt.executeQuery();

Authentication Middleware Design

Authentication middleware intercepts all requests and validates:

• JWT tokens for OAuth2 flows
• API keys for key-based authentication
• Session tokens for cookie-based flows

@Component
public class AuthenticationMiddleware {
    private final JwtService jwtService;
    private final ApiKeyValidator apiKeyValidator;

    public void validateRequest(HttpServletRequest request) {
        String authHeader = request.getHeader("Authorization");
        if (authHeader != null && authHeader.startsWith("Bearer ")) {
            String token = authHeader.substring(7);
            jwtService.validateToken(token);
        } else {
            String apiKey = request.getHeader("X-API-Key");
            apiKeyValidator.validate(apiKey);
        }
    }
}

Authorization Enforcement

RBAC (Role-Based Access Control) is implemented at the service layer:

@PreAuthorize("hasRole('ADMIN') or hasRole('API_USER')")
public void updateRateLimit(String apiKey, RateLimitConfig config) {
    // Only users with appropriate roles can modify rate limits
}

Sensitive Data Encryption

All sensitive data at rest is encrypted using AES-256:

• API keys are stored as SHA-256 hashes with salt
• Rate limit configurations are encrypted using a master key

public class EncryptionService {
    private final Cipher cipher;
    
    public String encrypt(String data) throws Exception {
        cipher.init(Cipher.ENCRYPT_MODE, secretKey);
        byte[] encrypted = cipher.doFinal(data.getBytes());
        return Base64.getEncoder().encodeToString(encrypted);
    }
}

PII Handling and Masking Strategy

PII data (like user identifiers) is masked in logs:

• API keys are truncated to last 4 characters
• User IDs are hashed with salt before logging

public class LogMasker {
    public static String maskApiKey(String apiKey) {
        return "****" + apiKey.substring(Math.max(0, apiKey.length() - 4));
    }
}

Secrets Management

API keys and encryption keys are managed through AWS Secrets Manager:

• Keys are rotated every 90 days
• Access is controlled via IAM policies
• Secrets are fetched at application startup and cached

@Component
public class SecretManager {
    private final AWSSecretsManager client;
    
    public String getSecret(String secretName) {
        GetSecretValueRequest request = new GetSecretValueRequest()
            .withSecretId(secretName);
        return client.getSecretValue(request).getSecretString();
    }
}

Audit Logging Design

All security-relevant events are logged with:

• Timestamp and user context
• Operation type and target resource
• Success/failure status and error details

{
  "timestamp": "2023-10-15T14:30:00Z",
  "userId": "user-123",
  "operation": "RATE_LIMIT_UPDATE",
  "resource": "apiKey:abc123",
  "status": "SUCCESS",
  "ipAddress": "192.168.1.100",
  "userAgent": "curl/7.64.1"
}

Retention policy: audit logs are retained for 90 days with daily backups to S3.

12. Testing Strategy

Unit Testing Scope and Coverage Targets

Unit testing will cover 85%+ of business logic with 90%+ line coverage. Core components include:

• Rate limiter service logic (token bucket, sliding window implementations)
• Configuration validation and parsing
• Request context processing (user ID, API key, endpoint mapping)
• Rate limit enforcement logic with boundary condition tests
• Mocked external dependencies (cache, database, event publisher)

Integration Testing Strategy

Integration tests will validate end-to-end behavior using:

• Testcontainers for Redis and database containers to ensure real infrastructure compatibility
• Embedded databases (H2/SQLite) for fast test execution in CI pipeline
• WireMock for external HTTP service stubbing (e.g., auth server, third-party APIs)
• Test fixtures for various rate limit configurations and user scenarios

Contract Testing

Contract testing will be implemented using Pact to:

• Validate API contract between rate limiter and downstream services (e.g., API gateway)
• Ensure that the rate limiter's response format matches consumer expectations
• Prevent breaking changes in the rate limiting behavior that could impact client applications

Performance Testing Tool and Scenarios

Performance testing will use k6 with key scenarios:

• Concurrent burst load: Simulate 1000 concurrent requests hitting the same endpoint within a second to test token bucket exhaustion
• Steady state throughput: Validate sustained load at 500 RPS over 30 seconds
• Edge case stress: Test with 10k concurrent users hitting rate limits to verify system stability and graceful degradation
• Latency under load: Ensure p95 latency stays below 100ms under normal conditions

Test Data Management Strategy

Test data will be managed using:

• Factory patterns for generating realistic test users, API keys, and rate limit configurations
• Fixtures for pre-defined scenarios like "user with 1000 requests per minute limit"
• Seeding scripts to populate test database with baseline data (e.g., default rate limit tiers, user roles)
• Test data isolation using database transaction rollback or dedicated test schemas to ensure test independence

// Example unit test for rate limit enforcement logic
@Test
public void testRateLimitExceeded() {
    // Given
    RateLimiterConfig config = new RateLimiterConfig(100, 1, TimeUnit.MINUTES);
    UserContext userContext = new UserContext("user-123", "api-key-456");
    
    // When
    for (int i = 0; i < 100; i++) {
        assertTrue(rateLimiter.isAllowed(userContext, "endpoint-1"));
    }
    
    // Then
    assertFalse(rateLimiter.isAllowed(userContext, "endpoint-1"));
}

13. Observability Design

Structured Logging Schema

All logs must follow a structured format with consistent fields to enable effective querying and analysis:

{
  "timestamp": "2023-10-27T14:30:00.000Z",
  "level": "INFO|WARN|ERROR",
  "service": "rate-limiter-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user-123",
  "request_id": "req-456",
  "endpoint": "/api/v1/users",
  "method": "GET",
  "ip_address": "192.168.1.1",
  "rate_limit_remaining": 45,
  "rate_limit_reset": 1698432000,
  "error_code": "RATE_LIMIT_EXCEEDED",
  "message": "Rate limit exceeded for user-123 on endpoint /api/v1/users",
  "error_details": {
    "limit": 50,
    "window": "1 minute",
    "current_requests": 51
  }
}

PII masking rules:

• IP addresses are masked to CIDR block (e.g., 192.168.1.0/24)
• User IDs are logged as hashed values (SHA-256) when not required for audit
• Request body content is never logged in production

Distributed Tracing Instrumentation

Using OpenTelemetry for end-to-end traceability:

Custom spans to add:

1. rate_limiter.check - measures rate limit check duration
2. rate_limiter.update - tracks token bucket updates
3. rate_limiter.reject - logs rejected requests with reasons

Attributes to capture:

Span currentSpan = Span.current();
currentSpan.setAttribute("rate_limit.user_id", userId);
currentSpan.setAttribute("rate_limit.endpoint", endpoint);
currentSpan.setAttribute("rate_limit.limit", limit);
currentSpan.setAttribute("rate_limit.window_seconds", windowSeconds);
currentSpan.setAttribute("rate_limit.remaining", remaining);

Metrics Design

RED Method Metrics (Rate, Errors, Duration):

Metric Name	Type	Labels	Description
rate_limiter_requests_total	Counter	endpoint, method, status_code, user_id	Total request count
rate_limiter_requests_rejected_total	Counter	endpoint, method, reason, user_id	Requests rejected due to rate limits
rate_limiter_requests_remaining_gauge	Gauge	endpoint, method, user_id	Remaining requests in current window
rate_limiter_check_duration_seconds	Histogram	endpoint, method	Time to check rate limits
rate_limiter_update_duration_seconds	Histogram	endpoint, method	Time to update rate limit state
rate_limiter_errors_total	Counter	error_type, endpoint, method	Internal errors during rate limiting

Health Check Endpoint Design

GET /health
Accept: application/json

Response:
{
  "status": "healthy",
  "timestamp": "2023-10-27T14:30:00.000Z",
  "service": "rate-limiter-service",
  "checks": {
    "database": "healthy",
    "cache": "healthy",
    "rate_limiting_engine": "healthy"
  }
}

Endpoints:

• /health - Overall service health (liveness probe)
• /ready - Service ready to serve requests (readiness probe)
• /live - Process health check
• /startup - Startup completion status

Alerting Thresholds

Metric	Warning Level	Critical Level	Escalation Path
rate_limiter_requests_rejected_total	> 1000/sec	> 5000/sec	On-call SRE team
rate_limiter_check_duration_seconds	p95 > 100ms	p99 > 500ms	SRE team
rate_limiter_errors_total	> 10/sec	> 50/sec	On-call SRE
rate_limiter_requests_remaining_gauge	< 10% of limit	< 5% of limit	SRE team
rate_limiter_cache_hit_ratio	< 80%	< 60%	SRE team

Alerts are configured with:

• Slack notifications for warning level
• PagerDuty escalation for critical level
• 5-minute alert cooldown period to prevent alert storms
• Service-level alerting with SLO-based thresholds

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• Rate limit enforcement granularity: Should rate limits be enforced at the API level, endpoint level, or user level? The choice affects both implementation complexity and user experience. For example, enforcing limits per endpoint allows for more granular control but increases system complexity.

• Burst handling strategy: How should systems handle bursty traffic? Should bursts be allowed with a token bucket algorithm or strictly capped? This decision impacts system responsiveness during traffic spikes.

• Graceful degradation: When rate limits are exceeded, should the system return a 429 (Too Many Requests) or attempt to queue requests? This affects client behavior and system reliability.

Known trade-offs and justifications

Trade-off	Option 1	Option 2	Justification
Memory vs. Persistence	In-memory rate limiters (e.g., Redis with TTL)	Database-backed rate limiters	In-memory provides better performance but loses state on restarts. Database persistence ensures reliability but introduces latency and complexity.
Accuracy vs. Performance	Token bucket algorithm with fine-grained tracking	Fixed window counter with coarse tracking	Token bucket provides better accuracy for burst handling but requires more complex state management. Fixed window is simpler and faster but can cause bursts to be miscounted.
Centralized vs. Distributed	Single instance rate limiter	Distributed rate limiter (e.g., Redis cluster)	Centralized is simpler to implement but becomes a bottleneck at scale. Distributed systems provide horizontal scalability but introduce network latency and consistency concerns.
Hard vs. Soft Limits	Enforce strict rate limits with 429 responses	Allow soft limits with queuing or throttling	Hard limits provide clear boundaries but can cause client failures. Soft limits improve user experience but increase system complexity and potential for abuse.

Risks and mitigation strategies

• Cache Invalidation Risk: If rate limit state is cached, invalidation can cause temporary over-limiting or under-limiting. Mitigation: Implement TTL-based cache invalidation with probabilistic early expiry to prevent stampedes.

• System-wide Outage Risk: A rate limiter failure can cause cascading failures across all API endpoints. Mitigation: Implement circuit breaker patterns and graceful degradation (e.g., allow requests when rate limiters are unavailable).

• Burst Denial Risk: Clients may be denied access during legitimate bursts. Mitigation: Implement burst tolerance with token bucket or sliding window algorithms that allow short bursts.

• Inconsistent Rate Limiting: In distributed systems, rate limits may not be consistent across nodes. Mitigation: Use a shared state store (e.g., Redis) with atomic operations to ensure consistency.

Technical debt acknowledged

• State Management Overhead: Maintaining rate limit state for all users and endpoints introduces significant overhead in memory and database load. This is acceptable for high-traffic systems but may not scale well in low-volume scenarios.

• Complexity of Burst Handling: Supporting burst tolerance adds significant complexity to the rate limiter logic, including token bucket implementation and concurrent access handling.

• Eventual Consistency in Distributed Systems: In distributed rate limiting, there may be eventual consistency issues when multiple nodes process requests simultaneously. This is acceptable for most use cases but requires careful monitoring and alerting.

• Configuration Management Overhead: Managing rate limit configurations per endpoint or user type requires a robust configuration management system to avoid misconfigurations and ensure consistency across environments.

• Audit Trail Limitations: The current design does not include detailed audit logging of rate limit violations or enforcement actions. This is a trade-off to reduce system overhead but may impact debugging and compliance requirements in regulated environments.