API Throttling Service Design and Implementation

This document presents a production-ready low-level design for API throttling services that handle high-volume traffic while maintaining system stability through intelligent rate-limiting mechanisms and distributed coordination patterns.

1. Problem Statement & Requirements

Clarifying questions

1. What scale expectations are there for concurrent API requests?

   • Are we talking about hundreds of thousands or millions of concurrent requests per second?

2. What consistency requirements exist for rate limits?

   • Should rate limiting be strictly enforced at the microsecond level or can there be a small tolerance window?

3. What are the key entities in this system and their relationships?

   • Is throttling applied per user, per API key, per IP address, or a combination?

4. What's the expected read/write ratio for rate limit data?

   • Will we be reading rate limits more frequently than updating them?

5. How sensitive is the system to latency?

   • Are there strict SLA requirements for throttling decisions (e.g., <10ms)?

6. What authentication model is in place?

   • Is it JWT-based, API key-based, or OAuth2?

7. Are there any external integrations or dependencies (e.g., Redis, Kafka)?

   • What infrastructure is available for caching and messaging?

Functional requirements

• Rate limit configuration per API key, user, or IP address
• Enforce rate limits in real-time for incoming API requests
• Support multiple rate limit types (e.g., requests per second, per minute, per hour)
• Provide detailed rate limit information in response headers (e.g., X-RateLimit-Limit, X-RateLimit-Remaining)
• Support burst rate limits (e.g., 100 requests per second with burst of 200)
• Support granular rate limit policies (e.g., per endpoint, per method, or per resource)
• Return appropriate HTTP status code (429 - Too Many Requests) when limits are exceeded
• Support dynamic rate limit updates without service restarts

Non-functional requirements

• P50 latency for rate limit check: <5ms
• P95 latency for rate limit check: <10ms
• P99 latency for rate limit check: <25ms
• Throughput target: 100,000+ requests per second
• Availability: 99.9% (5 minutes of downtime per year)
• Support for horizontal scaling across multiple instances

Constraints and assumptions

• Assumption: Rate limit data is stored in a centralized cache (e.g., Redis) for fast access
• Assumption: API keys and users are already managed by an external authentication service
• Assumption: The system will not store rate limit data in a database for performance reasons
• Constraint: Rate limit decisions must be made within 50ms to meet SLA requirements
• Constraint: The system should not block or delay API responses due to rate limit checks

Out of scope

• Rate limit policy management UI or dashboard
• Audit logging for rate limit violations (this is a separate concern)
• Support for complex rate limit policies (e.g., sliding window with custom logic)
• Rate limit enforcement for non-API traffic (e.g., WebSocket or direct TCP connections)
• Support for different rate limit algorithms (e.g., token bucket vs. leaky bucket)
• Rate limit enforcement for internal services or microservices communication

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for this throttling service due to:

• Strong typing and schema validation at compile time
• Efficient binary serialization with lower network overhead than REST
• First-class support for streaming and bidirectional communication
• Native support for versioning through proto file evolution
• Better tooling ecosystem (protoc-gen-java, grpc-java)

Endpoint Definitions

Method	Path	Description	Auth	Rate Limit
GetQuota	/throttling.v1.QuotaService/GetQuota	Fetches current quota for a client	JWT	100 RPS
UpdateQuota	/throttling.v1.QuotaService/UpdateQuota	Updates quota configuration	JWT + Admin Role	50 RPS
CheckQuota	/throttling.v1.QuotaService/CheckQuota	Validates if request can proceed	JWT	500 RPS
ResetQuota	/throttling.v1.QuotaService/ResetQuota	Resets quota for a client	JWT + Admin Role	100 RPS

Request and Response Schemas

GetQuotaRequest

{
  "client_id": "string",
  "quota_type": "string"
}

GetQuotaResponse

{
  "client_id": "string",
  "quota_type": "string",
  "limit": "int64",
  "remaining": "int64",
  "reset_time": "timestamp",
  "window_size": "duration"
}

UpdateQuotaRequest

{
  "client_id": "string",
  "quota_type": "string",
  "limit": "int64",
  "window_size": "duration"
}

UpdateQuotaResponse

{
  "success": "boolean",
  "updated_at": "timestamp"
}

CheckQuotaRequest

{
  "client_id": "string",
  "quota_type": "string",
  "request_count": "int32"
}

CheckQuotaResponse

{
  "allowed": "boolean",
  "remaining": "int64",
  "reset_time": "timestamp",
  "retry_after": "duration"
}

HTTP Status Codes and Error Response Format

Status Code	Description
200 OK	Successful operation
400 Bad Request	Invalid request format
401 Unauthorized	Missing or invalid auth
403 Forbidden	Insufficient permissions
429 Too Many Requests	Quota exceeded
500 Internal Server Error	Service failure

Standardized error response:

{
  "error": {
    "code": "string",
    "message": "string",
    "details": [
      {
        "field": "string",
        "message": "string"
      }
    ]
  }
}

API Versioning Strategy

Versioned using semantic versioning with gRPC package naming:

• throttling.v1 - Current stable version
• Future: throttling.v2 for breaking changes

Pagination Design

Cursor-based pagination is not applicable here as this service operates on individual quota entities rather than collections. Each client has a single quota configuration, so no pagination is required.

Authentication and Authorization

• Authentication: JWT tokens with audience validation (throttling-service)
• Authorization:
  • GetQuota, CheckQuota - Any authenticated client
  • UpdateQuota, ResetQuota - Admin role required via JWT claims

Rate Limiting per Endpoint

Endpoint	Limit	Period	Enforcement
GetQuota	100 RPS	1 second	In-memory token bucket
UpdateQuota	50 RPS	1 second	In-memory token bucket
CheckQuota	500 RPS	1 second	In-memory token bucket
ResetQuota	100 RPS	1 second	In-memory token bucket

Idempotency Design

For mutation endpoints (UpdateQuota, ResetQuota):

• Accept idempotency_key in request header
• Store operation state with key for 5 minutes
• If same key received, return cached result instead of reprocessing

public class IdempotencyKey {
    private String key;
    private String operationType;
    private String result;
    private Instant createdAt;
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    API_KEY {
        string id PK
        string key_hash
        string name
        string owner_id
        boolean is_active
        timestamp created_at
        timestamp updated_at
        timestamp expires_at
    }

    RATE_LIMIT_RULE {
        string id PK
        string api_key_id FK
        string resource_path
        integer max_requests
        integer window_seconds
        timestamp created_at
        timestamp updated_at
    }

    RATE_LIMIT_METRIC {
        string id PK
        string api_key_id FK
        string resource_path
        integer request_count
        timestamp window_start
        timestamp window_end
        timestamp created_at
    }

    API_KEY ||--o{ RATE_LIMIT_RULE : "has"
    API_KEY ||--o{ RATE_LIMIT_METRIC : "generates"
    RATE_LIMIT_RULE ||--o{ RATE_LIMIT_METRIC : "triggers"

Table / Collection Schema Definitions

api_keys table

CREATE TABLE api_keys (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    key_hash VARCHAR(255) NOT NULL UNIQUE,
    name VARCHAR(255) NOT NULL,
    owner_id VARCHAR(255) NOT NULL,
    is_active BOOLEAN DEFAULT TRUE,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    expires_at TIMESTAMP WITH TIME ZONE,
    
    INDEX idx_api_keys_owner_id (owner_id),
    INDEX idx_api_keys_key_hash (key_hash),
    INDEX idx_api_keys_is_active (is_active)
);

rate_limit_rules table

CREATE TABLE rate_limit_rules (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    api_key_id UUID NOT NULL REFERENCES api_keys(id) ON DELETE CASCADE,
    resource_path VARCHAR(512) NOT NULL,
    max_requests INTEGER NOT NULL CHECK (max_requests > 0),
    window_seconds INTEGER NOT NULL CHECK (window_seconds > 0),
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_rate_limit_rules_api_key_id (api_key_id),
    INDEX idx_rate_limit_rules_resource_path (resource_path),
    UNIQUE(api_key_id, resource_path)
);

rate_limit_metrics table

CREATE TABLE rate_limit_metrics (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    api_key_id UUID NOT NULL REFERENCES api_keys(id) ON DELETE CASCADE,
    resource_path VARCHAR(512) NOT NULL,
    request_count INTEGER NOT NULL DEFAULT 0,
    window_start TIMESTAMP WITH TIME ZONE NOT NULL,
    window_end TIMESTAMP WITH TIME ZONE NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    
    INDEX idx_rate_limit_metrics_api_key_id (api_key_id),
    INDEX idx_rate_limit_metrics_window_start (window_start),
    INDEX idx_rate_limit_metrics_resource_path (resource_path)
);

Primary Key Strategy

UUIDs are used for all primary keys to ensure:

• Global uniqueness across distributed systems
• No single point of failure in key generation
• Better sharding capability with hash-based distribution
• Protection against enumeration attacks (no sequential patterns)

Index Design

Index Name	Fields	Type	Purpose
idx_api_keys_owner_id	owner_id	Standard	Fast lookup by owner
idx_api_keys_key_hash	key_hash	Unique	Fast key validation
idx_api_keys_is_active	is_active	Standard	Efficient filtering of inactive keys
idx_rate_limit_rules_api_key_id	api_key_id	Standard	Fast rule lookup by key
idx_rate_limit_rules_resource_path	resource_path	Standard	Fast rule lookup by endpoint
idx_rate_limit_metrics_api_key_id	api_key_id	Standard	Fast metric lookup by key
idx_rate_limit_metrics_window_start	window_start	Standard	Efficient time-based queries
idx_rate_limit_metrics_resource_path	resource_path	Standard	Fast endpoint filtering

Data Partitioning / Sharding Strategy

• Shard by API key ID hash: Use consistent hashing to distribute keys across shards
• Partition window-based metrics by time: Use time-series partitioning (e.g., monthly) for rate_limit_metrics
• Shard by owner ID: For multi-tenant scenarios, shard by owner_id prefix or hash

Soft Delete vs. Hard Delete Decision

Soft delete is used for api_keys and rate_limit_rules because:

• Audit trail requirement: Need to track historical key usage and rules
• Compliance: Regulatory requirements may mandate data retention
• Recovery capability: Ability to restore keys if accidentally deleted
• Rule inheritance: Rules may be referenced in historical metrics

Audit Trail Design

All mutations to api_keys and rate_limit_rules are logged in an audit trail table:

CREATE TABLE audit_trail (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    operation VARCHAR(50) NOT NULL, -- CREATE, UPDATE, DELETE
    table_name VARCHAR(100) NOT NULL,
    record_id UUID NOT NULL,
    old_values JSONB,
    new_values JSONB,
    user_id VARCHAR(255),
    timestamp TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

Data Archival Strategy and Retention Policy

• Active data: api_keys and rate_limit_rules remain in primary storage
• Metrics archival: rate_limit_metrics older than 30 days are moved to cold storage (e.g., S3)
• Retention policy:
  • 7 days: Active metrics for real-time rate limiting
  • 30 days: Historical metrics for analytics and reporting
  • 1 year: Full audit trail (if required by compliance)
• Archival trigger: Scheduled job that moves expired metrics to archive storage

// Example archival logic (Java)
public class RateLimitMetricArchiver {
    private final RateLimitMetricRepository metricRepo;
    private final ArchiveStorageService archiveService;

    public void archiveExpiredMetrics() {
        LocalDateTime cutoff = LocalDateTime.now().minusDays(30);
        List<RateLimitMetric> expiredMetrics = metricRepo.findExpired(cutoff);
        
        for (RateLimitMetric metric : expiredMetrics) {
            archiveService.store(metric);
            metricRepo.delete(metric.getId());
        }
    }
}

4. Class Design & Service Layer

classDiagram
    class ThrottlingService {
        +throttle(request)
        +getRateLimitForUser(userId, resource)
        +updateRateLimit(userId, resource, limit)
        +resetRateLimit(userId, resource)
    }
    
    class RateLimit {
        -userId: String
        -resource: String
        -limit: int
        -window: Duration
        -current: int
        -resetTime: Instant
        +isAllowed(): boolean
        +consume(): void
    }
    
    class RateLimitRepository {
        +save(rateLimit)
        +findById(userId, resource)
        +update(rateLimit)
        +delete(userId, resource)
    }
    
    class RateLimitCalculator {
        +calculateRateLimit(user, resource): RateLimit
    }
    
    class ThrottlingConfig {
        -maxRequests: int
        -windowSeconds: int
        -resource: String
        -userId: String
    }
    
    class ThrottlingContext {
        -request: HttpRequest
        -userId: String
        -resource: String
        -rateLimit: RateLimit
    }
    
    class ThrottlingException {
        +getMessage(): String
        +getStatusCode(): int
    }
    
    class RateLimitEvent {
        -userId: String
        -resource: String
        -timestamp: Instant
        -action: String
    }
    
    class EventPublisher {
        +publish(event)
    }
    
    ThrottlingService --> RateLimit : uses
    ThrottlingService --> RateLimitRepository : persists
    ThrottlingService --> RateLimitCalculator : calculates
    ThrottlingService --> EventPublisher : publishes
    RateLimitCalculator --> ThrottlingConfig : configures
    ThrottlingContext --> RateLimit : contains
    ThrottlingService --> ThrottlingException : throws

stateDiagram-v2
    [*] --> Allowed
    Allowed --> Throttled : exceeds limit
    Throttled --> Allowed : reset time reached
    Throttled --> Throttled : re-attempt before reset

Service Class Structure and Responsibilities

The ThrottlingService is the core service responsible for enforcing rate limits across API resources. It coordinates between rate limit calculation, persistence, and event publishing.

Key responsibilities:

• Validate incoming requests against configured rate limits
• Calculate and enforce rate limit boundaries
• Manage rate limit state in persistence layer
• Publish throttling events for audit and monitoring
• Handle edge cases like burst limits and user resets

Interface / Abstract Class Design

public interface RateLimitRepository {
    RateLimit save(RateLimit rateLimit);
    Optional<RateLimit> findById(String userId, String resource);
    void update(RateLimit rateLimit);
    void delete(String userId, String resource);
}

public interface RateLimitCalculator {
    RateLimit calculateRateLimit(String userId, String resource, ThrottlingConfig config);
}

public interface EventPublisher {
    void publish(RateLimitEvent event);
}

Business Logic Encapsulation Patterns

Strategy Pattern: Rate limit calculation follows a strategy pattern to support different rate limiting algorithms (fixed window, sliding window, token bucket).

public class RateLimitStrategy {
    public enum Type {
        FIXED_WINDOW,
        SLIDING_WINDOW,
        TOKEN_BUCKET
    }
    
    public static RateLimitCalculator getCalculator(Type type) {
        switch (type) {
            case FIXED_WINDOW: return new FixedWindowCalculator();
            case SLIDING_WINDOW: return new SlidingWindowCalculator();
            case TOKEN_BUCKET: return new TokenBucketCalculator();
            default: throw new IllegalArgumentException("Unknown rate limit type");
        }
    }
}

public class FixedWindowCalculator implements RateLimitCalculator {
    @Override
    public RateLimit calculateRateLimit(String userId, String resource, ThrottlingConfig config) {
        Instant now = Instant.now();
        return new RateLimit(
            userId, resource, 
            config.getMaxRequests(), 
            Duration.ofSeconds(config.getWindowSeconds()),
            0, now.plusSeconds(config.getWindowSeconds())
        );
    }
}

Critical-Path Method Implementation

public class ThrottlingService {
    private final RateLimitRepository rateLimitRepository;
    private final RateLimitCalculator calculator;
    private final EventPublisher eventPublisher;
    
    public ThrottlingResult throttle(ThrottlingContext context) {
        String userId = context.getUserId();
        String resource = context.getResource();
        
        // Transaction boundary: all operations in single DB transaction
        return transactionTemplate.execute(status -> {
            try {
                // Read current rate limit or create new one
                RateLimit rateLimit = rateLimitRepository.findById(userId, resource)
                    .orElseGet(() -> {
                        ThrottlingConfig config = loadConfigForUser(userId, resource);
                        RateLimit newRateLimit = calculator.calculateRateLimit(userId, resource, config);
                        return rateLimitRepository.save(newRateLimit);
                    });
                
                // Check if we're in a new time window
                Instant now = Instant.now();
                if (rateLimit.getResetTime().isBefore(now)) {
                    rateLimit.reset();
                }
                
                // Validate if request is allowed
                if (!rateLimit.isAllowed()) {
                    // Publish throttling event for monitoring and alerting
                    eventPublisher.publish(new RateLimitEvent(
                        userId, resource, now, "THROTTLED"));
                    
                    throw new ThrottlingException(
                        "Rate limit exceeded for user: " + userId + 
                        " on resource: " + resource,
                        HttpStatus.TOO_MANY_REQUESTS.value());
                }
                
                // Consume the rate limit token
                rateLimit.consume();
                rateLimitRepository.update(rateLimit);
                
                // Publish success event for monitoring
                eventPublisher.publish(new RateLimitEvent(
                    userId, resource, now, "ALLOWED"));
                
                return new ThrottlingResult(true, rateLimit.getRemaining(), 
                    rateLimit.getResetTime());
                    
            } catch (Exception e) {
                // Ensure transaction rollback on any error
                status.setRollbackOnly();
                throw e;
            }
        });
    }
}

Transaction Boundary Design

All rate limit operations are wrapped in a single database transaction to ensure atomicity:

• Read current rate limit state
• Update rate limit counter and reset time if needed
• Publish audit events

Service Interface Contracts

public class ThrottlingResult {
    private final boolean allowed;
    private final long remaining;
    private final Instant resetTime;
    
    // Constructor, getters
}

public class ThrottlingContext {
    private final HttpRequest request;
    private final String userId;
    private final String resource;
    
    // Constructor, getters
}

public class ThrottlingException extends RuntimeException {
    private final int statusCode;
    
    public ThrottlingException(String message, int statusCode) {
        super(message);
        this.statusCode = statusCode;
    }
    
    public int getStatusCode() { return statusCode; }
}

Saga / Orchestration Pattern

For complex rate limiting scenarios involving external systems (e.g., third-party API limits), the service can integrate with a saga pattern for distributed transactions. However, in this design, all operations are local to the service.

Domain Object / Value Object Design

public class RateLimit {
    private final String userId;
    private final String resource;
    private final int limit;
    private final Duration window;
    private int current;
    private Instant resetTime;
    
    public RateLimit(String userId, String resource, int limit, Duration window, 
                     int current, Instant resetTime) {
        this.userId = userId;
        this.resource = resource;
        this.limit = limit;
        this.window = window;
        this.current = current;
        this.resetTime = resetTime;
    }
    
    public boolean isAllowed() {
        return current < limit;
    }
    
    public void consume() {
        current++;
    }
    
    public void reset() {
        this.current = 0;
        this.resetTime = Instant.now().plus(window);
    }
    
    public long getRemaining() {
        return limit - current;
    }
    
    // Getters and equals/hashCode
}

Key Design Decisions

1. Rate Limit Calculation Strategy: Fixed window algorithm for simplicity and predictability, with extensibility for sliding window or token bucket implementations.

2. Transaction Management: Single transaction per throttling operation ensures consistency between rate limit updates and event publishing.

3. Event Publishing: Audit events are published synchronously to ensure rate limit changes are logged for monitoring and alerting.

4. Idempotency: The service handles concurrent requests gracefully through database-level locking and atomic operations.

5. Error Handling: Custom throttling exceptions provide clear error messages and appropriate HTTP status codes for client consumption.

5. Concurrency & Thread Safety

Concurrency Model

The API throttling service will utilize a multi-threaded async model with Java's CompletableFuture and ExecutorService for handling concurrent requests. The service will leverage a thread pool with fixed size to manage request processing efficiently without overwhelming system resources.

Race Condition Prevention Strategy

Race conditions in the throttling service are primarily mitigated through:

1. Atomic operations for counter updates using AtomicLong or AtomicInteger
2. Synchronized blocks for critical sections where atomic operations are insufficient
3. Read-Write locks (ReentrantReadWriteLock) for concurrent read-heavy operations on rate limit configurations
4. Immutable data structures where possible to prevent modification during concurrent access

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected for rate limit bucket updates because:

• The system expects high throughput with low contention
• Rate limiting is a read-heavy operation with infrequent writes (bucket resets)
• Reduces lock contention and improves overall throughput
• Implements CAS (Compare-And-Swap) operations for atomic updates

Deadlock Prevention Strategy

Deadlock prevention is implemented through:

1. Fixed ordering of lock acquisition - all operations acquire locks in consistent order (e.g., by API key ID)
2. Timeout-based lock acquisition - using tryLock() with timeouts to prevent indefinite blocking
3. Hierarchical lock management - grouping related entities under consistent namespace prefixes

Idempotency Implementation for Concurrent Duplicate Requests

For concurrent duplicate requests, the service implements idempotency using:

public class ThrottlingService {
    private final Map<String, Semaphore> requestSemaphores = new ConcurrentHashMap<>();
    private final Map<String, AtomicLong> lastRequestTimestamps = new ConcurrentHashMap<>();

    public ThrottlingResult checkRateLimit(String apiKey, String endpoint) {
        String key = generateKey(apiKey, endpoint);
        Semaphore semaphore = requestSemaphores.computeIfAbsent(key, k -> new Semaphore(1));
        
        try {
            if (semaphore.tryAcquire(100, TimeUnit.MILLISECONDS)) {
                return processRequest(key, apiKey, endpoint);
            } else {
                // Handle concurrent request by returning rate limit exceeded
                return ThrottlingResult.RATE_LIMIT_EXCEEDED;
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            return ThrottlingResult.ERROR;
        } finally {
            semaphore.release();
        }
    }

    private ThrottlingResult processRequest(String key, String apiKey, String endpoint) {
        // Implementation details here
        return ThrottlingResult.ALLOWED;
    }
}

The idempotency strategy ensures that concurrent duplicate requests don't cause race conditions or inconsistent rate limit states, while maintaining the system's ability to process legitimate concurrent requests efficiently.

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant Client
    participant APIGateway
    participant ThrottlingService
    participant RateLimiter
    participant Redis

    Client->>APIGateway: Request to /api/users
    APIGateway->>ThrottlingService: Check rate limit for user
    ThrottlingService->>RateLimiter: Get current usage
    RateLimiter->>Redis: Get token count for user
    Redis-->>RateLimiter: Current tokens: 50
    RateLimiter->>ThrottlingService: Tokens available: 50
    ThrottlingService-->>APIGateway: Allow request
    APIGateway-->>Client: 200 OK

sequenceDiagram
    participant Client
    participant APIGateway
    participant ThrottlingService
    participant RateLimiter
    participant Redis

    Client->>APIGateway: Request to /api/users
    APIGateway->>ThrottlingService: Check rate limit for user
    ThrottlingService->>RateLimiter: Get current usage
    RateLimiter->>Redis: Get token count for user
    Redis-->>RateLimiter: Current tokens: 0
    RateLimiter->>ThrottlingService: Tokens available: 0
    ThrottlingService-->>APIGateway: Reject request
    APIGateway-->>Client: 429 Too Many Requests

sequenceDiagram
    participant Client
    participant APIGateway
    participant ThrottlingService
    participant RateLimiter
    participant Redis
    participant EventPublisher

    Client->>APIGateway: Request to /api/users
    APIGateway->>ThrottlingService: Check rate limit for user
    ThrottlingService->>RateLimiter: Get current usage
    RateLimiter->>Redis: Get token count for user
    Redis-->>RateLimiter: Current tokens: 50
    RateLimiter->>ThrottlingService: Tokens available: 50
    ThrottlingService->>RateLimiter: Deduct tokens from user
    RateLimiter->>Redis: Update token count for user
    Redis-->>RateLimiter: Token updated successfully
    RateLimiter->>EventPublisher: Publish rate limit event
    EventPublisher-->>RateLimiter: Event published
    ThrottlingService-->>APIGateway: Allow request
    APIGateway-->>Client: 200 OK

7. Caching Design

Cache Strategy Selection

For API throttling, we'll use a cache-aside pattern with time-based eviction. This approach provides the best balance between performance and consistency for rate limiting scenarios where strict real-time accuracy isn't required but low latency is critical.

Cache Key Design

Cache keys will follow a structured namespace pattern to ensure proper organization and prevent key collisions:

public class ThrottlingCacheKey {
    private static final String PREFIX = "throttle:";
    private static final String SEPARATOR = ":";
    
    public static String buildKey(String apiKey, String endpoint, String timeWindow) {
        return PREFIX + apiKey + SEPARATOR + endpoint + SEPARATOR + timeWindow;
    }
}

TTL Configuration

Data Type	TTL	Reason
Rate limit buckets	60 seconds	Aligns with minute-based rate limiting windows
API key metadata	1 hour	Reduces frequent database lookups for static config
Quota limits	24 hours	For daily quota tracking with reasonable cache freshness

Cache Eviction Policy

We'll use LRU (Least Recently Used) eviction policy with a maximum size constraint to prevent memory bloat:

public class ThrottlingCache {
    private final Cache<String, RateLimitBucket> cache;
    
    public ThrottlingCache() {
        this.cache = CacheBuilder.newBuilder()
            .maximumSize(100_000)
            .expireAfterWrite(60, TimeUnit.SECONDS)
            .recordStats()
            .build();
    }
}

Cache Stampede Prevention

To prevent cache stampedes during high-traffic periods, we'll implement probabilistic early expiry:

public class CacheStampedePrevention {
    private static final double STAMPED_EXPIRY_FACTOR = 0.8;
    
    public boolean shouldRefreshCache(String key, long currentTime) {
        // Check if cache entry is about to expire
        RateLimitBucket bucket = cache.getIfPresent(key);
        if (bucket != null) {
            long ttl = bucket.getExpiryTime() - currentTime;
            return ttl < (bucket.getTtl() * STAMPED_EXPIRY_FACTOR);
        }
        return true;
    }
}

Distributed Cache Architecture

We'll use Redis cluster with:

• 3 master nodes for high availability
• 3 replica nodes per master for read scaling
• Redis Sentinel for automatic failover
• Hash-based sharding to distribute keys across cluster nodes

Cache Invalidation Strategy

We'll implement a hybrid approach combining TTL-based and event-driven invalidation:

1. TTL-based: Automatic expiration based on time windows
2. Event-driven: Explicit invalidation when:
   • API key configuration changes
   • Rate limit thresholds are updated
   • User account status changes (suspended/active)

public class CacheInvalidator {
    private final RedisTemplate<String, Object> redisTemplate;
    
    public void invalidateForApiKey(String apiKey) {
        // Invalidate all rate limit buckets for this API key
        String pattern = "throttle:" + apiKey + ":*";
        Set<String> keys = redisTemplate.keys(pattern);
        redisTemplate.delete(keys);
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and database operations. Each entity has its dedicated repository interface and implementation.

public interface RateLimitRepository {
    void save(RateLimit rateLimit);
    Optional<RateLimit> findById(String id);
    List<RateLimit> findByApiKey(String apiKey);
    void update(RateLimit rateLimit);
    void delete(String id);
    List<RateLimit> findAll();
}

public class RateLimitRepositoryImpl implements RateLimitRepository {
    private final JdbcTemplate jdbcTemplate;
    private final ObjectMapper objectMapper;

    public RateLimitRepositoryImpl(JdbcTemplate jdbcTemplate, ObjectMapper objectMapper) {
        this.jdbcTemplate = jdbcTemplate;
        this.objectMapper = objectMapper;
    }

    @Override
    public void save(RateLimit rateLimit) {
        String sql = "INSERT INTO rate_limits (id, api_key, limit_type, limit_value, window_size, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?)";
        jdbcTemplate.update(sql, 
            rateLimit.getId(),
            rateLimit.getApiKey(),
            rateLimit.getLimitType().name(),
            rateLimit.getLimitValue(),
            rateLimit.getWindowSize(),
            rateLimit.getCreatedAt(),
            rateLimit.getUpdatedAt()
        );
    }

    @Override
    public Optional<RateLimit> findById(String id) {
        String sql = "SELECT * FROM rate_limits WHERE id = ?";
        try {
            return Optional.of(jdbcTemplate.queryForObject(sql, new RateLimitRowMapper(), id));
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }

    @Override
    public List<RateLimit> findByApiKey(String apiKey) {
        String sql = "SELECT * FROM rate_limits WHERE api_key = ?";
        return jdbcTemplate.query(sql, new RateLimitRowMapper(), apiKey);
    }

    @Override
    public void update(RateLimit rateLimit) {
        String sql = "UPDATE rate_limits SET limit_type = ?, limit_value = ?, window_size = ?, updated_at = ? WHERE id = ?";
        jdbcTemplate.update(sql,
            rateLimit.getLimitType().name(),
            rateLimit.getLimitValue(),
            rateLimit.getWindowSize(),
            rateLimit.getUpdatedAt(),
            rateLimit.getId()
        );
    }

    @Override
    public void delete(String id) {
        String sql = "DELETE FROM rate_limits WHERE id = ?";
        jdbcTemplate.update(sql, id);
    }

    @Override
    public List<RateLimit> findAll() {
        String sql = "SELECT * FROM rate_limits";
        return jdbcTemplate.query(sql, new RateLimitRowMapper());
    }
}

ORM vs. Raw Query Decisions

For the rate limiting service, raw SQL queries are preferred over ORM for performance-critical operations:

1. Performance: Raw queries provide better performance for high-frequency rate limit checks
2. Control: Direct control over query execution and optimization
3. Simplicity: Avoids ORM overhead for simple CRUD operations

However, ORM is used for complex data transformations and reporting queries where maintainability outweighs performance concerns.

Read vs. Write Separation

The service uses a read-write separated approach with CQRS pattern for better scalability:

public class RateLimitReadRepository {
    private final JdbcTemplate readJdbcTemplate;

    public RateLimitReadRepository(JdbcTemplate readJdbcTemplate) {
        this.readJdbcTemplate = readJdbcTemplate;
    }

    public RateLimit getRateLimitByApiKey(String apiKey) {
        String sql = "SELECT * FROM rate_limits WHERE api_key = ? AND deleted_at IS NULL";
        try {
            return readJdbcTemplate.queryForObject(sql, new RateLimitRowMapper(), apiKey);
        } catch (EmptyResultDataAccessException e) {
            return null;
        }
    }

    public List<RateLimit> getAllActiveRateLimits() {
        String sql = "SELECT * FROM rate_limits WHERE deleted_at IS NULL";
        return readJdbcTemplate.query(sql, new RateLimitRowMapper());
    }
}

public class RateLimitWriteRepository {
    private final JdbcTemplate writeJdbcTemplate;

    public RateLimitWriteRepository(JdbcTemplate writeJdbcTemplate) {
        this.writeJdbcTemplate = writeJdbcTemplate;
    }

    public void save(RateLimit rateLimit) {
        String sql = "INSERT INTO rate_limits (id, api_key, limit_type, limit_value, window_size, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?)";
        writeJdbcTemplate.update(sql, 
            rateLimit.getId(),
            rateLimit.getApiKey(),
            rateLimit.getLimitType().name(),
            rateLimit.getLimitValue(),
            rateLimit.getWindowSize(),
            rateLimit.getCreatedAt(),
            rateLimit.getUpdatedAt()
        );
    }

    public void update(RateLimit rateLimit) {
        String sql = "UPDATE rate_limits SET limit_type = ?, limit_value = ?, window_size = ?, updated_at = ? WHERE id = ?";
        writeJdbcTemplate.update(sql,
            rateLimit.getLimitType().name(),
            rateLimit.getLimitValue(),
            rateLimit.getWindowSize(),
            rateLimit.getUpdatedAt(),
            rateLimit.getId()
        );
    }

    public void softDelete(String id) {
        String sql = "UPDATE rate_limits SET deleted_at = NOW() WHERE id = ?";
        writeJdbcTemplate.update(sql, id);
    }
}

Connection Pool Configuration

Connection pool is configured with optimized settings for high throughput:

@Configuration
public class DatabaseConfig {
    @Bean
    public DataSource dataSource() {
        HikariConfig config = new HikariConfig();
        config.setJdbcUrl("jdbc:mysql://localhost:3306/rate_limiting");
        config.setUsername("user");
        config.setPassword("password");
        config.setMaximumPoolSize(20);
        config.setMinimumIdle(5);
        config.setConnectionTimeout(30000);
        config.setIdleTimeout(600000);
        config.setMaxLifetime(1800000);
        config.setLeakDetectionThreshold(60000);
        return new HikariDataSource(config);
    }
}

N+1 Query Prevention Strategy

To prevent N+1 query problems, eager loading and batch operations are implemented:

public class RateLimitBatchRepository {
    private final JdbcTemplate jdbcTemplate;

    public RateLimitBatchRepository(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }

    public Map<String, RateLimit> getRateLimitsByApiKeys(List<String> apiKeys) {
        if (apiKeys.isEmpty()) return Collections.emptyMap();

        String placeholders = String.join(",", Collections.nCopies(apiKeys.size(), "?"));
        String sql = "SELECT * FROM rate_limits WHERE api_key IN (" + placeholders + ") AND deleted_at IS NULL";
        
        List<RateLimit> rateLimits = jdbcTemplate.query(sql, apiKeys.toArray(), new RateLimitRowMapper());
        
        return rateLimits.stream()
            .collect(Collectors.toMap(RateLimit::getApiKey, Function.identity()));
    }
}

Batch Operation Design

Batch operations are implemented to handle bulk rate limit updates efficiently:

public class RateLimitBatchUpdateService {
    private final JdbcTemplate jdbcTemplate;

    public RateLimitBatchUpdateService(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }

    public void batchUpdateRateLimits(List<RateLimit> rateLimits) {
        String sql = "UPDATE rate_limits SET limit_type = ?, limit_value = ?, window_size = ?, updated_at = ? WHERE id = ?";
        
        jdbcTemplate.batchUpdate(sql, new BatchPreparedStatementSetter() {
            @Override
            public void setValues(PreparedStatement ps, int i) throws SQLException {
                RateLimit rateLimit = rateLimits.get(i);
                ps.setString(1, rateLimit.getLimitType().name());
                ps.setLong(2, rateLimit.getLimitValue());
                ps.setInt(3, rateLimit.getWindowSize());
                ps.setTimestamp(4, Timestamp.valueOf(rateLimit.getUpdatedAt()));
                ps.setString(5, rateLimit.getId());
            }

            @Override
            public int getBatchSize() {
                return rateLimits.size();
            }
        });
    }
}

Data Access Layer Interface Contracts

public interface RateLimitRepository {
    void save(RateLimit rateLimit);
    Optional<RateLimit> findById(String id);
    List<RateLimit> findByApiKey(String apiKey);
    void update(RateLimit rateLimit);
    void delete(String id);
    List<RateLimit> findAll();
    Map<String, RateLimit> getRateLimitsByApiKeys(List<String> apiKeys);
}

Transaction Management

Transaction boundaries are carefully defined to ensure data consistency:

@Service
@Transactional
public class RateLimitService {
    private final RateLimitRepository rateLimitRepository;
    private final RateLimitEventPublisher eventPublisher;

    public RateLimitService(RateLimitRepository rateLimitRepository, 
                           RateLimitEventPublisher eventPublisher) {
        this.rateLimitRepository = rateLimitRepository;
        this.eventPublisher = eventPublisher;
    }

    public RateLimit createRateLimit(RateLimit rateLimit) {
        // Transaction boundary starts here
        rateLimitRepository.save(rateLimit);
        eventPublisher.publishRateLimitCreated(rateLimit);
        // Transaction boundary ends here
        return rateLimit;
    }

    @Transactional(readOnly = true)
    public RateLimit getRateLimit(String id) {
        return rateLimitRepository.findById(id).orElse(null);
    }
}

Query Optimization Considerations

1. Index usage: Queries are optimized with proper indexing
2. Query plan analysis: Regular query plan reviews to ensure optimal execution
3. Result set size limits: Large result sets are paginated or limited

public class RateLimitQueryOptimizer {
    private final JdbcTemplate jdbcTemplate;

    public RateLimitQueryOptimizer(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }

    // Optimized query with proper indexing
    public List<RateLimit> getActiveRateLimitsByApiKey(String apiKey, int limit) {
        String sql = "SELECT * FROM rate_limits WHERE api_key = ? AND deleted_at IS NULL ORDER BY created_at DESC LIMIT ?";
        return jdbcTemplate.query(sql, new RateLimitRowMapper(), apiKey, limit);
    }
}

The repository layer is designed for high performance, scalability, and maintainability with careful attention to query optimization, connection management, and transaction boundaries.

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature — what triggers async vs. what stays synchronous, and why

The API throttling service operates primarily in a synchronous request-response model for real-time rate limiting decisions. However, we introduce asynchronous processing for:

1. Rate limit bucket replenishment - Using a background task to increment token buckets at fixed intervals (e.g., every 100ms) rather than on each request, reducing system load and avoiding contention.
2. Audit logging - Publishing detailed usage logs asynchronously to avoid blocking the main request path.
3. Alerting and notifications - Sending alerts when thresholds are breached or rate limits are approaching critical levels.
4. Metrics aggregation - Periodically aggregating and publishing usage statistics to monitoring systems.

The synchronous path handles immediate rate limiting decisions, while asynchronous flows handle operational concerns that don't block user requests.

Message schema design with a sample payload (Avro / Protobuf / JSON Schema — with versioning strategy)

We use JSON Schema for message payloads due to its simplicity and wide adoption in microservices environments. Versioning is handled through a version prefix in the message topic name.

{
  "version": "1.0",
  "type": "api_throttle_event",
  "timestamp": "2023-10-27T10:00:00Z",
  "payload": {
    "userId": "user-123",
    "apiKeyId": "key-456",
    "endpoint": "/api/v1/users/{id}",
    "method": "GET",
    "requestCount": 1,
    "rateLimitBucket": {
      "limit": 1000,
      "remaining": 999,
      "resetTime": "2023-10-27T10:01:00Z"
    },
    "isThrottled": false,
    "ipAddress": "192.168.1.1"
  }
}

Dead letter queue strategy and poison message handling

We implement a dead letter queue (DLQ) for messages that fail processing after maximum retries. Poison messages are identified by tracking failed attempts in a separate error tracking system.

• DLQ Topic: api-throttle-dlq
• Retry Limit: 3 attempts with exponential backoff
• Error Handling: Messages that fail after retries are moved to DLQ and logged with metadata for manual inspection
• Monitoring: Alerts triggered when DLQ queue size exceeds threshold (e.g., 100 messages)

At-least-once vs. exactly-once delivery — decision with justification for this feature

We choose at-least-once delivery semantics for the throttling service because:

1. Idempotency: Rate limit decisions are idempotent - reprocessing the same request won't change the outcome
2. Performance: Exactly-once requires distributed transaction coordination which adds significant complexity and latency
3. User Experience: A user making the same request twice will get the same throttling result, which is acceptable
4. Simpler Architecture: Avoids complexity of maintaining state across multiple systems

Retry policy for consumers (max attempts, backoff formula, non-retryable error classification)

public class RetryPolicy {
    private static final int MAX_RETRIES = 3;
    private static final long INITIAL_DELAY_MS = 1000;
    private static final double BACKOFF_MULTIPLIER = 2.0;
    private static final double JITTER_RANGE = 0.1; // ±10% jitter

    public static boolean shouldRetry(Exception exception) {
        return !(exception instanceof InvalidRequestException || 
                exception instanceof UnauthorizedAccessException);
    }

    public static long getDelay(int attempt) {
        long baseDelay = (long) (INITIAL_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempt));
        long jitter = (long) (baseDelay * (Math.random() * 2 * JITTER_RANGE - JITTER_RANGE));
        return baseDelay + jitter;
    }
}

Outbox pattern for reliable event publishing — include or explicitly justify omission

We do not use the outbox pattern in this design because:

1. Simplicity: The throttling service is a core component where reliability is ensured through message broker acknowledgment and retry mechanisms
2. Performance: Outbox pattern adds an extra database write operation per event, which would increase latency for high-throughput scenarios
3. Message broker reliability: Modern brokers like Kafka or RabbitMQ provide strong durability guarantees when properly configured

Instead, we rely on:

• Message broker acknowledgments
• Retry policies with exponential backoff
• Dead letter queue handling

Sequence diagram for all async flows (Mermaid)

sequenceDiagram
    participant Client
    participant ThrottleService
    participant MessageBroker
    participant AuditConsumer
    participant MetricsConsumer

    Client->>ThrottleService: API Request
    ThrottleService->>ThrottleService: Check Rate Limits
    ThrottleService->>MessageBroker: Publish Usage Event
    MessageBroker->>AuditConsumer: Async Audit Log
    MessageBroker->>MetricsConsumer: Async Metrics Update

    loop Retry Logic
        AuditConsumer-->>MessageBroker: Failed Processing
        MessageBroker->>MessageBroker: Retry with Exponential Backoff
    end

    MessageBroker->>MessageBroker: DLQ on Max Retry

10. Error Handling Design

Error Classification Hierarchy

The error handling strategy follows a hierarchical classification to enable proper routing and response generation:

Error
├── ValidationException (400)
│   ├── InvalidInputException
│   └── MissingFieldException
├── AuthorizationException (403)
│   ├── InsufficientPermissionException
│   └── InvalidTokenException
├── ResourceNotFoundException (404)
│   ├── ApiKeyNotFoundException
│   └── RateLimitConfigurationNotFoundException
├── RateLimitExceededException (429)
│   ├── ApiRateLimitExceeded
│   └── UserRateLimitExceeded
├── InfrastructureException (500)
│   ├── DatabaseException
│   ├── CacheException
│   └── ExternalServiceException
└── BusinessLogicException (400)
    ├── InvalidRateLimitConfiguration
    └── ConcurrentModificationException

Error Propagation Strategy

Errors follow a "bubble up" pattern with specific handling at each layer:

1. Repository Layer: Wrap database-specific exceptions in DatabaseException
2. Service Layer: Catch and rethrow domain-specific errors with appropriate HTTP status codes
3. Controller Layer: Catch all exceptions and return standardized JSON responses

Retry Strategy with Exponential Backoff and Jitter

For infrastructure errors (e.g., database, cache), the retry strategy uses exponential backoff with jitter:

public class RetryStrategy {
    private static final int MAX_RETRIES = 3;
    private static final long BASE_DELAY_MS = 1000;
    private static final double JITTER_FACTOR = 0.2;

    public <T> T executeWithRetry(Supplier<T> operation) {
        int attempts = 0;
        while (true) {
            try {
                return operation.get();
            } catch (InfrastructureException e) {
                if (attempts >= MAX_RETRIES) {
                    throw e;
                }
                long delay = calculateDelay(attempts);
                Thread.sleep(delay);
                attempts++;
            }
        }
    }

    private long calculateDelay(int attempt) {
        long delay = (long) Math.pow(2, attempt) * BASE_DELAY_MS;
        long jitter = (long) (delay * JITTER_FACTOR * (Math.random() - 0.5) * 2);
        return delay + jitter;
    }
}

Circuit Breaker Design

The circuit breaker pattern prevents cascading failures when external services are unavailable:

public class CircuitBreaker {
    private final int failureThreshold = 5;
    private final long timeout = 60000; // 1 minute
    private final int resetTimeout = 30000; // 30 seconds
    
    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private long lastFailureTime = 0;

    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }

    public <T> T execute(Supplier<T> operation) {
        switch (state) {
            case CLOSED:
                return executeWhenClosed(operation);
            case OPEN:
                if (System.currentTimeMillis() - lastFailureTime > resetTimeout) {
                    return executeWhenHalfOpen(operation);
                }
                throw new CircuitBreakerOpenException("Circuit is open");
            case HALF_OPEN:
                return executeWhenHalfOpen(operation);
        }
        throw new IllegalStateException("Invalid circuit state");
    }

    private <T> T executeWhenClosed(Supplier<T> operation) {
        try {
            T result = operation.get();
            failureCount = 0;
            return result;
        } catch (Exception e) {
            failureCount++;
            lastFailureTime = System.currentTimeMillis();
            if (failureCount >= failureThreshold) {
                state = CircuitState.OPEN;
            }
            throw e;
        }
    }

    private <T> T executeWhenHalfOpen(Supplier<T> operation) {
        try {
            T result = operation.get();
            state = CircuitState.CLOSED;
            failureCount = 0;
            return result;
        } catch (Exception e) {
            state = CircuitState.OPEN;
            lastFailureTime = System.currentTimeMillis();
            throw e;
        }
    }
}

Fallback Behavior Design

Fallbacks are implemented for external dependencies:

public class RateLimitFallback {
    private final Cache<String, RateLimit> fallbackCache;

    public RateLimit getRateLimit(String apiKey) {
        try {
            return fallbackCache.getIfPresent(apiKey);
        } catch (Exception e) {
            // Log and return default configuration
            logger.warn("Fallback cache unavailable for API key: {}", apiKey);
            return getDefaultRateLimit();
        }
    }

    private RateLimit getDefaultRateLimit() {
        return new RateLimit(1000, 60, "default");
    }
}

Error Logging Format and Alerting Thresholds

Structured logging follows this format:

{
  "timestamp": "2023-10-15T14:30:45.123Z",
  "trace_id": "123e4567-e89b-12d3-a456-426614174000",
  "span_id": "abcdef1234567890",
  "user_id": "user-123",
  "error_code": "RATE_LIMIT_EXCEEDED",
  "error_message": "API rate limit exceeded for key: api-key-123",
  "error_type": "RateLimitExceededException",
  "request_path": "/api/v1/throttle",
  "http_status": 429,
  "stack_trace": "...",
  "severity": "WARNING"
}

Alerting thresholds:

• Critical: More than 100 rate limit exceptions per minute
• Warning: More than 50 rate limit exceptions per minute
• Info: More than 10 infrastructure failures per hour

Error Response Format

All error responses follow a standardized JSON format:

{
  "error": {
    "code": "RATE_LIMIT_EXCEEDED",
    "message": "API rate limit exceeded for key: api-key-123",
    "http_status": 429,
    "timestamp": "2023-10-15T14:30:45.123Z",
    "request_id": "req-123"
  }
}

Error Handling in Controller Layer

@RestController
@RequestMapping("/api/v1/throttle")
public class ThrottlingController {
    
    @ExceptionHandler(RateLimitExceededException.class)
    public ResponseEntity<ErrorResponse> handleRateLimitExceeded(
            RateLimitExceededException ex) {
        ErrorResponse error = new ErrorResponse(
            "RATE_LIMIT_EXCEEDED",
            ex.getMessage(),
            HttpStatus.TOO_MANY_REQUESTS.value(),
            UUID.randomUUID().toString()
        );
        
        return ResponseEntity.status(HttpStatus.TOO_MANY_REQUESTS)
                           .body(error);
    }
    
    @ExceptionHandler(InfrastructureException.class)
    public ResponseEntity<ErrorResponse> handleInfrastructureError(
            InfrastructureException ex) {
        ErrorResponse error = new ErrorResponse(
            "INFRASTRUCTURE_ERROR",
            "Service temporarily unavailable. Please try again.",
            HttpStatus.INTERNAL_SERVER_ERROR.value(),
            UUID.randomUUID().toString()
        );
        
        // Log with severity
        logger.error("Infrastructure error occurred", ex);
        
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
                           .body(error);
    }
}

Error Handling in Service Layer

@Service
public class ThrottlingService {
    
    private final CircuitBreaker circuitBreaker = new CircuitBreaker();
    private final RetryStrategy retryStrategy = new RetryStrategy();
    
    public RateLimitResult checkRateLimit(String apiKey, String resource) {
        try {
            return circuitBreaker.execute(() -> {
                return retryStrategy.executeWithRetry(() -> {
                    // Check rate limits logic here
                    return checkRateLimitInternal(apiKey, resource);
                });
            });
        } catch (InfrastructureException e) {
            // Fallback to default rate limit
            logger.warn("Using fallback rate limit for API key: {}", apiKey);
            return new RateLimitResult(true, getDefaultRateLimit());
        } catch (RateLimitExceededException e) {
            throw e; // Re-throw rate limit exceptions
        } catch (Exception e) {
            logger.error("Unexpected error checking rate limit", e);
            throw new InfrastructureException("Failed to check rate limit", e);
        }
    }
}

Error Classification and Handling Summary

Error Type	HTTP Status	Retryable	Circuit Breaker	Fallback
ValidationException	400	No	No	No
AuthorizationException	403	No	No	No
ResourceNotFoundException	404	No	No	Yes (default)
RateLimitExceededException	429	No	No	No
InfrastructureException	500	Yes	Yes	Yes
BusinessLogicException	400	No	No	Yes (default)

This error handling design ensures robustness, proper error classification, and maintainable error responses while providing fallback mechanisms for resilience.

11. Security Design

Input Validation and Sanitization Strategy

All API inputs are validated using a combination of:

• Regex-based validation for format constraints (e.g., API key format, rate limit values)
• Length limits to prevent buffer overflow attacks (max 256 chars for strings, 1024 for payloads)
• Type checking to ensure correct data types
• Whitelisting for allowed characters and values (e.g., only alphanumeric + hyphens in resource IDs)

SQL Injection Prevention

• Prepared statements are used exclusively for all database queries
• Parameterized queries with no string concatenation in SQL generation
• ORM layer (Hibernate/MyBatis) is used to abstract away raw SQL construction
• Input sanitization for any dynamic query components (e.g., table names, column names) using a whitelist approach

Authentication Middleware Design

• JWT-based authentication for API key validation with:
  • 1-hour expiration time (configurable per client type)
  • Refresh token rotation with 7-day validity window
  • Secure HTTP-only cookies for web clients
• OAuth2 integration for third-party API access with:
  • Authorization code flow for web applications
  • Client credentials flow for machine-to-machine access
• API key validation performed at the entry point of all requests using a dedicated middleware component

Authorization Enforcement

• Role-Based Access Control (RBAC) implemented with:
  • Roles: admin, user, service_account
  • Permissions mapped to roles via a centralized permissions table
• Service-level authorization checks:
  • Admins can access all endpoints and modify any rate limit
  • Users can only view their own limits and modify their own keys
  • Service accounts can only access endpoints for which they're authorized

Sensitive Data Encryption at Rest and in Transit

• At-rest encryption using AES-256 with key rotation every 90 days
• In-transit encryption using TLS 1.3 with HSTS enabled (max age 31536000)
• Secrets storage in AWS Secrets Manager with:
  • Automatic rotation every 30 days
  • Encryption at rest using AWS KMS keys (KMS key rotation every 365 days)
• API key encryption using AES-256 with a dedicated key per client type

PII Handling and Masking Strategy

• PII data (user IDs, client names) is masked in logs and audit trails:
  • User IDs are hashed with salt using SHA-256
  • Client names are truncated to first 3 characters and masked with ***
• Log redaction implemented at the logging framework level:
  • All sensitive fields are excluded from structured logs
  • Only non-sensitive metadata (e.g., timestamp, operation type) is logged

Secrets Management

• AWS Secrets Manager for storing:
  • API keys and client secrets
  • Database credentials with automatic rotation
• Vault integration for:
  • Dynamic secrets (e.g., database credentials with TTL)
  • Service account tokens with rotation policies
• Secrets lifecycle management:
  • Rotation every 30 days for API keys
  • Automatic revocation of expired secrets
  • Audit trail of all secret access and modifications

Audit Logging Design

• Audit events captured for:
  • Rate limit changes (modified by, old value, new value)
  • API key creation/deletion
  • Access attempts (successful/failed)
• Event format includes:

  {
    "timestamp": "2023-10-27T14:30:00Z",
    "user_id": "hashed_user_id",
    "operation": "rate_limit_update",
    "resource_id": "api_key_abc123",
    "old_value": {
      "requests_per_minute": 100
    },
    "new_value": {
      "requests_per_minute": 200
    },
    "ip_address": "192.168.1.100",
    "user_agent": "Mozilla/5.0..."
  }

• Retention policy: 90 days for audit logs with daily backups to S3
• Compliance:
  • GDPR compliance with data minimization principles
  • Audit logs stored in encrypted format and accessible only to authorized personnel

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

• Service Layer: 85%+ line coverage, 100% branch coverage for core logic paths
• Repository Layer: 90%+ line coverage, focusing on query execution and error handling
• Business Logic: 95%+ code coverage with emphasis on edge cases and boundary conditions
• Validation Logic: 100% coverage for all input validation scenarios including malformed requests and invalid parameter combinations
• Rate Limiting Logic: 100% coverage for all rate limit states (normal, throttled, exceeded)
• Mocking Strategy: All external dependencies (database, cache, messaging) mocked with appropriate test doubles

Integration Testing Strategy

• Database Integration: Test using embedded H2 or PostgreSQL in-memory instances for schema validation and query correctness
• Cache Integration: Use Redis in-memory mode or mock cache implementations to verify caching behavior and consistency
• Messaging Integration: Utilize embedded Kafka or RabbitMQ containers for event-driven flow testing
• External Service Integration: Mock external services (e.g., authentication, billing) using WireMock or similar tools
• Cross-Service Integration: Validate interactions between throttling service and other services (e.g., auth, metrics) using test containers or contract testing

Contract Testing

• Pact Framework: Implement consumer-driven contracts between throttling service and API gateway or client services
• Contract Versioning: Maintain versioned contract files (v1, v2) to support backward compatibility during feature evolution
• Contract Validation: Run contract tests in CI pipeline to ensure service contracts remain valid with each change

Performance Testing Tool and Key Scenarios

• Tool: k6 (JavaScript-based load testing) with Grafana for visualization and alerting
• Key Scenarios:
  • Concurrent request load (1000+ concurrent requests) to validate rate limit enforcement
  • Burst traffic simulation (5x normal rate for 10 seconds) to test burst handling and token bucket behavior
  • High-throughput scenarios (50,000 requests/sec) to validate system limits and memory usage
  • Latency testing under various load conditions (p50/p95/p99 latency targets)
  • Memory leak detection during long-running test cycles (1 hour+)

Test Data Management Strategy

• Fixtures: Use YAML or JSON fixtures for predefined test data including API keys, rate limits, and user profiles
• Factories: Implement factory classes for generating test data with variations (e.g., different rate limits, user roles)
• Seeding: Use database seeding scripts to populate test environments with realistic data sets
• Data Isolation: Ensure each test runs in isolation using unique identifiers and proper cleanup between tests
• Test Data Rotation: Rotate test data periodically to prevent stale data from affecting test results

// Example unit test for rate limit enforcement logic
@Test
public void testRateLimitEnforcement() {
    // Given
    String apiKey = "test-api-key";
    int maxRequests = 100;
    long windowSeconds = 60;
    
    // When
    RateLimit rateLimit = new RateLimit(apiKey, maxRequests, windowSeconds);
    
    // Then
    assertEquals(maxRequests, rateLimit.getMaxRequests());
    assertEquals(windowSeconds, rateLimit.getWindowSeconds());
    assertNotNull(rateLimit.getTokens());
}

// Example integration test for database persistence
@Test
public void testRateLimitPersistence() {
    // Given
    String apiKey = "persist-test-key";
    int maxRequests = 50;
    
    // When
    RateLimit rateLimit = new RateLimit(apiKey, maxRequests, 300);
    rateLimitRepository.save(rateLimit);
    
    // Then
    Optional<RateLimit> saved = rateLimitRepository.findByApiKey(apiKey);
    assertTrue(saved.isPresent());
    assertEquals(maxRequests, saved.get().getMaxRequests());
}

13. Observability Design

Structured Logging Schema

The logging system will use structured JSON format to ensure consistency and enable efficient querying. All logs will include the following required fields:

{
  "timestamp": "2023-10-27T14:30:00.000Z",
  "level": "INFO|WARN|ERROR",
  "service": "api-throttling-service",
  "trace_id": "123e4567-e89b-12d3-a456-426614174000",
  "span_id": "abcdef1234567890",
  "user_id": "user-123",
  "entity_id": "api-key-abc123",
  "request_id": "req-xyz789",
  "method": "GET",
  "path": "/api/v1/throttle",
  "status_code": 200,
  "error_code": "THROTTLE_EXCEEDED",
  "error_message": "Rate limit exceeded for API key abc123",
  "duration_ms": 45,
  "ip_address": "192.168.1.1",
  "user_agent": "curl/7.64.1"
}

PII masking rules:

• User IDs and entity IDs will be masked in logs when not required for debugging
• IP addresses will be masked to CIDR block (e.g., 192.168.1.0/24)
• User agent strings will be truncated to first 50 characters

Distributed Tracing Instrumentation

Using OpenTelemetry for distributed tracing with custom spans:

Key operations to trace:

1. throttle_check - Main rate limiting decision point
2. rate_limit_fetch - Database fetch for rate limit configuration
3. token_bucket_update - Token bucket state update
4. api_key_validation - Authentication and authorization check

Span attributes to capture:

• throttle.api_key (masked)
• throttle.limit_type (requests/second, requests/hour)
• throttle.limit_value
• throttle.current_usage
• throttle.remaining_tokens
• throttle.reset_time
• throttle.is_allowed (boolean)
• throttle.error_code

Metrics Design

RED Method Metrics:

Metric Name	Type	Labels	Description
throttle_requests_total	Counter	api_key, limit_type, status	Total API requests processed
throttle_requests_blocked_total	Counter	api_key, limit_type, reason	Blocked requests due to throttling
throttle_request_duration_seconds	Histogram	api_key, limit_type	Request processing time
throttle_rate_limit_exceeded_total	Counter	api_key, limit_type	Rate limit violations
throttle_token_bucket_usage	Gauge	api_key, limit_type	Current token bucket usage
throttle_cache_hit_total	Counter	type	Cache hits vs misses

Additional Service Metrics:

• throttle_active_keys_count - Gauge - Current active API keys
• throttle_cache_miss_total - Counter - Cache misses for rate limit data
• throttle_error_total - Counter - Internal errors in throttling logic

Health Check Endpoint Design

GET /health
GET /ready  
GET /live
GET /startup

Response format:

{
  "status": "healthy|unhealthy",
  "timestamp": "2023-10-27T14:30:00.000Z",
  "service": "api-throttling-service",
  "version": "1.2.3",
  "checks": {
    "database": {
      "status": "healthy",
      "latency_ms": 25
    },
    "cache": {
      "status": "healthy",
      "latency_ms": 5
    },
    "metrics": {
      "status": "healthy"
    }
  }
}

Health check criteria:

• /live - Basic service liveness, responds within 100ms
• /ready - Service ready to serve traffic, DB and cache connectivity verified
• /health - Full health status including dependencies
• /startup - Service startup validation, validates all required configurations

Alerting Thresholds

Critical Alerts (require on-call escalation):

• throttle_rate_limit_exceeded_total > 100 in 5m (warn: 50, critical: 100)
• throttle_error_total > 5 in 1m (warn: 2, critical: 5)
• throttle_request_duration_seconds p99 > 1000ms (warn: 500ms, critical: 1000ms)
• throttle_cache_miss_total > 500 in 5m (warn: 100, critical: 500)

Warning Alerts (monitoring):

• throttle_requests_blocked_total > 10 in 1m (warn: 5, critical: 10)
• throttle_token_bucket_usage > 90% (warn: 80%, critical: 90%)
• throttle_cache_hit_total < 80% (warn: 70%, critical: 80%)

On-call escalation path:

1. First alert → Slack notification to on-call engineer
2. Second critical alert within 15m → SMS escalation to team lead
3. Third critical alert within 30m → Full team Slack channel notification

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• Rate limit enforcement granularity: Should limits be enforced per API key, per user, or per IP? This decision affects the complexity of the system and the level of control available to customers.
• Burst handling strategy: How should burst traffic be handled? Should we allow bursts up to a certain threshold or strictly enforce the rate limit even for bursts?
• Global vs. per-resource limits: Should we implement global rate limits or allow per-endpoint/resource limits? This impacts the flexibility of the service and the complexity of configuration.
• Customer tiering: Should we support different rate limit tiers (e.g., free, paid, enterprise) or enforce a flat rate limit across all customers?

Known trade-offs and justifications

• Memory vs. Persistence Trade-off: Using in-memory rate limit tracking provides better performance but loses state on service restarts. Storing limits in a database ensures persistence but introduces additional latency and complexity.
• Accuracy vs. Performance Trade-off: Real-time rate limit enforcement with distributed locks provides high accuracy but can introduce latency and contention. Token bucket or leaky bucket algorithms with local caches provide better performance at the cost of some accuracy.
• Consistency vs. Availability Trade-off: Strong consistency in rate limit tracking across distributed systems can impact availability during network partitions, while eventual consistency can lead to temporary over-limit scenarios.
• Granularity vs. Complexity Trade-off: Fine-grained rate limiting (per endpoint, per user, per IP) provides more control but increases system complexity and maintenance overhead.

Risks and mitigation strategies

• Cache stampede risk: When cache entries expire simultaneously, all requests may hit the database, causing a thundering herd. Mitigation includes probabilistic early expiry and distributed locks for cache warming.
• Race condition in rate limit calculation: Concurrent requests may cause race conditions in rate limit calculations. Mitigation involves optimistic locking or distributed atomic operations.
• System-wide rate limit exhaustion: If all rate limits are exhausted, the system may become unresponsive. Mitigation includes implementing circuit breaker patterns and fallback behaviors.
• Data inconsistency in distributed systems: Rate limit state may become inconsistent across nodes. Mitigation includes using distributed consensus protocols or eventual consistency with conflict resolution.

Technical debt acknowledged

• Hardcoded rate limit configurations: Current design assumes all rate limits are hardcoded, which makes the system inflexible for future configuration changes.
• Limited support for complex rate limit rules: The current implementation does not support advanced rate limit rules like sliding windows or composite limits, which may require future refactoring.
• No support for rate limit overrides: The system does not currently support per-user or per-client overrides, which may be needed for special customers or use cases.
• Limited observability: While basic metrics are included, more granular observability into individual rate limit violations and enforcement patterns would be beneficial for debugging and optimization.