API Gateway Low-Level Design for Scalable Microservices Architecture

An API Gateway serves as a critical entry point for microservices architectures, providing unified access control, traffic management, and cross-cutting concerns. This document presents a production-ready low-level design covering core components, data structures, and implementation strategies for building scalable API gateway systems.

1. Problem Statement & Requirements

Clarifying questions

1. What scale expectations are there for concurrent requests and throughput?
   • Are we targeting millions of requests per second or more modest scale?
2. What consistency and availability requirements exist for the API Gateway?
   • Is it a strict SLA requirement or more flexible?
3. What are the key entities and their relationships in the system?
   • Are we dealing with users, services, routes, or authentication tokens?
4. What is the expected read/write ratio for the gateway?
   • Is it primarily routing requests or also handling auth, rate limiting, etc.?
5. How latency-sensitive is the API Gateway?
   • Are there strict p99 latency targets (e.g., <50ms)?
6. What authentication and authorization model is in use?
   • JWT, OAuth2, API keys, or custom tokens?
7. Are there any external integrations or third-party services the gateway must interact with?
   • How are these managed (e.g., OAuth2 flows, service discovery)?

Functional requirements

• Request routing to backend services based on path and method
• Authentication and authorization of incoming requests (JWT, OAuth2, API keys)
• Rate limiting per user/service to prevent abuse
• Request/response transformation (e.g., header manipulation, payload conversion)
• Caching of responses to reduce backend load
• Monitoring and logging of all requests for observability
• Security headers injection (e.g., CORS, Content-Security-Policy)
• API versioning support (URL path, header, query param)
• Fallback responses for failed backend services
• Idempotency handling for critical mutations

Non-functional requirements

Requirement	Target
p50 latency	< 20ms
p95 latency	< 50ms
p99 latency	< 100ms
Throughput	> 50,000 requests/sec
Availability	99.9% (5-mins downtime max/year)
Error rate	< 0.01%
Concurrent requests	> 100,000 simultaneously

Constraints and assumptions

• Assumption: Backend services are stateless and can scale independently.
• Assumption: The gateway is deployed in a multi-zone, high-availability setup.
• Constraint: Gateway must support multiple authentication methods (JWT, OAuth2, API keys).
• Constraint: All requests must be processed within 100ms to meet SLA.
• Assumption: The system will use a centralized configuration store (e.g., Consul or etcd).
• Constraint: Gateway must support both synchronous and asynchronous request handling.

Out of scope

• Service discovery (e.g., integration with Eureka or Consul)
• API documentation generation (Swagger/OpenAPI)
• Full request body transformation (e.g., JSON to XML conversion)
• Custom business logic within the gateway (e.g., custom validation rules)
• Multi-region support or cross-cloud deployment
• Request body size limits enforcement (handled by backend services)
• API schema validation (e.g., JSON Schema, OpenAPI schema checks)

2. API Design

API Style Selection

For an API Gateway, I'll select gRPC with Protocol Buffers as the primary communication protocol. This choice provides:

• Strong typing and automatic code generation
• Efficient binary serialization (lower bandwidth usage)
• Built-in support for streaming and bidirectional communication
• First-class support for versioning through proto files
• Better performance characteristics for high-throughput scenarios

REST is considered but insufficient for the gateway's role as a centralized traffic control point with complex routing and transformation logic.

Complete Endpoint Definitions

Method	Path	Description	Auth
POST	/v1/gateway/routes	Create new route configuration	JWT
GET	/v1/gateway/routes/{id}	Get route by ID	JWT
PUT	/v1/gateway/routes/{id}	Update route configuration	JWT
DELETE	/v1/gateway/routes/{id}	Delete route configuration	JWT
GET	/v1/gateway/routes	List all routes with pagination	JWT
POST	/v1/gateway/health	Health check endpoint	None
POST	/v1/gateway/metrics	Retrieve gateway metrics	JWT
POST	/v1/gateway/config	Get current configuration	JWT

Request and Response Schemas

Route Creation Request

{
  "id": "uuid",
  "name": "string",
  "path": "/api/v1/users",
  "method": "GET|POST|PUT|DELETE",
  "targetService": {
    "name": "string",
    "url": "string"
  },
  "middleware": [
    {
      "type": "rate_limit",
      "config": {
        "requestsPerSecond": 100,
        "burst": 50
      }
    },
    {
      "type": "auth",
      "config": {
        "requiredScopes": ["read:user"]
      }
    }
  ],
  "timeoutMs": 5000,
  "retryPolicy": {
    "maxRetries": 3,
    "backoffMs": 100
  }
}

Route Creation Response

{
  "id": "uuid",
  "name": "string",
  "path": "/api/v1/users",
  "method": "GET|POST|PUT|DELETE",
  "targetService": {
    "name": "string",
    "url": "string"
  },
  "middleware": [
    {
      "type": "rate_limit",
      "config": {
        "requestsPerSecond": 100,
        "burst": 50
      }
    }
  ],
  "timeoutMs": 5000,
  "retryPolicy": {
    "maxRetries": 3,
    "backoffMs": 100
  },
  "createdAt": "timestamp",
  "updatedAt": "timestamp"
}

Error Response Format

{
  "error": {
    "code": "INVALID_ARGUMENT",
    "message": "Invalid input parameter: path",
    "details": [
      {
        "field": "path",
        "message": "Path must start with /"
      }
    ]
  }
}

HTTP Status Codes

Code	Description
200	Successful GET, PUT, PATCH
201	Successful POST
204	Successful DELETE
400	Bad Request - Invalid input
401	Unauthorized - Missing or invalid auth
403	Forbidden - Insufficient permissions
404	Not Found - Resource not found
429	Too Many Requests - Rate limit exceeded
500	Internal Server Error
503	Service Unavailable

API Versioning Strategy

Versioning will be done through the path: /v1/gateway/... with support for multiple versions:

• v1: Current stable version
• v2: Future version with breaking changes

Versioning will be enforced at the service level, allowing for independent evolution of different gateway components.

Pagination Design

Cursor-based pagination will be used with the following structure:

{
  "data": [ /* route objects */ ],
  "nextCursor": "string",
  "hasMore": true
}

Justification:

• Cursor-based is more efficient for large datasets and handles data changes better
• Provides consistent results even when items are inserted/deleted between queries

Authentication and Authorization

All administrative endpoints require JWT tokens with scopes:

• gateway:read - Read access to configuration
• gateway:write - Write access to configuration

API keys will be supported for service-to-service authentication with rate limiting.

Rate Limiting

Rate limiting per endpoint:

• Admin endpoints: 100 RPS (burst: 50)
• Service endpoints: 1000 RPS (burst: 200)
• Health check: 5000 RPS (burst: 1000)

Rate limiting will be implemented using the Token Bucket algorithm with distributed counters.

Idempotency Design

Idempotency will be handled via:

1. Idempotency-Key header in requests
2. Store request metadata with TTL of 5 minutes
3. For duplicate requests, return cached response if same key and parameters

// Example idempotency handling in gRPC service
public Response handleRequest(Request request, String idempotencyKey) {
    if (idempotencyKey != null) {
        Response cached = cache.get(idempotencyKey);
        if (cached != null) {
            return cached;
        }
    }
    
    Response result = processRequest(request);
    
    if (idempotencyKey != null) {
        cache.put(idempotencyKey, result, Duration.ofMinutes(5));
    }
    
    return result;
}

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    API_GATEWAY {
        string id PK
        string name
        string description
        string base_url
        string status
        timestamp created_at
        timestamp updated_at
    }

    ROUTE {
        string id PK
        string api_gateway_id FK
        string path
        string method
        string target_service
        string target_endpoint
        int timeout_ms
        bool is_active
        timestamp created_at
        timestamp updated_at
    }

    AUTH_CONFIG {
        string id PK
        string route_id FK
        string auth_type
        string provider
        string jwt_issuer
        string jwt_audience
        string jwks_url
        timestamp created_at
        timestamp updated_at
    }

    RATE_LIMIT_CONFIG {
        string id PK
        string route_id FK
        int requests_per_second
        int requests_per_minute
        int requests_per_hour
        string window_type
        timestamp created_at
        timestamp updated_at
    }

    MIDDLEWARE {
        string id PK
        string route_id FK
        string middleware_type
        string config
        int order
        timestamp created_at
        timestamp updated_at
    }

    API_GATEWAY ||--o{ ROUTE : "has"
    ROUTE ||--o{ AUTH_CONFIG : "has"
    ROUTE ||--o{ RATE_LIMIT_CONFIG : "has"
    ROUTE ||--o{ MIDDLEWARE : "has"

Table Schema Definitions

CREATE TABLE api_gateways (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL,
    description TEXT,
    base_url VARCHAR(1023) NOT NULL,
    status VARCHAR(50) NOT NULL DEFAULT 'active',
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE routes (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    api_gateway_id UUID NOT NULL REFERENCES api_gateways(id) ON DELETE CASCADE,
    path VARCHAR(1023) NOT NULL,
    method VARCHAR(10) NOT NULL,
    target_service VARCHAR(255) NOT NULL,
    target_endpoint VARCHAR(1023) NOT NULL,
    timeout_ms INTEGER NOT NULL DEFAULT 5000,
    is_active BOOLEAN NOT NULL DEFAULT TRUE,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(api_gateway_id, path, method)
);

CREATE TABLE auth_configs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    route_id UUID NOT NULL REFERENCES routes(id) ON DELETE CASCADE,
    auth_type VARCHAR(50) NOT NULL,
    provider VARCHAR(255),
    jwt_issuer VARCHAR(255),
    jwt_audience VARCHAR(255),
    jwks_url VARCHAR(1023),
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE rate_limit_configs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    route_id UUID NOT NULL REFERENCES routes(id) ON DELETE CASCADE,
    requests_per_second INTEGER NOT NULL DEFAULT 1000,
    requests_per_minute INTEGER NOT NULL DEFAULT 60000,
    requests_per_hour INTEGER NOT NULL DEFAULT 3600000,
    window_type VARCHAR(50) NOT NULL DEFAULT 'MINUTE',
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE middleware (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    route_id UUID NOT NULL REFERENCES routes(id) ON DELETE CASCADE,
    middleware_type VARCHAR(255) NOT NULL,
    config JSONB NOT NULL DEFAULT '{}',
    order INTEGER NOT NULL DEFAULT 0,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

Primary Key Strategy

UUIDs are used for all primary keys because:

• They provide global uniqueness without coordination between services
• They support horizontal scaling and sharding strategies
• They prevent enumeration attacks by making IDs unpredictable
• They eliminate the need for centralized ID generation services

Index Design

-- Composite indexes for common query patterns
CREATE INDEX idx_routes_api_gateway_path_method ON routes(api_gateway_id, path, method);
CREATE INDEX idx_routes_active ON routes(is_active);
CREATE INDEX idx_auth_configs_route_id ON auth_configs(route_id);
CREATE INDEX idx_rate_limit_configs_route_id ON rate_limit_configs(route_id);
CREATE INDEX idx_middleware_route_id ON middleware(route_id);

-- Partial indexes for performance optimization
CREATE INDEX idx_routes_active_path_method ON routes(api_gateway_id, path, method) WHERE is_active = TRUE;

-- Covering index for frequently accessed route data
CREATE INDEX idx_routes_covering ON routes(api_gateway_id, path, method, target_service, target_endpoint, timeout_ms, is_active);

Data Partitioning / Sharding Strategy

Sharding by API Gateway ID:

• Each API gateway is assigned to a specific shard
• Route lookups are performed within the correct shard using api_gateway_id
• This enables horizontal scaling across multiple gateway instances
• Data is partitioned by logical grouping (API gateways) rather than random distribution

Soft Delete vs. Hard Delete Decision

Soft delete is used for the following reasons:

• Audit trail preservation for compliance and debugging
• Ability to restore accidentally deleted configurations
• Support for versioning and rollback scenarios
• Reduced risk of data loss in production environments

Audit Trail Design

All table modifications are tracked with:

• created_at and updated_at timestamps
• Audit logs stored in a separate audit_logs table with:
  • entity_id
  • entity_type
  • operation_type (CREATE, UPDATE, DELETE)
  • changed_fields
  • old_values
  • new_values
  • user_id (when available)
  • timestamp

Data Archival Strategy and Retention Policy

• Active configurations: Retained indefinitely with automatic backups
• Historical audit logs:
  • 30 days in primary storage (hot tier)
  • 1 year in secondary storage (warm tier)
  • 5 years in cold storage (archive tier)
• Archival process: Automated nightly job that moves audit logs to appropriate storage tiers
• Compliance: Full audit trail retention for 7 years as required by regulations

4. Class Design & Service Layer

classDiagram
    class ApiGateway {
        +handleRequest()
        +validateRequest()
        +routeRequest()
        +applyRateLimiting()
        +applyAuthentication()
        +applyAuthorization()
        +publishAuditLog()
    }
    
    class RequestHandler {
        +handle()
    }
    
    class RouteManager {
        +loadRoutes()
        +findRoute()
        +validateRoute()
    }
    
    class AuthManager {
        +authenticate()
        +authorize()
        +validateToken()
    }
    
    class RateLimiter {
        +checkRateLimit()
        +updateUsage()
        +resetWindow()
    }
    
    class CircuitBreaker {
        +call()
        +onSuccess()
        +onFailure()
        +getState()
    }
    
    class EventPublisher {
        +publish()
        +queueEvent()
    }
    
    class ConfigManager {
        +loadConfig()
        +getConfig()
        +watchChanges()
    }
    
    class MetricsCollector {
        +collectMetrics()
        +incrementCounter()
        +recordDuration()
    }
    
    class RequestValidator {
        +validate()
        +sanitize()
        +transform()
    }
    
    ApiGateway --> RequestHandler : delegates to
    ApiGateway --> RouteManager : uses
    ApiGateway --> AuthManager : uses
    ApiGateway --> RateLimiter : uses
    ApiGateway --> CircuitBreaker : uses
    ApiGateway --> EventPublisher : publishes
    ApiGateway --> ConfigManager : reads config from
    ApiGateway --> MetricsCollector : collects metrics from
    ApiGateway --> RequestValidator : validates with
    
    class Route {
        +id
        +path
        +method
        +targetService
        +timeoutMs
        +rateLimit
    }
    
    class AuthConfig {
        +type
        +issuer
        +audience
        +jwksUri
    }
    
    class RateLimitConfig {
        +limit
        +windowMs
        +burst
    }
    
    class Token {
        +value
        +expiresAt
        +claims
    }
    
    Route --> AuthConfig : has auth config
    Route --> RateLimitConfig : has rate limit config
    AuthManager --> Token : validates token

stateDiagram-v2
    [*] --> Idle
    Idle --> Processing : request received
    Processing --> Validating : validate request
    Validating --> Routing : route found
    Routing --> Authenticated : auth successful
    Authenticated --> Authorized : authorization passed
    Authorized --> Forwarding : forward to service
    Forwarding --> Completed : response received
    Forwarding --> Failed : timeout/error
    Failed --> Retry : retry logic
    Retry --> Completed : retry succeeded
    Retry --> Failed : retry failed
    Failed --> Error : error handling
    Completed --> [*]
    Error --> [*]

Service Class Structure and Responsibilities

ApiGatewayService: Main entry point for all API gateway operations. Handles request routing, authentication, authorization, rate limiting, and metrics collection.

RouteManager: Manages route configuration and lookup. Responsible for loading routes from config store, finding matching routes, and validating route configurations.

AuthManager: Handles authentication and authorization logic. Validates JWT tokens, checks permissions, and manages token refresh logic.

RateLimiter: Implements rate limiting for API endpoints. Tracks usage per client and enforces limits based on configured policies.

CircuitBreaker: Implements circuit breaker pattern for downstream service calls to prevent cascading failures.

Interface / Abstract Class Design

public interface RequestHandler {
    CompletableFuture<HttpResponse> handleAsync(HttpRequest request);
}

public interface RouteResolver {
    Route resolveRoute(String path, String method);
}

public interface Authenticator {
    CompletableFuture<AuthenticationResult> authenticate(HttpRequest request);
}

public interface Authorizer {
    CompletableFuture<Boolean> authorize(AuthenticationResult auth, Route route);
}

public interface RateLimiter {
    CompletableFuture<RateLimitResult> checkRateLimit(String clientId, String routeId);
}

public interface CircuitBreaker {
    CompletableFuture<HttpResponse> callWithCircuitBreaker(
        String serviceId, 
        HttpRequest request, 
        Supplier<CompletableFuture<HttpResponse>> operation
    );
}

Business Logic Encapsulation Patterns

Strategy Pattern for Authentication: Using Strategy pattern to support multiple authentication methods (JWT, API Keys, OAuth2).

public interface AuthenticationStrategy {
    CompletableFuture<AuthenticationResult> authenticate(HttpRequest request);
    boolean supports(String authType);
}

public class JwtAuthenticationStrategy implements AuthenticationStrategy {
    private final JwtValidator jwtValidator;
    
    public JwtAuthenticationStrategy(JwtValidator jwtValidator) {
        this.jwtValidator = jwtValidator;
    }
    
    @Override
    public CompletableFuture<AuthenticationResult> authenticate(HttpRequest request) {
        String token = extractToken(request);
        if (token == null) {
            return CompletableFuture.failedFuture(new AuthenticationException("Missing JWT token"));
        }
        
        return jwtValidator.validate(token)
            .thenApply(claims -> new AuthenticationResult(claims.getSubject(), claims));
    }
    
    @Override
    public boolean supports(String authType) {
        return "Bearer".equalsIgnoreCase(authType);
    }
    
    private String extractToken(HttpRequest request) {
        String authHeader = request.getHeader("Authorization");
        if (authHeader != null && authHeader.startsWith("Bearer ")) {
            return authHeader.substring(7);
        }
        return null;
    }
}

public class AuthenticationManager {
    private final List<AuthenticationStrategy> strategies;
    
    public AuthenticationManager(List<AuthenticationStrategy> strategies) {
        this.strategies = strategies;
    }
    
    public CompletableFuture<AuthenticationResult> authenticate(HttpRequest request) {
        String authType = getAuthType(request);
        return strategies.stream()
            .filter(strategy -> strategy.supports(authType))
            .findFirst()
            .orElseThrow(() -> new AuthenticationException("Unsupported auth type: " + authType))
            .authenticate(request);
    }
    
    private String getAuthType(HttpRequest request) {
        String authHeader = request.getHeader("Authorization");
        if (authHeader != null) {
            return authHeader.split(" ")[0];
        }
        return "Unknown";
    }
}

Critical-Path Method Implementation

public class ApiGatewayService {
    private final RouteManager routeManager;
    private final AuthManager authManager;
    private final RateLimiter rateLimiter;
    private final CircuitBreaker circuitBreaker;
    private final EventPublisher eventPublisher;
    private final MetricsCollector metricsCollector;
    private final RequestValidator requestValidator;
    
    public CompletableFuture<HttpResponse> handleRequest(HttpRequest request) {
        return CompletableFuture.supplyAsync(() -> {
            long startTime = System.nanoTime();
            String traceId = generateTraceId();
            
            try {
                // Validate request
                HttpResponse validatedRequest = requestValidator.validate(request);
                
                // Find route
                Route route = routeManager.findRoute(validatedRequest.getPath(), validatedRequest.getMethod());
                
                // Apply rate limiting
                String clientId = getClientId(validatedRequest);
                RateLimitResult rateLimitResult = rateLimiter.checkRateLimit(clientId, route.getId());
                if (!rateLimitResult.isAllowed()) {
                    throw new RateLimitExceededException("Rate limit exceeded for client: " + clientId);
                }
                
                // Authenticate
                AuthenticationResult auth = authManager.authenticate(validatedRequest)
                    .thenCompose(authResult -> {
                        // Apply authorization if needed
                        if (route.isAuthorizationRequired()) {
                            return authManager.authorize(authResult, route)
                                .thenApply(authorized -> authorized ? authResult : 
                                    CompletableFuture.failedFuture(new AuthorizationException("Access denied")));
                        }
                        return CompletableFuture.completedFuture(authResult);
                    })
                    .join();
                
                // Forward to target service with circuit breaker
                HttpResponse response = circuitBreaker.callWithCircuitBreaker(
                    route.getTargetServiceId(),
                    validatedRequest,
                    () -> forwardToService(route, validatedRequest)
                ).join();
                
                // Publish audit log
                eventPublisher.publish(new AuditEvent(
                    traceId,
                    "REQUEST_HANDLED",
                    request.getPath(),
                    auth.getPrincipal(),
                    System.currentTimeMillis()
                ));
                
                // Collect metrics
                long duration = System.nanoTime() - startTime;
                metricsCollector.recordDuration("api_gateway_requests", duration);
                metricsCollector.incrementCounter("api_gateway_requests_total");
                
                return response;
                
            } catch (Exception e) {
                // Log error and publish audit event
                eventPublisher.publish(new AuditEvent(
                    traceId,
                    "REQUEST_ERROR",
                    request.getPath(),
                    null,
                    System.currentTimeMillis()
                ));
                
                metricsCollector.incrementCounter("api_gateway_errors_total");
                throw new GatewayException("Failed to handle request", e);
            }
        });
    }
    
    private CompletableFuture<HttpResponse> forwardToService(Route route, HttpRequest request) {
        // Implementation would make actual HTTP call to target service
        // This is a simplified version for demonstration
        return CompletableFuture.completedFuture(
            new HttpResponse(200, "Success", Collections.emptyMap(), null)
        );
    }
    
    private String getClientId(HttpRequest request) {
        // Extract client ID from authentication or headers
        return request.getHeader("X-Client-ID") != null ? 
            request.getHeader("X-Client-ID") : "anonymous";
    }
    
    private String generateTraceId() {
        return UUID.randomUUID().toString();
    }
}

Transaction Boundary Design

In API Gateway, the transaction boundary is defined by a single request processing lifecycle:

1. Request validation and sanitization
2. Route resolution and authorization
3. Rate limiting check
4. Authentication and authorization
5. Service call with circuit breaker protection
6. Response formatting and metrics collection

Each of these steps is atomic within the context of a single request, with proper error handling and rollback semantics where needed.

Service Interface Contracts

public interface ApiGatewayService {
    CompletableFuture<HttpResponse> handleRequest(HttpRequest request);
    void reloadConfiguration();
    HealthStatus getHealth();
}

Saga / Orchestration Pattern

For complex workflows involving multiple downstream services, the API gateway would implement a saga pattern to manage distributed transactions:

public class SagaManager {
    private final List<SagaStep> steps;
    
    public CompletableFuture<HttpResponse> executeSaga(SagaContext context) {
        return executeSteps(context, 0);
    }
    
    private CompletableFuture<HttpResponse> executeSteps(SagaContext context, int stepIndex) {
        if (stepIndex >= steps.size()) {
            return CompletableFuture.completedFuture(context.getResponse());
        }
        
        SagaStep step = steps.get(stepIndex);
        return step.execute(context)
            .thenCompose(result -> {
                if (result.isSuccess()) {
                    return executeSteps(context, stepIndex + 1);
                } else {
                    // Compensate previous steps
                    return compensateSteps(context, stepIndex - 1)
                        .thenCompose(compensated -> 
                            CompletableFuture.failedFuture(new SagaException("Saga failed at step: " + stepIndex)));
                }
            });
    }
    
    private CompletableFuture<Boolean> compensateSteps(SagaContext context, int stepIndex) {
        // Implementation for compensating failed steps
        return CompletableFuture.completedFuture(true);
    }
}

Domain Object / Value Object Design

public class Route {
    private final String id;
    private final String path;
    private final HttpMethod method;
    private final String targetServiceId;
    private final int timeoutMs;
    private final AuthConfig authConfig;
    private final RateLimitConfig rateLimitConfig;
    
    // Constructor, getters, equals, hashCode
}

public class AuthenticationResult {
    private final String principal;
    private final Map<String, Object> claims;
    
    // Constructor, getters
}

public class RateLimitResult {
    private final boolean allowed;
    private final long resetTimeMs;
    
    // Constructor, getters
}

public class HttpRequest {
    private final String path;
    private final HttpMethod method;
    private final Map<String, String> headers;
    private final byte[] body;
    
    // Constructor, getters
}

public class HttpResponse {
    private final int statusCode;
    private final String body;
    private final Map<String, String> headers;
    
    // Constructor, getters
}

5. Concurrency & Thread Safety

Concurrency Model

The API Gateway will operate in a highly concurrent environment with multiple simultaneous requests being processed. The concurrency model is based on:

• Event-driven architecture using Reactor pattern (Reactive Streams)
• Non-blocking I/O operations to handle thousands of concurrent connections efficiently
• Thread pool management with bounded executors for handling blocking operations
• Async/Await pattern for cooperative multitasking across request lifecycle

The gateway will use a single-threaded event loop model (similar to Netty or Vert.x) for I/O operations, with dedicated thread pools for CPU-intensive tasks like JWT validation and request transformation.

Race Condition Prevention Strategy

Race conditions are mitigated through:

1. Immutable objects for configuration and routing rules
2. Thread-safe collections (ConcurrentHashMap, CopyOnWriteArrayList) for runtime state
3. Atomic operations for counters and metrics updates
4. Synchronized access to shared mutable state using read-write locks where necessary
5. Immutable request/response contexts that are cloned when modified

Optimistic vs. Pessimistic Locking Selection

For the API Gateway, optimistic locking is preferred over pessimistic locking because:

• The gateway primarily handles read-heavy operations (routing, rate limiting, auth)
• Write operations are infrequent and mostly configuration updates
• High concurrency is expected with minimal contention
• Locking overhead would degrade performance for concurrent requests

Deadlock Prevention Strategy

Deadlock prevention mechanisms include:

1. Lock ordering - All locks acquired in consistent order across the system
2. Timeout-based acquisition - Maximum wait time for locks (50ms) to prevent indefinite blocking
3. No nested locking - Avoid acquiring multiple locks in different orders
4. Thread-safe configuration updates using atomic reference updates instead of synchronized blocks

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is implemented through:

1. Idempotency key extraction from request headers (X-Idempotency-Key)
2. In-memory cache with TTL for tracking recent requests (5 minutes)
3. Thread-safe hash map for storing request fingerprints
4. Request deduplication logic that checks for existing processing before starting new work

public class IdempotencyManager {
    private final Map<String, CompletableFuture<GatewayResponse>> activeRequests = 
        new ConcurrentHashMap<>();
    private final ScheduledExecutorService cleanupScheduler = 
        Executors.newScheduledThreadPool(1);
    
    public CompletableFuture<GatewayResponse> handleRequest(
            GatewayRequest request, 
            Function<GatewayRequest, CompletableFuture<GatewayResponse>> processor) {
        
        String idempotencyKey = extractIdempotencyKey(request);
        if (idempotencyKey != null) {
            return activeRequests.computeIfAbsent(idempotencyKey, key -> {
                CompletableFuture<GatewayResponse> future = processor.apply(request);
                future.thenRun(() -> activeRequests.remove(key));
                return future;
            });
        }
        
        return processor.apply(request);
    }
    
    private String extractIdempotencyKey(GatewayRequest request) {
        return request.getHeader("X-Idempotency-Key");
    }
}

Thread Safety in Core Components

1. Route matching - Immutable route tables with thread-safe lookup
2. Rate limiting - Atomic counters and synchronized token bucket implementations
3. Authentication - Thread-safe JWT parsing and validation with caching
4. Request transformation - Immutable transformations applied in sequence without side effects

Performance Considerations

• Minimal thread contention through lock-free data structures where possible
• Bounded thread pools to prevent resource exhaustion under load
• Async processing chains that don't block the main event loop
• Memory-efficient concurrent collections to reduce GC pressure

Concurrency Testing Strategy

Concurrency testing includes:

1. Stress tests with 10,000+ concurrent requests
2. Race condition detection using tools like ThreadSanitizer or JMH
3. Load testing with varying concurrent connection counts (100, 1000, 5000)
4. Stress testing of critical paths (auth, rate limiting, routing) under high concurrency

The design ensures that the API Gateway can handle thousands of concurrent requests without performance degradation or data corruption, maintaining high availability and strong consistency guarantees where required.

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant Client
    participant APIGateway
    participant AuthN
    participant AuthZ
    participant ServiceA
    participant ServiceB

    Client->>APIGateway: HTTP Request (GET /api/users/123)
    APIGateway->>AuthN: Validate JWT
    AuthN-->>APIGateway: AuthN Response (valid/invalid)
    alt Valid Token
        APIGateway->>AuthZ: Check Permissions
        AuthZ-->>APIGateway: AuthZ Response (allowed/denied)
        alt Permission Granted
            APIGateway->>ServiceA: Forward Request (GET /users/123)
            ServiceA-->>APIGateway: Response (User Data)
            APIGateway-->>Client: HTTP 200 OK
        else Permission Denied
            APIGateway-->>Client: HTTP 403 Forbidden
        end
    else Invalid Token
        APIGateway-->>Client: HTTP 401 Unauthorized
    end

sequenceDiagram
    participant Client
    participant APIGateway
    participant AuthN
    participant ServiceA
    participant ServiceB
    participant EventPublisher

    Client->>APIGateway: HTTP Request (POST /api/orders)
    APIGateway->>AuthN: Validate JWT
    AuthN-->>APIGateway: AuthN Response (valid)
    APIGateway->>ServiceA: Forward Request (POST /orders)
    ServiceA-->>APIGateway: Response (Order Created)
    APIGateway->>EventPublisher: Publish OrderCreated Event
    EventPublisher-->>APIGateway: Acknowledge Event
    APIGateway-->>Client: HTTP 201 Created

sequenceDiagram
    participant Client
    participant APIGateway
    participant ServiceA
    participant ServiceB
    participant ServiceC

    Client->>APIGateway: HTTP Request (GET /api/users/123/orders)
    APIGateway->>ServiceA: Call /users/123
    ServiceA-->>APIGateway: Response (User Data)
    APIGateway->>ServiceB: Call /orders?userId=123
    ServiceB-->>APIGateway: Response (Orders List)
    APIGateway->>ServiceC: Call /payments?orderId=456
    ServiceC-->>APIGateway: Response (Payment Data)
    APIGateway-->>Client: HTTP 200 OK (Aggregated Response)

7. Caching Design

Cache Strategy Selection

For API Gateway caching, we'll use cache-aside pattern with a distributed Redis cluster as the primary cache store. This approach provides:

• Flexibility: Cache can be invalidated or updated at the application level
• Consistency: Cache invalidation is explicit and controlled
• Scalability: Redis cluster supports horizontal scaling and high availability

Cache Key Design

Cache keys will follow a structured namespace pattern:

{cache_namespace}:{resource_type}:{resource_id}:{version}:{context_hash}

Example:

api:gateway:route:12345:v1:abc123
api:gateway:policy:auth:default:v1:xyz789

TTL Strategy

Data Type	TTL	Reason
Route Configuration	5 minutes	Frequent updates during deployment
Authentication Policy	1 hour	Security policies change less frequently
Rate Limit Configuration	15 minutes	Dynamic rate limiting adjustments
API Metadata	24 hours	Static metadata changes infrequently
Response Cache	5-30 seconds	Depends on upstream API freshness requirements

Eviction Policy

We'll use LRU (Least Recently Used) eviction policy with:

• TTL-based fallback: Items expire after configured time
• Memory pressure handling: Redis will evict least recently used items when memory threshold is reached (80% of max memory)
• Cache warming: Pre-warm frequently accessed routes during deployment

Cache Stampede Prevention

To prevent cache stampedes:

1. Probabilistic Early Expiry: 20% of TTL is used for early expiry (e.g., 4 minutes instead of 5)
2. Distributed Locking: Use Redis distributed locks for cache population
3. Cache Entry Padding: Add padding to cache keys to prevent simultaneous refreshes

Distributed Cache Architecture

We'll implement a Redis Cluster with:

• 3 master nodes for read/write operations
• 3 replica nodes for high availability
• Automatic failover capability
• Cross-zone replication for disaster recovery

Cache Invalidation Strategy

We'll use event-driven invalidation:

1. Explicit Invalidations: When route configuration changes, invalidate all related cache entries
2. TTL-based Invalidation: Automatic expiration based on configured TTLs
3. Cache Invalidation Events: Publish cache invalidation events to a message queue for distributed systems

Cache Implementation Example (Java)

@Component
public class ApiGatewayCache {
    private final RedisTemplate<String, Object> redisTemplate;
    private final ObjectMapper objectMapper;
    
    public ApiGatewayCache(RedisTemplate<String, Object> redisTemplate, 
                          ObjectMapper objectMapper) {
        this.redisTemplate = redisTemplate;
        this.objectMapper = objectMapper;
    }
    
    public RouteConfig getRouteConfig(String routeId) {
        String cacheKey = buildCacheKey("route", routeId, "v1");
        
        // Try to get from cache
        RouteConfig cached = (RouteConfig) redisTemplate.opsForValue().get(cacheKey);
        if (cached != null) {
            return cached;
        }
        
        // Cache miss - acquire distributed lock
        String lockKey = cacheKey + ":lock";
        if (tryAcquireLock(lockKey, 5000)) {
            try {
                // Double-check cache after lock acquisition
                cached = (RouteConfig) redisTemplate.opsForValue().get(cacheKey);
                if (cached != null) {
                    return cached;
                }
                
                // Fetch from source and populate cache
                RouteConfig route = fetchRouteFromSource(routeId);
                redisTemplate.opsForValue().set(cacheKey, route, 300, TimeUnit.SECONDS);
                return route;
            } finally {
                releaseLock(lockKey);
            }
        }
        
        // Fallback to direct fetch if lock acquisition fails
        return fetchRouteFromSource(routeId);
    }
    
    private boolean tryAcquireLock(String lockKey, long timeoutMs) {
        String lockValue = UUID.randomUUID().toString();
        Boolean acquired = redisTemplate.opsForValue()
            .setIfAbsent(lockKey, lockValue, timeoutMs, TimeUnit.MILLISECONDS);
        return acquired != null && acquired;
    }
    
    private void releaseLock(String lockKey) {
        String script = "if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end";
        redisTemplate.execute(new DefaultRedisScript<>(script, Long.class), 
                             Collections.singletonList(lockKey), "lockValue");
    }
    
    private String buildCacheKey(String namespace, String resourceId, String version) {
        return String.format("api:gateway:%s:%s:%s", namespace, resourceId, version);
    }
    
    private RouteConfig fetchRouteFromSource(String routeId) {
        // Implementation to fetch from configuration store or database
        return new RouteConfig(); // Placeholder
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain objects and database operations. Each repository encapsulates CRUD operations for specific entities while handling database-specific concerns like connection management, query building, and result mapping.

public interface ApiGatewayRepository {
    ApiRoute save(ApiRoute route);
    Optional<ApiRoute> findById(String id);
    List<ApiRoute> findAll();
    void deleteById(String id);
    List<ApiRoute> findByServiceId(String serviceId);
}

@Repository
public class ApiGatewayRepositoryImpl implements ApiGatewayRepository {
    private final JdbcTemplate jdbcTemplate;
    private final ObjectMapper objectMapper;

    public ApiGatewayRepositoryImpl(JdbcTemplate jdbcTemplate, ObjectMapper objectMapper) {
        this.jdbcTemplate = jdbcTemplate;
        this.objectMapper = objectMapper;
    }

    @Override
    public ApiRoute save(ApiRoute route) {
        String sql = """
            INSERT INTO api_routes (id, service_id, path, method, target_url, 
                                   rate_limit, auth_required, created_at, updated_at)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
            ON CONFLICT (id) DO UPDATE SET
                service_id = EXCLUDED.service_id,
                path = EXCLUDED.path,
                method = EXCLUDED.method,
                target_url = EXCLUDED.target_url,
                rate_limit = EXCLUDED.rate_limit,
                auth_required = EXCLUDED.auth_required,
                updated_at = EXCLUDED.updated_at
            """;
        
        jdbcTemplate.update(sql, route.getId(), route.getServiceId(), route.getPath(),
                           route.getMethod(), route.getTargetUrl(), route.getRateLimit(),
                           route.isAuthRequired(), route.getCreatedAt(), route.getUpdatedAt());
        
        return route;
    }

    @Override
    public Optional<ApiRoute> findById(String id) {
        String sql = "SELECT * FROM api_routes WHERE id = ?";
        try {
            return Optional.of(jdbcTemplate.queryForObject(sql, new ApiRouteRowMapper(), id));
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }

    @Override
    public List<ApiRoute> findAll() {
        return jdbcTemplate.query("SELECT * FROM api_routes ORDER BY created_at DESC", 
                                 new ApiRouteRowMapper());
    }

    @Override
    public void deleteById(String id) {
        jdbcTemplate.update("DELETE FROM api_routes WHERE id = ?", id);
    }

    @Override
    public List<ApiRoute> findByServiceId(String serviceId) {
        return jdbcTemplate.query("SELECT * FROM api_routes WHERE service_id = ? ORDER BY created_at DESC", 
                                 new ApiRouteRowMapper(), serviceId);
    }
}

ORM vs Raw Query Decisions

For this system, raw SQL queries are preferred over ORM for the following reasons:

1. Performance: Raw queries provide better performance by avoiding unnecessary object mapping overhead
2. Control: Direct control over query execution and optimization
3. Complex Queries: Complex JOINs and aggregations are easier to implement with raw SQL
4. Database-Specific Features: Ability to leverage database-specific features like PostgreSQL JSONB operations

However, for simpler read operations where performance isn't critical, we use JPA entities with Hibernate for type safety and maintainability.

Read vs Write Separation

The API Gateway uses a read-write separation strategy with CQRS pattern for better scalability:

// Read-optimized view model
public class ApiRouteView {
    private String id;
    private String serviceId;
    private String path;
    private String method;
    private String targetUrl;
    private int rateLimit;
    private boolean authRequired;
    private LocalDateTime createdAt;
    private LocalDateTime updatedAt;
}

// Write operations remain in the main repository
public class ApiRouteWriteRepository {
    public void save(ApiRoute route) { /* ... */ }
}

Connection Pool Configuration

Connection pool is configured with:

• Max pool size: 20 connections
• Min idle: 5 connections
• Max lifetime: 30 minutes
• Validation query: SELECT 1
• Idle timeout: 5 minutes

@Configuration
public class DatabaseConfig {
    @Bean
    public DataSource dataSource() {
        HikariConfig config = new HikariConfig();
        config.setJdbcUrl("jdbc:postgresql://localhost:5432/apigateway");
        config.setUsername("user");
        config.setPassword("password");
        config.setMaximumPoolSize(20);
        config.setMinimumIdle(5);
        config.setMaxLifetime(1800000); // 30 minutes
        config.setIdleTimeout(300000); // 5 minutes
        config.setConnectionTestQuery("SELECT 1");
        return new HikariDataSource(config);
    }
}

N+1 Query Prevention Strategy

To prevent N+1 query problems, we implement eager loading and batch operations:

public class ApiRouteBatchService {
    public List<ApiRouteView> getRoutesWithServices(List<String> serviceIds) {
        // Single query to fetch all routes for services
        String sql = """
            SELECT r.*, s.name as service_name 
            FROM api_routes r 
            JOIN services s ON r.service_id = s.id 
            WHERE r.service_id IN (?)
            """;
        
        // Using batch processing for better performance
        return jdbcTemplate.query(sql, new ApiRouteWithServiceRowMapper(), serviceIds);
    }
}

Batch Operation Design

Batch operations are implemented for bulk data processing:

public class BatchApiRouteRepository {
    public void batchSave(List<ApiRoute> routes) {
        String sql = """
            INSERT INTO api_routes (id, service_id, path, method, target_url, 
                                   rate_limit, auth_required, created_at, updated_at)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
            ON CONFLICT (id) DO UPDATE SET
                service_id = EXCLUDED.service_id,
                path = EXCLUDED.path,
                method = EXCLUDED.method,
                target_url = EXCLUDED.target_url,
                rate_limit = EXCLUDED.rate_limit,
                auth_required = EXCLUDED.auth_required,
                updated_at = EXCLUDED.updated_at
            """;
        
        // Using batch update for better performance
        jdbcTemplate.batchUpdate(sql, routes, routes.size(), 
            (ps, route) -> {
                ps.setString(1, route.getId());
                ps.setString(2, route.getServiceId());
                ps.setString(3, route.getPath());
                ps.setString(4, route.getMethod());
                ps.setString(5, route.getTargetUrl());
                ps.setInt(6, route.getRateLimit());
                ps.setBoolean(7, route.isAuthRequired());
                ps.setTimestamp(8, Timestamp.valueOf(route.getCreatedAt()));
                ps.setTimestamp(9, Timestamp.valueOf(route.getUpdatedAt()));
            });
    }
}

Data Access Layer Interface Contracts

public interface ApiRouteRepository {
    ApiRoute save(ApiRoute route);
    Optional<ApiRoute> findById(String id);
    List<ApiRoute> findAll();
    void deleteById(String id);
    List<ApiRoute> findByServiceId(String serviceId);
    Page<ApiRoute> findAll(Pageable pageable);
    List<ApiRoute> findByPathAndMethod(String path, String method);
}

The repository layer provides clean abstraction over database operations while maintaining performance through optimized queries and proper connection management. The design supports both high-volume read scenarios and write-heavy operations with appropriate caching strategies.

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The API Gateway serves as a central hub for routing requests to downstream services and managing cross-cutting concerns. For asynchronous processing, we identify two primary boundaries:

1. Synchronous Request-Response Flow: The core gateway functionality where incoming requests are validated, authenticated, rate-limited, and routed to appropriate backend services. This remains synchronous and is the primary concern for this design.

2. Asynchronous Event Processing: Secondary flows that occur after request completion, such as:

   • Audit logging of API calls (user, endpoint, timestamp, response time)
   • Analytics event publishing (API usage statistics, error rates)
   • Metrics collection for monitoring and alerting
   • Notification services (e.g., sending email on API rate limit exceeded)
   • Integration with external systems (e.g., billing, fraud detection)

These asynchronous flows are decoupled from the main request path to maintain low latency for the primary user-facing operations.

Message schema design with sample payload

We use JSON Schema for message format due to its human-readable nature and broad support in messaging systems. Each event will include a standardized structure:

{
  "version": "1.0",
  "event_type": "api_call_processed",
  "timestamp": "2023-10-27T14:30:00Z",
  "event_id": "uuid-12345",
  "payload": {
    "user_id": "user-123",
    "api_key": "key-abcde",
    "endpoint": "/v1/users/123",
    "method": "GET",
    "response_code": 200,
    "response_time_ms": 150,
    "ip_address": "192.168.1.1",
    "user_agent": "Mozilla/5.0...",
    "error_code": null,
    "error_message": null
  }
}

Versioning is handled through the version field in the root object, allowing for backward compatibility and evolution of event schemas.

Dead letter queue strategy and poison message handling

Poison messages are handled via a dead-letter queue (DLQ) approach:

1. Retry Limit: Each async consumer will attempt processing up to 3 times with exponential backoff (1s, 4s, 9s).
2. DLQ Configuration: Messages that fail after max retries are moved to a dedicated DLQ topic with a retention period of 7 days.
3. Monitoring: DLQ messages trigger alerts to on-call engineers for manual inspection and potential reprocessing.
4. Reprocessing: A separate batch job periodically inspects DLQ messages and attempts to reprocess them with updated configurations or manual overrides.

At-least-once vs. exactly-once delivery

We implement at-least-once delivery for the following reasons:

• Simplicity: Easier to implement and maintain than exactly-once semantics.
• Idempotency in downstream services: Most downstream systems can handle duplicate events gracefully (e.g., by checking if the operation was already performed).
• Performance: Avoids the overhead of maintaining state for deduplication across distributed systems.

For critical operations like billing or financial transactions, we ensure idempotency at the service layer rather than relying on message brokers for exactly-once guarantees.

Retry policy for consumers

Each async consumer follows this retry policy:

• Max Attempts: 3 retries
• Backoff Strategy: Exponential with jitter (1s, 4s, 9s)
• Jitter: ±20% randomization to prevent thundering herd
• Non-Retryable Errors:
  • Authentication/authorization failures (401, 403)
  • Invalid input data errors (400)
  • Resource not found (404)

These errors are immediately sent to DLQ without retry attempts.

Outbox pattern for reliable event publishing

We implement the outbox pattern to ensure that events are published reliably:

1. When an async event needs to be published, it is first written to a local database table (event_outbox) within the same transaction as the primary business logic.
2. A separate background worker polls this table and publishes messages to the message broker.
3. Once successfully published, the event is marked as processed in the outbox table.

This ensures that if a failure occurs during message publishing, the event will be retried on system restart or worker recovery.

Sequence diagram for async flows

sequenceDiagram
    participant G as API Gateway
    participant DB as Event Outbox DB
    participant MQ as Message Broker
    participant A as Audit Service
    participant M as Metrics Service

    G->>DB: Write event to outbox table (sync)
    DB-->>G: Acknowledge write
    G->>MQ: Publish event to audit topic
    MQ-->>A: Deliver audit event
    A->>DB: Mark as processed in outbox
    G->>MQ: Publish event to metrics topic
    MQ-->>M: Deliver metrics event
    M->>DB: Mark as processed in outbox

10. Error Handling Design

Error Classification Hierarchy

The API Gateway implements a three-tier error classification system:

1. Client Errors (4xx) - Request validation, authentication/authorization failures
2. Server Errors (5xx) - Gateway internal failures, downstream service issues
3. System Errors - Infrastructure problems, configuration issues

public enum ErrorType {
    VALIDATION_ERROR,
    AUTHENTICATION_ERROR,
    AUTHORIZATION_ERROR,
    SERVICE_UNAVAILABLE,
    TIMEOUT_ERROR,
    INTERNAL_SERVER_ERROR,
    CONFIGURATION_ERROR
}

Error Propagation Strategy

Errors follow a "bubble up" pattern with specific handling at each layer:

• API Layer: Catch and transform to standardized format
• Service Layer: Re-throw domain-specific exceptions
• Repository Layer: Wrap infrastructure errors in gateway-specific exceptions

Retry Strategy with Exponential Backoff and Jitter

For transient downstream service errors (503, 429, network timeouts):

public class RetryConfig {
    private int maxAttempts = 3;
    private long initialDelayMs = 1000;
    private double multiplier = 2.0;
    private long maxDelayMs = 30000;
    private double jitterRatio = 0.2;
    
    public long getDelay(int attempt) {
        long delay = (long) (initialDelayMs * Math.pow(multiplier, attempt));
        delay = Math.min(delay, maxDelayMs);
        
        // Add jitter (±20%)
        long jitter = (long) (delay * jitterRatio * (Math.random() - 0.5) * 2);
        return delay + jitter;
    }
}

Circuit Breaker Design

Uses the Hystrix-style circuit breaker pattern to prevent cascading failures:

public class CircuitBreaker {
    private final String serviceId;
    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private long lastFailureTime = 0;
    private final int failureThreshold = 5;
    private final long timeoutMs = 60000;
    private final long resetTimeoutMs = 30000;
    
    public boolean canExecute() {
        if (state == CircuitState.OPEN) {
            if (System.currentTimeMillis() - lastFailureTime > resetTimeoutMs) {
                state = CircuitState.HALF_OPEN;
            } else {
                return false;
            }
        }
        return state != CircuitState.OPEN;
    }
    
    public void recordSuccess() {
        if (state == CircuitState.HALF_OPEN) {
            state = CircuitState.CLOSED;
        }
        failureCount = 0;
    }
    
    public void recordFailure() {
        if (state == CircuitState.CLOSED) {
            failureCount++;
            lastFailureTime = System.currentTimeMillis();
            
            if (failureCount >= failureThreshold) {
                state = CircuitState.OPEN;
            }
        } else if (state == CircuitState.HALF_OPEN) {
            state = CircuitState.OPEN;
        }
    }
}

Fallback Behavior Design

For critical downstream services, fallbacks provide graceful degradation:

public class FallbackHandler {
    public ResponseEntity<?> handleServiceUnavailable(String serviceId) {
        switch (serviceId) {
            case "user-service":
                return ResponseEntity.ok().body(
                    Map.of("message", "User data temporarily unavailable", "fallback", true));
            case "payment-service":
                return ResponseEntity.ok().body(
                    Map.of("message", "Payment processing unavailable", "status", "pending"));
            default:
                return ResponseEntity.status(503).body(
                    Map.of("error", "Service temporarily unavailable"));
        }
    }
}

Error Logging Format and Alerting Thresholds

Structured logging with trace context:

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);
    
    public void logError(String traceId, String spanId, ErrorType errorType, 
                        String message, Exception ex) {
        Map<String, Object> logEntry = Map.of(
            "timestamp", Instant.now(),
            "trace_id", traceId,
            "span_id", spanId,
            "error_type", errorType.name(),
            "message", message,
            "error_code", extractErrorCode(ex),
            "stack_trace", ex != null ? ex.toString() : null,
            "service", "api-gateway"
        );
        
        logger.error("API Gateway Error: {}", logEntry);
    }
    
    private String extractErrorCode(Exception ex) {
        if (ex instanceof ValidationException) return "VALIDATION_ERROR";
        if (ex instanceof AuthenticationException) return "AUTHENTICATION_FAILED";
        if (ex instanceof AuthorizationException) return "AUTHORIZATION_FAILED";
        return "INTERNAL_ERROR";
    }
}

Alerting Thresholds

Metric	Warning Level	Critical Level	Alert Channel
Gateway 5xx errors	>100/minute	>500/minute	On-call
Circuit breaker open	1+ service	5+ services	Slack
Retry failures	>50/minute	>200/minute	Email
Timeout errors	>20/minute	>100/minute	PagerDuty

Error Response Format

Standardized JSON response with consistent structure:

{
  "error": {
    "code": "SERVICE_UNAVAILABLE",
    "message": "Payment service is temporarily unavailable",
    "timestamp": "2023-12-01T10:30:00Z",
    "trace_id": "abc123def456",
    "details": {
      "service": "payment-service",
      "retry_count": 3,
      "original_error": "TimeoutException"
    }
  }
}

Error Handling in Gateway Flow

public ResponseEntity<?> handleRequest(
    HttpServletRequest request, 
    String serviceId) {
    
    try {
        // Apply rate limiting and auth
        validateRequest(request);
        
        // Forward to service with retry logic
        return executeWithRetry(request, serviceId);
        
    } catch (RateLimitExceededException e) {
        return ResponseEntity.status(429)
            .body(createErrorResponse("RATE_LIMIT_EXCEEDED", 
                "Request rate limit exceeded"));
    } catch (AuthenticationException e) {
        return ResponseEntity.status(401)
            .body(createErrorResponse("AUTHENTICATION_FAILED", 
                "Authentication required"));
    } catch (AuthorizationException e) {
        return ResponseEntity.status(403)
            .body(createErrorResponse("AUTHORIZATION_FAILED", 
                "Insufficient permissions"));
    } catch (ServiceUnavailableException e) {
        return ResponseEntity.status(503)
            .body(createErrorResponse("SERVICE_UNAVAILABLE", 
                "Service temporarily unavailable"));
    } catch (Exception e) {
        return ResponseEntity.status(500)
            .body(createErrorResponse("INTERNAL_ERROR", 
                "Internal gateway error"));
    }
}

11. Security Design

Input Validation and Sanitization Strategy

All API inputs must undergo strict validation using a combination of schema validation (JSON Schema, Protobuf) and custom sanitization logic. For path parameters and query strings, we'll use regex patterns to ensure only expected characters are present (alphanumeric, hyphens, underscores). For JSON payloads, we'll validate against defined schemas with field-level constraints.

public class InputValidator {
    private static final Pattern PATH_PARAM_PATTERN = Pattern.compile("^[a-zA-Z0-9_-]+$");
    private static final Pattern EMAIL_PATTERN = Pattern.compile("^[A-Za-z0-9+_.-]+@([A-Za-z0-9.-]+\\.[A-Za-z]{2,})$");
    
    public static boolean isValidPathParameter(String value) {
        return value != null && PATH_PARAM_PATTERN.matcher(value).matches();
    }
    
    public static boolean isValidEmail(String email) {
        return email != null && EMAIL_PATTERN.matcher(email).matches();
    }
}

SQL Injection Prevention

All database queries must use parameterized prepared statements with ORM or direct JDBC prepared statements. Never construct SQL strings by concatenating user inputs. We'll implement a query builder that enforces parameterization.

Authentication Middleware Design

Authentication will be handled by a global middleware layer that:

1. Extracts authentication tokens from Authorization header
2. Validates JWT tokens using HS256 signing algorithm
3. Verifies token expiration and audience claims
4. Populates authenticated user context in request scope

@Component
public class AuthenticationMiddleware {
    private final JwtService jwtService;
    
    public void handle(HttpServletRequest request, HttpServletResponse response, 
                      HandlerExecutionChain executionChain) throws Exception {
        String token = extractToken(request);
        if (token != null && jwtService.validateToken(token)) {
            String userId = jwtService.extractUserId(token);
            request.setAttribute("userId", userId);
        } else {
            throw new AuthenticationException("Invalid or missing token");
        }
    }
}

Authorization Enforcement

RBAC (Role-Based Access Control) will be implemented at service layer boundaries. Each endpoint will have role requirements defined in annotations:

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface RequiresRole {
    String[] value();
}

@Service
public class UserService {
    @RequiresRole({"ADMIN", "USER"})
    public User getUserProfile(String userId) {
        // Implementation
    }
}

Sensitive Data Encryption

All sensitive data at rest will be encrypted using AES-256-GCM. For API keys and tokens, we'll use a key rotation strategy with 90-day rotation cycles. Data in transit will be encrypted using TLS 1.3.

PII Handling and Masking Strategy

PII data (email, phone numbers) will be masked in logs and responses when not required for the operation. Email addresses will show only domain part, phone numbers will show last 4 digits.

Secrets Management

API gateway secrets (JWT signing keys, external service credentials) will be managed using AWS Secrets Manager with automatic rotation. Secrets will be fetched at startup and cached with refresh mechanism.

Audit Logging Design

All security-relevant events will be logged with structured JSON format including:

• Timestamp
• User ID (if available)
• IP address
• Action type (login, token_refresh, access_denied)
• Outcome (success/failure)
• Session ID

{
  "timestamp": "2023-10-15T14:30:00Z",
  "userId": "user_12345",
  "ipAddress": "192.168.1.100",
  "action": "token_refresh",
  "outcome": "success",
  "sessionId": "session_abc123"
}

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

• Service Layer: 85% code coverage with 100% branch coverage for core business logic paths
• Repository Layer: 90% coverage with focus on query execution and error handling
• API Layer: 80% coverage for request/response validation and routing logic
• Security Middleware: 95% coverage for authentication, authorization, and rate limiting logic
• Event Handlers: 85% coverage for async processing and error recovery paths

Integration Testing Strategy

• Test Containers: Use Docker-based test containers for Redis, database instances, and external service mocks (e.g., OAuth2 provider)
• Embedded Databases: H2 for unit tests, PostgreSQL for integration tests to ensure schema compatibility
• WireMock: For simulating external HTTP services (payment gateways, notification services)
• Test Data Isolation: Each test runs in isolated database schema with clean state setup/teardown
• End-to-end Test Coverage: 30% of critical user journeys with real HTTP requests through the gateway

Contract Testing

• Pact Framework: For consumer-driven contract testing between gateway and downstream services
• Versioned Contracts: Maintain contract versions aligned with service release cycles (e.g., v1.0, v2.0)
• Contract Validation: Run contract verification as part of CI pipeline before service deployment
• Consumer Verification: Gateway consumer tests ensure that expected API responses match contract expectations

Performance Testing Tool and Key Scenarios

• k6: Load testing tool for simulating concurrent API requests with realistic traffic patterns
• Key Scenarios:
  • 10,000 concurrent requests with 50% read, 50% write operations
  • 95th percentile latency under 200ms for all endpoints
  • Throughput of 10,000 RPS with 99.9% success rate
  • Memory and CPU utilization under load (target: <50% CPU, <2GB RAM)
• Test Data: Synthetic data generation for 1M+ users, 100K+ API keys, 500+ services

Test Data Management Strategy

• Fixtures: Predefined JSON/YAML files for common test data (users, services, policies)
• Factories: Factory classes for generating realistic test data with varying attributes (e.g., valid/invalid API keys)
• Seeding: Database seeding scripts for test environments with consistent baseline data
• Data Cleanup: Automated cleanup of test data after each test suite run to prevent cross-contamination
• Snapshot Testing: For API response consistency validation using golden files for regression testing

// Example unit test for rate limiting logic
@Test
public void testRateLimitingPerApiKey() {
    String apiKey = "test-key-123";
    // Setup mock clock to simulate time progression
    Clock mockClock = Clock.fixed(Instant.now(), ZoneId.systemDefault());
    
    // Simulate 100 requests within rate limit window
    for (int i = 0; i < 100; i++) {
        assertTrue(rateLimiter.isAllowed(apiKey, mockClock));
    }
    
    // Next request should be denied
    assertFalse(rateLimiter.isAllowed(apiKey, mockClock));
    
    // Advance time beyond window
    mockClock = Clock.fixed(Instant.now().plusSeconds(60), ZoneId.systemDefault());
    assertTrue(rateLimiter.isAllowed(apiKey, mockClock));
}

13. Observability Design

Structured Logging Schema

The API Gateway implements structured logging with consistent schema across all log entries to enable effective monitoring and debugging. Each log entry includes the following required fields:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO",
  "trace_id": "1234567890abcdef1234567890abcdef",
  "span_id": "abcdef1234567890",
  "service_name": "api-gateway",
  "user_id": "user-12345",
  "request_id": "req-98765",
  "endpoint": "/api/v1/users/123",
  "method": "GET",
  "status_code": 200,
  "duration_ms": 156,
  "error_code": null,
  "error_message": null,
  "ip_address": "192.168.1.1",
  "user_agent": "Mozilla/5.0...",
  "request_body_size": 0,
  "response_body_size": 1234
}

PII masking rules:

• User IDs are logged as user-XXXX (masked)
• IP addresses are masked to 192.168.X.X
• User agent strings are truncated to first 50 characters
• Request/response bodies are logged only for error cases or when explicitly enabled for debugging

Distributed Tracing Instrumentation

The gateway integrates with OpenTelemetry to provide end-to-end tracing across service boundaries:

Custom Spans:

• gateway.request.processing: Measures total time from request reception to response
• gateway.route.matching: Time spent matching route and applying middleware
• gateway.authorization.check: Authorization check duration
• gateway.rate.limiting: Rate limiting evaluation time
• gateway.forward.to.upstream: Time spent waiting for upstream service response

Span Attributes Captured:

span.setAttribute("gateway.route.id", routeId);
span.setAttribute("gateway.upstream.host", upstreamHost);
span.setAttribute("gateway.user.role", userRole);
span.setAttribute("gateway.request.size.bytes", requestSizeBytes);
span.setAttribute("gateway.response.size.bytes", responseSizeBytes);

Metrics Design

RED Method Metrics (Rate, Errors, Duration):

Metric Name	Type	Labels	Description
gateway_requests_total	Counter	{method, endpoint, status_code}	Total request count
gateway_requests_errors_total	Counter	{method, endpoint, error_type}	Request errors count
gateway_requests_duration_seconds	Histogram	{method, endpoint}	Request processing time
gateway_rate_limit_exceeded_total	Counter	{method, endpoint}	Rate limit violations
gateway_upstream_errors_total	Counter	{upstream_service, error_code}	Upstream service failures
gateway_cache_hits_total	Counter	{cache_type}	Cache hit count
gateway_active_requests	Gauge	{method, endpoint}	Concurrent active requests

Additional Metrics:

• gateway_auth_failures_total (Counter): Authentication/authorization failures
• gateway_request_body_size_bytes (Histogram): Request body size distribution
• gateway_response_body_size_bytes (Histogram): Response body size distribution

Health Check Endpoint Design

The gateway exposes three health endpoints with distinct purposes:

GET /health     - Application liveness check (basic readiness)
GET /ready      - Service readiness (dependencies ready)
GET /live       - Full system health (all components healthy)
GET /startup    - Startup validation (initial configuration check)

Response Format:

{
  "status": "healthy",
  "timestamp": "2023-10-27T14:30:45.123Z",
  "components": {
    "database": {
      "status": "healthy",
      "latency_ms": 23
    },
    "redis_cache": {
      "status": "healthy",
      "latency_ms": 12
    },
    "upstream_service_a": {
      "status": "unhealthy",
      "latency_ms": 0,
      "error": "connection timeout"
    }
  },
  "version": "1.2.3",
  "build_time": "2023-10-27T14:30:45Z"
}

Alerting Thresholds

Critical Alerts (Escalation to On-Call):

• gateway_requests_errors_total rate > 50 errors/minute for 5 minutes
• gateway_requests_duration_seconds p99 > 5000ms for 10 minutes
• gateway_upstream_errors_total rate > 20 errors/minute for 5 minutes
• gateway_cache_hits_total drop below 80% of total requests for 15 minutes

Warning Alerts:

• gateway_requests_errors_total rate > 10 errors/minute for 5 minutes
• gateway_requests_duration_seconds p99 > 2000ms for 10 minutes
• gateway_cache_hits_total drop below 60% of total requests for 15 minutes

On-Call Escalation Path:

1. First alert → Slack channel notification
2. Second alert within 30 minutes → On-call engineer SMS
3. Third alert within 1 hour → Escalation to engineering lead

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

• API Gateway vs. Service Mesh: Should the gateway be a standalone component or integrated into service mesh architecture? This decision affects deployment complexity, observability scope, and feature set availability.
• Authentication/Authorization Strategy: Should the gateway handle all authentication (JWT, OAuth2) or delegate to downstream services? This impacts security model consistency and performance characteristics.
• Rate Limiting Granularity: Should rate limiting be enforced at the gateway level or distributed across microservices? This affects system-wide consistency vs. per-service flexibility.
• Caching Strategy: Should caching be centralized at the gateway or distributed across services? This impacts cache invalidation strategy and consistency trade-offs.

Known trade-offs and justifications

Trade-off	Option 1 (Gateway)	Option 2 (Service Level)	Justification
Consistency	Centralized enforcement	Distributed enforcement	Gateway ensures consistent policy application across all services, but adds single point of failure risk
Performance	Additional hop in request path	No additional latency	Gateway adds network hop but enables cross-cutting concerns to be handled once
Flexibility	Predefined policies	Per-service customization	Gateway provides uniformity but service-level handling allows for domain-specific logic
Scalability	Single point of contention	Distributed scaling	Gateway can become bottleneck under high load, but service-level approach scales better

Risks and mitigation strategies

• Single Point of Failure: The gateway becomes a critical system component that can bring down entire service ecosystem if unavailable.

  • Mitigation: Implement multi-zone deployment, circuit breaker patterns, and health check monitoring with automatic failover to secondary gateways.

• Performance Bottleneck: High traffic can overwhelm the gateway, causing latency spikes or request drops.

  • Mitigation: Use horizontal scaling with load balancing, implement efficient caching strategies, and optimize request processing pipeline.

• Security Vulnerabilities: Gateway acts as entry point for all traffic, making it an attractive target for attacks.

  • Mitigation: Implement comprehensive input validation, rate limiting, DDoS protection, and regular security audits.

• Complexity of Feature Management: Adding features like authentication, rate limiting, and caching increases system complexity.

  • Mitigation: Use modular plugin architecture with clear separation of concerns and feature flags for gradual rollout.

Technical debt acknowledged

• Tight Coupling: Gateway implementation tightly couples to specific service implementations, making future refactoring more difficult.
• Feature Bloat: Adding too many cross-cutting concerns to the gateway increases maintenance burden and reduces clarity.
• Monitoring Gaps: Gateway may not capture all relevant telemetry from downstream services, leading to incomplete observability.
• Versioning Complexity: Managing multiple versions of gateway features and service contracts creates operational overhead.
• Configuration Drift: Manual configuration management across environments can lead to inconsistent behavior and deployment issues.

The gateway design must balance between providing comprehensive cross-cutting functionality while maintaining flexibility for future evolution. Key areas of technical debt include over-engineering of features that may never be used, and insufficient abstraction to allow for future architectural changes without major refactoring.