Job Retry Scheduler Design and Implementation

Comprehensive low-level design for a robust job retry scheduling system with exponential backoff, circuit breaker patterns, and distributed coordination.

PublishedJune 15, 2025
Length10,554 words
Reading time53 min read

Backend LLD

A job retry scheduler is essential for building resilient distributed systems that can handle transient failures gracefully while maintaining system reliability and minimizing data loss through intelligent retry strategies and circuit breaker patterns.

1. Problem Statement & Requirements

Clarifying questions

What is the expected scale of jobs? Are we talking millions of jobs per day or just thousands?
What are the consistency requirements? Should retries be strictly ordered, or can they happen in any order?
What types of jobs are we dealing with? Are they all idempotent, or do some have side effects that must not be repeated?
What is the read/write ratio for job data? Is it mostly writes (job creation) or reads (status checks)?
What latency requirements exist? Should job status updates be real-time or acceptable with 1-5 second delays?
What authentication model is in place? Will all job scheduling be via API keys, or will OAuth2 be used?
Are there any external system integrations? Do jobs interact with third-party APIs or services that might affect retry behavior?

Functional requirements

Schedule jobs to be retried after a configurable delay (e.g., 5s, 1m, 5m)
Support exponential backoff for failed jobs
Allow manual override of retry delay (admin override capability)
Track job status (pending, in-progress, failed, completed)
Support job cancellation before retry window
Notify job owners or system admins on job failure (optional)
Maintain job metadata including retry count, last error, and next retry time
Support bulk job scheduling and status updates

Non-functional requirements

Latency targets: 95th percentile < 100ms for job status queries, 99th percentile < 500ms
Throughput: Handle 10,000+ job scheduling operations per second
Availability: 99.9% uptime with graceful degradation during outages
Scalability: Support horizontal scaling across multiple instances
Durability: Job state must survive system restarts and crashes

Constraints and assumptions

Jobs are created with a unique identifier (UUID)
Retry logic is handled by the same system that schedules jobs
Jobs must be retried within a specific time window (e.g., 1 hour max)
The system assumes jobs are idempotent or can be safely retried
No external dependencies for job scheduling (e.g., no integration with external CRON systems)
Jobs are processed in a single-threaded or low-concurrency environment

Out of scope

Integration with external job schedulers (e.g., Quartz, Celery)
Complex job dependencies or workflows (e.g., DAGs)
UI for job monitoring and management
Job prioritization or queuing strategies beyond retry scheduling
Cross-cluster job distribution or replication
Real-time job execution monitoring (only status tracking)
Job data encryption at rest or in transit
Multi-tenant job scheduling (single tenant only)

2. API Design

API Style Selection

gRPC with Protocol Buffers is selected for the Job Retry Scheduler due to:

Strong typing and automatic code generation
Efficient binary serialization with smaller payload sizes
Built-in support for streaming and bidirectional communication
Native support for versioning through proto package management
Better performance characteristics for high-throughput systems

Endpoint Definitions

Method	Path	Description	Auth
CreateJobRetry	`POST /v1/jobs/retry`	Schedule a new job retry with configurable backoff	JWT
GetJobRetry	`GET /v1/jobs/retry/{id}`	Retrieve job retry status and metadata	JWT
ListJobRetries	`GET /v1/jobs/retry`	List all scheduled job retries with pagination	JWT
CancelJobRetry	`DELETE /v1/jobs/retry/{id}`	Cancel a pending job retry	JWT
UpdateJobRetry	`PUT /v1/jobs/retry/{id}`	Modify retry configuration for an existing job	JWT
RetryNow	`POST /v1/jobs/retry/{id}/retry`	Force immediate retry of a scheduled job	JWT

Request and Response Schemas

CreateJobRetryRequest

{
  "jobId": "string",
  "maxRetries": 3,
  "backoffStrategy": {
    "type": "LINEAR",
    "initialDelayMs": 1000,
    "multiplier": 2.0
  },
  "retryableErrors": ["TIMEOUT", "NETWORK_ERROR"],
  "scheduleAt": "2023-12-01T10:00:00Z"
}

CreateJobRetryResponse

{
  "id": "uuid",
  "jobId": "string",
  "status": "PENDING",
  "scheduleAt": "2023-12-01T10:00:00Z",
  "createdAt": "2023-12-01T09:00:00Z",
  "updatedAt": "2023-12-01T09:00:00Z"
}

GetJobRetryResponse

{
  "id": "uuid",
  "jobId": "string",
  "status": "PENDING",
  "maxRetries": 3,
  "currentRetryCount": 1,
  "backoffStrategy": {
    "type": "LINEAR",
    "initialDelayMs": 1000,
    "multiplier": 2.0
  },
  "retryableErrors": ["TIMEOUT", "NETWORK_ERROR"],
  "scheduleAt": "2023-12-01T10:00:00Z",
  "createdAt": "2023-12-01T09:00:00Z",
  "updatedAt": "2023-12-01T09:00:00Z",
  "lastAttempt": {
    "attemptNumber": 1,
    "error": "TIMEOUT",
    "timestamp": "2023-12-01T09:30:00Z"
  }
}

HTTP Status Codes and Error Response Format

Status Code	Description
200	Success
201	Created
400	Bad Request (validation errors)
401	Unauthorized
403	Forbidden
404	Not Found
429	Rate Limited
500	Internal Server Error

Standardized error response:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid job ID format",
    "details": [
      {
        "field": "jobId",
        "message": "Must be a valid UUID"
      }
    ]
  }
}

API Versioning Strategy

Semantic versioning: /v1/ prefix for current stable API
Backward compatibility: All v1 endpoints remain functional
Version negotiation: Accept Accept: application/vnd.jobretry.v1+json header

Pagination Design

Cursor-based pagination is used because:

More efficient for large datasets (O(1) vs O(n))
Handles concurrent data changes gracefully
Simplifies client-side state management

Response format:

{
  "data": [...],
  "pagination": {
    "cursor": "eyJ0eXBlIjogIkxPQ0FUSU9OIiwgInZhbHVlIjogIjEwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw## 2. API Design

### API Style Selection

REST is selected for its simplicity, wide adoption, and ease of debugging. It provides clear semantics for job scheduling operations and integrates well with existing infrastructure.

### API Endpoints

| Method | Path | Description | Auth |
|--------|------|-------------|------|
| POST | `/v1/jobs` | Create a new job with retry configuration | JWT |
| GET | `/v1/jobs/{id}` | Retrieve a job by ID | JWT |
| PUT | `/v1/jobs/{id}` | Update retry configuration for a job | JWT |
| DELETE | `/v1/jobs/{id}` | Cancel a scheduled job | JWT |
| POST | `/v1/jobs/{id}/retry` | Force retry of a failed job | JWT |
| GET | `/v1/jobs/{id}/history` | Get retry history for a job | JWT |

### Request and Response Schemas

#### Create Job Request
```json
{
  "name": "data-processing-job",
  "description": "Process daily data batch",
  "schedule": {
    "type": "cron",
    "expression": "0 0 * * *"
  },
  "retryConfig": {
    "maxRetries": 3,
    "backoffStrategy": "exponential",
    "initialDelaySeconds": 10,
    "maxDelaySeconds": 300
  },
  "payload": {
    "dataUrl": "s3://bucket/data.csv"
  }
}

Create Job Response

{
  "id": "123e4567-e89b-12d3-a456-426614174000",
  "name": "data-processing-job",
  "status": "scheduled",
  "createdAt": "2023-05-15T10:00:00Z",
  "nextRunAt": "2023-05-15T11:00:00Z"
}

Retry Job Request

{
  "force": true,
  "retryCount": 1
}

Retry Job Response

{
  "jobId": "123e4567-e89b-12d3-a456-426614174000",
  "status": "retrying",
  "retries": 1
}

HTTP Status Codes and Error Response Format

Code	Description
200	Success
201	Created
400	Bad Request
401	Unauthorized
403	Forbidden
404	Not Found
429	Rate Limited
500	Internal Server Error

Standardized error response:

{
  "error": {
    "code": "INVALID_INPUT",
    "message": "Retry count must be between 0 and 3",
    "details": {
      "field": "retryCount"
    }
  }
}

API Versioning Strategy

Versioned using path prefix (/v1/jobs). Future versions will use /v2/jobs or similar.

Pagination Design

Cursor-based pagination is used for history endpoints to avoid issues with data changes during pagination.

Authentication and Authorization

OAuth2 with JWT tokens for authentication. RBAC is used for authorization:

admin role can manage all jobs
user role can manage only their own jobs

Rate Limiting

Rate limiting per endpoint:

Create/Update/Delete: 100 requests/minute
Get: 500 requests/minute

Rate limiting enforced using Redis with token bucket algorithm.

Idempotency Design

Idempotency keys are supported for job creation and retry operations:

X-Idempotency-Key header is used
Keys are stored in Redis with TTL of 1 hour
Duplicate requests return same response

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    JOB {
        UUID id PK
        String name
        String description
        String status
        DateTime createdAt
        DateTime updatedAt
        DateTime nextRunAt
        String payload
        UUID createdBy
    }

    RETRY_CONFIG {
        UUID jobId FK
        Integer maxRetries
        String backoffStrategy
        Integer initialDelaySeconds
        Integer maxDelaySeconds
    }

    JOB_HISTORY {
        UUID id PK
        UUID jobId FK
        String status
        String error
        DateTime executedAt
        Integer attemptNumber
    }

    JOB {
        UUID id PK
        String name
        String description
        String status
        DateTime createdAt
        DateTime updatedAt
        DateTime nextRunAt
        String payload
        UUID createdBy
    }

    RETRY_CONFIG {
        UUID jobId FK
        Integer maxRetries
        String backoffStrategy
        Integer initialDelaySeconds
        Integer maxDelaySeconds
    }

    JOB_HISTORY {
        UUID id PK
        UUID jobId FK
        String status
        String error
        DateTime executedAt
        Integer attemptNumber
    }

    JOB ||--o{ RETRY_CONFIG : has
    JOB ||--o{ JOB_HISTORY : has

Table Schema Definitions

Jobs Table

CREATE TABLE jobs (
    id UUID PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    description TEXT,
    status VARCHAR(50) NOT NULL DEFAULT 'scheduled',
    payload JSONB,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    next_run_at TIMESTAMP,
    created_by UUID NOT NULL,
    deleted_at TIMESTAMP
);

CREATE INDEX idx_jobs_status_next_run_at ON jobs(status, next_run_at);
CREATE INDEX idx_jobs_created_by ON jobs(created_by);

Retry Config Table

CREATE TABLE retry_configs (
    job_id UUID PRIMARY KEY REFERENCES jobs(id) ON DELETE CASCADE,
    max_retries INTEGER NOT NULL DEFAULT 3,
    backoff_strategy VARCHAR(50) NOT NULL DEFAULT 'exponential',
    initial_delay_seconds INTEGER NOT NULL DEFAULT 10,
    max_delay_seconds INTEGER NOT NULL DEFAULT 300,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

Job History Table

CREATE TABLE job_history (
    id UUID PRIMARY KEY,
    job_id UUID NOT NULL REFERENCES jobs(id) ON DELETE CASCADE,
    status VARCHAR(50) NOT NULL,
    error TEXT,
    executed_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    attempt_number INTEGER NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_job_history_job_id ON job_history(job_id);
CREATE INDEX idx_job_history_status ON job_history(status);

Primary Key Strategy

UUIDs are used as primary keys to support distributed systems and avoid contention in multi-region deployments.

Index Design

idx_jobs_status_next_run_at: For efficient job fetching by status and schedule time
idx_jobs_created_by: For user-based job filtering
idx_job_history_job_id: For job history lookups
idx_job_history_status: For filtering failed jobs

Data Partitioning Strategy

Jobs are partitioned by created_by to support multi-tenancy and reduce contention.

Soft Delete vs. Hard Delete

Soft delete is used to maintain data integrity for historical tracking and compliance.

Audit Trail Design

All job mutations are logged with:

User ID (created_by)
Timestamp
Operation type (create, update, delete)
Changes made

Data Archival Strategy

Jobs older than 1 year are archived to a separate table or storage system with reduced indexing.

4. Class Design & Service Layer

UML Class Diagram

classDiagram
    class JobSchedulerService {
        +createJob(CreateJobRequest request)
        +getJob(UUID id)
        +updateJob(UUID id, UpdateJobRequest request)
        +cancelJob(UUID id)
        +retryJob(UUID id, RetryJobRequest request)
        +getJobHistory(UUID id)
    }

    class JobRepository {
        +save(Job job)
        +findById(UUID id)
        +findByStatusAndNextRunAt(String status, DateTime nextRunAt)
        +update(Job job)
        +delete(UUID id)
    }

    class RetryConfigRepository {
        +save(RetryConfig config)
        +findByJobId(UUID jobId)
        +update(RetryConfig config)
    }

    class JobHistoryRepository {
        +save(JobHistory history)
        +findByJobId(UUID jobId)
    }

    class Job {
        -id: UUID
        -name: String
        -status: String
        -payload: String
        -retryConfig: RetryConfig
        -createdAt: DateTime
        -nextRunAt: DateTime
    }

    class RetryConfig {
        -jobId: UUID
        -maxRetries: Integer
        -backoffStrategy: String
        -initialDelaySeconds: Integer
        -maxDelaySeconds: Integer
    }

    class JobHistory {
        -id: UUID
        -jobId: UUID
        -status: String
        -error: String
        -executedAt: DateTime
        -attemptNumber: Integer
    }

    class BackoffStrategy {
        +calculateDelay(Integer attempt, RetryConfig config)
    }

    JobSchedulerService --> Job : uses
    JobSchedulerService --> JobRepository : accesses
    JobSchedulerService --> RetryConfigRepository : accesses
    JobSchedulerService --> JobHistoryRepository : accesses
    Job --> RetryConfig : has
    Job --> JobHistory : has
    BackoffStrategy <|-- ExponentialBackoffStrategy : implements
    BackoffStrategy <|-- LinearBackoffStrategy : implements

State Machine Diagram

stateDiagram-v2
    [*] --> scheduled
    scheduled --> running : scheduled time reached
    running --> success : completed successfully
    running --> failed : execution failed
    failed --> retrying : retry requested or max retries not reached
    retrying --> running : retry attempt
    retrying --> failed : max retries exceeded
    success --> [*]
    failed --> [*]

Service Class Structure

public interface JobSchedulerService {
    Job createJob(CreateJobRequest request);
    Job getJob(UUID id);
    Job updateJob(UUID id, UpdateJobRequest request);
    void cancelJob(UUID id);
    RetryResult retryJob(UUID id, RetryJobRequest request);
    List<JobHistory> getJobHistory(UUID id);
}

public class JobSchedulerServiceImpl implements JobSchedulerService {
    private final JobRepository jobRepository;
    private final RetryConfigRepository retryConfigRepository;
    private final JobHistoryRepository jobHistoryRepository;
    private final BackoffStrategyFactory backoffStrategyFactory;
    private final EventPublisher eventPublisher;
    
    public JobSchedulerServiceImpl(
        JobRepository jobRepository,
        RetryConfigRepository retryConfigRepository,
        JobHistoryRepository jobHistoryRepository,
        BackoffStrategyFactory backoffStrategyFactory,
        EventPublisher eventPublisher) {
        this.jobRepository = jobRepository;
        this.retryConfigRepository = retryConfigRepository;
        this.jobHistoryRepository = jobHistoryRepository;
        this.backoffStrategyFactory = backoffStrategyFactory;
        this.eventPublisher = eventPublisher;
    }
}

Business Logic Encapsulation

Strategy pattern is used for backoff strategies:

public interface BackoffStrategy {
    long calculateDelay(int attempt, RetryConfig config);
}

@Component
public class ExponentialBackoffStrategy implements BackoffStrategy {
    @Override
    public long calculateDelay(int attempt, RetryConfig config) {
        return Math.min(
            config.getInitialDelaySeconds() * (1L << attempt),
            config.getMaxDelaySeconds()
        );
    }
}

@Component
public class LinearBackoffStrategy implements BackoffStrategy {
    @Override
    public long calculateDelay(int attempt, RetryConfig config) {
        return Math.min(
            config.getInitialDelaySeconds() + (attempt * 10),
            config.getMaxDelaySeconds()
        );
    }
}

public class BackoffStrategyFactory {
    public BackoffStrategy getStrategy(String strategyType) {
        switch (strategyType.toLowerCase()) {
            case "exponential":
                return new ExponentialBackoffStrategy();
            case "linear":
                return new LinearBackoffStrategy();
            default:
                throw new IllegalArgumentException("Unknown backoff strategy: " + strategyType);
        }
    }
}

Critical-Path Method Implementation

@Transactional
public RetryResult retryJob(UUID jobId, RetryJobRequest request) {
    Job job = jobRepository.findById(jobId)
        .orElseThrow(() -> new JobNotFoundException("Job not found: " + jobId));
    
    if (!job.getStatus().equals("failed")) {
        throw new JobNotFailedException("Job is not in failed state: " + jobId);
    }
    
    // Validate retry count
    int maxRetries = job.getRetryConfig().getMaxRetries();
    if (request.getRetryCount() > maxRetries) {
        throw new InvalidRetryCountException("Retry count exceeds maximum: " + maxRetries);
    }
    
    // Create job history entry
    JobHistory history = new JobHistory();
    history.setId(UUID.randomUUID());
    history.setJobId(jobId);
    history.setStatus("retrying");
    history.setAttemptNumber(request.getRetryCount());
    history.setExecutedAt(Instant.now());
    
    jobHistoryRepository.save(history);
    
    // Update job status to retrying
    job.setStatus("retrying");
    job.setNextRunAt(Instant.now().plusSeconds(
        backoffStrategyFactory.getStrategy(job.getRetryConfig().getBackoffStrategy())
            .calculateDelay(request.getRetryCount(), job.getRetryConfig())
    ));
    
    jobRepository.update(job);
    
    // Publish retry event
    eventPublisher.publish(new JobRetryEvent(jobId, request.getRetryCount()));
    
    return new RetryResult(jobId, "retrying", request.getRetryCount());
}

Transaction Boundary Design

Each job operation (create, update, retry) is wrapped in a single database transaction to ensure data consistency.

Service Interface Contracts

All service methods are defined with clear input/output contracts and error handling specifications.

Saga Pattern for Distributed Transactions

Not applicable for this design as all operations are within a single database.

Domain Object Design

Job, RetryConfig, and JobHistory are designed as value objects with immutable properties where appropriate.

5. Concurrency & Thread Safety

Concurrency Model

Java's thread pool with @Async annotations for background job processing. Thread-safe operations are ensured through database-level locking and atomic updates.

Race Condition Prevention

Database-level row locking using SELECT FOR UPDATE during job status updates prevents race conditions in concurrent access scenarios.

Optimistic vs. Pessimistic Locking

Optimistic locking is used with version fields in the database to handle concurrent updates gracefully.

Deadlock Prevention Strategy

Database-level deadlock detection and automatic retry logic with exponential backoff prevents deadlocks in concurrent job operations.

Idempotency Implementation

Idempotency keys are stored in Redis with TTL to handle concurrent duplicate requests safely.

6. Sequence Diagrams for Key Flows

Happy Path Sequence Diagram

sequenceDiagram
    participant Client
    participant JobSchedulerService
    participant JobRepository
    participant RetryConfigRepository
    participant EventPublisher
    
    Client->>JobSchedulerService: POST /v1/jobs
    JobSchedulerService->>JobRepository: save(job)
    JobSchedulerService->>RetryConfigRepository: save(retryConfig)
    JobSchedulerService->>EventPublisher: publish(jobCreated)
    JobSchedulerService-->>Client: 201 Created

Error / Edge Case Sequence Diagram

sequenceDiagram
    participant Client
    participant JobSchedulerService
    participant JobRepository
    
    Client->>JobSchedulerService: POST /v1/jobs (invalid payload)
    JobSchedulerService->>JobRepository: validate request
    JobSchedulerService-->>Client: 400 Bad Request

Async Flow Sequence Diagram

sequenceDiagram
    participant JobSchedulerService
    participant EventPublisher
    participant JobExecutor
    participant JobRepository
    
    JobSchedulerService->>EventPublisher: publish(jobScheduled)
    EventPublisher->>JobExecutor: consume jobScheduled event
    JobExecutor->>JobRepository: update status to running
    JobExecutor->>JobRepository: update status to success/failed

7. Caching Design

Cache-Aside vs. Write-Through vs. Write-Behind

Cache-aside is selected for better control over cache lifecycle and more predictable performance.

Cache Key Design

Keys follow pattern: job:{id} for job data, job:history:{id} for history.

TTL Per Data Type

Data Type	TTL	Reason
Job	1 hour	Frequent reads, can be refreshed
Retry Config	24 hours	Stable configuration
Job History	7 days	Recent history needed

Cache Eviction Policy

LRU eviction policy with TTL-based expiration.

Cache Stampede Prevention

Distributed locks using Redis for cache warming and probabilistic early expiry to prevent stampedes.

Distributed Cache Architecture

Redis cluster with sentinel for high availability and automatic failover.

Cache Invalidation Strategy

Event-driven invalidation using Redis pub/sub for cache updates when job data changes.

8. Repository / Data Access Layer

Repository Pattern Implementation

Repository interfaces define data access contracts:

public interface JobRepository {
    Optional<Job> findById(UUID id);
    Job save(Job job);
    void update(Job job);
    List<Job> findByStatusAndNextRunAt(String status, Instant nextRunAt);
    void delete(UUID id);
}

ORM vs. Raw Query Decisions

Raw SQL queries are used for performance-critical operations like fetching jobs by status and next run time.

Read vs. Write Separation

Reads and writes are separated using read replicas for better scalability.

Connection Pool Configuration

HikariCP with:

Max pool size: 20
Idle timeout: 300 seconds
Connection timeout: 30 seconds

N+1 Query Prevention Strategy

Eager loading of retry config and job history using JOINs to prevent N+1 queries.

Batch Operation Design

Batch operations for bulk job status updates and history creation using batch inserts.

9. Asynchronous Processing & Messaging

Event-Driven Boundaries

Job creation and status updates are asynchronous to improve response times.

Message Schema Design

{
  "version": "1.0",
  "eventType": "job.scheduled",
  "payload": {
    "jobId": "123e4567-e89b-12d3-a456-426614174000",
    "scheduledAt": "2023-05-15T10:00:00Z"
  },
  "timestamp": "2023-05-15T10:00:00Z"
}

Dead Letter Queue Strategy

Messages that fail more than 3 times are moved to a dead letter queue for manual inspection.

At-Least-Once vs. Exactly-Once Delivery

At-least-once delivery is used with idempotency keys to handle duplicates gracefully.

Retry Policy for Consumers

Max retries: 3, exponential backoff with jitter (1s to 30s).

Outbox Pattern

Outbox pattern is implemented for reliable event publishing with database transactional consistency.

Async Flow Sequence Diagram

sequenceDiagram
    participant JobSchedulerService
    participant EventPublisher
    participant MessageBroker
    participant JobExecutor
    
    JobSchedulerService->>EventPublisher: publish(jobScheduled)
    EventPublisher->>MessageBroker: send event
    MessageBroker-->>JobExecutor: consume jobScheduled
    JobExecutor->>JobRepository: update status to running

10. Error Handling Design

Error Classification Hierarchy

Domain errors (e.g., job not found)
Infrastructure errors (e.g., database connection failure)
Validation errors (e.g., invalid input)

Error Propagation Strategy

Errors are propagated up the call stack with appropriate HTTP status codes.

Retry Strategy

Exponential backoff with jitter:

Delay = baseDelay * (2^attempt)
Jitter: ±20% of calculated delay

Circuit Breaker Design

Circuit breaker for external dependencies with:

Failure threshold: 5 failures
Timeout: 30 seconds
Half-open state after timeout

Fallback Behavior

Fallback to default retry configuration when external services are unavailable.

Error Logging Format

Structured logs with:

trace_id, span_id, user_id, job_id, error_code
PII masked in logs

11. Security Design

Input Validation and Sanitization

Validation using Bean Validation annotations with custom constraints.

SQL Injection Prevention

Prepared statements and parameterized queries to prevent SQL injection.

Authentication Middleware Design

JWT-based authentication middleware with role-based access control.

Authorization Enforcement

RBAC implementation using Spring Security annotations:

@PreAuthorize("hasRole('ADMIN') or @jobAuthorizationService.isOwner(jobId)")

Sensitive Data Encryption

AES-256 encryption for sensitive job payloads stored in database.

PII Handling and Masking

PII data is masked in logs and audit trails using redaction rules.

Secrets Management

AWS Secrets Manager integration for storing sensitive configurations.

Audit Logging Design

All job mutations are logged with:

User ID
Timestamp
Operation type
Changes made

12. Testing Strategy

Unit Testing Scope and Coverage Targets

90% code coverage for service layer with mocks for external dependencies.

Integration Testing Strategy

Testcontainers for database integration testing, embedded Redis for caching tests.

Contract Testing

Pact framework for contract testing between job scheduler and consumers.

Performance Testing Tool

k6 for load testing with scenarios:

Concurrent job creation (1000 jobs)
High-volume retry operations (500 concurrent retries)

Test Data Management Strategy

Factory pattern for generating test data, fixtures for common scenarios.

13. Observability Design

Structured Logging Schema

Log fields:

trace_id (UUID)
span_id (UUID)
user_id (UUID)
job_id (UUID)
error_code (String)
PII masked in logs

Distributed Tracing Instrumentation

OpenTelemetry integration with:

Custom spans for job execution
Attributes: job_id, attempt_number, status

Metrics Design at Service Level

RED metrics:

Rate: job creation rate per minute
Errors: job failure count
Duration: job execution time

Health Check Endpoint Design

{
  "status": "healthy",
  "components": {
    "database": "healthy",
    "cache": "healthy",
    "message-broker": "healthy"
  }
}

Alerting Thresholds

Job failure rate > 5% over 5 minutes → Critical
Cache hit ratio < 80% → Warning
Database connection pool usage > 90% → Warning

14. Open Questions, Risks & Trade-offs

Decisions Requiring Product / Engineering Alignment

Whether to support job dependencies or just scheduling
How to handle time zone differences in cron expressions

Known Trade-offs and Justifications

Using UUIDs for primary keys increases storage overhead but provides better distribution
Caching trade-off between performance and consistency

Risks and Mitigation Strategies

Risk of job duplication due to concurrent access → Mitigated with database locking
Risk of cache stampede → Mitigated with distributed locks and probabilistic expiry

Technical Debt Acknowledged

Current implementation assumes single database instance; sharding strategy needs future design consideration

3. Data Model Design

Entity-Relationship Diagram

erDiagram
    JOB {
        string id PK
        string name
        string description
        string status
        string payload
        string error_message
        int max_retries
        int current_retry_count
        int retry_delay_seconds
        timestamp created_at
        timestamp updated_at
        timestamp scheduled_at
        timestamp last_attempted_at
        string owner_id
        string job_type
    }

    JOB_ATTEMPT {
        string id PK
        string job_id FK
        timestamp attempted_at
        string status
        string error_message
        int attempt_number
        string result_data
    }

    JOB_RETRY_CONFIG {
        string id PK
        string job_type
        int max_retries
        int delay_seconds
        timestamp created_at
        timestamp updated_at
    }

    JOB_DEPENDENCY {
        string job_id FK
        string depends_on_job_id FK
        string dependency_type
    }

    JOB <-- JOB_ATTEMPT : has_many
    JOB <-- JOB_RETRY_CONFIG : has_one
    JOB <-- JOB_DEPENDENCY : has_many

Table Schema Definitions

`jobs` table

CREATE TABLE jobs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL,
    description TEXT,
    status VARCHAR(50) NOT NULL DEFAULT 'PENDING',
    payload JSONB NOT NULL,
    error_message TEXT,
    max_retries INTEGER NOT NULL DEFAULT 3,
    current_retry_count INTEGER NOT NULL DEFAULT 0,
    retry_delay_seconds INTEGER NOT NULL DEFAULT 30,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    scheduled_at TIMESTAMP WITH TIME ZONE NOT NULL,
    last_attempted_at TIMESTAMP WITH TIME ZONE,
    owner_id VARCHAR(255),
    job_type VARCHAR(100) NOT NULL,
    
    INDEX idx_jobs_status (status),
    INDEX idx_jobs_scheduled_at (scheduled_at),
    INDEX idx_jobs_owner_id (owner_id),
    INDEX idx_jobs_job_type (job_type),
    INDEX idx_jobs_created_at (created_at)
);

`job_attempts` table

CREATE TABLE job_attempts (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    job_id UUID NOT NULL REFERENCES jobs(id) ON DELETE CASCADE,
    attempted_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    status VARCHAR(50) NOT NULL,
    error_message TEXT,
    attempt_number INTEGER NOT NULL,
    result_data JSONB,
    
    INDEX idx_job_attempts_job_id (job_id),
    INDEX idx_job_attempts_attempted_at (attempted_at),
    INDEX idx_job_attempts_status (status)
);

`job_retry_configs` table

CREATE TABLE job_retry_configs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    job_type VARCHAR(100) NOT NULL UNIQUE,
    max_retries INTEGER NOT NULL DEFAULT 3,
    delay_seconds INTEGER NOT NULL DEFAULT 30,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    
    INDEX idx_job_retry_configs_job_type (job_type)
);

`job_dependencies` table

CREATE TABLE job_dependencies (
    job_id UUID NOT NULL REFERENCES jobs(id) ON DELETE CASCADE,
    depends_on_job_id UUID NOT NULL REFERENCES jobs(id) ON DELETE CASCADE,
    dependency_type VARCHAR(50) NOT NULL DEFAULT 'SEQUENTIAL',
    
    PRIMARY KEY (job_id, depends_on_job_id),
    INDEX idx_job_dependencies_depends_on_job_id (depends_on_job_id)
);

Primary Key Strategy

UUID is selected for primary keys due to:

Distributed system compatibility (no coordination needed for ID generation)
Better sharding characteristics across multiple database instances
Prevention of sequential ID predictability (security concern)
Support for eventual consistency patterns

Index Design Justification

idx_jobs_status - For efficient querying of jobs by status (active, failed, completed)
idx_jobs_scheduled_at - For fetching jobs ready for execution in the scheduler
idx_jobs_owner_id - For owner-based job filtering and audit trails
idx_jobs_job_type - For efficient lookup of jobs by type to apply retry configurations
idx_jobs_created_at - For time-based analytics and job aging monitoring
idx_job_attempts_job_id - For fetching all attempts for a specific job
idx_job_attempts_attempted_at - For time-based querying of job attempts
idx_job_attempts_status - For filtering attempts by status (success, failure)
idx_job_retry_configs_job_type - For fast lookup of retry configurations by job type

Data Partitioning / Sharding Strategy

Partition jobs by job_type and created_at:

Shard by job_type: Different job types can have different retry strategies and processing patterns
Time-based partitioning: Jobs created in the same time window can be grouped together for batch processing
Hash-based sharding: Use consistent hashing on job_type to distribute load across database shards

Soft Delete vs. Hard Delete Decision

Soft delete is chosen because:

Audit trail requirement for compliance and debugging
Ability to recover from accidental deletions
Support for historical job analysis and reporting
Compliance with data retention policies

Audit Trail Design

All changes to jobs are tracked through:

created_at and updated_at timestamps
owner_id for ownership tracking
Job status transitions are audited through job_attempts table
All modifications to retry configuration are logged in a separate audit table

Data Archival Strategy and Retention Policy

Active jobs: Keep in primary database (last 30 days)
Completed jobs: Archive to cold storage after 90 days
Failed jobs: Keep in primary database for 7 days or until resolved
Job attempts: Keep for 30 days to support debugging and analytics
Archival process: Scheduled nightly job that moves completed jobs to archive storage (e.g., S3 with lifecycle policies)

// Sample Java entity for job model
public class Job {
    private String id;
    private String name;
    private String description;
    private JobStatus status;
    private String payload;
    private String errorMessage;
    private int maxRetries;
    private int currentRetryCount;
    private int retryDelaySeconds;
    private LocalDateTime createdAt;
    private LocalDateTime updatedAt;
    private LocalDateTime scheduledAt;
    private LocalDateTime lastAttemptedAt;
    private String ownerId;
    private String jobType;
    
    // Getters and setters...
}

4. Class Design & Service Layer

classDiagram
    class JobSchedulerService {
        +scheduleRetry(JobRequest)
        +processScheduledJobs()
        +cancelRetry(String jobId)
        +getJobStatus(String jobId)
    }
    
    class JobRequest {
        -String id
        -String payload
        -int maxRetries
        -int currentRetry
        -long scheduledTime
        -String status
        -String failureReason
        +createRetryJob()
    }
    
    class JobRepository {
        +save(JobRequest)
        +findById(String id)
        +findByStatusAndScheduledTime(String status, long timestamp)
        +update(JobRequest)
        +delete(String id)
    }
    
    class RetryScheduler {
        -ScheduledExecutorService scheduler
        -JobRepository repository
        +scheduleRetry(JobRequest)
        +processScheduledJobs()
    }
    
    class JobProcessor {
        -JobRepository repository
        -EventPublisher eventPublisher
        +processJob(JobRequest)
        +handleFailure(JobRequest, Exception)
    }
    
    class EventPublisher {
        -MessagingService messagingService
        +publish(JobEvent)
    }
    
    class JobEvent {
        -String jobId
        -String eventType
        -String payload
        -long timestamp
    }
    
    JobSchedulerService --> JobRequest : creates
    JobSchedulerService --> JobRepository : persists
    JobSchedulerService --> RetryScheduler : schedules
    RetryScheduler --> JobRepository : reads/writes
    RetryScheduler --> JobProcessor : processes
    JobProcessor --> EventPublisher : publishes
    EventPublisher --> MessagingService : sends
    
    class JobStatus {
        PENDING
        PROCESSING
        SUCCESS
        FAILED
        CANCELLED
    }

stateDiagram-v2
    [*] --> PENDING
    PENDING --> PROCESSING : scheduled
    PROCESSING --> SUCCESS : success
    PROCESSING --> FAILED : max retries exceeded
    PROCESSING --> PENDING : retry scheduled
    FAILED --> CANCELLED : cancelled
    PENDING --> CANCELLED : cancelled

Service Layer Architecture

The service layer implements the core business logic for job retry scheduling with a clean separation of concerns:

JobSchedulerService - Main entry point for job scheduling and status management
RetryScheduler - Handles scheduling of retry jobs using ScheduledExecutorService
JobProcessor - Processes actual job execution and failure handling
EventPublisher - Publishes job lifecycle events for observability

Business Logic Patterns

Strategy Pattern Implementation

For retry delay calculation, we implement a strategy pattern:

public interface RetryDelayStrategy {
    long calculateDelay(int currentRetry, int maxRetries);
}

@Component
public class ExponentialBackoffStrategy implements RetryDelayStrategy {
    private static final long BASE_DELAY_MS = 1000;
    private static final double MULTIPLIER = 2.0;
    
    @Override
    public long calculateDelay(int currentRetry, int maxRetries) {
        if (currentRetry <= 0) return BASE_DELAY_MS;
        
        long delay = (long) (BASE_DELAY_MS * Math.pow(MULTIPLIER, currentRetry - 1));
        return Math.min(delay, MAX_RETRY_DELAY_MS);
    }
}

@Component
public class LinearBackoffStrategy implements RetryDelayStrategy {
    private static final long BASE_DELAY_MS = 5000;
    
    @Override
    public long calculateDelay(int currentRetry, int maxRetries) {
        return BASE_DELAY_MS * currentRetry;
    }
}

Critical-Path Method Implementation

@Transactional
public void processJob(JobRequest job) {
    try {
        // Acquire pessimistic lock on job record to prevent concurrent processing
        JobRequest lockedJob = jobRepository.findByIdForUpdate(job.getId());
        
        // Validate job state
        if (!"PROCESSING".equals(lockedJob.getStatus())) {
            throw new IllegalStateException("Job not in processing state: " + job.getId());
        }
        
        // Execute the actual job logic
        executeJobLogic(lockedJob);
        
        // Update job status to SUCCESS
        lockedJob.setStatus("SUCCESS");
        lockedJob.setProcessedAt(System.currentTimeMillis());
        jobRepository.save(lockedJob);
        
        // Publish success event
        JobEvent event = new JobEvent(
            job.getId(), 
            "JOB_SUCCESS", 
            job.getPayload(),
            System.currentTimeMillis()
        );
        eventPublisher.publish(event);
        
    } catch (Exception e) {
        // Handle job failure with retry logic
        handleJobFailure(job, e);
    }
}

private void handleJobFailure(JobRequest job, Exception exception) {
    JobRequest updatedJob = jobRepository.findByIdForUpdate(job.getId());
    
    // Increment retry count
    int newRetryCount = updatedJob.getCurrentRetry() + 1;
    
    if (newRetryCount <= updatedJob.getMaxRetries()) {
        // Schedule next retry
        long retryDelay = retryDelayStrategy.calculateDelay(newRetryCount, updatedJob.getMaxRetries());
        long nextScheduledTime = System.currentTimeMillis() + retryDelay;
        
        updatedJob.setCurrentRetry(newRetryCount);
        updatedJob.setScheduledTime(nextScheduledTime);
        updatedJob.setStatus("PENDING");
        updatedJob.setFailureReason(exception.getMessage());
        
        jobRepository.save(updatedJob);
        
        // Schedule retry in the future
        retryScheduler.scheduleRetry(updatedJob);
        
        // Publish retry event
        JobEvent event = new JobEvent(
            job.getId(),
            "JOB_RETRY_SCHEDULED",
            job.getPayload(),
            System.currentTimeMillis()
        );
        eventPublisher.publish(event);
        
    } else {
        // Max retries exceeded - mark as failed
        updatedJob.setStatus("FAILED");
        updatedJob.setFailureReason(exception.getMessage());
        jobRepository.save(updatedJob);
        
        // Publish failure event
        JobEvent event = new JobEvent(
            job.getId(),
            "JOB_FAILED",
            job.getPayload(),
            System.currentTimeMillis()
        );
        eventPublisher.publish(event);
    }
}

Transaction Boundary Design

Each job processing operation is wrapped in a single database transaction:

Scope: Entire job execution and status update
Isolation Level: READ_COMMITTED with FOR UPDATE locking for concurrent safety
Boundary: Transaction begins when job enters processing state, commits after successful completion or final failure

Service Interface Contracts

public interface JobSchedulerService {
    JobRequest scheduleRetry(JobRequest jobRequest);
    void cancelRetry(String jobId);
    JobStatus getJobStatus(String jobId);
    List<JobRequest> getScheduledJobs();
}

Saga Pattern for Distributed Transactions

For distributed systems, we implement a saga pattern using job state tracking:

public class JobSaga {
    private String jobId;
    private List<SagaStep> steps;
    private SagaStatus status;
    
    public void executeStep(SagaStep step) {
        try {
            step.execute();
            step.setStatus("COMPLETED");
            updateSagaStatus();
        } catch (Exception e) {
            rollbackSteps();
            step.setStatus("FAILED");
            updateSagaStatus();
        }
    }
}

Domain Object Design

@Entity
@Table(name = "job_requests")
public class JobRequest {
    @Id
    @GeneratedValue(strategy = GenerationType.UUID)
    private String id;
    
    @Column(length = 10000)
    private String payload;
    
    private int maxRetries = 3;
    private int currentRetry = 0;
    
    @Column(name = "scheduled_time")
    private long scheduledTime;
    
    @Enumerated(EnumType.STRING)
    private JobStatus status = JobStatus.PENDING;
    
    @Column(name = "failure_reason")
    private String failureReason;
    
    @Column(name = "processed_at")
    private Long processedAt;
    
    @CreatedTimestamp
    private Instant createdAt;
    
    @UpdateTimestamp
    private Instant updatedAt;
}

Value Object Design

public class RetryConfig {
    private int maxRetries;
    private long initialDelayMs;
    private double backoffMultiplier;
    private long maxDelayMs;
    
    // Getters and setters
}

Thread Safety Considerations

State Management: Job status transitions are protected by database-level locking
Scheduler Thread Safety: ScheduledExecutorService is thread-safe for scheduling operations
Idempotency: Job IDs are UUIDs to prevent duplicate processing
Race Condition Prevention: FOR UPDATE queries ensure atomic state updates

5. Concurrency & Thread Safety

Concurrency Model

The job retry scheduler operates in a multi-threaded environment where multiple worker threads may be processing jobs concurrently. The system uses a thread pool executor with configurable number of worker threads to handle job scheduling and execution. Each job processing task runs in a separate thread, ensuring that long-running or blocking operations don't block the main scheduler thread.

Race Condition Prevention Strategy

Race conditions are mitigated through:

Atomic operations for job status updates using database-level atomic operations or optimistic locking
Database transaction boundaries to ensure consistency during job state transitions
Thread-safe data structures for in-memory job tracking (e.g., ConcurrentHashMap)
Locking mechanisms when updating shared job state across multiple threads

Optimistic vs. Pessimistic Locking Selection

Optimistic locking is selected because:

The system handles high throughput with many concurrent job operations
Job status updates are infrequent and typically involve small data sets
Pessimistic locking would introduce unnecessary contention and reduce system throughput
Database-level optimistic locking (version field) provides better performance characteristics

Deadlock Prevention Strategy

Deadlock prevention is implemented through:

Fixed ordering of database locks - All job processing operations acquire database locks in a consistent order
Timeout configuration - Database transaction timeouts prevent indefinite waiting
Lock acquisition timeout - Maximum wait time for lock acquisition before retrying or failing
Hierarchical job dependency management - Jobs are processed in dependency order to avoid circular dependencies

Idempotency Implementation for Concurrent Duplicate Requests

Idempotency is handled through:

Job ID uniqueness - Each job has a unique identifier that prevents duplicate processing
Database constraints - Unique indexes on job identifiers prevent concurrent creation of same job
State-based idempotency - Job state checks before processing to avoid reprocessing completed jobs
Idempotency keys - For external job creation, idempotency keys are used to ensure same operation is not processed multiple times

// Example of idempotent job processing with thread safety
public class JobRetryScheduler {
    private final Map<String, JobState> jobCache = new ConcurrentHashMap<>();
    private final Object lock = new Object();
    
    public void processJob(String jobId, JobRequest request) {
        // Check if job already processed (idempotency)
        if (jobCache.containsKey(jobId)) {
            JobState state = jobCache.get(jobId);
            if (state.isCompleted() || state.isFailed()) {
                return; // Already processed, skip
            }
        }
        
        synchronized (lock) {
            // Double-check locking pattern to prevent race conditions
            if (!jobCache.containsKey(jobId)) {
                try {
                    // Process job with database transaction
                    processJobWithTransaction(jobId, request);
                    jobCache.put(jobId, JobState.COMPLETED);
                } catch (Exception e) {
                    jobCache.put(jobId, JobState.FAILED);
                    throw new JobProcessingException("Failed to process job: " + jobId, e);
                }
            }
        }
    }
    
    private void processJobWithTransaction(String jobId, JobRequest request) {
        // Transaction boundary - all database operations in single transaction
        // Using Spring @Transactional or similar
        jobRepository.updateJobStatus(jobId, JobStatus.PROCESSING);
        
        try {
            // Perform actual job processing logic
            executeJobLogic(request);
            
            // Update job status to completed
            jobRepository.updateJobStatus(jobId, JobStatus.COMPLETED);
        } catch (Exception e) {
            // Handle retry logic and update status to failed or queued for retry
            jobRepository.updateJobStatus(jobId, JobStatus.FAILED);
            throw e;
        }
    }
}

6. Sequence Diagrams for Key Flows

sequenceDiagram
    participant Client
    participant JobSchedulerService
    participant JobRepository
    participant JobExecutor
    participant EventPublisher

    Client->>JobSchedulerService: Submit job with retry config
    JobSchedulerService->>JobRepository: Save job with initial status
    JobSchedulerService->>EventPublisher: Publish job scheduled event
    JobSchedulerService-->>Client: Acknowledge job submission

    loop Retry Loop
        JobExecutor->>JobRepository: Check for pending retry jobs
        JobRepository-->>JobExecutor: Return job details
        JobExecutor->>JobExecutor: Execute job logic
        alt Success
            JobExecutor->>JobRepository: Update job status to completed
            JobExecutor->>EventPublisher: Publish job completed event
        else Failure
            JobExecutor->>JobRepository: Increment retry count
            JobExecutor->>JobRepository: Update next retry time
            JobExecutor->>EventPublisher: Publish job failed event
        end
    end

sequenceDiagram
    participant Client
    participant JobSchedulerService
    participant JobRepository
    participant JobExecutor
    participant EventPublisher

    Client->>JobSchedulerService: Submit job with max retries=3
    JobSchedulerService->>JobRepository: Save job with status=PENDING
    JobSchedulerService->>EventPublisher: Publish job scheduled event

    loop Retry Attempts (3)
        JobExecutor->>JobRepository: Check for retry jobs
        JobRepository-->>JobExecutor: Return job with status=PENDING
        JobExecutor->>JobExecutor: Execute job logic (fails)
        JobExecutor->>JobRepository: Update retry count=1
        JobExecutor->>JobRepository: Update next retry time (exponential backoff)
        JobExecutor->>EventPublisher: Publish job failed event

        JobExecutor->>JobRepository: Check for retry jobs
        JobExecutor->>JobRepository: Return job with status=PENDING
        JobExecutor->>JobExecutor: Execute job logic (fails)
        JobExecutor->>JobRepository: Update retry count=2
        JobExecutor->>JobRepository: Update next retry time (exponential backoff)
        JobExecutor->>EventPublisher: Publish job failed event

        JobExecutor->>JobRepository: Check for retry jobs
        JobExecutor->>JobRepository: Return job with status=PENDING
        JobExecutor->>JobExecutor: Execute job logic (fails)
        JobExecutor->>JobRepository: Update retry count=3
        JobExecutor->>JobRepository: Update status to FAILED
        JobExecutor->>EventPublisher: Publish job failed event
    end

    JobSchedulerService-->>Client: Return final job status (FAILED)

sequenceDiagram
    participant Client
    participant JobSchedulerService
    participant JobRepository
    participant JobExecutor
    participant MessageBroker
    participant EventPublisher

    Client->>JobSchedulerService: Submit async job with retry config
    JobSchedulerService->>JobRepository: Save job with status=PENDING
    JobSchedulerService->>EventPublisher: Publish job scheduled event
    JobSchedulerService-->>Client: Acknowledge job submission

    JobExecutor->>JobRepository: Poll for pending jobs
    JobRepository-->>JobExecutor: Return job details
    JobExecutor->>MessageBroker: Publish job execution event
    MessageBroker-->>JobExecutor: Acknowledge event published

    JobExecutor->>JobRepository: Update job status to PROCESSING
    JobExecutor->>MessageBroker: Publish job processing event

    alt Success
        JobExecutor->>JobRepository: Update job status to COMPLETED
        JobExecutor->>EventPublisher: Publish job completed event
    else Failure
        JobExecutor->>JobRepository: Increment retry count
        JobExecutor->>JobRepository: Update next retry time (exponential backoff)
        JobExecutor->>EventPublisher: Publish job failed event
    end

    loop Retry Loop
        JobExecutor->>JobRepository: Check for pending retry jobs
        JobRepository-->>JobExecutor: Return job details
        JobExecutor->>MessageBroker: Publish retry job execution event
        MessageBroker-->>JobExecutor: Acknowledge event published

        JobExecutor->>JobRepository: Update job status to PROCESSING
        JobExecutor->>MessageBroker: Publish retry job processing event

        alt Success
            JobExecutor->>JobRepository: Update job status to COMPLETED
            JobExecutor->>EventPublisher: Publish job completed event
        else Failure
            JobExecutor->>JobRepository: Increment retry count
            JobExecutor->>JobRepository: Update next retry time (exponential backoff)
            JobExecutor->>EventPublisher: Publish job failed event
        end
    end

7. Caching Design

Cache Strategy Selection

For job retry scheduling, I'll use a cache-aside pattern with write-through behavior for critical job state updates. This approach provides the best balance between performance and consistency for a retry scheduler system.

Rationale:

Cache-aside allows us to control when and how data is cached
Write-through ensures job state consistency between cache and database
Avoids cache stampede with probabilistic early expiry strategy
Supports the high-read, low-write nature of job retry systems

Cache Key Design

Cache keys will follow a structured namespace pattern:

job_retry_scheduler:{jobId}:{retryAttempt}
job_retry_scheduler:config:{jobId}
job_retry_scheduler:stats:{jobId}

TTL Strategy

Data Type	TTL	Reason
Job retry state	30 minutes	Jobs should be rescheduled within this window
Retry configuration	1 hour	Configuration changes are infrequent
Job statistics	24 hours	Aggregated stats can be refreshed periodically

Cache Eviction Policy

LRU (Least Recently Used) with TTL-based expiration as primary policy. This ensures:

Frequently accessed job states remain in cache
Memory pressure is managed automatically
Old retry attempts don't accumulate indefinitely

Cache Stampede Prevention

Probabilistic Early Expiry: When cache misses occur, we'll use a probabilistic approach to avoid stampede:

public class CacheStampedePrevention {
    private static final double STAMPED_PROBABILITY = 0.1;
    
    public boolean shouldRefreshCache(String key) {
        return Math.random() < STAMPED_PROBABILITY;
    }
    
    public void handleCacheMiss(String key, Supplier<Object> dataLoader) {
        if (shouldRefreshCache(key)) {
            // Acquire distributed lock before cache refresh
            try (DistributedLock lock = lockManager.acquire(key, 5000)) {
                // Double-check if another thread already cached it
                Object cached = cache.get(key);
                if (cached == null) {
                    Object data = dataLoader.get();
                    cache.put(key, data, TTL);
                }
            }
        } else {
            // Serve stale data or use default
            Object cached = cache.get(key);
            if (cached != null) {
                return cached;
            }
        }
    }
}

Distributed Cache Architecture

Redis Cluster with Sentinel for high availability:

Cluster mode: Horizontal scaling across multiple nodes
Sentinel monitoring: Automatic failover capability
Replication: 3x replication for data durability
Sharding strategy: Hash-based sharding by job ID

Cache Invalidation Strategy

Event-driven invalidation with a hybrid approach:

Immediate invalidation: When job state changes (success, failure, retry)
TTL-based cleanup: Automatic expiration for stale entries
Explicit purge: Admin-triggered cache clearing

@Component
public class JobRetryCacheInvalidator {
    
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;
    
    @EventListener
    public void handleJobStateChange(JobStateChangedEvent event) {
        String jobId = event.getJobId();
        
        // Invalidate all related cache entries
        Set<String> keysToInvalidate = new HashSet<>();
        keysToInvalidate.add("job_retry_scheduler:" + jobId + ":*");
        keysToInvalidate.add("job_retry_scheduler:config:" + jobId);
        
        // Use Redis pipeline for batch invalidation
        redisTemplate.delete(keysToInvalidate);
    }
}

Cache Read-Through Pattern

@Service
public class JobRetryCacheService {
    
    private final RedisTemplate<String, Object> redisTemplate;
    private final JobRetryRepository jobRetryRepository;
    
    public JobRetryState getJobRetryState(String jobId, int retryAttempt) {
        String cacheKey = "job_retry_scheduler:" + jobId + ":" + retryAttempt;
        
        // Try cache first
        JobRetryState cached = (JobRetryState) redisTemplate.opsForValue().get(cacheKey);
        if (cached != null) {
            return cached;
        }
        
        // Cache miss - fetch from DB and cache
        JobRetryState dbState = jobRetryRepository.findByJobIdAndAttempt(jobId, retryAttempt);
        if (dbState != null) {
            // Cache with TTL
            redisTemplate.opsForValue().set(cacheKey, dbState, 30, TimeUnit.MINUTES);
            return dbState;
        }
        
        return null;
    }
}

8. Repository / Data Access Layer

Repository Pattern Implementation

The repository pattern is implemented to abstract data access logic and provide a clean separation between domain logic and persistence concerns. Each entity has its own repository interface and implementation.

public interface JobRepository {
    Job save(Job job);
    Optional<Job> findById(UUID id);
    List<Job> findByStatus(JobStatus status, Pageable pageable);
    List<Job> findByScheduledTimeBefore(Instant now, Pageable pageable);
    void updateStatus(UUID id, JobStatus status);
    void updateRetryCount(UUID id, int retryCount);
    void delete(UUID id);
}

@Repository
public class JobRepositoryImpl implements JobRepository {
    private final JdbcTemplate jdbcTemplate;
    private final ObjectMapper objectMapper;

    public JobRepositoryImpl(JdbcTemplate jdbcTemplate, ObjectMapper objectMapper) {
        this.jdbcTemplate = jdbcTemplate;
        this.objectMapper = objectMapper;
    }

    @Override
    public Job save(Job job) {
        String sql = """
            INSERT INTO jobs (id, name, payload, status, retry_count, max_retries, 
                             scheduled_time, created_at, updated_at) 
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
            """;
        
        jdbcTemplate.update(sql, 
            job.getId(), 
            job.getName(),
            objectMapper.writeValueAsString(job.getPayload()),
            job.getStatus().name(),
            job.getRetryCount(),
            job.getMaxRetries(),
            job.getScheduledTime(),
            job.getCreatedAt(),
            job.getUpdatedAt()
        );
        
        return job;
    }

    @Override
    public Optional<Job> findById(UUID id) {
        String sql = "SELECT * FROM jobs WHERE id = ?";
        try {
            return Optional.of(jdbcTemplate.queryForObject(sql, new JobRowMapper(), id));
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }

    @Override
    public List<Job> findByStatus(JobStatus status, Pageable pageable) {
        String sql = """
            SELECT * FROM jobs 
            WHERE status = ? 
            ORDER BY scheduled_time ASC 
            LIMIT ? OFFSET ?
            """;
        
        return jdbcTemplate.query(sql, new JobRowMapper(), 
            status.name(), 
            pageable.getPageSize(), 
            pageable.getOffset());
    }

    @Override
    public List<Job> findByScheduledTimeBefore(Instant now, Pageable pageable) {
        String sql = """
            SELECT * FROM jobs 
            WHERE scheduled_time <= ? 
            AND status IN ('PENDING', 'FAILED') 
            ORDER BY scheduled_time ASC 
            LIMIT ? OFFSET ?
            """;
        
        return jdbcTemplate.query(sql, new JobRowMapper(), 
            now, 
            pageable.getPageSize(), 
            pageable.getOffset());
    }

    @Override
    public void updateStatus(UUID id, JobStatus status) {
        String sql = "UPDATE jobs SET status = ?, updated_at = ? WHERE id = ?";
        jdbcTemplate.update(sql, status.name(), Instant.now(), id);
    }

    @Override
    public void updateRetryCount(UUID id, int retryCount) {
        String sql = "UPDATE jobs SET retry_count = ?, updated_at = ? WHERE id = ?";
        jdbcTemplate.update(sql, retryCount, Instant.now(), id);
    }

    @Override
    public void delete(UUID id) {
        String sql = "DELETE FROM jobs WHERE id = ?";
        jdbcTemplate.update(sql, id);
    }
}

ORM vs. Raw Query Decisions

Raw SQL queries are preferred over ORM for the following reasons:

Performance: Direct control over generated SQL for better optimization
Complex Queries: Complex filtering and pagination scenarios benefit from raw queries
Database-Specific Features: Leverage PostgreSQL-specific features like JSONB operations when needed

For simple CRUD operations, the JdbcTemplate is used with explicit SQL statements to maintain control over query execution and performance characteristics.

Read vs. Write Separation

The system uses a read-write separation strategy with:

Write Operations: Direct database writes via primary database connection
Read Operations: Read replicas for read-heavy operations like job status polling

public interface JobReadRepository {
    List<Job> findByStatus(JobStatus status, Pageable pageable);
    Optional<Job> findById(UUID id);
}

@Repository
public class JobReadRepositoryImpl implements JobReadRepository {
    private final JdbcTemplate readJdbcTemplate;

    public JobReadRepositoryImpl(@Qualifier("readDataSource") JdbcTemplate readJdbcTemplate) {
        this.readJdbcTemplate = readJdbcTemplate;
    }

    @Override
    public List<Job> findByStatus(JobStatus status, Pageable pageable) {
        String sql = """
            SELECT * FROM jobs 
            WHERE status = ? 
            ORDER BY scheduled_time ASC 
            LIMIT ? OFFSET ?
            """;
        
        return readJdbcTemplate.query(sql, new JobRowMapper(), 
            status.name(), 
            pageable.getPageSize(), 
            pageable.getOffset());
    }

    @Override
    public Optional<Job> findById(UUID id) {
        String sql = "SELECT * FROM jobs WHERE id = ?";
        try {
            return Optional.of(readJdbcTemplate.queryForObject(sql, new JobRowMapper(), id));
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }
}

Connection Pool Configuration

# application.yml
spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      validation-timeout: 5000
      leak-detection-threshold: 60000

N+1 Query Prevention Strategy

The job scheduler design prevents N+1 queries by:

Batch Loading: Jobs are loaded in batches with all required information
Eager Loading: All job dependencies are fetched in single queries
Pre-joined Queries: Complex joins are used to fetch related data

// Example of batch job loading with pre-fetched dependencies
public List<Job> loadJobsWithDependencies(List<UUID> jobIds) {
    String sql = """
        SELECT j.*, 
               json_agg(jd.data) as dependencies
        FROM jobs j
        LEFT JOIN job_dependencies jd ON j.id = jd.job_id
        WHERE j.id IN (?)
        GROUP BY j.id
        """;
    
    return jdbcTemplate.query(sql, new JobWithDependenciesRowMapper(), jobIds);
}

Batch Operation Design

Batch operations are implemented for high-volume job processing:

public class BatchJobProcessor {
    private static final int BATCH_SIZE = 1000;

    public void processJobsInBatches(List<Job> jobs) {
        for (int i = 0; i < jobs.size(); i += BATCH_SIZE) {
            int end = Math.min(i + BATCH_SIZE, jobs.size());
            List<Job> batch = jobs.subList(i, end);
            
            processBatch(batch);
        }
    }

    private void processBatch(List<Job> batch) {
        String sql = """
            INSERT INTO jobs (id, name, payload, status, retry_count, max_retries, 
                             scheduled_time, created_at, updated_at) 
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
            """;
        
        jdbcTemplate.batchUpdate(sql, batch, BATCH_SIZE, (ps, job) -> {
            ps.setObject(1, job.getId());
            ps.setString(2, job.getName());
            ps.setString(3, objectMapper.writeValueAsString(job.getPayload()));
            ps.setString(4, job.getStatus().name());
            ps.setInt(5, job.getRetryCount());
            ps.setInt(6, job.getMaxRetries());
            ps.setTimestamp(7, Timestamp.from(job.getScheduledTime()));
            ps.setTimestamp(8, Timestamp.from(job.getCreatedAt()));
            ps.setTimestamp(9, Timestamp.from(job.getUpdatedAt()));
        });
    }
}

Data Access Layer Transaction Management

All data access operations are wrapped in appropriate transaction boundaries:

@Service
@Transactional
public class JobService {
    private final JobRepository jobRepository;
    
    public void scheduleJob(Job job) {
        // Transaction boundary includes all data operations
        Job saved = jobRepository.save(job);
        // Additional business logic that must be atomic
    }
}

Read Replica Configuration

@Configuration
public class DataSourceConfig {
    @Bean("readDataSource")
    @ConfigurationProperties(prefix = "spring.datasource.read")
    public DataSource readDataSource() {
        return HikariDataSourceFactory.createDataSource();
    }
    
    @Bean("writeDataSource")
    @ConfigurationProperties(prefix = "spring.datasource.write")
    public DataSource writeDataSource() {
        return HikariDataSourceFactory.createDataSource();
    }
}

Repository Interface Contracts

public interface JobRepository {
    Job save(Job job);
    Optional<Job> findById(UUID id);
    List<Job> findByStatus(JobStatus status, Pageable pageable);
    List<Job> findByScheduledTimeBefore(Instant now, Pageable pageable);
    void updateStatus(UUID id, JobStatus status);
    void updateRetryCount(UUID id, int retryCount);
    void delete(UUID id);
}

The repository layer is designed to handle high-throughput job scheduling operations with proper transaction management, read-write separation for performance optimization, and batch processing capabilities for bulk operations. The implementation follows clean architecture principles with clear separation of concerns between data access and business logic.

9. Asynchronous Processing & Messaging

Event-driven boundaries in this feature

The job retry scheduler operates primarily in a synchronous context for core scheduling decisions but leverages asynchronous processing for:

Job execution failures - When a job fails, it's asynchronously scheduled for retry based on configured policies
Retry scheduling - The actual scheduling of retry attempts happens asynchronously to avoid blocking the main execution flow
Notification and monitoring - Retry events, failures, and system status changes are published asynchronously for external systems to consume
Cleanup operations - Expired or completed retry jobs are asynchronously cleaned up from the system

The synchronous boundary is maintained for immediate job submission and status queries, while asynchronous flows handle the operational overhead of retry management.

Message schema design with sample payload

{
  "version": "1.0",
  "event_type": "job_retry_scheduled",
  "timestamp": "2023-10-15T14:30:00Z",
  "job_id": "b7a2c9f1-8d4e-4a3b-9f2e-1c5d6e7f8a9b",
  "retry_count": 2,
  "next_retry_at": "2023-10-15T14:45:00Z",
  "retry_policy": {
    "max_retries": 5,
    "backoff_strategy": "exponential",
    "initial_delay_seconds": 30,
    "max_delay_seconds": 3600
  },
  "failure_reason": "Database connection timeout",
  "job_metadata": {
    "job_type": "data_processing",
    "priority": "high"
  }
}

Dead letter queue strategy and poison message handling

The system implements a dead-letter queue (DLQ) for retry jobs that exceed maximum retry limits or fail consistently:

Poison Message Detection: Jobs that fail after reaching max retries are moved to DLQ
DLQ Queue Configuration: Separate queue with extended TTL (e.g., 7 days) and separate monitoring
Manual Intervention: DLQ messages are flagged for manual review by system administrators
Retry Limit Exceeded: Jobs that fail with same error pattern after 3 consecutive attempts go to DLQ immediately

At-least-once vs. exactly-once delivery

At-least-once delivery is selected for the job retry scheduler because:

Retry scheduling is idempotent - rescheduling same job with same parameters is safe
The system maintains state to prevent duplicate processing
Business logic can handle duplicate retry events gracefully
Simpler implementation and better performance characteristics

Retry policy for consumers

public class RetryPolicy {
    private int maxAttempts = 3;
    private long initialDelayMs = 1000;
    private long maxDelayMs = 30000;
    private double backoffMultiplier = 2.0;
    
    public long getDelay(int attempt) {
        if (attempt >= maxAttempts) return -1; // Stop retrying
        long delay = (long) (initialDelayMs * Math.pow(backoffMultiplier, attempt));
        return Math.min(delay, maxDelayMs);
    }
}

Retry strategy:

Max attempts: 3
Backoff formula: Exponential with base 2 (1s, 2s, 4s)
Max delay: 30 seconds
Non-retryable errors: Network timeouts, invalid job configuration, permanent failures

Outbox pattern for reliable event publishing

The outbox pattern is implemented to ensure reliable event publishing:

public class JobRetryEventOutbox {
    private final JdbcTemplate jdbcTemplate;
    
    @Transactional
    public void publishRetryEvent(JobRetryEvent event) {
        // Store event in outbox table
        String sql = "INSERT INTO job_retry_outbox (event_data, status, created_at) VALUES (?, ?, ?)";
        jdbcTemplate.update(sql, 
            objectMapper.writeValueAsString(event), 
            "PENDING", 
            LocalDateTime.now());
        
        // Publish to message queue in same transaction
        messageQueue.publish(event);
    }
}

Sequence diagram for async flows

sequenceDiagram
    participant JobScheduler
    participant JobExecutor
    participant MessageQueue
    participant RetryProcessor
    participant EventStore
    
    JobScheduler->>JobExecutor: Submit job execution
    JobExecutor-->>JobScheduler: Execution failed
    JobScheduler->>EventStore: Store retry event
    JobScheduler->>MessageQueue: Publish retry scheduled event
    MessageQueue-->>RetryProcessor: Async retry event
    RetryProcessor->>JobScheduler: Schedule retry job
    JobScheduler->>EventStore: Update job status

10. Error Handling Design

Error Classification Hierarchy

Error
├── ValidationException (400)
│   ├── InvalidInputException
│   └── ConstraintViolationException
├── DomainException (409)
│   ├── JobAlreadyExistsException
│   ├── JobNotFoundException
│   └── JobStateException
├── InfrastructureException (500)
│   ├── DatabaseException
│   ├── ExternalServiceException
│   └── TimeoutException
└── AuthorizationException (401/403)
    ├── UnauthorizedAccessException
    └── InsufficientPermissionException

Error Propagation Strategy

Domain and Service Layer: Catch and wrap infrastructure errors, rethrow domain-specific exceptions
Controller Layer: Handle and translate to appropriate HTTP responses
Cross-cutting Concerns: Log at error level, include stack trace for debugging

Retry Strategy with Exponential Backoff and Jitter

public class RetryConfig {
    private static final int MAX_RETRIES = 5;
    private static final long INITIAL_DELAY_MS = 1000;
    private static final double BACKOFF_MULTIPLIER = 2.0;
    private static final double JITTER_RANGE = 0.1; // ±10% jitter
    
    public static long calculateDelay(int attempt) {
        double delay = INITIAL_DELAY_MS * Math.pow(BACKOFF_MULTIPLIER, attempt);
        double jitter = delay * (Math.random() * 2 * JITTER_RANGE - JITTER_RANGE);
        return Math.round(delay + jitter);
    }
    
    public static boolean shouldRetry(int attempt, Exception exception) {
        return attempt < MAX_RETRIES && 
               (exception instanceof DatabaseException || 
                exception instanceof ExternalServiceException);
    }
}

Circuit Breaker Design

public class CircuitBreaker {
    private final int failureThreshold = 5;
    private final long timeout = 30000; // 30 seconds
    private final int resetTimeout = 60000; // 1 minute
    
    private volatile CircuitState state = CircuitState.CLOSED;
    private int failureCount = 0;
    private long lastFailureTime = 0;
    
    public enum CircuitState {
        CLOSED, OPEN, HALF_OPEN
    }
    
    public boolean canExecute() {
        switch (state) {
            case CLOSED:
                return true;
            case OPEN:
                if (System.currentTimeMillis() - lastFailureTime > resetTimeout) {
                    state = CircuitState.HALF_OPEN;
                    return true;
                }
                return false;
            case HALF_OPEN:
                return true;
        }
        return false;
    }
    
    public void recordSuccess() {
        if (state == CircuitState.HALF_OPEN) {
            state = CircuitState.CLOSED;
        }
    }
    
    public void recordFailure() {
        lastFailureTime = System.currentTimeMillis();
        failureCount++;
        
        if (failureCount >= failureThreshold) {
            state = CircuitState.OPEN;
        }
    }
}

Fallback Behavior Design

public class RetryService {
    private final CircuitBreaker circuitBreaker = new CircuitBreaker();
    
    public RetryResult executeWithFallback(RetryOperation operation) {
        try {
            if (circuitBreaker.canExecute()) {
                return operation.execute();
            } else {
                // Fallback to default behavior or cached result
                return fallback();
            }
        } catch (Exception e) {
            circuitBreaker.recordFailure();
            throw new InfrastructureException("Retry operation failed", e);
        }
    }
    
    private RetryResult fallback() {
        // Return default retry configuration or cached state
        return new RetryResult(false, "Fallback: Using default retry policy");
    }
}

Error Logging Format

public class ErrorLogger {
    private static final Logger logger = LoggerFactory.getLogger(ErrorLogger.class);
    
    public void logError(String operation, Exception exception, Map<String, Object> context) {
        Map<String, Object> errorContext = new HashMap<>(context);
        errorContext.put("error_code", getErrorCode(exception));
        errorContext.put("error_type", exception.getClass().getSimpleName());
        errorContext.put("stack_trace", exception.getStackTrace());
        
        logger.error("Operation failed: {}", operation, errorContext);
    }
    
    private String getErrorCode(Exception exception) {
        if (exception instanceof ValidationException) return "VALIDATION_ERROR";
        if (exception instanceof DomainException) return "DOMAIN_ERROR";
        if (exception instanceof InfrastructureException) return "INFRASTRUCTURE_ERROR";
        return "UNKNOWN_ERROR";
    }
}

Alerting Thresholds

Metric	Warning Threshold	Critical Threshold	Alert Channel
Retry failures per minute	10	50	Slack/Email
Circuit breaker open	1	5	On-call
Database timeouts	5	20	PagerDuty
External service errors	10	50	Slack/Email

Retryable vs Non-Retryable Errors

public class RetryableErrorClassifier {
    private static final Set<Class<? extends Exception>> RETRYABLE_EXCEPTIONS = 
        Set.of(DatabaseException.class, ExternalServiceException.class, TimeoutException.class);
    
    private static final Set<Class<? extends Exception>> NON_RETRYABLE_EXCEPTIONS = 
        Set.of(ValidationException.class, AuthorizationException.class);
    
    public static boolean isRetryable(Exception exception) {
        return RETRYABLE_EXCEPTIONS.stream()
            .anyMatch(clazz -> clazz.isInstance(exception));
    }
    
    public static boolean isNonRetryable(Exception exception) {
        return NON_RETRYABLE_EXCEPTIONS.stream()
            .anyMatch(clazz -> clazz.isInstance(exception));
    }
}

11. Security Design

Input validation and sanitization strategy

All API inputs are validated using a combination of:

Apache Commons Validator for standard validation (email, URL, numeric ranges)
Custom regex patterns for domain-specific constraints (job IDs, queue names)
Length limits to prevent buffer overflow attacks (max 255 chars for string fields)
Type coercion to ensure inputs match expected data types

public class JobRequestValidator {
    private static final Pattern JOB_ID_PATTERN = Pattern.compile("^[a-zA-Z0-9_-]{1,255}$");
    
    public void validateJobRequest(JobRequest request) {
        if (request.getJobId() == null || !JOB_ID_PATTERN.matcher(request.getJobId()).matches()) {
            throw new ValidationException("Invalid job ID format");
        }
        
        if (request.getRetryCount() < 0 || request.getRetryCount() > 100) {
            throw new ValidationException("Retry count must be between 0 and 100");
        }
    }
}

SQL injection prevention

All database interactions use parameterized queries exclusively:

JPA/Hibernate with named parameters for all dynamic queries
Prepared statements for raw SQL operations
No string concatenation in query building logic

Authentication middleware design

Authentication is handled via JWT tokens with the following flow:

OAuth2 Bearer token in Authorization header
Token validation using RS256 algorithm with public key verification
Role-based access control through JWT claims (admin, scheduler, user)
Token expiration handling with automatic refresh for valid sessions

Authorization enforcement

RBAC implementation:

Role hierarchy: admin > scheduler > user
Resource-based permissions:
- Admin: full CRUD access to all jobs and retry configurations
- Scheduler: create/update/delete own jobs, read all jobs
- User: read-only access to jobs they own

@PreAuthorize("hasRole('ADMIN') or (hasRole('SCHEDULER') and @jobAuthorizationService.canAccessJob(principal, #jobId))")
public void updateRetryConfig(String jobId, RetryConfig config) {
    // Implementation
}

Sensitive data encryption at rest and in transit

At rest:

AES-256-GCM encryption for sensitive job parameters and retry configurations
Key rotation every 90 days using AWS KMS or similar HSM
Encrypted job payloads stored in database fields with encrypted_data prefix

In transit:

TLS 1.3 required for all API endpoints
HSTS headers enabled to prevent protocol downgrade attacks
Mutual TLS for internal service-to-service communication

PII handling and masking strategy

PII fields (user IDs, email addresses) are encrypted at rest using AES-256
Masking in logs: user IDs shown as user_****1234
Audit trail logs only include non-sensitive job metadata (job name, status, retry count)
Data retention policy: PII data deleted after 30 days of job completion

Secrets management

AWS Secrets Manager integration for storing:
- Database connection strings
- External API keys (e.g., Slack, email service)
- Encryption keys for sensitive data
Automatic rotation every 90 days with zero downtime
Least privilege access - only necessary services can read specific secrets

Audit logging design

Events logged:

Job creation, update, deletion
Retry configuration changes
Manual retry triggers
Authentication/authorization failures

Log format:

{
  "timestamp": "2023-12-01T10:30:45.123Z",
  "trace_id": "abc123def456",
  "span_id": "xyz789uvw012",
  "user_id": "user_****1234",
  "job_id": "job_****5678",
  "action": "retry_config_updated",
  "ip_address": "10.0.0.1",
  "user_agent": "scheduler-client/2.1.0",
  "status": "success"
}

Retention policy: Audit logs retained for 90 days with daily backups to S3 with encryption.

12. Testing Strategy

Unit Testing Scope and Coverage Targets (%)

Service Layer: 85% code coverage with focus on business logic, error paths, and boundary conditions
Repository Layer: 90% coverage for all data access methods with mock database interactions
Domain Objects: 100% coverage for value objects and immutable entities
Exception Handling: 100% coverage for all custom exception scenarios
Edge Cases: 95% coverage for retry logic edge cases (max retries, exponential backoff, failure states)
Idempotency: 100% coverage for concurrent duplicate request handling

Integration Testing Strategy

Database Integration: Test with embedded H2 or PostgreSQL in-memory instances for all CRUD operations and transaction boundaries
External Service Mocking: Use WireMock or Mockito to simulate external API failures and network issues
Event System Testing: Test with embedded Kafka or RabbitMQ (or mock) to validate async flows and message consumption
Caching Layer: Integration tests with Redis stub or in-memory cache to validate cache hit/miss scenarios and TTL behavior
Transaction Boundaries: Use database transaction rollback to ensure data consistency in multi-step operations

Contract Testing

API Contract Testing: Use Pact for consumer-driven contract testing between job scheduler and downstream services
Event Schema Contract Testing: Use Avro schema registry with contract testing to ensure event payload consistency across services
External API Contracts: Validate all external service responses with contract tests to prevent breaking changes

Performance Testing Tool

Tool: k6 (JavaScript-based load testing) for simulating concurrent job scheduling and retry operations
Key Scenarios:
- High-concurrency job creation (1000+ concurrent requests)
- Retry rate stress testing (50% of jobs failing, max retries = 3)
- Memory leak detection under long-running retry cycles
- Latency distribution testing for job scheduling under load (p50/p95/p99 targets)
Metrics Tracked:
- Throughput (jobs/sec)
- Latency (p50/p95/p99 response times)
- Error rates (retry failures, system errors)
- Resource utilization (CPU, memory, DB connections)

Test Data Management Strategy

Fixtures: Predefined job configurations and retry states for consistent test environments
Factories: Factory methods to generate diverse job types (success, failure, timeout) with varying retry configurations
Seeding: Automated database seeding scripts to populate test environments with realistic job data (e.g., 10k jobs with different retry patterns)
Test Data Isolation: Use separate test databases per environment to prevent cross-test contamination
Data Cleanup: Automated cleanup procedures after each test suite run to maintain test environment purity

// Example unit test for retry logic with Mockito
@Test
public void testJobRetryWithExponentialBackoff() {
    Job job = createTestJob();
    when(jobRepository.findById(any())).thenReturn(Optional.of(job));
    
    // Simulate job failure and retry
    jobService.scheduleJob(job);
    
    verify(jobRepository, times(3)).save(any());
}

13. Observability Design

Structured Logging Schema

All logs must follow a consistent schema to enable effective querying and analysis:

{
  "timestamp": "2023-10-27T14:30:45.123Z",
  "level": "INFO|WARN|ERROR",
  "service": "job-scheduler-service",
  "trace_id": "12345678-1234-1234-1234-123456789012",
  "span_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "user_id": "user-123",
  "job_id": "job-456",
  "operation": "retry_job",
  "message": "Job retry scheduled for job-456",
  "error_code": "JOB_RETRY_FAILED",
  "error_details": {
    "exception_type": "java.lang.IllegalArgumentException",
    "stack_trace": "..."
  },
  "duration_ms": 150,
  "metadata": {
    "retry_count": 2,
    "next_retry_at": "2023-10-27T15:00:00.000Z"
  }
}

PII masking rules:

User IDs and job IDs are logged in full for debugging but may be redacted in production logs
Sensitive fields like passwords or tokens are never logged

Distributed Tracing Instrumentation

Using OpenTelemetry for distributed tracing with custom spans:

Key Operations to Trace:

job_retry_scheduler.process - Main job retry processing
job_retry_scheduler.fetch_jobs - Fetching jobs from database
job_retry_scheduler.schedule_retry - Scheduling retry in queue
job_retry_scheduler.execute_retry - Actual job execution

Span Attributes:

job.id: Unique identifier for the job being retried
job.type: Type of job (e.g., "email_notification", "payment_processing")
retry.count: Current retry attempt number
retry.delay.ms: Delay in milliseconds before next retry
job.status: Current job status (QUEUED, PROCESSING, FAILED)
error.code: Error code if operation failed

Metrics Design

RED Method Metrics:

Metric Name	Type	Labels	Description
`job_retry_scheduler.job_count`	Gauge	job_type, status	Current count of jobs by type and status
`job_retry_scheduler.retry_attempts`	Counter	job_type, retry_status	Total retry attempts (success/failure)
`job_retry_scheduler.retry_processing_duration`	Histogram	job_type, operation	Processing time for retry operations
`job_retry_scheduler.retry_queue_size`	Gauge	queue_name	Size of retry queues
`job_retry_scheduler.failed_job_count`	Counter	job_type, failure_reason	Count of failed jobs
`job_retry_scheduler.retry_backoff_duration`	Histogram	job_type, retry_count	Time spent in backoff

Example Metrics Output:

job_retry_scheduler.job_count{job_type="email_notification", status="failed"} 12
job_retry_scheduler.retry_attempts{job_type="payment_processing", retry_status="success"} 45
job_retry_scheduler.retry_processing_duration{job_type="email_notification", operation="process"} 150ms
job_retry_scheduler.retry_queue_size{queue_name="high_priority"} 23

Health Check Endpoint Design

Endpoints:

/health - Readiness check for core dependencies
/ready - Liveness check for service availability
/live - Basic service status
/startup - Startup validation

Response Format:

{
  "status": "healthy|unhealthy",
  "timestamp": "2023-10-27T14:30:45.123Z",
  "service": "job-scheduler-service",
  "checks": {
    "database": {
      "status": "healthy",
      "latency_ms": 5
    },
    "message_queue": {
      "status": "healthy",
      "latency_ms": 12
    },
    "cache": {
      "status": "healthy",
      "latency_ms": 3
    }
  },
  "version": "1.2.3"
}

Alerting Thresholds

Critical Alerts (Escalation to On-Call):

job_retry_scheduler.retry_attempts counter increases > 100 in 5 minutes
job_retry_scheduler.failed_job_count counter increases > 50 in 10 minutes
job_retry_scheduler.retry_processing_duration p99 > 5000ms for 10 minutes
job_retry_scheduler.retry_queue_size > 1000 for 5 minutes

Warning Alerts (Monitoring Team):

job_retry_scheduler.retry_processing_duration p99 > 2000ms for 15 minutes
job_retry_scheduler.failed_job_count counter increases > 20 in 15 minutes
job_retry_scheduler.job_count with status="failed" > 100 for 5 minutes

On-Call Escalation Path:

Critical alerts → On-call engineer (SMS + Slack)
Warning alerts → Monitoring team (Slack)
Service degradation → Engineering lead (Slack + email)

Log Rotation and Retention

Log files rotated daily at 12:00 AM UTC
Keep 30 days of logs in production environments
Compress rotated logs using gzip compression
Archive logs older than 90 days to long-term storage (S3 or similar)
Retention policy enforced via log management system (e.g., ELK stack)

14. Open Questions, Risks & Trade-offs

Decisions requiring product / engineering alignment

Retry policy granularity: Should retry policies be defined at the job level, queue level, or system-wide? This requires alignment between teams on whether to support per-job configuration vs. unified policies.
Maximum retry limits: What's the acceptable upper bound for retries? Too high and we risk system overload; too low and legitimate transient failures may not be handled.
Job categorization strategy: Should we implement job priority tiers (e.g., critical vs. background) with different retry behaviors? This affects system resource allocation and SLA expectations.
Notification strategy: When should system administrators be alerted about failed jobs? Should all failures trigger alerts or only those exceeding retry limits?

Known trade-offs and justifications

Trade-off	Option A	Option B	Justification
Retry vs. Dead Letter Queue	Immediate retry on failure	Move to DLQ after N attempts	Option A provides better fault tolerance but risks system overload; Option B prevents system thrashing but may lose jobs
Synchronous vs. Asynchronous Retry	Retry immediately in same thread	Queue for later processing	Option A provides faster recovery but blocks the calling thread; Option B enables better scalability and avoids blocking
Memory vs. Persistent Retry State	Keep retry metadata in-memory	Store retry state in database	Option A offers better performance but loses state on system restart; Option B provides durability but adds I/O overhead
Fixed vs. Exponential Backoff	Fixed 5-second retry intervals	Exponential backoff with jitter	Option A is simpler to implement and debug; Option B prevents thundering herd and system overload

Risks and mitigation strategies

System Overload Risk: High-volume job failures could overwhelm system resources. Mitigation: Implement circuit breaker patterns, queue prioritization, and configurable retry rate limits per job type.
Infinite Retry Loop Risk: Jobs that fail consistently could create infinite retry cycles. Mitigation: Implement maximum retry limits, exponential backoff with jitter, and alerting on jobs exceeding retry thresholds.
Data Consistency Risk: Concurrent retries might cause data integrity issues. Mitigation: Use optimistic locking, transactional boundaries, and idempotency patterns for retry operations.
Resource Exhaustion Risk: Retry mechanisms consuming excessive memory or database connections. Mitigation: Implement bounded retry queues, connection pooling limits, and proper resource cleanup.

Technical debt acknowledged

Hardcoded retry policies: Current design assumes fixed retry configurations. Future technical debt includes supporting dynamic policy updates without system restarts.
Limited visibility into retry behavior: Current implementation provides basic metrics but lacks detailed insights into why retries occur and their impact on system performance.
No job dependency management: Retry logic doesn't account for job dependencies or cascading failures, which could lead to inefficient resource usage during retry chains.
Inconsistent error handling across job types: Different job types may have varying retry strategies, but the current design doesn't support granular per-type handling without code duplication.
No support for job-specific retry configuration: The system currently uses global retry policies, but individual jobs may require different retry behaviors that aren't currently supported.