Design a Job Scheduler — System Design Interview Practice

Interview prompt

Design a distributed job scheduler for one-time and recurring tasks that provides durable schedules, accurate due-time dispatch, worker leases, retries, backoff, and tenant fairness.

• Make the schedule and execution state authoritative, with leases and fencing preventing two workers from owning one attempt.
• Partition due work by time bucket and tenant, use priority and fairness, and avoid scanning millions of future jobs on every tick.
• Define at-least-once execution, idempotency keys, retry/backoff, misfire policy, concurrency limits, and cancellation.
• Explain clock skew, scheduler failover, worker loss, recurring-job drift, dead letters, quotas, and auditability.

Requirements and scale assumptions

• Create, update, pause, resume, and delete one-time or recurring schedules with timezone and misfire policies.
• Claim due jobs, execute attempts, heartbeat leases, retry failures, dead-letter exhausted jobs, and expose run history.
• Support tenant quotas, priority, concurrency limits, idempotent cancellation, audit records, and safe replay.
• Dispatch jobs within a five-second p95 due-time tolerance under normal load.
• Handle 50 million schedules through time buckets, partitioned queues, and tenant-aware worker pools.
• Never lose a committed schedule; make dispatch, lease renewal, retry, and completion idempotent.
• Preserve due work and recover leases when schedulers, workers, or downstream task dependencies fail.
• Support 50 million schedules, 5 million daily executions, and bursty top-of-hour workloads.
• Partition by tenant and due-time bucket; shard hot tenants and spread recurring jobs with jitter.
• Retain schedules, attempts, leases, and audit history while keeping ready queues and worker heartbeats bounded.

• Schedule volume: 50M schedules — Volume drives time buckets, queue sharding, fairness, and due-work indexing.
• Dispatch tolerance: p95 <=5s — Measures due time to first leased attempt, separate from task runtime.
• Durable boundary: Committed before async — The source of truth is Schedule jobs for future execution; Support one-time and recurring jobs.
• Async boundary: At-least-once workers — Keep Priority queue for job scheduling, Distributed lock for job execution, Heartbeat mechanism for worker health off the synchronous path.

Key entities

• ResourceSpecresourceId, tenantId, desiredState, version, policyVersion, updatedAt

  Versioned desired state for a job scheduler managed resource.

• OperationoperationId, resourceId, requestHash, step, attempt, status

  Durable job scheduler reconciliation operation with per-step progress.

• PolicyVersionpolicyId, scope, version, rules, effectiveAt, status

  Auditable job scheduler policy evaluated before provisioning or mutation.

• ReconciliationCheckpointresourceId, provider, observedVersion, cursor, lastError, updatedAt

  Provider-specific job scheduler observation and recovery cursor.

Data flow

1. 1. Accept a desired-state commandThe job scheduler control plane authenticates the tenant, validates policy and quotas, checks the expected version, and records the desired state.
2. 2. Plan a safe operationA planner turns job scheduler desired state into ordered, bounded steps with dependency checks, blast-radius limits, and rollback metadata.
3. 3. Reconcile providers asynchronouslyWorkers apply job scheduler operations through provider adapters, persist checkpoints, rate-limit calls, and treat unknown outcomes as observable state.
4. 4. Publish observed healthThe serving projection joins desired and observed job scheduler state with operation status, policy version, freshness, and actionable errors.
5. 5. Recover and auditRetries, dead letters, drift detection, and operator approvals repair job scheduler resources without losing the original command or provider evidence.

Deep dives and trade-offs

• Desired versus observed stateKeep job scheduler desired state separate from provider-observed state and show both to operators. Make every reconciliation step conditional and resumable so a worker crash does not restart unsafe effects. Version policy and resource state so old operations cannot overwrite newer intent.
• Provider failures and unknown outcomesUse provider-specific idempotency tokens and query-after-timeout behavior for job scheduler operations. Bound retries with exponential backoff, circuit breakers, and per-provider quotas. Route irreconcilable drift to an approval or quarantine path instead of retrying forever.
• Blast radius and operationsPartition job scheduler work by tenant, region, cluster, or resource class and cap concurrent mutations. Audit who changed desired state, which policy allowed it, and what provider evidence was observed. Alert on drift age, operation backlog, failed steps, policy denials, and stale observations.
• Push versus pull reconciliationUse event triggers for fast response and periodic scans for missed events, drift, and recovery. A push-only job scheduler controller silently misses changes when a provider event is lost.
• Central control plane versus provider-native controllersKeep policy, intent, and audit centralized while isolating provider-specific application logic behind adapters. A monolithic controller becomes hard to scale and couples unrelated provider failure domains.
• Automatic repair versus approvalAutomate low-risk, reversible job scheduler changes and require approval for destructive or high-blast-radius operations. Full automation without policy or blast-radius controls can turn a transient signal into a widespread outage.