SYSTEM DESIGN · QUESTION 114
How would you design a distributed job scheduler?
On this page
I would clarify scheduling precision, recurrence, job duration, retries, and execution guarantees. Store job definitions and run state durably, have schedulers claim due work with leases, and dispatch it through queues to workers. Use idempotency for retries, heartbeats for long jobs, and a dead-letter path. Separate “scheduled,” “started,” and “completed” states, and define what happens after scheduler failover.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A scheduler decides when work should run, while workers perform it. In a distributed service, several schedulers may see the same due job, and a worker can crash midway. Durable state, leases, and idempotent execution help prevent lost or repeatedly duplicated work.
A USEFUL COMPARISON
Picture it this way
A dispatch board assigns a task to one worker and records when the assignment expires. If the worker disappears, another can claim it. The task itself may have started before the failure, so retries need a way to avoid repeating harmful side effects.
FOLLOW THE JOURNEY
Claim due jobs with leases and dispatch them to workers
Show durable schedule state, claiming, execution, heartbeat, and retry.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Recovery must be visible and safe.
A scheduler finds due jobs, atomically claims each with a lease, queues execution to workers, records heartbeats and completion, and retries expired claims safely.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Recovery must be visible and safe.
UNDER THE HOOD
How it works, step by step
Persist schedule
Store recurrence, next run time, ownership, and job configuration.
Why it matters: Schedulers need durable state across restarts.
Claim work atomically
Use a lease or conditional update for due jobs.
Why it matters: Multiple scheduler instances must not dispatch the same run concurrently.
Execute and heartbeat
Workers report progress and renew leases for long jobs.
Why it matters: The scheduler must distinguish slow work from lost workers.
Complete or retry
Record results, apply bounded retry policy, or route failures for review.
Why it matters: Recovery must be visible and safe.
OUTSIDE THE TEXTBOOK
A practical example
A nightly report job becomes due while two scheduler instances are active. A conditional claim grants one instance a lease and queues the run. The worker periodically renews its lease; if it crashes, another worker can retry after expiry, using an idempotent output key to avoid duplicate reports.
GO A LITTLE DEEPER
Technical details that matter
- Leases need clock and expiration assumptions; fencing tokens can prevent a stale worker from writing after losing ownership.
- Exactly-once execution is difficult; design jobs for at-least-once delivery with idempotent effects where possible.
- Recurring schedule semantics need timezone, daylight-saving, missed-run, and overlap policies.
MAKE A GOOD CHOICE
When is this useful?
- Recurring reports, data pipelines, and background workflows.
- Distributed task execution with durable retries.
- Coordinating scheduled work across multiple service instances.
A single-process scheduler can be sufficient for low-volume internal jobs if durability and failover are not requirements.
WATCH OUT FOR
Common misunderstandings
- Assuming a lease guarantees the old worker has stopped.
- Retrying side effects without idempotency.
- Ignoring timezone and overlapping-run behavior.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Leases, heartbeats, and recovery add state complexity.
- Long-running tasks may need checkpointing or compensation.
- Scheduler precision can be affected by queue and worker capacity.
WHAT TO REMEMBER
Key takeaways
- 1Persist job state and atomically claim due work.
- 2Assume retries and worker loss can happen.
- 3Use idempotency, leases, and explicit recurrence semantics.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
What is a lease used for?
It grants temporary ownership of work and allows another worker to recover it after expiration, though stale workers may require fencing to prevent writes.
Can a scheduler guarantee exactly-once execution?
Usually not across arbitrary failures and side effects. Design for retries and idempotent outcomes, and state the delivery guarantee.
SAY IT OUT LOUD
How to explain this in an interview
“I would store job definitions and run state durably, have schedulers atomically claim due jobs with leases, and dispatch them to workers through queues. Workers report heartbeats and completion; expired or failed runs follow bounded retry policies. Since retries can happen, jobs should be idempotent, and schedule semantics must define time zones and missed runs.”
Connect this question to the roadmap topic it supports.