Design a distributed job scheduler that runs millions of scheduled tasks reliably — how would you build it?

#Why plain cron doesn't survive production scale

A single crontab works until it doesn't: it runs on exactly one machine, so that machine is a single point of failure; it has no durable record of what fired and what didn't across a restart; and it has no way to coordinate with a second instance, so the naive fix for "what if that machine dies" — running two — causes every job to fire twice. None of that is acceptable once you're scheduling millions of jobs across a fleet: you need durable storage of due times, horizontal scalability of both storage and dispatch, and a way to guarantee that a job that's due gets claimed by exactly one dispatcher, even with multiple dispatcher processes running concurrently.

#Requirements worth stating explicitly

Restate design goals before jumping to a solution, because they justify every subsequent choice: at-least-once delivery (never promise exactly-once at scale — design the execution side to be idempotent? instead), horizontal scalability in both the number of stored jobs and the dispatch throughput, fault tolerance so a crashed scheduler node loses nothing, and low dispatch latency so jobs fire close to their due time rather than drifting by minutes under load.

#Storage: a time-bucketed job table

Instead of one giant, unordered table of jobs, partition due jobs by time into buckets — for example, one bucket per minute. A relational table indexed on (due_bucket, status) lets a dispatcher cheaply ask "give me every pending job whose bucket is now or earlier" without scanning the whole table. An in-memory structure like a Redis sorted set, scored by Unix timestamp, works as a fast staging layer in front of the same idea: ZADD to schedule, ZRANGEBYSCORE 0 now to find what's due. The database table remains the durable source of truth; the sorted set is an accelerant, not a replacement.

#Leader election to avoid double dispatch

Running one scheduler process is a single point of failure; running several without coordination means the same due job gets claimed and dispatched by more than one node. The fix that keeps the design simple is to elect exactly one active dispatcher among a fleet of standby replicas, using a lease with a short TTL held in something like Redis or etcd — whichever node holds the lease is the one allowed to scan for due jobs and enqueue them, and it must renew the lease before it expires or another node takes over. Losing leadership mid-scan is expected and must be safe: a job claimed via a row-level lock (or SKIP LOCKED semantics) but never successfully enqueued should be recoverable by whichever node becomes leader next.

#Reference implementation

The sketch below shows the two pieces that make this design work: leadership acquisition via a lease, and a dispatch loop that claims due jobs safely under concurrent access using SELECT ... FOR UPDATE SKIP LOCKED (so two nodes racing for leadership during a handoff never claim the same row), then hands them to a queue for actual execution.

Wrap long lines
package scheduler

import (
	"context"
	"database/sql"
	"time"

	"github.com/redis/go-redis/v9"
)

const leaseKey = "scheduler:leader"
const leaseTTL = 10 * time.Second

// tryAcquireLeadership attempts to become (or remain) the active dispatcher.
// It's safe to call repeatedly — SetNX only succeeds for a node that doesn't
// already hold the lease, and the current leader renews via SetXX below.
func tryAcquireLeadership(ctx context.Context, rdb *redis.Client, nodeID string) (bool, error) {
	ok, err := rdb.SetNX(ctx, leaseKey, nodeID, leaseTTL).Result()
	if err != nil {
		return false, err
	}
	if ok {
		return true, nil
	}
	current, err := rdb.Get(ctx, leaseKey).Result()
	if err != nil {
		return false, err
	}
	if current == nodeID {
		_, err = rdb.Expire(ctx, leaseKey, leaseTTL).Result()
		return true, err
	}
	return false, nil
}

// dispatchDueJobs runs on the leader only. It claims a batch of due jobs
// under SKIP LOCKED so a concurrent claimer (during a leadership handoff)
// never double-claims a row, then pushes each to the work queue.
func dispatchDueJobs(ctx context.Context, db *sql.DB, queue Enqueuer, batchSize int) error {
	tx, err := db.BeginTx(ctx, nil)
	if err != nil {
		return err
	}
	defer tx.Rollback()

	rows, err := tx.QueryContext(ctx, `
		SELECT id, payload, attempt
		FROM jobs
		WHERE status = 'pending' AND due_at <= now()
		ORDER BY due_at
		LIMIT $1
		FOR UPDATE SKIP LOCKED
	`, batchSize)
	if err != nil {
		return err
	}

	var claimed []Job
	for rows.Next() {
		var j Job
		if err := rows.Scan(&j.ID, &j.Payload, &j.Attempt); err != nil {
			rows.Close()
			return err
		}
		claimed = append(claimed, j)
	}
	rows.Close()

	for _, j := range claimed {
		if _, err := tx.ExecContext(ctx,
			`UPDATE jobs SET status = 'dispatched', dispatched_at = now() WHERE id = $1`, j.ID); err != nil {
			return err
		}
	}
	if err := tx.Commit(); err != nil {
		return err
	}

	for _, j := range claimed {
		if err := queue.Enqueue(ctx, j.ID, j.Payload, j.Attempt); err != nil {
			// Not fatal: the job stays 'dispatched' and a reconciliation
			// sweep re-queues anything stuck past a visibility timeout.
			continue
		}
	}
	return nil
}

The transaction does two things in one shot: it claims rows (so a losing leader that crashes mid-loop leaves no ambiguity — the rows are either committed as dispatched or the whole claim rolled back) and it marks them before they're actually pushed to the queue, which means a crash between commit and enqueue leaves jobs in dispatched limbo. A background reconciliation sweep that requeues anything stuck in dispatched past a visibility timeout (mirroring how SQS or a similar queue handles unacked messages) closes that gap.

#Idempotency on the execution side

Because delivery is at-least-once — the queue can redeliver, the reconciliation sweep can re-enqueue, a worker can crash after doing the work but before acking — the worker itself must treat every execution as possibly a retry. The standard technique is an idempotency ledger: before running a job's side effects, the worker inserts a row keyed by (job_id, attempt-independent idempotency_key) into a table with a unique constraint, and only proceeds if the insert succeeds. A duplicate delivery hits the unique constraint, finds the effect already recorded, and skips straight to acknowledging the message.