Queues & Background Processing in Depth · Background Job Processing
Scheduled Work and Distributed Cron
Run periodic jobs once, reliably, across many instances.
Periodic jobs (nightly reports, cleanups) must run exactly one instance per schedule across replicas. Options: a single scheduler process (Celery beat, BullMQ job schedulers), Kubernetes CronJobs with concurrencyPolicy: Forbid, distributed locks (ShedLock, Redis locks with fencing), or managed schedulers (EventBridge Scheduler, Cloud Scheduler). Make jobs idempotent, record runs, alert on missed runs (dead man's switch), and use UTC.
Scheduling options
| Option | Runs once? | Missed-run handling | Ops |
|---|---|---|---|
| cron on every app server | No (N times) | None | Bad |
| Single scheduler process | Yes (if one instance) | Depends | Low |
| K8s CronJob (Forbid) | Yes | startingDeadlineSeconds | Low |
| Distributed lock (ShedLock) | Yes | Next run | Low |
| Managed scheduler | Yes | Retries, DLQ | None |
apiVersion: batch/v1
kind: CronJob
metadata: { name: nightly-invoices }
spec:
schedule: "15 2 * * *"
timeZone: "UTC"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 3600
jobTemplate:
spec:
backoffLimit: 2
template:
spec:
restartPolicy: OnFailure
containers: [{ name: job, image: registry/app:1.42.0, args: ["python", "-m", "jobs.nightly_invoices"] }]
Common pitfalls
- Schedulers started inside every web worker.
- Local-time schedules breaking on DST.
- No alert when a job silently stops running.
Key points
- One run per schedule
- Idempotent jobs, UTC
- Dead man's switch alerts
Further reading: Kubernetes: CronJob
Practice questions
What does concurrencyPolicy: Forbid do on a Kubernetes CronJob?
Answer
It skips a new run if the previous one is still running, preventing overlapping executions.
What's a dead man's switch for cron jobs?
Answer
The job pings a monitoring service on success; if no ping arrives within the expected window, an alert fires — catching jobs that silently stopped running.
Open in the app to mark lessons complete, take notes and track your progress.