Priorities, Delays and Deduplication — Queues & Background Processing in Depth

Q: Why are separate queues often better than broker priorities?

They're predictable (dedicated capacity per class), easy to monitor and scale independently, and avoid starvation surprises.

Q: Why must a delayed job re-check state when it runs?

The situation may have changed (the cart was purchased, the reminder cancelled); acting on stale assumptions causes wrong actions.

Separate priorities with separate queues/worker pools (simplest and most predictable) or broker priorities (RabbitMQ priority queues, BullMQ priorities). Delayed jobs (retry later, reminders) use broker features (SQS DelaySeconds ≤ 15 min, RabbitMQ delayed-message plugin, BullMQ delay) or a scheduled-jobs table for long delays. Deduplicate enqueues with job IDs (BullMQ jobId, SQS FIFO dedup IDs) to avoid duplicate work from double submissions.

Scheduling patterns

Need	Approach
Urgent vs bulk	Separate queues + dedicated workers (e.g., emails.transactional vs emails.bulk)
Delay < 15 min	Broker delay (SQS DelaySeconds, BullMQ delay)
Delay hours/days	Scheduled table + poller, or workflow engine timer
Avoid duplicate jobs	Deterministic job IDs / dedup IDs
Cancel scheduled job	Store job id; delete or check a 'cancelled' flag on run

# BullMQ: delayed, prioritized, deduplicated
await reminders.add("cart-abandoned", { cartId }, {
  delay: 60 * 60 * 1000,            // 1 hour
  priority: 5,                      // lower number = higher priority
  jobId: `cart-abandoned:${cartId}`, // dedupe: re-adding is a no-op while it exists
});

Common pitfalls

One queue for transactional and bulk work.
Long delays held in memory-only schedulers.
Delayed jobs that don't re-check state when they run.

Key points

Separate queues for priorities
Right tool for delay length
Deterministic job IDs

Scheduling patterns

Common pitfalls

Key points

Practice questions