Process background jobs with BullMQ

Approach

Use BullMQ (Redis-backed) with a Queue in the API to enqueue jobs and a separate Worker process to handle them. Configure attempts with exponential backoff, concurrency, job IDs for deduplication, and cleanup of completed jobs. Make handlers idempotent.

Solution

// queue.ts (shared)
import { Queue, Worker, type Job } from "bullmq";
const connection = { url: process.env.REDIS_URL! };

export const emailQueue = new Queue<{ userId: string; template: string }>("emails", {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: "exponential", delay: 5_000 },
    removeOnComplete: { age: 24 * 3600 },
    removeOnFail: { age: 7 * 24 * 3600 },
  },
});

// In the API, after the DB commit:
await emailQueue.add("welcome", { userId: user.id, template: "welcome" }, { jobId: `welcome:${user.id}` });  // dedupe

// worker.ts (separate process)
const worker = new Worker("emails", async (job: Job<{ userId: string; template: string }>) => {
  const user = await users.get(job.data.userId);
  if (await sentEmails.exists(job.id!)) return;           // idempotent
  await mailer.send(user.email, job.data.template);
  await sentEmails.record(job.id!);
}, { connection, concurrency: 10 });

worker.on("failed", (job, err) => logger.warn({ jobId: job?.id, err }, "email job failed"));
process.on("SIGTERM", async () => { await worker.close(); process.exit(0); });

Pitfalls

• Running workers inside the web process (they compete for CPU and die with deploys).
• Enqueuing before the DB transaction commits.
• Redis without persistence (AOF) losing jobs on restart.

Variations

• Repeatable/cron jobs with queue.upsertJobScheduler.
• Managed queues (SQS, Cloud Tasks) to avoid operating Redis.

Further reading: BullMQ documentation