Review a Node.js Batch Processor

1. Unbounded concurrency

Critical, line 6-13

Promise.all(keys.map(...)) starts all 200,000 downloads and thumbnail calls at once. That exhausts sockets and file descriptors, gets the job rate-limited, and floods the thumbnail service that the live website also depends on.

Fix: Limit concurrency (for example 10 to 20 at a time) with a pool (p-limit, p-map with concurrency), and make the limit configurable.

2. Memory grows with the batch size

Major, line 8

Every in-flight download is a full image buffer in memory. With unbounded concurrency the process will run out of memory long before finishing.

Fix: Bounded concurrency fixes most of it; stream large objects instead of buffering, and process keys page by page instead of listing all of them first.

3. One failure fails everything, and progress is lost

Major, line 6

Promise.all rejects on the first error. The job stops reporting, the other operations keep running unobserved, and a rerun starts from scratch.

Fix: Use Promise.allSettled or per-item try/catch, record failures for retry, and make the job resumable (skip keys whose thumbnail already exists or track a checkpoint).

4. Unhandled rejection and exit code

Major, line 17

run() is not awaited or caught. A failure becomes an unhandled rejection; depending on the Node version the process crashes with a stack trace or exits 0, so the scheduler may think the job succeeded.

Fix: run().catch((err) => { console.error(err); process.exitCode = 1; });

5. No retries or progress reporting

Minor, line 8-11

Transient network errors are common over 200,000 operations, and there is no way to see how far a 3-hour job has got.

Fix: Retry transient failures with backoff, and log progress every N items (or emit metrics).

Promise.all over an unbounded list is one of the most common Node.js production mistakes. Interviewers want to hear about concurrency limits, memory, partial failure, and how the job reports success to whatever schedules it.

const pLimit = require("p-limit");
const { listImageKeyPages, download, upload, exists } = require("./storage");
const { makeThumbnail } = require("./thumbnail-service");
const { withRetry } = require("./retry");

const limit = pLimit(Number(process.env.THUMB_CONCURRENCY ?? 16));

async function processKey(key) {
  if (await exists(`thumbs/${key}`)) return "skipped"; // resumable
  const image = await withRetry(() => download(key));
  const thumb = await withRetry(() => makeThumbnail(image));
  await withRetry(() => upload(`thumbs/${key}`, thumb));
  return "done";
}

async function run() {
  const failed = [];
  let processed = 0;
  for await (const page of listImageKeyPages(1000)) {
    const outcomes = await Promise.allSettled(page.map((key) => limit(() => processKey(key))));
    outcomes.forEach((o, i) => { if (o.status === "rejected") failed.push(page[i]); });
    processed += page.length;
    console.log(`processed ${processed}, failed ${failed.length}`);
  }
  if (failed.length) throw new Error(`${failed.length} images failed: ${failed.slice(0, 10).join(", ")}...`);
}

run().catch((err) => {
  console.error(err);
  process.exitCode = 1;
});