SQS: Partial Batch Failures and Visibility Timeouts

Scenario

Example app: an order pipeline: the API puts order messages on an SQS queue, and a Lambda function consumes them in batches of 10 to reserve stock and send confirmations.

What you see in production

Customers receive duplicate confirmation emails, sometimes five or six. Stock counts drift downward. The queue's age-of-oldest-message metric keeps rising, and one malformed message has been "in flight" for hours.

The naive code

export const handler = async (event) => {
  for (const record of event.Records) {
    const order = JSON.parse(record.body);
    await reserveStock(order);            // one failure throws...
    await sendConfirmation(order);
  }
};                                         // ...and the whole batch returns to the queue

The function timeout is 60 seconds; the queue's visibility timeout is the default 30 seconds.

Reproduce it

function process(batch, reportPartialFailures) {
  const sent = [], failed = [];
  for (const m of batch) {
    if (m.poison) { failed.push(m.id); if (!reportPartialFailures) break; continue; }
    sent.push(m.id);
  }
  if (!reportPartialFailures && failed.length) return { sent, retry: batch.map((m) => m.id) };   // whole batch back
  return { sent, retry: failed };                                                               // only failures back
}

const batch = Array.from({ length: 10 }, (_, i) => ({ id: `m${i + 1}`, poison: i === 6 }));
for (const partial of [false, true]) {
  let emails = 0, queue = batch, attempts = 0;
  while (queue.length && attempts < 4) {            // maxReceiveCount = 4, then the DLQ
    const { sent, retry } = process(queue, partial);
    emails += sent.length;
    queue = batch.filter((m) => retry.includes(m.id));
    attempts++;
  }
  console.log(partial ? "partial batch responses:" : "throw on first error:  ", "emails sent =", emails, "| sent to DLQ:", queue.map((m) => m.id));
}

throw on first error:   emails sent = 24 | sent to DLQ: [ 'm1', 'm2', 'm3', 'm4', 'm5', 'm6', 'm7', 'm8', 'm9', 'm10' ]
partial batch responses: emails sent = 9 | sent to DLQ: [ 'm7' ]

Root cause

With SQS, Lambda deletes messages only when the invocation succeeds. If the handler throws, every message in the batch becomes visible again after the visibility timeout and is redelivered, including those already processed. A message that can never succeed (a poison message) keeps failing until it exceeds the queue's maxReceiveCount, and if there is no dead-letter queue it blocks forever. Separately, a visibility timeout shorter than the function timeout lets a second consumer receive a message while the first is still working on it.

The fix

• Turn on ReportBatchItemFailures in the event source mapping and return the IDs of failed messages, so only those are retried. Powertools' batch processing utility does this for you.
• Configure a dead-letter queue with a sensible maxReceiveCount (for example 3 to 5) and alarm on it.
• Set the queue's visibility timeout to at least six times the function timeout (AWS's guidance), to allow for retries within the event source mapping.
• Make processing idempotent: SQS standard queues deliver at least once, so duplicates still happen.

import { BatchProcessor, EventType, processPartialResponse } from "@aws-lambda-powertools/batch";
const processor = new BatchProcessor(EventType.SQS);

const recordHandler = async (record) => {
  const order = JSON.parse(record.body);
  await reserveStockOnce(order);       // idempotent by order ID
  await sendConfirmationOnce(order);
};

export const handler = async (event, context) =>
  processPartialResponse(event, recordHandler, processor, { context });   // returns { batchItemFailures }

Use the event source mapping's maximum concurrency setting to cap how many function instances consume the queue, protecting downstream systems.

Trade-offs

• Smaller batches mean less rework after failures but more invocations; batching windows trade latency for efficiency.
• FIFO queues preserve order per message group, so one failing message blocks the rest of its group: decide whether ordering is truly needed.
• A DLQ needs an owner, alarms and a redrive process, or it just hides failures.

Detect it in production

• ApproximateAgeOfOldestMessage rising, NumberOfMessagesReceived much higher than NumberOfMessagesSent (redeliveries).
• Messages arriving in the DLQ; alarm on any.
• Duplicate side effects in downstream systems (same order ID processed twice).

Key points

• A thrown error retries the whole batch; use partial batch responses.
• Always configure a DLQ with a sensible max receive count.
• Visibility timeout at least 6x the function timeout.
• At-least-once delivery: make processing idempotent; cap concurrency to protect downstreams.

Further reading: Using Lambda with Amazon SQS · Handling errors for an SQS event source