Why can one bad record stop processing for part of a DynamoDB stream?

Stream event sources preserve per-shard order, so a failed batch is retried without moving past it until it succeeds, the record expires (24 hours) or retry limits are hit. Everything behind it in that shard waits.

Which event source mapping settings limit the damage of a poison record?

BisectBatchOnFunctionError to isolate it, MaximumRetryAttempts and MaximumRecordAgeInSeconds to bound retries, an on-failure destination to capture discarded records, and ReportBatchItemFailures to resume after the last good record.

Which metric tells you a stream consumer is falling behind?

IteratorAge: the age of the last record processed. A steadily rising value means the consumer is blocked or too slow.

DynamoDB Streams: Ordering and Poison Records — Node.js Serverless on AWS Lambda

Scenario

Example app: a DynamoDB Streams consumer keeps a search index and a cache in sync whenever products change.

What you see in production

Search results go stale for hours for a subset of products. The function's error count is high and the IteratorAge metric for the stream climbs steadily. Other products update normally.

The naive code

export const handler = async (event) => {
  for (const record of event.Records) {
    const product = unmarshall(record.dynamodb.NewImage);
    await searchIndex.upsert(product);        // throws for one malformed product
  }
};

The event source mapping uses defaults: retry until the record expires, no failure destination.

Reproduce it

A poison record blocks its shardTry it yourself

const shardA = ["p1", "p2", "BAD", "p4", "p5"];
const shardB = ["p6", "p7", "p8"];

function run(shard, { bisect, maxRetries }) {
  const indexed = [];
  let retries = 0, discarded = [];
  let i = 0;
  while (i < shard.length) {
    if (shard[i] === "BAD") {
      if (maxRetries !== Infinity && retries >= maxRetries) { discarded.push(shard[i]); i++; retries = 0; continue; }
      retries++;
      if (!bisect && retries > 50) return { indexed, blockedAt: shard[i], note: "still retrying..." };
      continue;
    }
    indexed.push(shard[i]); i++;
  }
  return { indexed, discardedToDestination: discarded };
}
console.log("defaults, shard A:", JSON.stringify(run(shardA, { bisect: false, maxRetries: Infinity })));
console.log("defaults, shard B:", JSON.stringify(run(shardB, { bisect: false, maxRetries: Infinity })));
console.log("bisect + 2 retries + destination, shard A:", JSON.stringify(run(shardA, { bisect: true, maxRetries: 2 })));

Output

defaults, shard A: {"indexed":["p1","p2"],"blockedAt":"BAD","note":"still retrying..."}
defaults, shard B: {"indexed":["p6","p7","p8"],"discardedToDestination":[]}
bisect + 2 retries + destination, shard A: {"indexed":["p1","p2","p4","p5"],"discardedToDestination":["BAD"]}

Root cause

Stream event sources deliver records in order per shard. If a batch fails, Lambda retries the same batch and does not move on, to preserve ordering, until the records expire from the stream (24 hours for DynamoDB Streams) or the configured retry limits are reached. A single poison record therefore blocks everything behind it in that shard, while other shards continue, which is why only some items go stale.

The fix

Configure the event source mapping for failure, and process records idempotently:

BisectBatchOnFunctionError: split failing batches to isolate the bad record.
MaximumRetryAttempts and MaximumRecordAgeInSeconds: bound how long a failure can block the shard.
On-failure destination (SQS or SNS): receive metadata about discarded records for investigation and replay.
ReportBatchItemFailures: report the first failed record's sequence number so processing resumes from there instead of repeating the whole batch.

Write consumers that tolerate duplicates and out-of-date events (compare a version or timestamp before overwriting the index), because retries and bisection re-deliver records.

Trade-offs

Discarding records after limited retries keeps the stream moving but means some updates need a reconciliation path (replay from the destination or a periodic full sync).
Per-shard ordering only: events for different items can arrive in any relative order.
Parallelization factor can increase throughput per shard but must keep per-key ordering in mind.

Detect it in production

IteratorAge alarms: a rising value means the consumer is falling behind or blocked.
Error counts concentrated on repeated sequence numbers.
Messages arriving in the on-failure destination.

Key points

Streams retry the same batch and block the shard until success or expiry.
Bisect, cap retries and record age, and set an on-failure destination.
Report partial batch failures; make consumers idempotent and version-aware.
Alarm on IteratorAge.

Further reading: Using Lambda with DynamoDB Streams · Lambda event source mappings: error handling