What does Powertools' process_partial_response return, and why does it matter?

It returns a batchItemFailures list with the IDs of messages that failed, so Lambda deletes the successful messages and only the failed ones are retried, instead of the whole batch.

Why use the invoice ID rather than the SQS message ID as the idempotency key?

Duplicates can come from the producer sending the same request twice, which creates different SQS message IDs. The business key (invoice ID) identifies the action itself, so any duplicate is recognised.

What does idempotent_function store, and where?

It stores a record per idempotency key, with its status (in progress or completed) and the function's result, in a persistence layer such as a DynamoDB table with a TTL, returning the stored result for repeated keys.

Batch Processing and Idempotency with Powertools — Python Serverless on AWS Lambda

Scenario

Example app: a Python function consumes an SQS queue of send-invoice messages in batches of 10, renders a PDF and emails it.

What you see in production

Customers receive the same invoice email two or three times. When one malformed message appears, the whole batch is retried repeatedly until it lands in the DLQ, along with nine valid messages.

The naive code

import json

def handler(event, context):
    for record in event["Records"]:
        msg = json.loads(record["body"])
        pdf = render_invoice(msg["invoice_id"])      # raises for one bad message
        send_email(msg["customer_email"], pdf)        # earlier messages already emailed

Reproduce it

Partial batch failures and idempotent sendsTry it yourself

def run(batch, partial, idempotent, rounds=3):
    sent, done = [], set()
    queue = list(batch)
    for _ in range(rounds):
        failed = []
        for m in queue:
            if m.get("bad"):
                failed.append(m)
                if not partial:
                    break
                continue
            if idempotent and m["id"] in done:
                continue                       # already processed: skip the side effect
            sent.append(m["id"]); done.add(m["id"])
        queue = queue if (failed and not partial) else failed
        if not failed:
            break
    return len(sent), sorted(set(sent))

batch = [{"id": f"inv-{i}"} for i in range(1, 6)] + [{"id": "inv-bad", "bad": True}] + [{"id": "inv-7"}]
print("throw on error, no idempotency:", run(batch, partial=False, idempotent=False))
print("partial failures + idempotency:", run(batch, partial=True, idempotent=True))

Output

throw on error, no idempotency: (15, ['inv-1', 'inv-2', 'inv-3', 'inv-4', 'inv-5'])
partial failures + idempotency: (6, ['inv-1', 'inv-2', 'inv-3', 'inv-4', 'inv-5', 'inv-7'])

Root cause

When an SQS-triggered handler raises, the entire batch returns to the queue and is delivered again, so already-processed messages repeat their side effects. SQS standard queues are also at-least-once, so duplicates happen even without errors.

The fix

Use Powertools batch processing to report only failed messages, and the idempotency utility to make each message's side effect happen once:

from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, process_partial_response
from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer, IdempotencyConfig, idempotent_function)

processor = BatchProcessor(event_type=EventType.SQS)
persistence = DynamoDBPersistenceLayer(table_name="idempotency")
config = IdempotencyConfig(event_key_jmespath="invoice_id", expires_after_seconds=24 * 3600)

@idempotent_function(data_keyword_argument="message", persistence_store=persistence, config=config)
def send_invoice(message: dict) -> dict:
    pdf = render_invoice(message["invoice_id"])
    send_email(message["customer_email"], pdf)
    return {"sent": message["invoice_id"]}

def record_handler(record):
    send_invoice(message=record.json_body)

def handler(event, context):
    config.register_lambda_context(context)
    return process_partial_response(event=event, record_handler=record_handler, processor=processor, context=context)

Enable ReportBatchItemFailures on the event source mapping, configure a DLQ with a sensible maxReceiveCount, and set the queue's visibility timeout to at least six times the function timeout. (The Node.js course covers these queue settings in depth.)

Trade-offs

The idempotency table adds a DynamoDB read and write per message and needs a TTL.
Choose idempotency keys that represent the business action (invoice ID), not the SQS message ID, which differs when a producer sends the same request twice.
Partial batch responses do not help FIFO ordering: a failing message still blocks its message group.

Detect it in production

Duplicate-send counts per invoice; DLQ depth; ApproximateAgeOfOldestMessage.
Powertools idempotency metrics/log entries for replayed keys.
Messages received much more often than sent (redeliveries).

Key points

A raised exception retries the whole SQS batch; use Powertools batch processing.
SQS is at-least-once: make side effects idempotent by business key.
DLQ, max receive count and visibility timeout configured deliberately.

Further reading: Powertools for AWS Lambda (Python): Batch processing · Powertools for AWS Lambda (Python): Idempotency