Acknowledgements, Visibility Timeouts and Prefetch — Queues & Background Processing in Depth

Consumers acknowledge messages after processing; unacknowledged messages are redelivered. SQS hides received messages for a visibility timeout (extend it for long jobs); RabbitMQ tracks unacked deliveries per channel with prefetch (QoS) limits; Kafka uses offset commits per partition (commit after processing for at-least-once). Prefetch too high → one slow consumer hoards messages; too low → throughput suffers.

In-flight message handling

Broker	In-flight tracking	Redelivery trigger	Tuning knob
SQS	Visibility timeout	Timeout expires without delete	VisibilityTimeout, ChangeMessageVisibility
RabbitMQ	Unacked per channel	Channel/connection closes or nack	basic.qos prefetch
Kafka	Committed offset	Consumer restarts from last commit	max.poll.records, commit strategy
Redis Streams	Pending entries list	XCLAIM/XAUTOCLAIM after idle	COUNT, idle time

msgs = sqs.receive_message(QueueUrl=url, MaxNumberOfMessages=10, WaitTimeSeconds=20, VisibilityTimeout=60)
for m in msgs.get("Messages", []):
    with heartbeat(lambda: sqs.change_message_visibility(QueueUrl=url, ReceiptHandle=m["ReceiptHandle"], VisibilityTimeout=60), every=30):
        process(json.loads(m["Body"]))
    sqs.delete_message(QueueUrl=url, ReceiptHandle=m["ReceiptHandle"])     # ack after processing

const ch = await conn.createChannel();
await ch.prefetch(10);                              // at most 10 unacked per consumer
await ch.consume("emails", async (msg) => {
  try { await sendEmail(JSON.parse(msg!.content.toString())); ch.ack(msg!); }
  catch (e) { ch.nack(msg!, false, false); }        // to dead-letter exchange
});

Common pitfalls

Visibility timeout shorter than processing time (duplicate processing).
Auto-ack modes losing messages on crash.
Huge prefetch with slow handlers.

Key points

Ack after processing
Extend visibility for long jobs
Tune prefetch to handler speed

Further reading: SQS: Visibility timeout

Practice questions

What happens if an SQS job runs longer than the visibility timeout?

Answer

The message becomes visible again and another consumer may process it concurrently, causing duplicate work unless the timeout is extended or the handler is idempotent.

Why can a high RabbitMQ prefetch hurt?

Answer

A consumer grabs many messages it can't process quickly, starving other consumers and delaying those messages.