Messaging & Queues Interview Questions

Summary

Queues decouple producers from consumers in time and scale: work can be processed asynchronously, absorb traffic spikes (buffering), retry on failure, and scale consumers independently. They turn slow or unreliable steps (emails, image processing, third-party calls) into background work.

Longer Answer

They also enable event-driven architectures, where multiple services react to the same event. Costs: eventual consistency, more moving parts, duplicate and out-of-order delivery to handle, and monitoring of backlogs.

Key Points

• Asynchronous processing
• Buffer spikes
• Retries and fault isolation
• Independent scaling
• Adds eventual consistency

Summary

At-most-once may lose messages but never duplicates (ack before processing); at-least-once never loses messages but may deliver duplicates (ack after processing); exactly-once is achievable only within limited boundaries (e.g., Kafka transactions within Kafka) and in practice means at-least-once delivery plus idempotent processing.

Longer Answer

Most systems choose at-least-once and make consumers idempotent: deduplicate by message ID, use upserts, or record processed IDs in the same transaction as the side effect.

Key Points

• At-most-once: possible loss
• At-least-once: possible duplicates
• Exactly-once = at-least-once + idempotency
• Ack after processing

Summary

Give each message a unique ID and record processed IDs (in the same transaction as the business change, with a unique constraint), design operations as upserts or absolute state changes ('set status to paid' rather than 'add 10'), and check current state before acting.

Longer Answer

Example: an inbox table processed_messages(message_id PRIMARY KEY); insert it and apply the change in one DB transaction; on duplicate key, skip. Expire old IDs after the redelivery window.

Code Example

BEGIN;
INSERT INTO processed_messages (message_id) VALUES ($1);  -- fails on duplicate → skip
UPDATE orders SET status = 'paid' WHERE id = $2 AND status = 'pending';
COMMIT;

Key Points

• Unique message IDs
• Processed-IDs table in same transaction
• Upserts / absolute updates
• Check state first

Summary

Kafka is a distributed, partitioned, replicated log: messages are retained and consumers track offsets, enabling replay, high throughput and ordered processing per partition. RabbitMQ is a message broker with flexible routing (exchanges, queues), per-message acks and work-queue semantics. SQS is a fully managed queue (standard or FIFO) with minimal operations.

Longer Answer

Use Kafka for event streaming, event sourcing, analytics pipelines and many consumer groups replaying data; RabbitMQ for task queues and complex routing; SQS/SNS or Google Pub/Sub for simple managed messaging on cloud. Redis Streams and NATS JetStream are lighter alternatives.

Key Points

• Kafka: retained log, replay, partitions
• RabbitMQ: routing, acks, task queues
• SQS: managed simplicity
• Choose by retention and routing needs

Comparison / Tradeoffs

Summary

Retry transient failures with exponential backoff and jitter up to a limit; after max attempts, move the message to a dead-letter queue (DLQ) for inspection and replay instead of blocking the queue or losing it. Poison messages (that always fail) must not block other messages.

Longer Answer

Distinguish transient (timeouts, 503) from permanent errors (validation), sending permanent ones straight to the DLQ. Alert on DLQ growth and provide tooling to replay after fixes.

Key Points

• Backoff + jitter
• Max attempts → DLQ
• Don't block on poison messages
• Alert and replay tooling

Summary

Writing to a database and publishing to a broker as two separate operations can leave them inconsistent if one fails (order saved but event never sent, or event sent but transaction rolled back). Avoid it with the transactional outbox, change data capture, or by making the broker the source of truth (event sourcing).

Longer Answer

The outbox writes the event in the same DB transaction; a relay publishes it afterwards with at-least-once semantics.

Key Points

• Two writes, no shared transaction
• Transactional outbox
• CDC (Debezium)
• At-least-once after