Sign in Get started

Free course · 20 lessons · Intermediate

Queues & Background Processing in Depth

Messaging models, message design, ordering, acknowledgements, broker choice, job frameworks by stack, retries and DLQs, poison messages, priorities and scheduling, scaling consumers, durable workflows and operations.

Start the course

1. Messaging Foundations

Queue semantics, message design, ordering, acknowledgements and broker choice.

  1. Queues, Pub/Sub and Streams — Three messaging models and what each is for.
  2. Designing Messages — Envelopes, IDs, schemas, size limits and CloudEvents.
  3. Ordering Guarantees — What ordering brokers give you and how to design around it.
  4. Acknowledgements, Visibility Timeouts and Prefetch — How brokers track in-flight messages.
  5. Choosing a Message Broker — Kafka, RabbitMQ, SQS/SNS, Google Pub/Sub, Redis Streams and NATS.

2. Background Job Processing

Frameworks, retries, dead letters, poison messages, priorities and schedules.

  1. Background Job Frameworks by Stack — Celery, BullMQ, Sidekiq, Laravel Queues, Spring/JobRunr and friends.
  2. Retries and Dead-Letter Queues — Retry transient failures, park permanent ones.
  3. Poison Messages and Head-of-Line Blocking — Stop one bad message from blocking a partition or queue.
  4. Priorities, Delays and Deduplication — Run urgent work first, defer work and avoid duplicate jobs.
  5. Scheduled Work and Distributed Cron — Run periodic jobs once, reliably, across many instances.

3. Scaling Consumers

Concurrency, lag, autoscaling and partitioning.

  1. Consumer Concurrency and Throughput — Workers, threads, prefetch and partition-bound parallelism.
  2. Lag, Queue Age and Backpressure — Measure how far behind consumers are and react.
  3. Autoscaling Workers on Queue Metrics — Scale consumers with KEDA, cloud autoscalers and serverless triggers.
  4. Partitioning and Hot Keys — Choose partition keys and handle skew.

4. Workflows and Operations

Durable workflows, managed orchestration, sagas with queues and operating messaging.

  1. Durable Workflows with Temporal — Long-running, reliable business processes as code.
  2. Managed Orchestration: Step Functions and Friends — Serverless state machines for cloud-native workflows.
  3. Sagas with Messaging — Coordinate multi-service transactions using events or commands.
  4. Operating Messaging Systems — Monitoring, replay, schema governance and runbooks.

5. Capstone

Design a job platform and fix a backlog incident.

  1. Capstone: Design a Notification Job Pipeline — Build reliable, prioritized background processing for millions of notifications.
  2. Capstone: Debug a Kafka Lag Incident — Find out why consumers fell hours behind and fix it.