Queues & Background Processing in Depth · Capstone
Capstone: Design a Notification Job Pipeline
Build reliable, prioritized background processing for millions of notifications.
Scenario
A SaaS sends transactional emails (password resets, receipts), marketing campaigns (5 million recipients), push notifications and nightly digests. Today a marketing blast delays password resets by an hour, failed sends vanish, and workers crash on malformed templates.
Requirements
- Password resets delivered within 1 minute
- No lost messages; failures visible and replayable
- Campaigns don't affect transactional traffic
- Provider rate limits respected
Constraints
- Email provider: 1,000 emails/s; push provider: 5,000/s
- Team of 4, prefers managed services
Deliverables
- Design queues, priorities and workers.
- Handle failures and poison messages.
- Define SLO monitoring.
Draft each deliverable below before revealing the model answer and rubric.
Further reading: Amazon SQS Developer Guide
Deliverables
Design queues, priorities and workers.
Model answer
Separate queues: email.transactional, email.bulk, push, digest (SQS or RabbitMQ). Dedicated worker pools autoscaled on oldest-message age (KEDA), transactional pool always warm. Producer writes intent via outbox; campaign fan-out job enqueues batches into bulk queue at a controlled rate. Workers use a shared token bucket (Redis) to respect provider limits with transactional getting reserved capacity (e.g., 200/s).
Rubric
Strong answers isolate transactional traffic physically (queues + workers + reserved rate), not just with priority flags.
Handle failures and poison messages.
Model answer
Classify errors: provider 429/5xx → retry with exponential backoff + jitter (max 8 attempts); invalid template/address → DLQ immediately with reason. DLQ per queue with alerts and a redrive tool; idempotent sends keyed by notification ID; template validation at publish time to stop poison messages early.
Rubric
Look for transient vs permanent classification, DLQ ownership and producer-side validation.
Define SLO monitoring.
Model answer
Alert on oldest-message age: transactional > 60 s (page), bulk > 2 h (ticket); DLQ depth > 0 on transactional (page); provider error rate; dashboards per queue for ingress/egress, retries, and rate-limit waits.
Rubric
Age-based alerts tied to business SLOs are key.
Open in the app to mark lessons complete, take notes and track your progress.