Queues & Background Processing in Depth · Messaging Foundations
Designing Messages
Envelopes, IDs, schemas, size limits and CloudEvents.
Good messages are self-describing and evolvable: a unique message/event ID (for dedupe), type and schema version, timestamp, source, correlation/causation IDs and trace context, plus a payload with the data consumers need. Keep messages small (brokers have limits: SQS 256 KB, Kafka default ~1 MB) — store large blobs in object storage and send a reference (claim check pattern). CloudEvents standardizes the envelope across brokers.
Envelope fields
| Field | Why |
|---|---|
| id | Deduplication, tracing a single event |
| type + version | Routing and schema evolution |
| time / source | Debugging, ordering hints |
| correlation / causation id | Following workflows across services |
| traceparent | Distributed tracing across async hops |
| Reference to large payload | Claim check for blobs > broker limit |
{
"specversion": "1.0",
"id": "evt_01J9ZQ8M4K2R",
"type": "com.acme.orders.order.paid.v1",
"source": "/services/orders",
"subject": "ord_8812",
"time": "2026-10-11T09:15:02Z",
"datacontenttype": "application/json",
"traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
"correlationid": "req_7f3a9c",
"data": {"order_id": "ord_8812", "total": {"amount": 2400, "currency": "EUR"}, "invoice_pdf": "s3://invoices/ord_8812.pdf"}
}
Common pitfalls
- No message ID (consumers can't dedupe).
- Huge payloads hitting broker limits.
- Breaking schema changes without versioning.
Key points
- IDs, types, versions, trace context
- Small messages; claim check for blobs
- CloudEvents for a common envelope
Further reading: CloudEvents
Practice questions
What is the claim check pattern?
Answer
Store a large payload in external storage (e.g., S3) and send only a reference in the message; consumers fetch the payload when needed.
Why include correlation IDs in messages?
Answer
They link messages belonging to the same business transaction or request across services, making debugging and tracing workflows possible.
Open in the app to mark lessons complete, take notes and track your progress.