Observing Distributed Systems — Distributed Systems for Backend Engineers

Debugging distributed systems requires following a request across services and async hops: distributed tracing (OpenTelemetry, W3C traceparent propagated through HTTP and message headers), correlation IDs in logs, span links for batch/async work, and service maps built from traces. Measure end-to-end latency of async workflows (event created → processed) and per-hop queue time.

Context propagation

Hop type	Propagation	Span relationship
HTTP / gRPC	traceparent header	Child span
Message queue	traceparent in message headers/attributes	Child or link
Batch processing many messages	Each message's context	Span links to all producers
Scheduled job	New trace	Link to triggering event if any

Python (OpenTelemetry + Kafka headers)

from opentelemetry import propagate, trace
tracer = trace.get_tracer("orders")

def publish(producer, topic, event):
    headers = {}
    propagate.inject(headers)                                     # adds traceparent
    producer.send(topic, value=event, headers=[(k, v.encode()) for k, v in headers.items()])

def consume(record):
    ctx = propagate.extract({k: v.decode() for k, v in record.headers})
    with tracer.start_as_current_span("process order.paid", context=ctx):
        handle(record.value)

Common pitfalls

Traces that stop at the first queue.
Logs without trace IDs.
Measuring only synchronous latency for async workflows.

Key points

Propagate context through queues
Span links for batches
Measure end-to-end async latency

Further reading: OpenTelemetry: Messaging semantic conventions

Practice questions

How do you continue a trace across a Kafka message?

Answer

Inject the trace context (traceparent) into message headers when producing and extract it when consuming to start the consumer span with that parent.

When are span links better than parent-child spans?

Answer

When one operation processes many messages from different traces (batching), linking to each producer span instead of picking one parent.