Distributed Tracing with OpenTelemetry

A trace is a tree of spans (name, timing, attributes, status) linked by a trace ID propagated across service calls via the W3C traceparent header. Use OpenTelemetry auto-instrumentation for HTTP servers/clients, databases and messaging, add manual spans around important business steps, and export via OTLP to a collector. Use head sampling for cost control and tail sampling in the collector to keep all errors and slow traces.

# OpenTelemetry Collector: keep errors and slow traces, sample the rest
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Common pitfalls

• Broken context propagation through queues or thread pools.
• Span names with IDs (high cardinality).
• 100% sampling at high volume.

Key points

• Traces = trees of spans with propagated context
• Auto-instrument, then add business spans
• Tail sampling keeps interesting traces

Further reading: OpenTelemetry: Traces