Telemetry Pipelines and Observability Cost

At scale, send telemetry through a pipeline — agents/sidecars → OpenTelemetry Collector (or Vector) → backends — to batch, enrich (Kubernetes metadata), redact, sample and route data (e.g., all errors to the expensive tool, the rest to cheap storage). Observability costs grow with log volume, metric cardinality and trace volume; govern them with budgets per team, retention tiers and regular cleanup.

# otel-collector.yaml (excerpt)
receivers:
  otlp: { protocols: { grpc: {}, http: {} } }
processors:
  batch: {}
  memory_limiter: { check_interval: 1s, limit_percentage: 80 }
  attributes/redact:
    actions:
      - { key: http.request.header.authorization, action: delete }
      - { key: user.email, action: hash }
exporters:
  otlphttp/vendor: { endpoint: https://otlp.vendor.example }
service:
  pipelines:
    traces: { receivers: [otlp], processors: [memory_limiter, attributes/redact, batch], exporters: [otlphttp/vendor] }

Common pitfalls

• Every service exporting directly to vendors (no central control).
• No memory limits in collectors (OOM under bursts).
• Paying to index data nobody queries.

Key points

• Collector pipelines: batch, enrich, redact, sample, route
• Govern cardinality and volume
• Retention tiers

Further reading: OpenTelemetry Collector