Structured Logging That Scales — Observability and Operations

Emit logs as structured JSON with consistent field names (service, env, level, message, trace_id, user_id, duration_ms), log events rather than prose, and use levels deliberately. Redact secrets and personal data at the source. Control cost with sampling of high-volume debug/info logs, retention tiers, and dropping noise in the pipeline (OpenTelemetry Collector, Vector, Fluent Bit).

{"ts":"2026-10-10T09:12:44.123Z","level":"warn","service":"checkout","env":"prod",
 "message":"payment retry","trace_id":"4bf92f3577b34da6","order_id":"ord_8812",
 "attempt":2,"provider":"acme-pay","error":"timeout","duration_ms":3004}

Common pitfalls

Logging full request bodies, tokens or card data.
Inconsistent field names across services (userId vs user_id).
ERROR level for expected business outcomes, causing alert noise.

Key points

JSON with a shared schema
Redact at the source
Sample and tier for cost

Further reading: OpenTelemetry semantic conventions

Practice questions

Why adopt a shared log schema across services?

Answer

Consistent field names let you query, join and build dashboards across services; otherwise every investigation needs per-service query syntax.

How do you keep logging costs under control?

Answer

Drop or sample noisy low-value logs, keep errors and audit logs, index only useful fields, and move older logs to cheaper storage tiers.