Logging & Observability Interview Questions

Summary

Logs record discrete events with context, metrics are numeric time series aggregated over time, and traces follow a single request across services. Together they let you detect, diagnose and explain problems in production.

Longer Answer

Metrics (request rate, error rate, latency percentiles, CPU) are cheap to store and great for dashboards and alerts. They tell you that something is wrong. Traces show the path and timing of one request through services, databases and queues, so they tell you where. Logs carry rich detail about individual events (errors, decisions, inputs), so they tell you why.

Modern systems correlate the three with shared identifiers (trace ID in log lines, exemplars linking metrics to traces). Some teams add profiles (continuous profiling) and events as further signals. Observability is about being able to ask new questions of your system without shipping new code.

Key Points

• Metrics: detect (cheap, aggregated)
• Traces: locate (per-request path and timing)
• Logs: explain (detailed events)
• Correlate with trace IDs
• Profiles are an emerging fourth signal

Comparison / Tradeoffs

Summary

Structured logging emits each log event as machine-readable key-value data (usually JSON) instead of free-form text, so logs can be filtered, aggregated and queried reliably.

Longer Answer

A plain line like User 42 failed login from 1.2.3.4 must be parsed with fragile regexes. A structured event {"event":"login_failed","user_id":42,"ip":"1.2.3.4","level":"warn"} can be queried directly (event=login_failed AND ip=...), aggregated into counts, and enriched automatically with request IDs, service name, version and environment.

Most languages have structured loggers: Pino (Node), structlog/python-json-logger (Python), Monolog with JSON formatter (PHP), Logback with JSON encoders (Java). Keep a stable schema (consistent field names) across services.

Code Example

{"ts":"2026-10-10T09:15:02Z","level":"warn","service":"checkout","version":"1.8.2",
 "event":"payment_declined","order_id":"ord_123","user_id":42,"reason":"insufficient_funds",
 "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","duration_ms":182}

Key Points

• JSON key-value events
• Queryable and aggregatable
• Consistent field names across services
• Automatic context enrichment (request ID, service, version)

Summary

A correlation ID is a unique identifier assigned to a request at the edge and attached to every log line, outbound call and message it causes, so you can reconstruct everything that happened for one request across services.

Longer Answer

Generate it at the first entry point (or accept an incoming X-Request-ID / W3C traceparent from a trusted proxy), store it in request-scoped context (AsyncLocalStorage in Node, contextvars in Python, MDC in Java, a request attribute in PHP), add it to every log line automatically, forward it in headers to downstream HTTP calls, and include it in queue message metadata. Return it in the response so users and support can quote it. With OpenTelemetry, the trace ID serves this purpose.

Code Example

# Python (FastAPI middleware + contextvars)
import uuid, contextvars, logging

request_id = contextvars.ContextVar("request_id", default="-")

class RequestIdFilter(logging.Filter):
    def filter(self, record):
        record.request_id = request_id.get()
        return True

@app.middleware("http")
async def assign_request_id(request, call_next):
    rid = request.headers.get("x-request-id") or str(uuid.uuid4())
    token = request_id.set(rid)
    try:
        response = await call_next(request)
    finally:
        request_id.reset(token)
    response.headers["x-request-id"] = rid
    return response

Key Points

• Assigned at the edge
• Stored in request-scoped context
• Auto-added to logs
• Forwarded in headers and message metadata
• Returned to clients for support

Summary

Distributed tracing records a request as a tree of timed spans across services; OpenTelemetry (OTel) is the vendor-neutral standard (APIs, SDKs, the OTLP protocol and the Collector) for producing and exporting traces, metrics and logs.

Longer Answer

Each span has a name, start/end time, attributes and a parent; context (trace ID, span ID) propagates between services via the W3C traceparent header. Auto-instrumentation libraries create spans for HTTP servers/clients, databases and queues without code changes, and you add custom spans for important business steps. The OTel Collector receives, processes (sampling, redaction) and exports data to backends like Jaeger, Tempo, Honeycomb or Datadog, so you can switch vendors without re-instrumenting.

Key Points

• Spans form a per-request tree
• W3C traceparent propagation
• Auto-instrumentation + custom spans
• Collector decouples apps from vendors
• Sampling controls cost

Summary

They're checklists for what to measure: RED (Rate, Errors, Duration) for request-driven services; USE (Utilization, Saturation, Errors) for resources like CPU, disks and pools; and Google's Four Golden Signals (latency, traffic, errors, saturation) for user-facing systems.

Longer Answer

RED gives a consistent dashboard per service and endpoint. USE helps find resource bottlenecks (a DB connection pool at 100% utilization with a growing wait queue is saturated). Golden Signals combine both views. Measure latency as percentiles (p50, p95, p99), not averages, and separate successful from failed request latency.

Key Points

• RED for services
• USE for resources
• Golden signals for user-facing systems
• Percentiles, not averages

Summary

An SLI is a measured indicator of service quality (e.g., % of requests under 300 ms), an SLO is the target for it (99.9% over 30 days), and the error budget is the allowed failure (0.1%) that teams can 'spend' on releases and experiments.

Longer Answer

SLOs align engineering with user experience. When the error budget is being burned too fast, teams slow down releases and prioritize reliability. Alert on burn rate (how fast the budget is being consumed, using multi-window rules) rather than on every spike, which reduces noisy pages. An SLA is a contractual promise to customers, usually looser than the internal SLO.

Key Points

• SLI = measurement
• SLO = target
• Error budget = allowed unreliability
• Burn-rate alerts reduce noise
• SLA = external contract

Summary

Never log secrets (passwords, tokens, API keys, session cookies), full payment card data, or unnecessary personal data. Prevent it with redaction in the logging library, allow-listed fields, code review and automated scanning of logs.

Longer Answer

Leaked credentials in logs are a common breach path because logs are widely accessible and long-lived. Configure redaction paths (e.g., Pino redact, structlog processors, Monolog processors), avoid dumping whole request bodies or headers, and mask identifiers where possible. Comply with GDPR/PCI retention rules and restrict log access.

Key Points

• No secrets, tokens, card data
• Minimize PII
• Library-level redaction
• Restricted access and retention policies

Summary

Confirm the scope on dashboards (which endpoints, regions, customers), correlate with recent changes (deploys, flags, traffic), use traces to find the slow span, check dependency and resource saturation, then mitigate first (rollback, scale, disable feature) and find root cause after.

Longer Answer

A practical sequence: (1) check SLO/RED dashboards and when it started; (2) look for deploy or config markers; (3) open slow traces (exemplars from the latency histogram) to see whether time is spent in the DB, an external API, lock contention or the app itself; (4) check saturation: DB connections, thread pools, CPU throttling, GC pauses, queue backlog; (5) mitigate with the fastest safe action; (6) write a blameless postmortem with follow-ups.

Key Points

• Scope and timing first
• Correlate with changes
• Traces to find the slow span
• Check saturation of pools/CPU/GC
• Mitigate before root-causing