Observability for LLM Features — Integrating LLM APIs

Instrument every call with: model, prompt ID/version, input/output/cache tokens, cost, latency (time to first token and total), stop reason, tool calls, errors and the request ID from the provider. Use OpenTelemetry GenAI semantic conventions (gen_ai.* attributes) so traces work across vendors and tools. Add quality signals: user feedback (thumbs), automated eval sampling of production traffic, and business outcomes. Redact sensitive content in traces, or store content separately with access controls.

What to measure

Signal	Example	Use
Latency	TTFT, total duration	UX SLOs, model/effort choice
Tokens & cost	gen_ai.usage.input_tokens / output_tokens	Budgets, per-tenant attribution
Cache efficiency	cache read vs write tokens	Prompt caching health
Errors	429, overload, refusals, validation failures	Reliability, alerts
Tool usage	Calls per request, failures	Agent behavior, abuse detection
Quality	Feedback rate, sampled eval score	Regression detection

Python (OpenTelemetry)

with tracer.start_as_current_span("chat claude-opus-5-5") as span:
    span.set_attribute("gen_ai.operation.name", "chat")
    span.set_attribute("gen_ai.request.model", MODEL)
    span.set_attribute("app.prompt_id", PROMPT_ID)
    response = client.messages.create(model=MODEL, max_tokens=4096, messages=messages)
    span.set_attribute("gen_ai.response.model", response.model)
    span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)
    span.set_attribute("gen_ai.response.finish_reasons", [response.stop_reason])
    span.set_attribute("app.provider_request_id", response._request_id)

Common pitfalls

Logging full prompts with secrets/PII to a shared log system.
No per-feature cost visibility.
Only measuring latency, never quality.

Key points

Trace every call with gen_ai attributes
Cost per feature/tenant
Production quality sampling

Further reading: OpenTelemetry GenAI semantic conventions

Practice questions

What is time to first token (TTFT) and why track it?

Answer

The delay until the first streamed token arrives; it drives perceived responsiveness in chat UIs more than total duration.

Why log the provider's request ID?

Answer

It lets the provider's support trace a specific failing request and correlates your logs with theirs.