Add Observability to LLM and Agent Applications

## TL;DR

LLM and agent observability starts with one trace per user request. Inside that
trace, record the retrieval steps, model calls, tool calls, retries, fallbacks,
and final response path. Add metrics for latency, errors, token usage, cost, and
tool outcomes. Use structured logs for decisions that need human debugging.

The hard part is not adding more telemetry. The hard part is deciding what to
keep, what to redact, and which signals actually help maintainers debug a
production incident.

## Prerequisites & Requirements

- [ ] {"task": "Request boundaries", "description": "The app can identify one user request, task, job, or conversation turn"}
- [ ] {"task": "Telemetry backend", "description": "Traces, metrics, and logs can be exported to a collector or observability service"}
- [ ] {"task": "Privacy policy", "description": "Prompt, completion, retrieval, and tool data retention is defined before export"}
- [ ] {"task": "Failure fixtures", "description": "Tests or staging traffic cover model errors, tool errors, retries, and timeouts"}

## Core Concepts Explained

### Traces show the agent path

Traces explain what happened during one request. For an LLM app, the trace should
connect the incoming request, retrieval, prompt assembly, model call, tool calls,
retry decisions, and final response.

### Metrics show system health

Metrics answer operational questions across many requests: latency, error rate,
model-call volume, token usage, cost, queue depth, retry count, fallback rate,
and tool success rate.

### Logs explain individual decisions

Logs should be structured, redacted, and tied to trace ids. Use them for events
that need narrative detail: chosen tool, rejected tool output, fallback reason,
validation failure, policy decision, or user-visible error.

### GenAI attributes make traces comparable

OpenTelemetry's GenAI semantic conventions define common attributes for
generative AI systems. Using shared names for provider, model, operation, usage,
and request metadata makes traces easier to query across providers and tools.

By convention, a GenAI span is named `{gen_ai.operation.name} {gen_ai.request.model}`
(for example, `chat claude-3`), so traces stay readable even when several
providers and models run side by side.

### GenAI semantic-convention attributes

These are the OpenTelemetry GenAI span attributes most relevant to LLM and agent
instrumentation. Provider and model attributes are recommended on every model
span; token usage and `error.type` should be set when available.

| Attribute | Type | Example value | What it captures |
| --- | --- | --- | --- |
| `gen_ai.operation.name` | string | `chat`, `embeddings`, `execute_tool`, `invoke_agent` | The kind of GenAI operation the span represents |
| `gen_ai.provider.name` | string | `anthropic`, `openai`, `aws.bedrock`, `gcp.vertex_ai` | The provider or platform serving the request |
| `gen_ai.request.model` | string | `claude-3` | The model requested by the caller |
| `gen_ai.response.model` | string | `claude-3-0613` | The model that actually produced the response |
| `gen_ai.request.temperature` | double | `0.7` | Sampling temperature requested |
| `gen_ai.request.max_tokens` | int | `100` | Maximum tokens requested for the completion |
| `gen_ai.request.top_p` | double | `1.0` | Nucleus-sampling parameter requested |
| `gen_ai.usage.input_tokens` | int | `100` | Tokens consumed by the prompt/input |
| `gen_ai.usage.output_tokens` | int | `180` | Tokens produced in the completion |
| `gen_ai.response.id` | string | `chatcmpl-123` | Provider-assigned response identifier |
| `gen_ai.response.finish_reasons` | string[] | `["stop"]`, `["length"]` | Why generation stopped |
| `error.type` | string | provider error code or exception name | Set when the operation fails |

For aggregate health, the conventions also define client metrics:
`gen_ai.client.operation.duration` (histogram, unit `s`) for operation latency
and `gen_ai.client.token.usage` (histogram, unit `{token}`) for input/output
token counts. Both carry the operation name, provider name, and request model as
attributes, so latency and token cost can be sliced by model the same way traces
are.

### Instrumenting a model span

The example below creates a model-call span and sets GenAI attributes by their
semantic-convention names. Use your provider SDK in place of `call_model`; the
attribute names stay the same across providers, which is the point of the shared
conventions.

```python
from opentelemetry import trace

tracer = trace.get_tracer("llm.app")

def traced_chat(request_model: str, messages: list) -> dict:
    # Span name follows the GenAI convention: "{operation} {model}".
    with tracer.start_as_current_span(f"chat {request_model}") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", "anthropic")
        span.set_attribute("gen_ai.request.model", request_model)
        span.set_attribute("gen_ai.request.max_tokens", 1024)
        span.set_attribute("gen_ai.request.temperature", 0.2)

        try:
            response = call_model(request_model, messages)
        except Exception as exc:
            # Conditionally required when the operation fails.
            span.set_attribute("error.type", type(exc).__name__)
            span.record_exception(exc)
            span.set_status(trace.StatusCode.ERROR)
            raise

        usage = response["usage"]
        span.set_attribute("gen_ai.response.model", response["model"])
        span.set_attribute("gen_ai.response.id", response["id"])
        span.set_attribute("gen_ai.response.finish_reasons", [response["stop_reason"]])
        span.set_attribute("gen_ai.usage.input_tokens", usage["input_tokens"])
        span.set_attribute("gen_ai.usage.output_tokens", usage["output_tokens"])
        return response
```

Note that prompt and completion text are deliberately not placed on the span;
only ids, the model, finish reason, and token counts are recorded. Capture raw
message content only through an approved debug path, per your redaction policy.

## Step-by-Step Implementation Guide

1. **Pick the root span.** Create one root span for each user request,
   conversation turn, background job, or agent task. Put the request id,
   environment, route, tenant or workspace hash, and app version on that span.

2. **Trace the model boundary.** Add child spans around each model call. Capture
   model provider, model name, operation name, prompt version, latency, status,
   retry count, fallback use, and token usage when available.

3. **Trace retrieval and context assembly.** Add spans for vector search,
   database lookups, document fetches, reranking, and prompt assembly. Store
   counts, ids, and scores rather than raw documents unless your privacy policy
   explicitly allows content capture.

4. **Trace tool calls separately.** Each tool call should have its own span with
   tool name, target system, status, latency, retry count, and error class. Link
   the tool span to the model decision that requested it.

5. **Emit operational metrics.** Track request latency, model latency, tool
   latency, error counts, token usage, cost estimates, timeout counts, retry
   counts, fallback counts, and queue depth for async agents.

6. **Use structured logs for decisions.** Log compact events such as
   `tool_selected`, `tool_rejected`, `schema_validation_failed`,
   `retrieval_empty`, `fallback_model_used`, and `human_review_required`. Include
   trace ids so logs and traces can be joined.

7. **Redact before export.** Remove or hash emails, access tokens, file names,
   account ids, raw documents, prompt text, completions, and tool outputs unless
   the team has approved retention for that field.

8. **Sample deliberately.** Keep full traces for errors, timeouts, high-latency
   requests, new releases, and expensive model calls. Sample routine successful
   traffic if volume or privacy risk is high.

9. **Build an incident view.** A useful dashboard answers: which model failed,
   which tool failed, where latency grew, whether retries helped, whether cost
   spiked, and whether a release changed behavior.

## Reusing Existing Agent Telemetry

If your agents run on top of Claude Code, you do not have to instrument the host
from scratch. Claude Code can export OpenTelemetry data directly: set
`CLAUDE_CODE_ENABLE_TELEMETRY=1`, choose exporters with `OTEL_METRICS_EXPORTER`
and `OTEL_LOGS_EXPORTER` (both support `otlp`, `console`, or `none`; metrics also
support `prometheus`), and point `OTEL_EXPORTER_OTLP_ENDPOINT` at your collector.
It emits metrics such as `claude_code.token.usage`, `claude_code.cost.usage`,
`claude_code.session.count`, and `claude_code.tool.execution`, plus
`claude_code.api_error` and `claude_code.api_request` for model-call health. Send
those into the same collector as your own GenAI spans so host usage and your
application traces share one backend.

## Observability Checklist

- [ ] {"task": "Trace coverage", "description": "Requests include model, retrieval, tool, retry, fallback, and response spans"}
- [ ] {"task": "Metrics coverage", "description": "Latency, errors, token usage, cost, retry, and tool outcome metrics exist"}
- [ ] {"task": "Log joins", "description": "Structured logs carry trace ids or request ids"}
- [ ] {"task": "GenAI attributes", "description": "Provider, model, operation, status, and usage fields follow shared semantic names where possible"}
- [ ] {"task": "Redaction boundary", "description": "Sensitive prompt, completion, retrieval, and tool data is removed or explicitly retained"}
- [ ] {"task": "Sampling policy", "description": "Errors and high-risk paths keep enough detail while routine traffic is sampled"}
- [ ] {"task": "Incident dashboard", "description": "Maintainers can diagnose model, retrieval, tool, latency, and cost failures quickly"}

## What to Alert On

Alert on symptoms that a maintainer can act on:

- Model error rate or timeout rate above normal.
- Tool failure rate, validation failures, or repeated retries.
- Retrieval returning empty or low-confidence context for important routes.
- Token usage or cost estimates rising sharply after a release.
- Queue depth, job age, or agent task duration crossing a service target.
- Fallback model usage increasing unexpectedly.

Avoid alerting on every individual model refusal, low-confidence answer, or
sampled trace gap unless it maps to a clear action.

## Troubleshooting

- **Traces are too noisy**: keep the request, model, retrieval, and tool spans,
  then drop internal helper spans that do not explain behavior.
- **Telemetry contains too much user data**: export ids, counts, hashes, scores,
  and prompt versions by default; capture raw content only in approved debug
  paths.
- **Costs are hard to explain**: record model name, token usage, retry count,
  fallback model, and request route on model spans.
- **Tool failures are invisible**: put every external action in its own span and
  log the validation or error class.
- **Sampling hides incidents**: always keep error, timeout, high-cost, and
  high-latency traces, then sample ordinary successful requests.

## Duplicate Check

This guide is vendor-neutral and focuses on the observability architecture for
LLM and agent applications. Existing entries cover specific observability,
evaluation, and tracing tools; this guide is distinct because it explains the
signals, spans, metrics, logs, redaction, and sampling strategy that can be used
with those tools.

## References

- OpenTelemetry traces - https://opentelemetry.io/docs/concepts/signals/traces/
- OpenTelemetry metrics - https://opentelemetry.io/docs/concepts/signals/metrics/
- OpenTelemetry logs - https://opentelemetry.io/docs/concepts/signals/logs/
- OpenTelemetry GenAI semantic conventions - https://opentelemetry.io/docs/specs/semconv/gen-ai/
- OpenTelemetry sampling - https://opentelemetry.io/docs/concepts/sampling/
- OpenTelemetry JavaScript instrumentation - https://opentelemetry.io/docs/languages/js/instrumentation/
- OpenTelemetry Python instrumentation - https://opentelemetry.io/docs/languages/python/instrumentation/
- OpenTelemetry documentation home - https://opentelemetry.io/docs/
- Claude Code monitoring and OpenTelemetry export - https://code.claude.com/docs/en/monitoring-usage