LLM Observability and Tracing
LLM tracing and AI observability in production: span models, cost tracking, PII redaction, quality overlays, and prompt monitoring alerts for production drift.
- Read time
- 16 min
- Level
- Intermediate
- Updated
- 2026-08-25
- Components
- Trace model · Span collector · PII redactor
LLM observability is the production tracing system that records each request as a tree of spans — model calls, retrieval, tools, guardrails — with tokens, cost, latency, redacted prompts, and a quality overlay so prompt monitoring can alert on drift rather than on CPU alone.
Key takeaways
- 01
The trace is the unit of work: one user request, nested spans for retrieval, tools, model, and filters, with a stable trace ID.
- 02
Log tokens, cost, model version, prompt version, and tool names on every span or you cannot explain a bill or a regression.
- 03
PII redaction happens before the observability store; raw prompts in a vendor SaaS without a contract is a data incident.
- 04
A quality overlay (sampled judges, groundedness, schema fail rate) is what makes traces useful for AI, not just for latency.
- 05
Alert on trend breaks — cost per request, empty-retrieval rate, judge scores — not on single ugly completions.
LLM tracing: the span model
LLM tracing treats one user-visible request as a trace: a tree of spans for routing, retrieval, each model call, each tool, guardrails, and the final render. An AI observability platform is that model plus storage, search, and alerts — not a dashboard of token counts with no parent-child structure. Prompt monitoring means you can open a bad answer and see which prompt version, which chunks, and which tool args produced it.
Propagate trace IDs into every downstream call, including provider APIs and your own gateways. If a retry or a fallback model fires, it is a child span of the same trace, not a new request. Without that, incident review is a pile of unrelated log lines.
- Trace: one user request, stable ID, user and session refs
- Spans: retrieve, generate, tool, guardrail, judge
- Attributes: model, prompt hash, tokens, cost, latency
- Links: parent span, retry of, fallback from
What to collect — and what to redact
Collect enough to replay a failure: prompt template ID and hash (not always the fully expanded prompt), retrieved document IDs, tool names and redacted arguments, model name and pinned version, token in/out, cache hits, finish reason, and guardrail verdicts. Cost is a first-class attribute; retries and judges are usually the silent multipliers.
PII redaction is part of the collector, not a later cleanup job. Strip secrets, account numbers, and named entities according to policy before spans leave the VPC. Store raw payloads only in a tightly controlled store if legal requires it; default to hashes, IDs, and redacted previews. Observability that cannot enter the client's cloud is the wrong shape for this work.
Quality overlay on top of ops metrics
Latency and error rate tell you the system is up. They do not tell you it is answering. Overlay sampled quality: schema validity, citation presence, groundedness judges, task success for agents. Attach those scores to the same traces so a dip in faithfulness can be sliced by prompt version, retriever, and model.
Sample deliberately. High-cost or high-risk traces (writes, complaints, empty retrieval) should be sampled at a higher rate than routine FAQ hits. Async judges must not sit on the user path; they write back to the trace after the response has shipped.
Prompt monitoring and drift alerts
Alert on distribution shifts, not on one rude completion. Useful pages: empty-retrieval rate, tool-error rate, cost per successful task, p95 latency, judge-score moving average, schema-fail rate, guardrail block rate. A provider model update that arrives unpinned often shows up as a quality sag with unchanged traffic.
Wire alerts to owners who can roll back a prompt or pin a model, not to a generic inbox. Include the trace-search link for the window. If nobody can act in ten minutes, it is a report, not an alert.
Failure modes and when not to build a platform
Common failure: logging full prompts with PII into a third-party store. Second: traces without prompt or model versions, so you cannot correlate a regression. Third: 100% LLM-as-judge on live traffic, which doubles cost and still needs calibration. Fourth: dashboards nobody opens because they show tokens, not tasks.
Do not stand up an "AI observability platform" before you have a trace ID and five attributes on the critical path. A structured log plus a golden-set CI gate beats a vacant vendor UI. Also skip shipping raw chain-of-thought to logs if policy forbids storing reasoning traces.
- PII and secrets in prompt logs
- Missing prompt/model version on spans
- Judges on the user-critical path
- Metrics that cannot slice a regression
Operating traces: retention, access, eval loop
Retention should match incident and eval needs: hot search for days, sampled archives for months. Access is role-based; customer-support traces may be more sensitive than public-web traces. Feed low-scoring traces into a triage queue and then into the golden set. That loop — production miss → labeled case → CI gate — is the reason to collect traces, not a heat map of tokens.