How to Set Up LLM Observability
Set up LLM observability: trace LLM calls, monitor prompts in production, token and cost metrics, quality scores, drift alerts, and PII redaction in the logs.
To set up LLM observability, emit a trace for every turn (retrieval, model, tools), attach token and cost metrics, overlay sampled quality scores, alert on spend and drift, and redact PII before you store or export prompts.
What you’ll build
- End-to-end traces: user turn → retrieval → model → tools → verifier, with token counts
- Dashboards for cost, latency, cache hit rate, and quality on the same request id
- Alerts on spend spikes, groundedness drops, and schema-failure retries
- PII redaction before logs leave the app, including screenshots of prompts in the vendor UI
- A weekly quality sample that is itself traced, so judges do not become a silent second bill
Before you start
- 01A unique request id on every user turn, propagated into model, tool, and retrieval spans
- 02A place to store traces that is not the model provider's training corpus
- 03A redaction policy for prompts, tool args, and retrieved chunks before they hit disk
- 04A small golden set so 'quality' is a number, not a vibe in Slack
- 05On-call ownership for burn-rate and quality alerts, not only HTTP 500s
Key takeaways
- 01
LLM observability is traces plus cost plus quality, not a prompt playground.
- 02
Propagate one request id through retrieval, the model, tools, and the verifier or you cannot debug a bad answer.
- 03
Redact PII at the app boundary; 'the vendor is SOC2' is not a redaction policy.
- 04
Alert on cost per successful task and quality floors, not on raw token counters.
- 05
Tag eval and red-team traffic so it does not page humans or pollute user dashboards.
Trace LLM calls as a tree, not a log line
A chat completion is not the unit of work. The unit is the user turn: retrieval, one or more model calls, tool rounds, cache lookups, and maybe a judge. LLM observability starts when those are child spans under one parent with a request id the support agent can paste. If you only log 'POST /v1/chat/completions 200,' you cannot explain an answer.
Name spans after the product, not the vendor: retrieve.hybrid, model.route.small, tool.refund.dry_run, verify.faithfulness. Attach the pinned model revision, tool schema hash, prompt template version, and cache hit boolean. When quality drops next Tuesday, those attributes tell you what changed without a git bisect of YAML.
Token, cost, and latency on the same id
Copy token counts off the provider response, including cached and reasoning tokens if they exist. Multiply by a price table you own and version, because list prices change. Split cost by surface and by purpose (user, eval, red-team). Cost per successful task is the number finance wants; tokens per request is a debugging clue.
Latency belongs on the same trace: time to first token, time to tool, time to final. A cheap model that times out and retries twice is not cheap. Cache hit rate sits next to these. If you cannot break out prompt-cache versus semantic-cache hits, you will tune the wrong cache.
- Per span: tokens in/out/cached, model id, dollars, cache hit
- Per turn: cost, TTFT, tool rounds, schema retries
- Per surface: cost per successful task vs the quality floor
- Per purpose: user vs eval vs red-team, so invoices match dashboards
Quality scores, drift alerts, monitor prompts in production
To monitor prompts in production you need two views: the redacted prompt text on the trace (for debugging a single miss) and aggregate quality over time (for drift). Sample. One hundred percent LLM-as-judge is a second product on the bill and a privacy surface. Start at 5–10% plus every thumbs-down.
Drift is a quality or mix shift, not a 500. Alert when groundedness, schema-ok, or task pass rate drops below the floor for a route, or when the mix of 'hard' routed turns spikes and the small model is failing. Slow drifts (refusal rate, cache freshness) go to a weekly review. Fast drifts (cost ×3, retry storm) page.
PII redaction in logs
Treat prompts, retrieved chunks, tool arguments, and completions as regulated data. Redact before the trace backend, before the vendor playground export, and before a screenshot lands in Slack. Store a reversible copy only in a locked store if legal requires it, with a separate ACL from the observability UI.
Redaction misses happen. Ban PAN-like patterns, well-known secret prefixes, and your employee-id regex. Do not log full retrieved HR docs because 'the chunk might help debug.' Hash the chunk id; the index still has the text for people who are allowed to see it. If a vendor's tracing product trains on customer prompts, it is not your tracing product.
On-call, CI, and what not to build
The runbook should start at the request id: open the trace, see which span failed, see whether the route, the retriever, or the tool is new. CI should replay the golden set with tracing on, so a prompt change that doubles cost fails the build. Tag synthetic traffic. A red-team suite that looks like 10k angry users is how you burn the night team.
You do not need a new platform if you already have OpenTelemetry and a metrics store. You need span conventions, a price table, a redactor, and a quality overlay. A dedicated LLM UI is optional. It is not a substitute for those four.
Implementation partner, not another SaaS bill
ReinforcedX sets up LLM observability in your cloud: trace conventions, redaction, cost overlay, quality sample, and runbooks. We are not an observability vendor and we do not mark up tokens. You own the traces, evals, and dashboards. Model-agnostic — the same span model works in front of Anthropic, OpenAI, Google, or vLLM.
Four weeks: conventions and redaction, traces on one surface, quality overlay and alerts, then handover. Thirty days on-call is for the first false page. If you already have OTel, week 2 is short. If prompts currently live in an unredacted Slack channel, week 1 is that.
Step-by-step build
- 1
Pick a trace model and a request id
Use OpenTelemetry-style spans: one parent per user turn, children for retrieval, each model call, each tool, and the verifier; put the same id on application logs.
- 2
Record tokens, cache hits, and dollars
On every model span log input, output, cached, and reasoning tokens, the pinned model id, and a cost estimate from your price table — not the provider's marketing page.
- 3
Redact before persist
Run detectors for PAN, secrets, and names you care about on prompts, completions, tool args, and retrieved text; store hashes of raw prompts if you must join later.
- 4
Overlay quality on a sample
Score a sampled slice with deterministic checks plus a judge; write groundedness, schema-ok, and task-pass back onto the parent span.
- 5
Alert on drift and burn
Page on quality floor breaches, retry storms, and cost-per-task spikes; ticket on slower drift (cache hit rate, refusal rate) with a weekly owner.
- 6
Close the loop in CI and on-call
Fail deploys when the golden set drops; give on-call a runbook that starts at the trace, not at the prompt file; review a weekly sample of thumbs-down.
Common pitfalls
The mistakes that show up in real deployments — each one costs a week if you learn it the hard way.
Logging prompts to a third-party SaaS without redaction
The most common LLM observability incident is a full customer prompt in someone else's cloud. Redact first, then export. If the vendor trains on logs, do not send them.
Metrics without traces
Average tokens/request will not tell you which tool loop spent $40. You need a trace per turn with child spans, or you will tune the wrong surface.
Quality as a vibe channel
A Slack screenshot is not a score. Overlay sampled faithfulness, task pass rate, and thumbs-down on the same timeline as cost, or a cheap route will ship unnoticed.
Alerting on every token
Noisy alerts get muted. Alert on cost per successful task, p95 latency, schema-retry storms, and a quality floor — not on each completion.
Forgetting eval traffic is production traffic
Nightly judges and red-team jobs show up on the invoice and can look like a user spike. Tag those traces with a purpose field or you will page the on-call at 02:00.