LLM Observability & Monitoring

In the previous lesson, a large language model (LLM) answer path called tracking-answerer-v1 refused to promise a delivery date that wasn't present in a FastShip record. It also emitted a trace: evidence version, route, and first failed stage. One safe trace proves one request behaved correctly. It doesn't tell you whether the service is still safe across live traffic.

Suppose a prompt rollback silently bypasses that gate for two customers. The endpoint still returns 200. Answers remain fluent. Latency stays fast. Only the trace events reveal that unsupported ETAs escaped into served messages.

Monitoring reduces many trace events into rates, percentiles, and alerts. Observability lets you start from an alert and reconstruct why a particular request failed. An LLM application needs both: aggregates to detect a change and request evidence to diagnose it.

Start with the event your gate already produced

Avoid beginning with a dashboard wish list. Begin with the decision boundary from the serving path:

This chapter uses synthetic events from the same delivery-update path as the prior lab. The last two events model a regression; they aren't claims about real production traffic.

The first cell creates six requests. Four are safe outcomes from a strict gate: a supported answer or a blocked factual claim. Two come from a regressed template that served an invented ETA.

Notice the distinction: an abstention isn't automatically a failure. When evidence doesn't establish an ETA, abstention is the safe product behavior. The event you need to page on is an unsupported factual claim that was actually served.

Turn request evidence into quality metrics

An aggregate must preserve the invariant it claims to monitor. A generic "answer quality score" could hide two unsafe delivery promises inside many pleasant responses. For this answer path, start with metrics directly derived from claim-gate outcomes:

The next cell aggregates the synthetic production window.

The count does more than say "quality is down." Both escaped claims first failed at serving_gate, while the earlier strict template safely abstained on the same invented-ETA case. That evidence points at a gate or template rollback before anyone tunes retrieval.

Measure responsiveness without hiding correctness

Quality is the first invariant here, but customers also notice sluggish answers. For a streamed response:

• Time to first token (TTFT) is the delay before the customer sees output.
• Post-first-token throughput measures how quickly the remaining output arrives.
• End-to-end latency includes the entire response.

Be precise with token arithmetic. If TTFT marks arrival of the first token, only output_tokens - 1 tokens arrive in the time after TTFT. Counting all output tokens in that interval slightly overstates throughput.

This window is fast and unsafe. A latency-only dashboard would celebrate the exact release that should be rolled back.

Don't hide malformed timing events behind a tiny fallback denominator. If a multi-token response reports total_ms <= ttft_ms, fix the instrumentation before trusting its throughput.

OpenTelemetry (OTel) publishes generative-AI semantic conventions for spans and metrics. Its current metrics distinguish client first-chunk latency (gen_ai.client.operation.time_to_first_chunk) from model-server first-token latency (gen_ai.server.time_to_first_token). This lab's ttft_ms is an application-owned customer-visible field, so document exactly where your timer starts and stops. The convention is marked Development; pin the convention emitted by your instrumentation instead of treating its field set as permanent.^[1][2]

Keep standard telemetry and product decisions together

OTel's generative-AI span attributes describe model operations and usage. The current inference-span convention requires both gen_ai.operation.name and gen_ai.provider.name; gen_ai.request.model is conditionally required when available. The synthetic service below uses a custom provider name because no real provider is involved. Use the convention's well-known provider value when one applies. Your application still needs custom attributes for its own invariant: which evidence version supported the answer, whether the answer route blocked a claim, and where the first failure occurred.^[3]

Prompt, output-message, and system-instruction attributes are opt-in in the OTel convention because they may contain sensitive content. Store stable identifiers and safe outcome attributes by default; make raw text capture a deliberate, governed exception.^[3]

gen_ai.* attributes let tracing backends recognize the model call. shopflow.* attributes explain the business decision. Neither requires putting a customer's full message into a durable trace.

Attribute cost with a versioned rate card

Cost is part of monitoring, but embedding current provider price claims in application logic or a tutorial makes both go stale quickly. Production systems should store usage counts on the trace and evaluate them with a versioned rate card. The rates below are synthetic fixture values, chosen only to exercise the calculation.

If a provider later changes cached-token pricing or adds a service-tier surcharge, old traces remain interpretable: usage was recorded once, while the rate-card version states how the estimate was computed.

Store durable debugging evidence without customer text

A trace may contain an order identifier, email address, or full customer request. That text can help an incident investigation, but it shouldn't be your default long-lived record.

For this workflow, durable logs can retain:

The record below keeps a redacted preview for an incident exemplar while leaving raw payload storage unset.

The regex is intentionally narrow: it makes the fixture readable, but it isn't a production scrubber. In production, minimize collected attributes, review what instrumentation libraries emit, and apply centrally managed Collector processors or an equivalent scrub pipeline before durable storage.^[4]

Alert on decisions engineers can act on

Not every metric deserves a page. A long answer or higher cost might warrant investigation. An unsupported delivery promise that escaped a hard gate violates the product invariant and should page immediately.

Thresholds must come from a service contract, not a generic article. This lab's synthetic policy says:

• Any unsafe served delivery claim is a page.
• TTFT above 500 ms is a warning for this interactive answer path.
• Estimated cost above the fixture budget is a warning, using the rate card named in the event window.

The page is actionable because it says which invariant broke. It doesn't page merely because a judge score drifted or because latency was a little noisy.

Six synthetic requests are enough to demonstrate alert logic, not to set a production latency or cost budget. In a real launch, define those warning thresholds from representative traffic and revisit them as workload shape changes. The zero-tolerance safety invariant is different: a known unsupported delivery promise must not be served.

Attach an exemplar and a first investigation step

An on-call engineer shouldn't begin by searching arbitrary logs. For an invariant page, attach one violating trace with its prompt template and source version, then state the immediate check.

The request still needs deeper review, but investigation is now narrow: compare the template or gate configuration that served the claim with the strict version that abstained on the same failure case.

Extend the contract when answers call tools

The central example is a factual answer gate. If the product later allows an agent to create return labels or message a carrier, the same discipline applies to tool actions. Log state transitions you control: tool name, redacted parameter hash, result status, progress marker, retry count, and stop reason. Don't depend on hidden reasoning text.

Preserve the evidence for the next fix

An alert tells you something is wrong now. The next engineering question is whether a proposed fix is better. That requires an experiment record linking the failing window, proposed template, policy, and rerun results.

The monitoring system doesn't approve the proposed fix. It creates a precise starting point for the next controlled run: what broke, which candidate claims to fix it, and which metric must pass.

Mastery check

Key concepts

• Monitoring aggregates and request-level observability
• Safety metrics derived from claim-gate decisions
• First-failure-stage attribution
• TTFT and correctly computed post-first-token throughput
• Standard model-call attributes plus product-specific decision attributes
• Versioned cost estimates
• Durable redacted debugging records
• Actionable pages with exemplar traces
• Tool-loop evidence for agent paths
• Experiment evidence handoff

Evaluation rubric

• Starts from a factual-serving invariant instead of generic dashboard metrics
• Treats justified abstention as safe behavior rather than a blanket failure
• Detects unsafe served claims even when latency is healthy
• Computes post-first-token throughput without counting the first token twice
• Keeps sensitive prompt and answer text out of default durable attributes
• Uses a versioned cost card rather than hard-coded current-provider assertions
• Pages with an exemplar trace and a concrete first check
• Records tool progress without relying on chain-of-thought logs
• Blocks a candidate fix until a measured experiment validates it

Follow-up questions

Common pitfalls

Response success is mistaken for answer safety

Symptom: Dashboards stay green while customers receive invented ETAs. Cause: Monitoring captures HTTP status and latency but not claim-gate outcomes. Fix: Emit and aggregate the unsafe-served-claim invariant beside performance metrics.

Abstention pages the on-call engineer

Symptom: A strict gate produces noisy incidents whenever evidence is missing. Cause: Safe withholding and unsafe serving were collapsed into one "bad answer" metric. Fix: Separate unsafe-serve, abstention, and supported-serve rates.

Streaming speed is calculated incorrectly

Symptom: Generation throughput looks slightly better than it was. Cause: The first token is counted in an interval that begins after first-token arrival. Fix: Divide remaining output tokens by post-TTFT duration, or define a different measured interval explicitly. Reject malformed timing events instead of masking them with a fallback denominator.

Durable logs collect raw customer text

Symptom: Debugging records retain order identifiers and email addresses unnecessarily. Cause: Raw prompts were treated as ordinary metadata. Fix: Store redacted previews and stable identifiers by default; govern any raw-payload access separately.

The alert cannot start an investigation

Symptom: A page says "quality decreased" and on-call searches broad logs manually. Cause: The alert has no exemplar trace, template version, evidence version, or first failed stage. Fix: Attach a violating trace and the first runbook check to every invariant alert.