LLM Observability & Monitoring

deploy-answerer-v1, a large language model (LLM) answer path, refused to promise a deploy approval that wasn't present in a ReleaseOps record. It also emitted a trace: evidence version, route, and first failed stage. One safe trace proves one request behaved correctly. It doesn't tell you whether the service is still safe across live traffic.

Suppose a prompt rollback silently bypasses that gate for two operators. The endpoint still returns 200. Answers remain fluent. Latency stays fast. Only the trace events reveal that unsupported deploy approvals escaped into served messages.

Monitoring reduces many trace events into rates, percentiles, and alerts. Observability lets you start from an alert and reconstruct why a particular request failed. An LLM application needs both: aggregates to detect a change and request evidence to diagnose it.

Start with the event your gate already produced

Avoid beginning with a dashboard wish list. Begin with the decision boundary from the serving path:

The examples use synthetic events from the same deploy-policy answer path as the prior lab. The last two events model a regression; they aren't claims about real production traffic.

The first cell creates six requests. Four are safe outcomes from a strict gate: a supported answer or a blocked factual claim. Two come from a regressed template that served an invented deploy approval.

Notice the distinction: an abstention isn't automatically a failure. When evidence doesn't establish a deploy approval, abstention is the safe product behavior. The event you need to page on is an unsupported factual claim that was served.

Turn request evidence into quality metrics

An aggregate must preserve the invariant it claims to monitor. A generic "answer quality score" could hide two unsafe deploy claims inside many pleasant responses. For this answer path, start with metrics directly derived from claim-gate outcomes:

The next cell aggregates the synthetic production window.

The counts do more than say "quality is down." All four unsupported drafts first failed at claim_generation. The strict template safely withheld two; the regressed template let two escape through serving_gate. That evidence points at a gate or template rollback before anyone tunes retrieval.

Measure responsiveness without hiding correctness

Quality is the first invariant here, but operators also notice sluggish answers. For a streamed response:

• Time to first token (TTFT) is the delay before the operator sees output.
• Post-first-token throughput measures how quickly the remaining output arrives.
• End-to-end latency includes the entire response.

Be precise with token arithmetic. If TTFT marks arrival of the first token, only output_tokens - 1 tokens arrive in the time after TTFT. Counting all output tokens in that interval slightly overstates throughput.

The fixture also reports p95 TTFT: 95% of measured requests begin output at or below that first-token delay. With only six requests, nearest-rank p95 is the slowest observation. This teaches the calculation, not a production budget.

This window is fast and unsafe. A latency-only dashboard would celebrate the exact release that should be rolled back.

Don't hide malformed timing events behind a tiny fallback denominator. If a multi-token response reports total_ms <= ttft_ms, fix the instrumentation before trusting its throughput.

OpenTelemetry (OTel) semantic conventions 1.41.0 distinguish client first-chunk latency (gen_ai.client.operation.time_to_first_chunk) from model-server first-token latency (gen_ai.server.time_to_first_token). This lab's ttft_ms is an application-owned operator-visible field, so document exactly where your timer starts and stops. The GenAI convention is still marked Development; pin the convention emitted by your instrumentation instead of treating its field set as permanent.^{[1]Reference 1Semantic conventions for generative AI systemshttps://opentelemetry.io/docs/specs/semconv/gen-ai/[2]Reference 2OpenTelemetry GenAI metrics, v1.41.0https://github.com/open-telemetry/semantic-conventions/blob/v1.41.0/docs/gen-ai/gen-ai-metrics.md}

Parent-child span hierarchy for agent tracing

A simple request-response model is a flat timeline. But a tool-using agent, RAG pipeline, or multi-step assistant behaves like a tree: a root user operation spawns retrieval queries, multiple LLM thinking loops, and external tool execution spans.

To reconstruct what happened, OpenTelemetry uses parent-child span relationships. The root span represents the user-facing request, while child spans capture nested operations such as retrieval, model generation, and tool execution.^{[3]Reference 3OpenTelemetry Specification.https://opentelemetry.io/docs/specs/otel}

Here is how you initialize and link these spans programmatically using the OpenTelemetry API:

Keep standard telemetry and product decisions together

In semantic conventions 1.41.0, the inference-span attributes require both gen_ai.operation.name and gen_ai.provider.name; gen_ai.request.model is conditionally required when available. The synthetic service below uses a custom provider name because no real provider is involved. Use the convention's well-known provider value when one applies. Your application still needs custom attributes for its own invariant: which evidence version supported the answer, whether the answer route blocked a claim, where the first failure occurred, and which protection boundary leaked it.^{[4]Reference 4OpenTelemetry GenAI span definitions, v1.41.0https://github.com/open-telemetry/semantic-conventions/blob/v1.41.0/model/gen-ai/spans.yaml}

Prompt, output-message, and system-instruction attributes are opt-in in the OTel convention because they may contain sensitive content. Store stable identifiers and safe outcome attributes by default; make raw text capture a deliberate, governed exception.^{[4]Reference 4OpenTelemetry GenAI span definitions, v1.41.0https://github.com/open-telemetry/semantic-conventions/blob/v1.41.0/model/gen-ai/spans.yaml}

gen_ai.* attributes let tracing backends recognize the model call. platform.* attributes explain the business decision. Neither requires putting an operator's full message into a durable trace.

Attribute cost with a versioned rate card

Cost is part of monitoring, but embedding current provider price claims in application logic or a tutorial makes both go stale quickly. Production systems should store usage counts on the trace and evaluate them with a versioned rate card. The rates below are synthetic fixture values, chosen only to exercise the calculation.

If a provider later changes cached-token pricing or adds a service-tier surcharge, old traces remain interpretable: usage was recorded once, while the rate-card version states how the estimate was computed. Reject impossible counters rather than clamping them into a plausible-looking cost.

Store durable debugging evidence without operator text

A trace may contain an incident identifier, email address, or full operator request. That text can help an incident investigation, but it shouldn't be your default long-lived record.

For this workflow, durable logs can retain:

The record below keeps a redacted preview for an incident exemplar while leaving raw payload storage unset.

The regex is intentionally narrow: it makes the fixture readable, but it isn't a production scrubber. In production, minimize collected attributes, review what instrumentation libraries emit, and apply centrally managed Collector processors or an equivalent scrub pipeline before durable storage.^{[5]Reference 5Handling sensitive datahttps://opentelemetry.io/docs/security/handling-sensitive-data/}

Alert on decisions engineers can act on

Not every metric deserves a page. A long answer or higher cost might warrant investigation. An unsupported deploy claim that escaped a hard gate violates the product invariant and should page immediately.

Thresholds must come from a service-level objective (SLO) or another owned service contract, not a generic article. This lab's synthetic policy says:

• Any unsafe served deploy claim is a page.
• TTFT above 500 ms is a warning for this interactive answer path.
• Estimated cost above the fixture budget is a warning, using the rate card named in the event window.

The page is actionable because it says which invariant broke. It doesn't page merely because a judge score drifted or because latency was a little noisy.

Six synthetic requests are enough to demonstrate alert logic, not to set a production latency or cost budget. In a real launch, define those warning thresholds from representative traffic and revisit them as workload shape changes. The zero-tolerance safety invariant is different: a known unsupported deploy claim must not be served.

Attach an exemplar and a first investigation step

An on-call engineer shouldn't begin by searching arbitrary logs. For an invariant page, attach one violating trace with its prompt template and source version, then state the immediate check.

The request still needs deeper review, but investigation is now narrow: generation produced an unsupported deploy approval, and the serving gate let it escape. Compare the template or gate configuration with the strict version that abstained on the same failure case.

Extend the contract when answers call tools

The central example is a factual answer gate. If the product later allows an agent to create open rollback tickets or trigger CI jobs, the same discipline applies to tool actions. Log state transitions you control: tool name, redacted parameter hash, result status, progress marker, retry count, and stop reason. Don't depend on hidden reasoning text.

Preserve the evidence for the next fix

An alert tells you something is wrong now. The next engineering question is whether a proposed fix is better. That requires an experiment record linking the failing window, proposed template, policy, and rerun results.

The monitoring system doesn't approve the proposed fix. It creates a precise starting point for the next controlled run: what broke, which candidate claims to fix it, and which metric must pass.

Mastery check

Mastery outcomes

Evaluation rubric

• Starts from a factual-serving invariant instead of generic dashboard metrics
• Treats justified abstention as safe behavior rather than a blanket failure
• Detects unsafe served claims even when latency is healthy
• Keeps the earliest defect separate from the protection boundary that leaked it
• Computes post-first-token throughput without counting the first token twice
• Keeps sensitive prompt and answer text out of default durable attributes
• Uses a versioned cost card rather than hard-coded current-provider assertions
• Pages with an exemplar trace and a concrete first check
• Records tool progress without relying on chain-of-thought logs
• Blocks a candidate fix until a measured experiment validates it

Follow-up questions

Common pitfalls

Response success is mistaken for answer safety

• Symptom: Dashboards stay green while operators receive invented deploy approvals.
• Cause: Monitoring captures HTTP status and latency but not claim-gate outcomes.
• Fix: Emit and aggregate the unsafe-served-claim invariant beside performance metrics.

Abstention pages the on-call engineer

• Symptom: A strict gate produces noisy incidents whenever evidence is missing.
• Cause: Safe withholding and unsafe serving were collapsed into one "bad answer" metric.
• Fix: Separate unsafe-serve, abstention, and supported-serve rates.

Streaming speed is calculated incorrectly

• Symptom: Generation throughput looks slightly better than it was.
• Cause: The first token is counted in an interval that begins after first-token arrival.
• Fix: Divide remaining output tokens by post-TTFT duration, or define a different measured interval explicitly. Reject malformed timing events instead of masking them with a fallback denominator.

Root cause and containment escape are collapsed

• Symptom: An incident starts at the serving gate but loses where the unsupported draft originated.
• Cause: One field is used for both the earliest defect and the boundary that leaked it.
• Fix: Record first_failed_stage and escape_stage separately, then attach both to the exemplar.

Durable logs collect raw operator text

• Symptom: Debugging records retain incident identifiers and email addresses unnecessarily.
• Cause: Raw prompts were treated as ordinary metadata.
• Fix: Store redacted previews and stable identifiers by default; govern any raw-payload access separately.

The alert can't start an investigation

• Symptom: A page says "quality decreased" and on-call searches broad logs manually.
• Cause: The alert has no exemplar trace, template version, evidence version, first failed stage, or escape stage.
• Fix: Attach a violating trace and the first runbook check to every invariant alert.