LearnApplied LLM EngineeringLLM Observability & Monitoring

⚙️MediumMLOps & Deployment

LLM Observability & Monitoring

Turn claim-level answer traces into production metrics, actionable alerts, privacy-safe debugging records, and reproducible incident evidence.

20 min read

Learning path

Step 78 of 196 in the full curriculum

Hallucination Detection & Mitigation Experiment Tracking with MLflow and W&B

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

At 09:12 in this synthetic deploy-policy window, the dashboard for a large language model (LLM) service looks healthy: every endpoint response is 200, first-token latency stays below the 500 ms warning, and cost stays under the fixture budget. Two answers still contain unsupported deploy approvals. Status and latency show completion, not safety. A trace that joins claim-gate outcome to served route exposes the failure.

The last lesson built incident-answerer-v1: a claim gate that refused to invent a recovery time missing from the versioned incident record. Each request also wrote a trace with evidence version, route, and first failed stage. Keep that contract on the parallel deploy-answerer-v1 path, then watch a window. One trace answers what happened to one request; window metrics answer whether failure rate or latency shifted.

Observability joins those views. Monitoring compresses many events into rates, percentiles, and alerts. Request evidence lets you start from an alert and reconstruct why one call failed. You need both because a fast 200 can still be unsafe.

TTFT bar chart for six deploy-answer requests on a 0 to 500 millisecond axis. Requests 201 through 204 stay safe at 172, 181, 188, and 205 ms. Requests 205 and 206 are unsafe serves at 218 and 240 ms. Every bar sits under the 500 ms warning, which is the chart maximum. — Every first-token delay sits under the 500 ms warning. Color, not height, marks the two unsafe serves. A latency dashboard would miss the rollback.

The gate already emits the fields you need

Start with serving decision, not dashboard. Prior lesson already recorded fields that let you ask both “what happened?” and “where did containment fail?”:

Serving question	Event field	Why it matters
What answer route was chosen?	`route`	Shows `serve`, `shorten`, `review`, or `abstain` behavior
Did an unsupported claim reach the operator?	`unsafe_claim_served`	Direct safety invariant
Where did a request first go wrong?	`first_failed_stage`	Separates evidence, generation, and serving faults
Which protection boundary leaked a known failure?	`escape_stage`	Distinguishes root cause from failed containment
Which record and prompt were used?	`evidence_version`, `prompt_template_id`	Makes a failure reproducible
Was the request usable and affordable?	token and timing fields	Measures experience and cost beside quality

These are synthetic events from parallel deploy-policy path, built on prior lab's trace contract. Four requests show strict gate containing an unsupported draft; two show a regressed template serving an invented deploy approval. They aren't claims about production traffic.

Before running cell, predict page signal: abstentions should remain safe, while any unsafe_claim_served=True row should be visible as an unsafe serve. Six rows make that distinction inspectable.

trace-events.py

from dataclasses import dataclass, replace

@dataclass(frozen=True)
class TraceEvent:
    request_id: str
    case_id: str
    evidence_version: str | None
    prompt_template_id: str
    route: str
    first_failed_stage: str
    escape_stage: str | None
    unsafe_claim_served: bool
    input_tokens: int
    cached_input_tokens: int
    output_tokens: int
    ttft_ms: int
    total_ms: int

current_window = [
    TraceEvent("req_201", "clean_policy", "deploy-policy@v17", "deploy-answerer-v1",
               "serve", "passed", None, False, 188, 80, 27, 172, 410),
    TraceEvent("req_202", "invented_approval", "deploy-policy@v17", "deploy-answerer-v1",
               "abstain", "claim_generation", None, False, 196, 80, 20, 181, 395),
    TraceEvent("req_203", "wrong_freeze_status", "deploy-policy@v17", "deploy-answerer-v1",
               "abstain", "claim_generation", None, False, 192, 80, 21, 188, 402),
    TraceEvent("req_204", "missing_policy", None, "deploy-answerer-v1",
               "abstain", "evidence_admission", None, False, 160, 64, 15, 205, 382),
    TraceEvent("req_205", "invented_approval", "deploy-policy@v17", "deploy-answerer-v1.1-regression",
               "serve", "claim_generation", "serving_gate", True, 198, 80, 29, 218, 430),
    TraceEvent("req_206", "invented_approval", "deploy-policy@v17", "deploy-answerer-v1.1-regression",
               "serve", "claim_generation", "serving_gate", True, 201, 80, 30, 240, 455),
]

print(f"events={len(current_window)}")
for event in current_window:
    verdict = "UNSAFE SERVE" if event.unsafe_claim_served else "safe route"
    print(f"{event.request_id}: {event.route:7} {verdict}")

Output

events=6
req_201: serve   safe route
req_202: abstain safe route
req_203: abstain safe route
req_204: abstain safe route
req_205: serve   UNSAFE SERVE
req_206: serve   UNSAFE SERVE

An abstention isn't automatically a failure. When evidence doesn't establish a deploy approval, withholding is safe product behavior. Page-worthy event is an unsupported factual claim reaching an operator.

Rows show event-level evidence. Next, turn them into a window metric while keeping one violating request attached for diagnosis.

Diagram showing One request, Retrieve policy, Generate claims, and Serving gate. — One request, Retrieve policy, Generate claims, and Serving gate.

Keep signal boundaries clear. Traces answer which request and which step. Metrics answer how often and how badly across a window. Durable logs keep reduced, privacy-safe evidence after trace retention. One artifact can't replace all three.

Turn request evidence into quality metrics

Metric is useful only when its numerator matches invariant it claims to protect. A generic "answer quality score" could hide two unsafe deploy claims inside many pleasant responses. Start with counters derived directly from claim-gate outcomes:

Metric	Numerator / denominator	Interpretation
Unsafe-serve rate	Requests with any unsupported served claim / requests	Operator-facing factual safety failure
Safe-route rate	Requests without unsafe served claims / requests	Gate effectiveness, including justified abstention
Abstention rate	Abstained requests / requests	Product usefulness signal, not a safety failure by itself
Failure-stage count	Requests by first failed stage	Where engineers should investigate first
Escape-stage count	Unsafe requests by leaked protection boundary	Where containment failed after the first defect

Before running cell, predict summary: 2 of 6 requests should page as unsafe, 3 should abstain safely, and all 4 unsupported drafts should point to claim_generation. Cell makes those counts reproducible.

quality-window.py

from collections import Counter
from dataclasses import dataclass

@dataclass(frozen=True)
class QualityWindow:
    requests: int
    unsafe_serves: int
    abstentions: int
    failure_stages: Counter
    escape_stages: Counter

    @property
    def unsafe_serve_rate(self) -> float:
        return self.unsafe_serves / self.requests

    @property
    def safe_route_rate(self) -> float:
        return (self.requests - self.unsafe_serves) / self.requests

def quality_window(events: list[TraceEvent]) -> QualityWindow:
    return QualityWindow(
        requests=len(events),
        unsafe_serves=sum(event.unsafe_claim_served for event in events),
        abstentions=sum(event.route == "abstain" for event in events),
        failure_stages=Counter(event.first_failed_stage for event in events),
        escape_stages=Counter(
            event.escape_stage for event in events if event.escape_stage is not None
        ),
    )

quality = quality_window(current_window)
print(f"unsafe_serve_rate={quality.unsafe_serve_rate:.1%}")
print(f"safe_route_rate={quality.safe_route_rate:.1%}")
print(f"abstentions={quality.abstentions}")
print(f"claim_generation_failures={quality.failure_stages['claim_generation']}")
print(f"serving_gate_escapes={quality.escape_stages['serving_gate']}")

assert quality.unsafe_serves == 2
assert quality.failure_stages["claim_generation"] == 4
assert quality.escape_stages["serving_gate"] == 2

Output

unsafe_serve_rate=33.3%
safe_route_rate=66.7%
abstentions=3
claim_generation_failures=4
serving_gate_escapes=2

The counts do more than say "quality is down." All four unsupported drafts first failed at claim_generation. The strict template safely withheld two; the regressed template let two escape through serving_gate. That evidence points at a gate or template rollback before anyone tunes retrieval.

Read those stage fields as a small diagnosis tree. If first_failed_stage points at claim generation and escape_stage is empty, generation failed but gate contained it. If escape_stage is set, follow that boundary first, then return to root stage. One pair of fields separates mitigation from root-cause work.

Split of four claim-generation defects in the six-request window. Left: req_202 and req_203 stay contained under deploy-answerer-v1, route abstain, no escape stage. Right: req_205 and req_206 escape through serving_gate on v1.1-regression, route serve, and page. — Four unsupported drafts share the same first failed stage. The serving gate is what turns two of them into a page and leaves the other two as safe abstentions.

Measure responsiveness without hiding correctness

Quality is first invariant here, but operators also notice sluggish answers. For a streamed response:

Time to first token (TTFT) is the delay before the operator sees output.
Post-first-token throughput measures how quickly the remaining output arrives.
End-to-end latency includes the entire response.

Be precise with token arithmetic. If TTFT marks arrival of first token, only output_tokens - 1 tokens arrive in time after TTFT. Counting all output tokens in that interval slightly overstates throughput. For req_206, predict before running cell: 30 output tokens mean 29 arrive after first, over 215 ms.

The fixture also reports p95 TTFT: 95% of measured requests begin output at or below that first-token delay. With only six requests, nearest-rank p95 is the slowest observation. This teaches the calculation, not a production budget.

streaming-latency.py

import math

def nearest_rank_percentile(values: list[int], percentile: float) -> int:
    ordered = sorted(values)
    rank = max(1, math.ceil(percentile * len(ordered)))
    return ordered[rank - 1]

def post_first_token_tps(event: TraceEvent) -> float:
    remaining_tokens = max(event.output_tokens - 1, 0)
    if remaining_tokens == 0:
        return 0.0
    remaining_ms = event.total_ms - event.ttft_ms
    if remaining_ms <= 0:
        raise ValueError("total_ms must exceed ttft_ms after the first token")
    return remaining_tokens / (remaining_ms / 1000)

p95_ttft_ms = nearest_rank_percentile([event.ttft_ms for event in current_window], 0.95)
example_tps = post_first_token_tps(current_window[-1])
malformed_timing = replace(current_window[-1], total_ms=240)

print(f"p95_ttft_ms={p95_ttft_ms}")
print(f"req_206_post_first_tps={example_tps:.1f}")
print(f"quality_safe={quality.unsafe_serves == 0}")
try:
    post_first_token_tps(malformed_timing)
except ValueError as error:
    print(f"invalid_timing={error}")
else:
    raise AssertionError("malformed timing event should fail")

assert p95_ttft_ms == 240
assert quality.unsafe_serves > 0

Output

p95_ttft_ms=240
req_206_post_first_tps=134.9
quality_safe=False
invalid_timing=total_ms must exceed ttft_ms after the first token

This window is fast and unsafe. A latency-only dashboard would celebrate release that needs rollback.

Treat malformed timing as instrumentation failure. If a multi-token response reports total_ms <= ttft_ms, repair timer before trusting throughput. A fallback denominator would turn bad telemetry into a believable number.

OpenTelemetry (OTel) GenAI conventions remain Development and now live in a dedicated repository rather than a frozen 1.41.x release. Pin convention your instrumentation emits.^{[1]Reference 1OpenTelemetry GenAI Semantic Conventionshttps://github.com/open-telemetry/semantic-conventions-genai}

Last versioned snapshot and current docs use gen_ai.client.operation.time_to_first_chunk for client first-chunk latency and gen_ai.server.time_to_first_token for model-server first-token latency. Streaming spans can also record gen_ai.response.time_to_first_chunk in seconds.^{[2]Reference 2OpenTelemetry GenAI metrics, v1.41.0https://github.com/open-telemetry/semantic-conventions/blob/v1.41.0/docs/gen-ai/gen-ai-metrics.md}^{[3]Reference 3OpenTelemetry GenAI span definitions, v1.41.0https://github.com/open-telemetry/semantic-conventions/blob/v1.41.0/model/gen-ai/spans.yaml}

This lab's ttft_ms is an application-owned, operator-visible field. Write down where timer starts and stops before comparing it with a server metric.

One request is a tree of spans

A flat event row can page, but it can't show which nested step invented deploy approval. Retrieval, generation, and serving gate have different owners. Without one parent trace, debugging becomes timestamp correlation and guesswork.

OpenTelemetry encodes that causal shape with parent-child spans: one trace id for user request and child spans for nested work.^{[4]Reference 4OpenTelemetry Specification.https://opentelemetry.io/docs/specs/otel} Cell rebuilds req_205 as a tiny in-process tracer, so structure is visible without an SDK. Inference spans follow {operation} {model}; retrieval spans follow {operation} {data_source.id}; serving gate keeps a product name because it isn't a GenAI operation.^{[3]Reference 3OpenTelemetry GenAI span definitions, v1.41.0https://github.com/open-telemetry/semantic-conventions/blob/v1.41.0/model/gen-ai/spans.yaml}

Predict tree before running cell: request is root; retrieval, chat, and serving_gate are siblings. Chat should carry first failure; serving gate should carry escape.

request-span-tree.py

from contextlib import contextmanager
from dataclasses import dataclass

@dataclass
class Span:
    name: str
    parent: str | None
    attributes: dict[str, str | int | bool]

class Tracer:
    def __init__(self) -> None:
        self.spans: list[Span] = []
        self._stack: list[str] = []

    @contextmanager
    def start(self, name: str, **attributes: str | int | bool):
        parent = self._stack[-1] if self._stack else None
        self.spans.append(Span(name, parent, dict(attributes)))
        self._stack.append(name)
        try:
            yield
        finally:
            self._stack.pop()

tracer = Tracer()
unsafe = next(event for event in current_window if event.unsafe_claim_served)
with tracer.start("deploy-answer.request", **{"platform.request.id": unsafe.request_id}):
    with tracer.start(
        "retrieval deploy-policy",
        **{
            "gen_ai.operation.name": "retrieval",
            "platform.evidence.version": unsafe.evidence_version or "",
        },
    ):
        pass
    with tracer.start(
        "chat deploy-assistant-prod",
        **{
            "gen_ai.operation.name": "chat",
            "gen_ai.provider.name": "platform.internal",
            "gen_ai.request.model": "deploy-assistant-prod",
            "gen_ai.usage.input_tokens": unsafe.input_tokens,
            "gen_ai.usage.output_tokens": unsafe.output_tokens,
            "platform.failure.stage": unsafe.first_failed_stage,
        },
    ):
        pass
    with tracer.start(
        "serving_gate",
        **{
            "platform.answer.route": unsafe.route,
            "platform.answer.unsafe_claim_served": unsafe.unsafe_claim_served,
            "platform.failure.escape_stage": unsafe.escape_stage or "",
        },
    ):
        pass

parent_of = {span.name: span.parent for span in tracer.spans}
print(f"root={unsafe.request_id}")
for span in tracer.spans:
    print(f"{span.name} parent={parent_of[span.name]}")

assert unsafe.request_id == "req_205"
assert parent_of == {
    "deploy-answer.request": None,
    "retrieval deploy-policy": "deploy-answer.request",
    "chat deploy-assistant-prod": "deploy-answer.request",
    "serving_gate": "deploy-answer.request",
}
assert tracer.spans[-1].attributes["platform.answer.unsafe_claim_served"] is True

Output

root=req_205
deploy-answer.request parent=None
retrieval deploy-policy parent=deploy-answer.request
chat deploy-assistant-prod parent=deploy-answer.request
serving_gate parent=deploy-answer.request

Chat span owns first failed stage. Serving-gate span owns escape. Together they explain same unsafe row without putting operator message on trace.

Keep standard telemetry and product decisions together

Standard fields identify model operation. On inference span, gen_ai.operation.name and gen_ai.provider.name are required. gen_ai.request.model is required when model id is available. Synthetic service uses custom provider because no real vendor is involved. Use well-known provider value when one applies.^{[3]Reference 3OpenTelemetry GenAI span definitions, v1.41.0https://github.com/open-telemetry/semantic-conventions/blob/v1.41.0/model/gen-ai/spans.yaml}

Product fields answer different questions: which evidence version supported answer, whether route blocked claim, where first failure occurred, and which protection boundary leaked it. Keep those custom attributes beside standard telemetry.

Prompt, output-message, and system-instruction attributes are opt-in in OTel because they may contain sensitive content. Store stable identifiers and safe outcomes by default; raw text capture needs deliberate governance.^{[3]Reference 3OpenTelemetry GenAI span definitions, v1.41.0https://github.com/open-telemetry/semantic-conventions/blob/v1.41.0/model/gen-ai/spans.yaml}

Before running cell, predict what durable attribute map omits: raw input messages, output messages, and system instructions. Keep enough IDs and stages to reproduce decision path without copying operator text.

trace-attributes.py

def trace_attributes(event: TraceEvent) -> dict[str, str | int | bool]:
    attributes: dict[str, str | int | bool] = {
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": "platform.internal",
        "gen_ai.request.model": "deploy-assistant-prod",
        "gen_ai.usage.input_tokens": event.input_tokens,
        "gen_ai.usage.cache_read.input_tokens": event.cached_input_tokens,
        "gen_ai.usage.output_tokens": event.output_tokens,
        "platform.answer.route": event.route,
        "platform.answer.unsafe_claim_served": event.unsafe_claim_served,
        "platform.failure.stage": event.first_failed_stage,
        "platform.prompt.template_id": event.prompt_template_id,
    }
    if event.evidence_version is not None:
        attributes["platform.evidence.version"] = event.evidence_version
    if event.escape_stage is not None:
        attributes["platform.failure.escape_stage"] = event.escape_stage
    return attributes

unsafe_attributes = trace_attributes(current_window[-1])
print(f"operation={unsafe_attributes['gen_ai.operation.name']}")
print(f"template={unsafe_attributes['platform.prompt.template_id']}")
print(f"failed_stage={unsafe_attributes['platform.failure.stage']}")
print(f"escape_stage={unsafe_attributes['platform.failure.escape_stage']}")
print(f"cached_input_tokens={unsafe_attributes['gen_ai.usage.cache_read.input_tokens']}")
print(f"raw_messages_logged={'gen_ai.input.messages' in unsafe_attributes}")

assert unsafe_attributes["platform.answer.unsafe_claim_served"] is True
assert unsafe_attributes["platform.failure.escape_stage"] == "serving_gate"
assert unsafe_attributes["gen_ai.provider.name"] == "platform.internal"
assert unsafe_attributes["gen_ai.usage.cache_read.input_tokens"] == 80
assert "gen_ai.input.messages" not in unsafe_attributes
assert "gen_ai.output.messages" not in unsafe_attributes
assert "gen_ai.system_instructions" not in unsafe_attributes

Output

operation=chat
template=deploy-answerer-v1.1-regression
failed_stage=claim_generation
escape_stage=serving_gate
cached_input_tokens=80
raw_messages_logged=False

gen_ai.* attributes let tracing backends recognize model call, including recommended cache-read counts when provider reports them. platform.* attributes explain business decision. Neither requires putting operator's full message into durable trace.

Attribute cost with a versioned rate card

Cost belongs in same window, but baking today's provider prices into application logic or tutorial makes both go stale. Store usage counts on trace, including cached input when provider reports it, then evaluate them with versioned rate card. Rates below are synthetic fixture values chosen to exercise calculation.

Predict req_206 before running cell: 201 total input minus 80 cached leaves 121 uncached; fixture rates should produce $0.000279. Second malformed event should be rejected, not clamped.

versioned-cost.py

@dataclass(frozen=True)
class RateCard:
    version: str
    input_per_million: float
    cached_input_per_million: float
    output_per_million: float

def estimate_cost(event: TraceEvent, card: RateCard) -> float:
    if event.input_tokens < 0 or event.output_tokens < 0:
        raise ValueError("token counts must be non-negative")
    if not 0 <= event.cached_input_tokens <= event.input_tokens:
        raise ValueError("cached_input_tokens must be between 0 and input_tokens")
    uncached_input = event.input_tokens - event.cached_input_tokens
    return (
        uncached_input * card.input_per_million
        + event.cached_input_tokens * card.cached_input_per_million
        + event.output_tokens * card.output_per_million
    ) / 1_000_000

card = RateCard(
    version="internal-fixture-2026-05-27",
    input_per_million=1.00,
    cached_input_per_million=0.10,
    output_per_million=5.00,
)
cost_by_request = {
    event.request_id: estimate_cost(event, card)
    for event in current_window
}
total_cost = sum(cost_by_request.values())
malformed_usage = replace(current_window[-1], cached_input_tokens=202)

print(f"rate_card={card.version}")
print(f"window_cost_usd={total_cost:.6f}")
print(f"req_206_cost_usd={cost_by_request['req_206']:.6f}")
try:
    estimate_cost(malformed_usage, card)
except ValueError as error:
    print(f"invalid_usage={error}")
else:
    raise AssertionError("malformed usage event should fail")

assert card.version.startswith("internal-fixture")
assert total_cost > 0

Output

rate_card=internal-fixture-2026-05-27
window_cost_usd=0.001427
req_206_cost_usd=0.000279
invalid_usage=cached_input_tokens must be between 0 and input_tokens

Versioning separates observation from repricing. If provider changes cached-token pricing or adds service-tier surcharge, old traces keep usage while rate-card version explains estimate. Reject impossible counters rather than clamping them into plausible-looking cost.

Store durable debugging evidence without operator text

A trace may contain an incident identifier, email address, or full operator request. That context can help solve today's incident, but it shouldn't become default long-lived record.

For this workflow, durable logs can retain:

Retain by default	Avoid by default
Request ID, evidence version, prompt template ID	Full operator message
Route, first failure stage, escape stage, claim-verdict counts	Full generated answer
Token counts, latency, rate-card version	Unredacted incident or contact details
Redacted preview when needed	Raw retrieved documents

Record below keeps redacted preview for incident exemplar while leaving raw payload storage unset. Predict what survives: IDs, versions, route, stages, and safe preview. Full message and raw payload stay out.

durable-log-record.py

import re

@dataclass(frozen=True)
class DurableRecord:
    request_id: str
    prompt_template_id: str
    evidence_version: str | None
    route: str
    failed_stage: str
    escape_stage: str | None
    redacted_preview: str
    raw_payload_ref: str | None
    scrub_policy: str

def redact_operator_text(text: str) -> str:
    text = re.sub(r"#[A-Z0-9]+", "[INCIDENT_ID]", text)
    return re.sub(r"\b[\w.+-]+@[\w.-]+\.\w+\b", "[EMAIL]", text)

def durable_record(event: TraceEvent, operator_text: str) -> DurableRecord:
    return DurableRecord(
        request_id=event.request_id,
        prompt_template_id=event.prompt_template_id,
        evidence_version=event.evidence_version,
        route=event.route,
        failed_stage=event.first_failed_stage,
        escape_stage=event.escape_stage,
        redacted_preview=redact_operator_text(operator_text),
        raw_payload_ref=None,
        scrub_policy="platform-pii-v1",
    )

record = durable_record(
    current_window[-1],
    "Email me deploy approval for incident #INC10234 at [email protected]",
)
print(record.redacted_preview)
print(f"raw_payload_stored={record.raw_payload_ref is not None}")
print(f"scrub_policy={record.scrub_policy}")
print(f"escape_stage={record.escape_stage}")

assert "#INC10234" not in record.redacted_preview
assert "[email protected]" not in record.redacted_preview

Output

Email me deploy approval for incident [INCIDENT_ID] at [EMAIL]
raw_payload_stored=False
scrub_policy=platform-pii-v1
escape_stage=serving_gate

Regex is intentionally narrow: it makes fixture readable, but it isn't a production scrubber. Before durable storage, minimize collected attributes, review what instrumentation libraries emit, and apply centrally managed Collector processors or equivalent scrub pipeline.^{[5]Reference 5Handling sensitive datahttps://opentelemetry.io/docs/security/handling-sensitive-data/}

That boundary matters operationally. A trace is short-lived investigation context; durable log is deliberately reduced evidence for later diagnosis. Copying everything into both raises privacy and storage risk without adding a new answer to incident question.

Three-step privacy path for req_206. Raw operator text includes incident INC10234 and bruno@example.com and is not stored. platform-pii-v1 replaces those values with INCIDENT_ID and EMAIL markers. The durable record keeps template, evidence version, route, claim-generation failure, and serving-gate escape, with raw_payload_ref set to None. — The durable req_206 record keeps template, evidence, route, and both failure stages. The incident id, email, and raw payload never become the stored record.

Alert on decisions engineers can act on

Not every metric deserves a page. A long answer or higher cost might warrant investigation. An unsupported deploy claim that escaped hard gate violates product invariant and should page immediately.

Thresholds belong to a service-level objective (SLO) or another owned service contract, not a generic article. This lab's synthetic policy says:

Any unsafe served deploy claim is a page.
TTFT above 500 ms is a warning for this interactive answer path.
Estimated cost above the fixture budget is a warning, using the rate card named in the event window.

Predict findings before running cell: two unsafe serves should produce one page; p95 TTFT is 240 ms, below 500; cost is below $0.002. Result should contain one action, not three noisy warnings.

actionable-alerts.py

@dataclass(frozen=True)
class MonitorPolicy:
    policy_id: str
    max_unsafe_serves: int
    max_p95_ttft_ms: int
    max_window_cost_usd: float

def evaluate_alerts(
    events: list[TraceEvent],
    quality: QualityWindow,
    cost_usd: float,
    policy: MonitorPolicy,
) -> list[str]:
    p95 = nearest_rank_percentile([event.ttft_ms for event in events], 0.95)
    findings: list[str] = []
    if quality.unsafe_serves > policy.max_unsafe_serves:
        findings.append(
            f"PAGE unsafe_claim_served={quality.unsafe_serves}/{quality.requests}"
        )
    if p95 > policy.max_p95_ttft_ms:
        findings.append(f"WARN p95_ttft_ms={p95}")
    if cost_usd > policy.max_window_cost_usd:
        findings.append(f"WARN window_cost_usd={cost_usd:.6f}")
    return findings

policy = MonitorPolicy(
    policy_id="deploy-grounding-slo-v1",
    max_unsafe_serves=0,
    max_p95_ttft_ms=500,
    max_window_cost_usd=0.002,
)
findings = evaluate_alerts(current_window, quality, total_cost, policy)

for finding in findings:
    print(finding)
print(f"latency_warning={any('ttft' in item for item in findings)}")
print(f"cost_warning={any('cost' in item for item in findings)}")

assert findings == ["PAGE unsafe_claim_served=2/6"]

Output

PAGE unsafe_claim_served=2/6
latency_warning=False
cost_warning=False

Page is actionable because it names broken invariant. It doesn't page merely because judge score drifted or latency was noisy.

Six synthetic requests demonstrate alert logic, not production latency or cost budget. Set warning thresholds from representative traffic and revisit them as workload shape changes. Zero-tolerance safety invariant is different: known unsupported deploy claim must not be served.

Attach an exemplar and a first investigation step

On-call engineer shouldn't begin by searching arbitrary logs. For invariant page, attach one violating trace with prompt template and source version, then state immediate check. Read evidence in order: invariant breach, first failed stage, escape stage, version IDs, first check. That sequence separates root defect from containment leak.

runbook-card.py

@dataclass(frozen=True)
class IncidentCard:
    policy_id: str
    summary: str
    exemplar_request_id: str
    template_id: str
    evidence_version: str | None
    first_failed_stage: str
    escape_stage: str | None
    first_check: str

def incident_card(events: list[TraceEvent], policy: MonitorPolicy) -> IncidentCard:
    exemplar = next(event for event in events if event.unsafe_claim_served)
    return IncidentCard(
        policy_id=policy.policy_id,
        summary="Unsupported deploy claim reached operator response.",
        exemplar_request_id=exemplar.request_id,
        template_id=exemplar.prompt_template_id,
        evidence_version=exemplar.evidence_version,
        first_failed_stage=exemplar.first_failed_stage,
        escape_stage=exemplar.escape_stage,
        first_check="Compare serving-gate template with deploy-answerer-v1.",
    )

card_view = incident_card(current_window, policy)
print(f"exemplar={card_view.exemplar_request_id}")
print(f"template={card_view.template_id}")
print(f"first_stage={card_view.first_failed_stage}")
print(f"escape_stage={card_view.escape_stage}")
print(f"first_check={card_view.first_check}")

assert card_view.first_failed_stage == "claim_generation"
assert card_view.escape_stage == "serving_gate"

Output

exemplar=req_205
template=deploy-answerer-v1.1-regression
first_stage=claim_generation
escape_stage=serving_gate
first_check=Compare serving-gate template with deploy-answerer-v1.

Request still needs deeper review, but investigation is narrow: generation produced unsupported deploy approval; serving gate let it escape. Compare template or gate configuration with strict version that abstained on same failure case.

Extend the contract when answers call tools

Central example is factual answer gate. If product later lets agent open rollback tickets or trigger CI jobs, apply same discipline to tool actions. OTel names those spans execute_tool {gen_ai.tool.name}.^{[3]Reference 3OpenTelemetry GenAI span definitions, v1.41.0https://github.com/open-telemetry/semantic-conventions/blob/v1.41.0/model/gen-ai/spans.yaml}

Tool arguments and results are opt-in because they can carry secrets. This lab logs redacted parameter hash instead of raw payload. Keep state you control: tool name, hash, result status, progress marker, retry count, and stop reason. Don't depend on hidden reasoning text.

Predict loop decision before running cell: same tool and hash aren't enough to prove a stall; repeated progress marker is missing evidence of movement.

Two consecutive lookup_deploy_status steps compared on tool name, redacted parameter hash deploy-redacted:v1, and progress marker. When the marker stays scan:v17, stalled_repeat is true and the route is stop_and_review. When the marker becomes scan:v18, the same tool and hash continue. — The same tool and parameter hash are not a loop by themselves. The progress marker has to stay frozen before the monitor stops the agent for review.

tool-loop-monitor.py

@dataclass(frozen=True)
class ToolStep:
    tool_name: str
    params_hash: str
    progress_marker: str

def stalled_repeat(steps: list[ToolStep]) -> bool:
    if len(steps) < 2:
        return False
    last, previous = steps[-1], steps[-2]
    return (
        last.tool_name == previous.tool_name
        and last.params_hash == previous.params_hash
        and last.progress_marker == previous.progress_marker
    )

steps = [
    ToolStep("lookup_deploy_status", "deploy-redacted:v1", "scan:v17"),
    ToolStep("lookup_deploy_status", "deploy-redacted:v1", "scan:v17"),
]

print(f"tool={steps[-1].tool_name}")
print(f"stalled_repeat={stalled_repeat(steps)}")
print("route=stop_and_review" if stalled_repeat(steps) else "route=continue")

assert stalled_repeat(steps) is True

Output

tool=lookup_deploy_status
stalled_repeat=True
route=stop_and_review

Preserve the evidence for the next fix

An alert tells you something is wrong now. The next engineering question is whether a proposed fix is better. That requires an experiment record linking the failing window, proposed template, policy, and rerun results.

experiment-handoff.py

@dataclass(frozen=True)
class ExperimentHandoff:
    incident_policy_id: str
    failing_template_id: str
    exemplar_request_id: str
    evidence_version: str | None
    candidate_template_id: str
    required_metric: str
    promotion_status: str

handoff = ExperimentHandoff(
    incident_policy_id=card_view.policy_id,
    failing_template_id=card_view.template_id,
    exemplar_request_id=card_view.exemplar_request_id,
    evidence_version=card_view.evidence_version,
    candidate_template_id="deploy-answerer-v1.2-fix",
    required_metric="unsafe_claim_served == 0 on regression and holdout windows",
    promotion_status="BLOCKED_PENDING_EVALUATION",
)

print(f"candidate={handoff.candidate_template_id}")
print(f"required_metric={handoff.required_metric}")
print(f"promotion={handoff.promotion_status}")

assert handoff.promotion_status == "BLOCKED_PENDING_EVALUATION"

Output

candidate=deploy-answerer-v1.2-fix
required_metric=unsafe_claim_served == 0 on regression and holdout windows
promotion=BLOCKED_PENDING_EVALUATION

Monitoring doesn't approve proposed fix. It creates precise starting point for next controlled run: what broke, which candidate claims to fix it, and which metric must pass.

Keep handoff small: incident policy, failing template, exemplar request, evidence version, candidate, and required safety metric. Experiment tracking can consume those identifiers without copying raw traces or inventing new failure label.

Sample judges. Keep the hard gate unsampled

Paging path assumes claim-gate events already exist on every request. Other production surfaces have hard gates on only a subset of traffic, or none at all. Choose sampling by signal class:

Signal class	Sampling habit	Why
Hard safety invariants (unsafe serve, policy violation)	Always-on counters and exemplars	One miss must page; keep the page path unsampled
Expensive judges / LLM-as-judge quality scores	Head sample a fixed rate (for example 1 to 10%), or tail-sample errors and high latency first	Cost grows with volume; pin the judge version like experiment tracking
High-cardinality attributes (raw prompts, full tool payloads)	Default off; sample only under an investigation flag	Protects privacy and trace storage
Agent tool loops	Always log controlled state deltas (tool name, param hash, progress); sample full nested spans at a lower rate if needed	Loop detection needs the state markers more than every attribute

Treat continuous online evaluation as a separate, versioned scorer on sampled set, not a substitute for hard gate. Always-on invariant still pages; sample feeds trends and holdout comparison for next experiment.

Next Step

Continue to Experiment Tracking with MLflow and W&B

You can now page on an unsafe serve and hand off the failing template, exemplar, and required metric. Next you'll freeze those identifiers into comparable runs so `deploy-answerer-v1.2-fix` earns promotion instead of inheriting it from the incident.

PreviousHallucination Detection & Mitigation

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

OpenTelemetry GenAI Semantic Conventions

OpenTelemetry Authors · 2026

https://github.com/open-telemetry/semantic-conventions-genai

OpenTelemetry GenAI metrics, v1.41.0

OpenTelemetry Authors · 2026

https://github.com/open-telemetry/semantic-conventions/blob/v1.41.0/docs/gen-ai/gen-ai-metrics.md

OpenTelemetry GenAI span definitions, v1.41.0

OpenTelemetry Authors · 2026

https://github.com/open-telemetry/semantic-conventions/blob/v1.41.0/model/gen-ai/spans.yaml

OpenTelemetry Specification.

OpenTelemetry Authors. · 2023

https://opentelemetry.io/docs/specs/otel

Handling sensitive data

OpenTelemetry Authors · 2026

https://opentelemetry.io/docs/security/handling-sensitive-data/

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.

LLM Observability & Monitoring

The gate already emits the fields you need

Turn request evidence into quality metrics

An answer returns HTTP 200 in 500 ms but cites the wrong runbook. Which signal catches the failure?

Measure responsiveness without hiding correctness

One request is a tree of spans

Why should retrieval, generation, and the serving gate share one root request trace?

Keep standard telemetry and product decisions together

Attribute cost with a versioned rate card

Store durable debugging evidence without operator text

Alert on decisions engineers can act on

Attach an exemplar and a first investigation step

An alert says only `grounding_rate < 95%`. What evidence makes it actionable?

Extend the contract when answers call tools

Preserve the evidence for the next fix

Sample judges. Keep the hard gate unsampled

Mastery Check

Discussion

An answer returns HTTP 200 in 500 ms but cites the wrong runbook. Which signal catches the failure?

An answer returns HTTP 200 in 500 ms but cites the wrong runbook. Which signal catches the failure?

Why should retrieval, generation, and the serving gate share one root request trace?

Why should retrieval, generation, and the serving gate share one root request trace?

An alert says only grounding_rate < 95%. What evidence makes it actionable?

An alert says only grounding_rate < 95%. What evidence makes it actionable?

Mastery Check

Discussion

An alert says only `grounding_rate < 95%`. What evidence makes it actionable?