Personalize this lesson
Adapt explanations and teaching visuals to your background and preferred voice.
Imagine an assistant claiming a deployment is approved when the ReleaseOps database has no approval record. An engineer reverts the prompt template in Git, but the serving image still loads a newer compiled prompt artifact. The retrieved evidence has changed too. Which combination is actually answering users?
Reverting one component while others remain new can create an untested combination. Software and classical ML releases also depend on code, data, configuration, and infrastructure. An LLM application makes those dependencies especially visible: model identity, prompt state, tool contracts, retrieval data, policy, and runtime all affect its behavior. A release manifest records the intended combination. Hashing that manifest identifies its bytes; retention, artifact verification, and serving controls must make the declared combination real.
Our fictional deploy-answerer handles ReleaseOps questions. A supplied incident fixture, req_205, represents an unsupported approval claim for payment-tests and SHIP-4412. Recovery should activate the evaluated, retained combination and check that it works under current conditions. An older release isn't automatically safe forever.
The candidate stands in for a compiled DSPy program paired with an updated serving image. This lab doesn't generate that artifact, run DSPy search, query a hosted model, or deploy infrastructure. Its identifiers, support labels, and metric summaries are supplied fixtures; the Python exercises local state transitions. A frozen September 2 clock makes freshness checks reproducible.

A release is more than model weights
deploy-answerer declares a hosted answer model, a releaseops-evidence-classifier-v2 gate, and prompt state. In a real system, admission rules and authorization determine which records may support an answer. A classifier or prompt can make mistakes; the host must check the resulting claims and routes. A partial rollback can leave a combination that was never evaluated.
Call that complete set a release bundle. An immutable manifest identifies the code, artifacts, and configuration intended to serve together. Attach evaluation reports to its identity rather than inserting report contents into the hashed manifest: a report must name the release it evaluated without creating a circular hash dependency. Data, code, configuration, and serving infrastructure otherwise drift along separate paths, leaving reviewers to guess which change caused a behavior shift.[1][2]
| Bundle coordinate | Role in serving stack | Risk if decoupled |
|---|---|---|
| Model artifact or provider snapshot | Identifies the declared model | A floating identifier can resolve to a newer model |
| Prompt state & compile provenance | Identifies instructions, demos, and their build | Reverting a source template may leave a different saved JSON loaded |
| Tool & input/output contracts | Declares compatible interfaces | Schema drift can reject arguments or misinterpret state |
| Retrieval corpus & index | Identifies evidence and retrieval settings | Changed chunks or indexes can alter admitted evidence |
| Admission & safety policy | Declares serving and refusal rules | A changed rule can admit unsupported claims or reject useful answers |
| Runtime & generation config | Identifies code, tokenizer, and effective settings | Runtime changes can affect parsing, latency, and outputs; temperature zero doesn't guarantee determinism |
| Evaluation suite & scorer | Identifies the evaluation contract | A changed scorer makes old thresholds or comparisons inapplicable |
Treat the manifest as a version tuple, not as one model number. Its coordinates cover prompt state, evidence, policy, schema, evaluator, runtime, and decoding. Change one coordinate and the release identity must change, even when weights stay put.
Compatibility sits inside that tuple. A candidate can pass answer-quality checks and still fail when its input, output, or tool schema no longer matches the router. Pin those contracts, test them with the same release ID, and diagnose an interface failure separately from a model-quality regression.
We'll use two bundles. stable uses the hand-edited deploy-answerer-v1 template. candidate swaps in the compiled program and the image that loads it. Other fields stay fixed. This narrows the intended change to two interacting components; it doesn't isolate the prompt's causal effect from the image's effect or eliminate model-output variation.
Names and short digest-like strings below are fixture labels, not downloadable artifacts or valid OCI digests. In a real manifest, pin artifact bytes with full digests or immutable object versions, then verify them before use.[3] Hashing a filename doesn't freeze the file behind it: overwriting v1.json would change behavior without changing this manifest. Also include relevant retrieval/embedding settings, output and tool contracts, and effective runtime options; this compact schema doesn't enumerate every production dependency.
from dataclasses import asdict, dataclass, fields, replace
import hashlib
import json
@dataclass(frozen=True)
class ReleaseBundle:
service: str
answer_model: str
evidence_gate: str
prompt_compile_run: str
tokenizer: str
prompt_version: str
policy_version: str
corpus_version: str
serving_image: str
input_schema: str
eval_suite: str
evaluator_version: str
decoding: str
def __post_init__(self) -> None:
if any(type(getattr(self, field.name)) is not str or not getattr(self, field.name).strip() for field in fields(self)):
raise ValueError("release fields must be nonempty strings")
stable = ReleaseBundle(
service="deploy-answerer",
answer_model="hosted-model@2026-04-01",
evidence_gate="releaseops-evidence-classifier-v2@sha256:41d8",
prompt_compile_run="manual-template-v1",
tokenizer="deploy-tokenizer@sha256:91aa",
prompt_version="deploy-answerer-v1.json",
policy_version="[email protected]",
corpus_version="releaseops-records@sha256:corpus-5",
serving_image="registry.example/deploy-answerer@sha256:image-a",
input_schema="deploy-answer.v2",
eval_suite="deploy-grounding-suite@sha256:suite-7",
evaluator_version="claim-evidence-eval-v2",
decoding="temperature=0",
)
candidate = replace(
stable,
prompt_compile_run="dspy-mipro-v2-run-052",
prompt_version="compiled_a_hosted-2026-04-01.json",
serving_image="registry.example/deploy-answerer@sha256:image-b",
)
print(f"stable_prompt={stable.prompt_version}")
print(f"candidate_prompt={candidate.prompt_version}")
print(f"candidate_compile_run={candidate.prompt_compile_run}")stable_prompt=deploy-answerer-v1.json
candidate_prompt=compiled_a_hosted-2026-04-01.json
candidate_compile_run=dspy-mipro-v2-run-052A human-readable name such as v3 helps people find a release, but it can't prove what went into it. Before running the next cell, predict the result of changing only prompt_version: the release ID should change, because the canonical manifest includes that field.
def release_id(bundle: ReleaseBundle) -> str:
payload = json.dumps(asdict(bundle), sort_keys=True, separators=(",", ":"))
digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()
return f"{bundle.service}@sha256:{digest}"
stable_id = release_id(stable)
candidate_id = release_id(candidate)
prompt_patch_id = release_id(
replace(candidate, prompt_version="compiled_a_hosted-2026-04-01.b.json")
)
print(f"stable={stable_id}")
print(f"candidate={candidate_id}")
print(f"prompt_patch={prompt_patch_id}")
print(f"prompt_patch_is_new_release={prompt_patch_id != candidate_id}")
# A name in a manifest is not a digest of the file behind that name.
from pathlib import Path
from tempfile import TemporaryDirectory
with TemporaryDirectory() as directory:
prompt_file = Path(directory) / stable.prompt_version
prompt_file.write_text('{"instructions":"first fixture"}')
first_bytes_digest = hashlib.sha256(prompt_file.read_bytes()).hexdigest()
prompt_file.write_text('{"instructions":"changed fixture"}')
second_bytes_digest = hashlib.sha256(prompt_file.read_bytes()).hexdigest()
print("same_manifest_id_after_overwrite=", release_id(stable) == stable_id, sep="")
print("prompt_bytes_changed=", first_bytes_digest != second_bytes_digest, sep="")stable=deploy-answerer@sha256:32b8ed409b8e6e47313599b0b26d8481b57c0343bc55f6bcb89ba6b0fc9b8127
candidate=deploy-answerer@sha256:e31f2b6d7a2cec362416c2725a8ea4bce109b6e649cff9caabbdc181dfe03738
prompt_patch=deploy-answerer@sha256:f94cb4d1892f93eb15b91d508281d6e3603800da98428dd88c9a49df4cb4e310
prompt_patch_is_new_release=True
same_manifest_id_after_overwrite=True
prompt_bytes_changed=TrueThe registry uses the full SHA-256 digest; figures abbreviate it for display. The overwrite demonstration shows the boundary: a manifest naming the same file can retain its ID while the prompt bytes change. Pin those bytes too. A hash doesn't establish truthful metadata or trusted build provenance. Equivalent JSON from another language needs the same serialization convention to produce the same digest.
Why should a one-line prompt correction produce a new release ID?
Answer
The prompt can change served behavior even when model weights don't move. Reusing the old release ID would make traces and rollbacks lie about what users saw.
Artifacts stay fixed; aliases move
A registry record stores an immutable bundle. An alias such as production or canary is a mutable pointer used by traffic. Retaining the record lets recovery select an older bundle without rebuilding it. Actual recovery also needs ready serving capacity and routers that observe the new pointer.
MLflow's Model Registry workflow uses aliases and tags to organize model versions, and its documentation marks fixed Model Stages as deprecated.[4] The distinction matters: a model version preserves history, while an alias names the version a loader should resolve now.
An MLflow model alias still points to one registered model version. It doesn't automatically include the prompt, corpus, policy, schema, or serving image in this release bundle. Treat it as one component pointer, or register a wrapper artifact whose manifest resolves the complete bundle. A movable model alias isn't complete release identity.
An alias update changes metadata. Serving processes may cache its old target, retain a loaded artifact, or have requests already in flight. Recovery also needs retained artifacts, compatible state, ready capacity, and verified routing. Keeping the previous evaluated release available can shorten that path; neither an alias nor a registry promises a universal recovery time.

The small registry below enforces that rule. register() keeps a deep copy of each bundle, and move_alias() accepts only a registered ID. Before running it, predict the alias state: registering candidate should increase the record count while production still resolves to stable.
from copy import deepcopy
class ReleaseRegistry:
def __init__(self) -> None:
self._bundles: dict[str, ReleaseBundle] = {}
self._aliases: dict[str, str] = {}
def register(self, bundle: ReleaseBundle) -> str:
bundle_id = release_id(bundle)
existing = self._bundles.get(bundle_id)
if existing is not None and existing != bundle:
raise ValueError("release digest collision")
self._bundles[bundle_id] = deepcopy(bundle)
return bundle_id
def move_alias(self, alias: str, bundle_id: str, *, expected: str | None = None) -> None:
if type(alias) is not str or not alias.strip():
raise ValueError("alias must be a nonempty string")
if bundle_id not in self._bundles:
raise KeyError(f"unregistered release: {bundle_id}")
if self._aliases.get(alias) != expected:
raise ValueError("stale alias update")
self._aliases[alias] = bundle_id
def resolve(self, alias: str) -> str:
return self._aliases[alias]
def bundle(self, bundle_id: str) -> ReleaseBundle:
return deepcopy(self._bundles[bundle_id])
registry = ReleaseRegistry()
assert registry.register(stable) == stable_id
assert registry.register(candidate) == candidate_id
registry.move_alias("production", stable_id)
print(f"registered={len(registry._bundles)}")
print(f"production={registry.resolve('production')}")
print(f"candidate_registered={candidate_id in registry._bundles}")registered=2
production=deploy-answerer@sha256:32b8ed409b8e6e47313599b0b26d8481b57c0343bc55f6bcb89ba6b0fc9b8127
candidate_registered=TrueTry moving production to candidate without supplying expected=stable_id: the registry rejects the stale update instead of overwriting an existing target. This check is sequential Python, not a distributed lock. A production control plane must perform the compare-and-swap write atomically, authenticate the actor, and record an audit event. Tracking a monotonically increasing alias revision alongside its target catches ABA problems where an alias moves away and returns to the same release during a deployment race.
Promotion begins with controlled evidence
Registration makes a bundle addressable, not approved. Continuous delivery for machine learning adds evaluation gates and monitoring to ordinary build-and-deploy practices because a valid artifact can still produce unacceptable behavior.[5]
The gate should speak the same language as the observability, evaluation, and experiment lessons that came before it. Reuse the grounded evidence contract instead of inventing one generic quality score:
supported_evidence_f1scores the fixture's binary route decision against reference labels. With SERVE as the positive class, : TP is a supported request served, FP is an unsupported request served, and FN is a supported request abstained. A real evaluator must define invalid outputs and undefined-denominator behavior too.unsupported_serve_countmust remain zero in the frozen high-risk suite. That's thereq_205failure: serving a deploy-approval claim ReleaseOps didn't support.rollback_slice_f1checks the rollback questions separately. Naming a slice in the suite doesn't enforce its floor; the gate must inspect the result.p95_latency_ms(p95 latency) keeps a behaviorally acceptable release inside its response budget.- Schema, evaluation-suite, and evaluator identifiers check the declared contract. Timestamps, case counts, support-review coverage, and latency coverage check supplied report metadata; they don't prove the checks actually ran.
evaluation_reportpreserves the report artifact for a reviewer or incident responder to inspect later.
The supplied candidate summary has 0.93 F1, 0.91 rollback-slice F1, zero unsupported serves, and 472 ms p95. Our exercise requires 200 evaluated cases with complete support and latency checks, plus a report completed within 24 hours. These values aren't measured model results or universal thresholds. Route F1 isn't a score for generated answer truthfulness; the unsupported-claim counter requires a separate claim-support review in a real system. Predict why a strong report from evaluator v3 must still fail.
from datetime import datetime, timedelta, timezone
import math
NOW = datetime(2026, 9, 2, 12, 0, tzinfo=timezone.utc)
@dataclass(frozen=True)
class OfflineEvidence:
release_id: str
eval_suite: str
evaluator_version: str
input_schema: str
evaluation_report: str
supported_evidence_f1: float
unsupported_serve_count: int
p95_latency_ms: int
rollback_slice_f1: float
evaluated_count: int
support_checked_count: int
latency_sample_count: int
completed_at: datetime
@dataclass(frozen=True)
class Decision:
allowed: bool
reason: str
def finite_between(value: object, low: float, high: float) -> bool:
if type(value) not in (int, float):
return False
try:
return math.isfinite(value) and low <= value <= high
except OverflowError:
return False
def fresh(completed_at: datetime, as_of: datetime, max_age: timedelta) -> bool:
if not isinstance(completed_at, datetime) or not isinstance(as_of, datetime):
return False
if completed_at.utcoffset() is None or as_of.utcoffset() is None:
return False
return timedelta(0) <= as_of - completed_at <= max_age
def offline_gate(
bundle: ReleaseBundle, evidence: OfflineEvidence, as_of: datetime
) -> Decision:
if type(bundle) is not ReleaseBundle or type(evidence) is not OfflineEvidence:
return Decision(False, "invalid release or evidence record")
if evidence.release_id != release_id(bundle):
return Decision(False, "evidence belongs to another release")
if evidence.eval_suite != bundle.eval_suite:
return Decision(False, "evaluation suite changed")
if evidence.evaluator_version != bundle.evaluator_version:
return Decision(False, "evaluator version changed")
if evidence.input_schema != bundle.input_schema:
return Decision(False, "schema mismatch")
if type(evidence.evaluation_report) is not str or not evidence.evaluation_report.strip():
return Decision(False, "evaluation report missing")
if not fresh(evidence.completed_at, as_of, timedelta(hours=24)):
return Decision(False, "report is stale, future-dated, or lacks a timezone")
if type(evidence.evaluated_count) is not int or evidence.evaluated_count != 200:
return Decision(False, "suite requires 200 evaluated cases")
if any(type(count) is not int or count != evidence.evaluated_count for count in (
evidence.support_checked_count, evidence.latency_sample_count
)):
return Decision(False, "support or latency coverage incomplete")
if not all(finite_between(score, 0, 1) for score in (
evidence.supported_evidence_f1, evidence.rollback_slice_f1
)) or not finite_between(evidence.p95_latency_ms, 0, math.inf):
return Decision(False, "invalid metric")
if type(evidence.unsupported_serve_count) is not int or not 0 <= evidence.unsupported_serve_count <= evidence.support_checked_count:
return Decision(False, "unsupported count must be a valid integer")
if evidence.supported_evidence_f1 < 0.92:
return Decision(False, "supported_evidence_f1 below 0.92")
if evidence.rollback_slice_f1 < 0.90:
return Decision(False, "rollback slice below 0.90")
if evidence.unsupported_serve_count != 0:
return Decision(False, "unsupported answer was served")
if evidence.p95_latency_ms > 500:
return Decision(False, "p95 latency exceeds 500 ms")
return Decision(True, "offline gate passed")
candidate_offline = OfflineEvidence(
release_id=candidate_id,
eval_suite=candidate.eval_suite,
evaluator_version=candidate.evaluator_version,
input_schema=candidate.input_schema,
evaluation_report="reports/candidate-suite-7-redacted.json",
supported_evidence_f1=0.93,
unsupported_serve_count=0,
p95_latency_ms=472,
rollback_slice_f1=0.91,
evaluated_count=200,
support_checked_count=200,
latency_sample_count=200,
completed_at=NOW - timedelta(hours=2),
)
weaker_candidate = replace(candidate_offline, supported_evidence_f1=0.89)
changed_evaluator = replace(candidate_offline, evaluator_version="claim-evidence-eval-v3")
print(f"candidate={offline_gate(candidate, candidate_offline, NOW)}")
print(f"weak_metric={offline_gate(candidate, weaker_candidate, NOW)}")
print(f"changed_evaluator={offline_gate(candidate, changed_evaluator, NOW)}")candidate=Decision(allowed=True, reason='offline gate passed')
weak_metric=Decision(allowed=False, reason='supported_evidence_f1 below 0.92')
changed_evaluator=Decision(allowed=False, reason='evaluator version changed')Passing the offline gate permits the next experiment. It doesn't replace production yet. Create a canary alias while production still points to the known-good bundle. Creating that pointer alone sends no requests; traffic routing comes later.
The gate assumes an authenticated evaluator supplied the parsed report. It doesn't fetch the artifact, verify its digest, or prove that 200 distinct cases received valid support reviews and latency measurements. Ingestion must verify the retained report, case membership, review semantics, and actual runtime identity before trusting its summary. Latency coverage here includes terminal failures and timeouts, not only successful responses. The 24-hour rule bounds reliance on a report while the operating environment may change; it doesn't invalidate mathematics overnight.
def open_canary(
bundle: ReleaseBundle, evidence: OfflineEvidence, as_of: datetime
) -> Decision:
decision = offline_gate(bundle, evidence, as_of)
if decision.allowed:
registry.move_alias("canary", release_id(bundle))
return decision
canary_decision = open_canary(candidate, candidate_offline, NOW)
print(f"canary_opened={canary_decision.allowed}")
print(f"canary={registry.resolve('canary')}")
print(f"production_still_stable={registry.resolve('production') == stable_id}")canary_opened=True
canary=deploy-answerer@sha256:e31f2b6d7a2cec362416c2725a8ea4bce109b6e649cff9caabbdc181dfe03738
production_still_stable=TrueManaged models need a documented pin
For self-hosted weights, verifying a content hash identifies checkpoint bytes; inference code, configuration, and hardware can still affect outputs. For a managed API, use the strongest version identifier the provider documents and capture the effective request settings.[6] The fixture hosted-model@2026-04-01 is an invented stand-in, not an available model.
A floating alias can change without a deployment in your application. A dated identifier reduces that source of drift according to the provider's contract; its spelling isn't proof of immutable service behavior, deterministic output, or permanent availability. OpenAI explicitly recommends pinned versions plus evaluations, and publishes snapshot retirements.[6][7] Monitor regression cases and retirement notices, and evaluate identifier/configuration changes as new candidates. A retired or unavailable stable endpoint cannot be your usable rollback target.
Replay a failed production trace against the candidate
Frozen fixtures test failures you thought to write down. Trace replay starts with a recorded failure: reuse its request and recorded tool responses against another release. Holding those inputs fixed removes two sources of variation. It doesn't guarantee deterministic model output, even with temperature zero.
The regression question is narrow: does the candidate repeat a recorded failure when relevant inputs are held fixed? Replay can complement offline and shadow evaluation; it isn't required to occupy one universal funnel stage. Substitute recorded tool results rather than executing production writes. Replay still consumes inference compute when a real model runs.
Apply the question to fixture req_205. Its encoded deploy-answerer-v1.1-regression branch serves an unsupported approval; the encoded stable and candidate branches honor the supplied support label. Compare those branches while checking the declared model, corpus, and evidence-gate names.
The next cell is a deterministic policy-branch simulation. A supplied boolean represents whether the claim is supported; no recorded request body, evidence bytes, tool results, or generated answer are replayed. The seed and temperature are returned as metadata, not applied to inference. The checks compare declared corpus, model, and evidence-gate names, then distinguish an encoded regressed branch from an encoded corrected branch.
@dataclass(frozen=True)
class RecordedTrace:
request_id: str
origin_release: str
prompt_version: str
evidence_version: str
evidence_gate: str
tool_sequence: tuple[str, ...]
evidence_supports_claim: bool
served_unsupported_claim: bool
temperature: float = 0.0
seed: int = 0
model_snapshot: str = ""
def prompt_would_serve(bundle: ReleaseBundle, evidence_supports_claim: bool) -> bool:
if type(evidence_supports_claim) is not bool:
raise ValueError("fixture support label must be a boolean")
# Historical v1.1 bypassed the gate; known-good v1 and the candidate don't.
if bundle.prompt_version == "deploy-answerer-v1.1-regression.json":
return True
if bundle.prompt_version in {stable.prompt_version, candidate.prompt_version}:
return evidence_supports_claim
raise ValueError("no simulated policy for this prompt")
def replay_against(trace: RecordedTrace, bundle: ReleaseBundle) -> dict[str, object]:
if trace.evidence_version != bundle.corpus_version:
raise ValueError("replay evidence snapshot changed")
if trace.model_snapshot != bundle.answer_model:
raise ValueError("replay model pin changed")
if trace.evidence_gate != bundle.evidence_gate:
raise ValueError("replay evidence gate changed")
would_serve = prompt_would_serve(bundle, trace.evidence_supports_claim)
reproduces = would_serve and not trace.evidence_supports_claim
return {
"candidate_release": release_id(bundle),
"prompt_version": bundle.prompt_version,
"tool_sequence": trace.tool_sequence,
"pins": (trace.temperature, trace.seed, trace.model_snapshot),
"candidate_reproduces_failure": reproduces,
}
historical_incident = replace(
stable,
prompt_compile_run="regressed-template-v1.1",
prompt_version="deploy-answerer-v1.1-regression.json",
serving_image="registry.example/deploy-answerer@sha256:image-regressed",
)
failed_trace = RecordedTrace(
request_id="req_205",
origin_release=release_id(historical_incident),
prompt_version=historical_incident.prompt_version,
evidence_version="releaseops-records@sha256:corpus-5",
evidence_gate=stable.evidence_gate,
tool_sequence=("lookup_release", "fetch_record", "answer"),
evidence_supports_claim=False,
served_unsupported_claim=True,
temperature=0.0,
seed=7,
model_snapshot=stable.answer_model,
)
historical_replay = replay_against(failed_trace, historical_incident)
stable_replay = replay_against(failed_trace, stable)
candidate_replay = replay_against(failed_trace, candidate)
print(f"origin_release={failed_trace.origin_release}")
print(f"historical_reproduces_failure={historical_replay['candidate_reproduces_failure']}")
print(f"stable_reproduces_failure={stable_replay['candidate_reproduces_failure']}")
print(f"replayed_against={candidate_replay['candidate_release']}")
print(f"candidate_prompt={candidate_replay['prompt_version']}")
print(f"recorded_step_names={candidate_replay['tool_sequence']}")
print(f"pins={candidate_replay['pins']}")
print(f"candidate_reproduces_failure={candidate_replay['candidate_reproduces_failure']}")
assert historical_replay["candidate_reproduces_failure"] is True
assert stable_replay["candidate_reproduces_failure"] is False
assert candidate_replay["candidate_reproduces_failure"] is Falseorigin_release=deploy-answerer@sha256:efe48fbcb5a2fb27baf7d386061cbf9383afd3f9dfade71e4ba0a27b3140c966
historical_reproduces_failure=True
stable_reproduces_failure=False
replayed_against=deploy-answerer@sha256:e31f2b6d7a2cec362416c2725a8ea4bce109b6e649cff9caabbdc181dfe03738
candidate_prompt=compiled_a_hosted-2026-04-01.json
recorded_step_names=('lookup_release', 'fetch_record', 'answer')
pins=(0.0, 7, 'hosted-model@2026-04-01')
candidate_reproduces_failure=FalseThe output verifies our simulated policy branch. It says nothing about the real compiled program's quality. A production replay harness must load that program, substitute recorded tool inputs and outputs, check unexpected tool calls, and score the generated answer. A candidate can choose a different tool sequence; don't silently supply an unrelated recorded result just to finish the replay.
Retain the request, evidence snapshot, actual runtime identity, and supported sampling controls. OpenAI's compatibility guidance explicitly describes model outputs as variable and recommends pinned versions plus evaluations.[6] Run repeated comparisons when outputs vary, and judge the distribution of failures rather than demanding byte-for-byte reproduction. Keep past incidents in the regression suite, while recognizing that passing known cases can't rule out new failures.
Shadow evaluation must be read-only
Replay gives us one known failure. Offline fixtures still can't cover every real request shape. A shadow sends a sanitized copy of a production request to the candidate while the stable release alone supplies the user-visible answer. It reveals latency or evidence-support problems without exposing candidate text to customers.
The safety boundary is strict: shadow traffic can't send messages or perform production writes. Use recorded tool results or approved read-only operations; its output is evaluation data. The envelope below redacts a ship identifier, but doesn't implement a real queue or authorization boundary.
Before running it, inspect the flags that must stay false: side effects must be disabled and candidate output must stay hidden. Here the recorded stable response abstained, and the simulated candidate should abstain too. Agreement is one comparison, not a correctness score: two releases can agree on a bad answer.
import re
@dataclass(frozen=True)
class ShadowEnvelope:
candidate_release: str
stable_release: str
sanitized_text: str
side_effects_enabled: bool
response_visible_to_user: bool
evidence_supports_claim: bool
observed_stable_route: str
@dataclass(frozen=True)
class ShadowComparison:
route_agreement: bool
candidate_route_supported: bool
candidate_would_serve: bool
stable_route: str
candidate_route: str
dropped: bool
def make_shadow(
text: str, evidence_supports_claim: bool, observed_stable_route: str
) -> ShadowEnvelope:
if type(text) is not str or type(evidence_supports_claim) is not bool:
raise ValueError("need text and a boolean fixture support label")
if observed_stable_route not in {"serve", "abstain"}:
raise ValueError("unknown stable route")
sanitized = re.sub(r"SHIP-\d+", "[SHIP_ID]", text)
return ShadowEnvelope(
candidate_release=registry.resolve("canary"),
stable_release=registry.resolve("production"),
sanitized_text=sanitized,
side_effects_enabled=False,
response_visible_to_user=False,
evidence_supports_claim=evidence_supports_claim,
observed_stable_route=observed_stable_route,
)
def compare_shadow(envelope: ShadowEnvelope, queue_full: bool = False) -> ShadowComparison:
if any(type(value) is not bool for value in (
queue_full, envelope.side_effects_enabled, envelope.response_visible_to_user,
envelope.evidence_supports_claim,
)) or envelope.observed_stable_route not in {"serve", "abstain"}:
raise ValueError("invalid shadow metadata or route")
if queue_full or envelope.side_effects_enabled or envelope.response_visible_to_user:
return ShadowComparison(False, False, False, "dropped", "dropped", True)
# Stable already answered the user; shadow only scores the candidate copy.
stable_route = envelope.observed_stable_route
candidate_bundle = registry.bundle(envelope.candidate_release)
candidate_would_serve = prompt_would_serve(
candidate_bundle, envelope.evidence_supports_claim
)
candidate_route = "serve" if candidate_would_serve else "abstain"
route_agreement = candidate_route == stable_route
candidate_route_supported = candidate_route == "abstain" or envelope.evidence_supports_claim
return ShadowComparison(
route_agreement=route_agreement,
candidate_route_supported=candidate_route_supported,
candidate_would_serve=candidate_would_serve,
stable_route=stable_route,
candidate_route=candidate_route,
dropped=False,
)
shadow = make_shadow(
"Is payment-tests approved for SHIP-4412 without a ReleaseOps record?",
evidence_supports_claim=False,
observed_stable_route="abstain",
)
comparison = compare_shadow(shadow)
dropped = compare_shadow(shadow, queue_full=True)
print(f"shadow_text={shadow.sanitized_text}")
print(f"candidate={shadow.candidate_release}")
print(f"side_effects_enabled={shadow.side_effects_enabled}")
print(f"response_visible={shadow.response_visible_to_user}")
print(f"candidate_route={comparison.candidate_route}")
print(f"route_agreement={comparison.route_agreement}")
print(f"candidate_route_supported={comparison.candidate_route_supported}")
print(f"dropped_when_queue_full={dropped.dropped}")
assert comparison.candidate_would_serve is False
assert comparison.route_agreement is True
assert comparison.candidate_route_supported is True
assert dropped.dropped is Trueshadow_text=Is payment-tests approved for [SHIP_ID] without a ReleaseOps record?
candidate=deploy-answerer@sha256:e31f2b6d7a2cec362416c2725a8ea4bce109b6e649cff9caabbdc181dfe03738
side_effects_enabled=False
response_visible=False
candidate_route=abstain
route_agreement=True
candidate_route_supported=True
dropped_when_queue_full=TrueThe shadow comparison uses the same boolean policy simulation, not sanitized text or a generated answer. candidate_route_supported means the encoded route doesn't serve against a false fixture support label; it doesn't establish answer truthfulness or overall safety. The one-pattern redaction is also a fixture. A real path needs schema-aware data minimization for identifiers and secrets before queues, logs, or candidate services. Redaction can change retrieval or routing; compare releases on equivalent sanitized inputs or account for that difference. Isolate shadow capacity and bound its cost.
At the service boundary, put the envelope on a bounded worker queue and count dropped or failed comparisons. The compare step is the other half of shadow: after the candidate answers the sanitized copy, record route agreement, claim safety, and latency against the stable path. Feed shadow_drop_rate from dropped or timed-out comparisons. Don't start an untracked background task in request scope and assume its evaluation record will survive a process restart.
side_effects_enabled=False in this teaching envelope is metadata, not an authorization boundary. The worker still needs a read-only tool allowlist and credentials that can't write production state. Reject a shadow request that asks for a side effect.
Why must a shadow evaluation of an agent candidate be read-only?
Answer
Shadow traffic duplicates production inputs. If the candidate can send messages, modify records, or call irreversible tools, evaluation itself creates duplicate side effects instead of passive evidence.
Canary traffic is visible and sticky
Shadow results can justify limited exposure, not automatic promotion. A canary returns candidate responses to a limited share of users. For this multi-turn product, our rollout policy pins each conversation to one release during normal widening. Switching releases can alter behavior or make stored state incompatible; products with independent requests may use a different assignment unit.
Use deterministic hashing of a conversation ID to choose new conversations consistently across workers. Python's built-in hash() is process-dependent, so it can't provide that bucket. Hashing alone isn't enough: when exposure widens from 1% to 10%, a higher threshold could move an existing conversation from stable to candidate. Persist the first resolved release ID for the conversation lifetime.
Predict the widening case before running the router. A thread assigned to stable at 10% should still resolve to stable after the threshold becomes 100%; an abort must be able to override that assignment.
def bucket(conversation_id: str) -> int:
if type(conversation_id) is not str or not conversation_id.strip():
raise ValueError("conversation ID must be a nonempty string")
digest = hashlib.sha256(conversation_id.encode("utf-8")).hexdigest()
return int(digest[:8], 16) % 100
conversation_assignments: dict[str, str] = {}
aborted_releases: set[str] = set()
def assigned_release(conversation_id: str, canary_percent: int) -> str:
if type(conversation_id) is not str or not conversation_id.strip():
raise ValueError("conversation ID is required")
if type(canary_percent) is not int or not 0 <= canary_percent <= 100:
raise ValueError("canary percentage must be an integer from 0 to 100")
existing = conversation_assignments.get(conversation_id)
if existing is not None and existing not in aborted_releases:
return existing
alias = "canary" if bucket(conversation_id) < canary_percent else "production"
bundle_id = registry.resolve(alias)
if bundle_id in aborted_releases:
bundle_id = registry.resolve("production")
if bundle_id in aborted_releases:
raise RuntimeError("no healthy release is available")
conversation_assignments[conversation_id] = bundle_id
return bundle_id
canary_thread = next(
f"thread-{index}" for index in range(1000) if bucket(f"thread-{index}") < 10
)
assignments = [assigned_release(canary_thread, canary_percent=10) for _ in range(3)]
stable_thread = next(
f"thread-{index}" for index in range(1000) if bucket(f"thread-{index}") >= 10
)
stable_before_widen = assigned_release(stable_thread, canary_percent=10)
stable_after_widen = assigned_release(stable_thread, canary_percent=100)
print(f"canary_thread={canary_thread}")
print(f"bucket={bucket(canary_thread)}")
print(f"same_release_each_turn={len(set(assignments)) == 1}")
print(f"assigned_to_candidate={assignments[0] == candidate_id}")
print(f"existing_stable_thread_pinned_after_widen={stable_before_widen == stable_after_widen == stable_id}")canary_thread=thread-6
bucket=6
same_release_each_turn=True
assigned_to_candidate=True
existing_stable_thread_pinned_after_widen=TrueThe in-memory dictionary is a local fixture. For this policy, a production router needs shared persisted assignments, atomic first assignment, a defined expiry, and resolved release IDs in traces. Hashing chooses a bucket; persistence preserves the release when thresholds or aliases change. Incompatible conversation/tool-state schemas can make a release switch fail, so test compatibility rather than assuming any switch necessarily crashes.
A 10% threshold assigns roughly 10% of newly initiated conversations, not 10% of raw requests. Active conversations dominate request volume until they conclude. When exposure widens from 10% to 25% or 100%, existing stable conversations finish their lifecycles on stable.
An abort must override failed-release assignments. Applying that change across routers takes propagation and cache handling; it isn't instantaneous. If state is incompatible with fallback, pause, reset, or migrate it under a reviewed policy. Don't claim recovery until traces show the restored release handling requests.

A canary window answers a new question
Offline evidence covers frozen examples. A live window asks how candidate behaves under the current traffic mix. Bind it to both the release and rollout attempt; yesterday's attempt isn't evidence for today's exposure. Our exercise requires an interval ending within five minutes and at least 1,000 requests with complete support, latency, and transport checks. It also requires 100 answerable requests, including 50 answerable rollback requests, and at least 90% useful answers in each of those groups. These are declared teaching policies, not statistical guarantees or universal thresholds.
An answerable request has admitted records that support a useful answer under the reviewed task rubric. The useful-serve counts require acceptable delivered answers, not just SERVE routes. Errors, empty responses, and unnecessary abstentions don't count as useful. A rollout that refuses everything may avoid unsupported claims while failing the product's purpose.
The controller needs three outcomes: PASS permits advancing to the next ramp tier, HOLD pauses to collect more trustworthy evidence, and ABORT trips the circuit breaker to remove a defective candidate. Missing telemetry isn't a passing zero. Conversely, one confirmed unsupported serve triggers an immediate abort, even before reaching the sample minimum. In a deployed system, missing telemetry also requires a strict escalation timeout so a canary can't linger in production without active monitoring.
Read the two windows in the figure as an operational verdict. The 1% sample clears this contract; the 10% sample fails even though its latency, error rate, and usefulness floors pass. A positive unsupported counter overrides those passing metrics. Zero unsupported serves alone still can't establish acceptable usefulness.
@dataclass(frozen=True)
class LiveWindow:
name: str
release_id: str
rollout_id: str
started_at: datetime
ended_at: datetime
request_count: int
support_checked_count: int
unsupported_serve_count: int
latency_sample_count: int
p95_latency_ms: int
transport_checked_count: int
error_count: int
shadow_drop_rate: float
supported_request_count: int
useful_supported_serve_count: int
rollback_supported_request_count: int
rollback_useful_serve_count: int
@property
def error_rate(self) -> float:
return self.error_count / self.request_count
@dataclass(frozen=True)
class LiveDecision:
status: str
reason: str
def live_gate(
window: LiveWindow, expected_release: str, expected_rollout: str, as_of: datetime
) -> LiveDecision:
if type(window) is not LiveWindow:
return LiveDecision("HOLD", "invalid window record")
if (window.release_id, window.rollout_id) != (expected_release, expected_rollout):
return LiveDecision("HOLD", "window belongs to another release or rollout")
counts = (window.request_count, window.support_checked_count, window.unsupported_serve_count)
if any(type(n) is not int or n < 0 for n in counts):
return LiveDecision("HOLD", "invalid count")
if not window.unsupported_serve_count <= window.support_checked_count <= window.request_count:
return LiveDecision("HOLD", "inconsistent counts")
if window.unsupported_serve_count > 0:
return LiveDecision("ABORT", "unsupported serve count must remain zero")
if (
not fresh(window.ended_at, as_of, timedelta(minutes=5))
or not isinstance(window.started_at, datetime)
or window.started_at.utcoffset() is None
or window.started_at >= window.ended_at
):
return LiveDecision("HOLD", "invalid or stale observation interval")
if any(type(count) is not int or count < 0 for count in (
window.latency_sample_count, window.transport_checked_count, window.error_count
)):
return LiveDecision("HOLD", "invalid measurement count")
if not (window.error_count <= window.transport_checked_count <= window.request_count
and window.latency_sample_count <= window.request_count):
return LiveDecision("HOLD", "inconsistent measurement counts")
if window.request_count == 0:
return LiveDecision("HOLD", "fewer than 1000 requests observed")
if not (
finite_between(window.p95_latency_ms, 0, math.inf)
and finite_between(window.shadow_drop_rate, 0, 1)
):
return LiveDecision("HOLD", "invalid metric")
if window.p95_latency_ms > 550:
return LiveDecision("ABORT", "live latency exceeded 550 ms")
if window.error_rate > 0.01:
return LiveDecision("ABORT", "error rate exceeded 1%")
if window.shadow_drop_rate > 0.02:
return LiveDecision("HOLD", "shadow telemetry incomplete")
if window.support_checked_count != window.request_count:
return LiveDecision("HOLD", "support evaluation incomplete")
if window.latency_sample_count != window.request_count or window.transport_checked_count != window.request_count:
return LiveDecision("HOLD", "latency or transport coverage incomplete")
if window.request_count < 1_000:
return LiveDecision("HOLD", "fewer than 1000 requests observed")
usefulness_counts = (
window.supported_request_count, window.useful_supported_serve_count,
window.rollback_supported_request_count, window.rollback_useful_serve_count,
)
if any(type(count) is not int or count < 0 for count in usefulness_counts):
return LiveDecision("HOLD", "invalid usefulness count")
if not (
window.useful_supported_serve_count <= window.supported_request_count <= window.request_count
and window.rollback_useful_serve_count <= window.rollback_supported_request_count <= window.supported_request_count
and window.rollback_useful_serve_count <= window.useful_supported_serve_count
and window.useful_supported_serve_count - window.rollback_useful_serve_count <= window.supported_request_count - window.rollback_supported_request_count
):
return LiveDecision("HOLD", "inconsistent usefulness counts")
if window.supported_request_count < 100 or window.rollback_supported_request_count < 50:
return LiveDecision("HOLD", "too few answerable or rollback cases")
if (10 * window.useful_supported_serve_count < 9 * window.supported_request_count
or 10 * window.rollback_useful_serve_count < 9 * window.rollback_supported_request_count):
return LiveDecision("ABORT", "useful-answer floor below 90%")
return LiveDecision("PASS", "live window passed")
rollout_id = "rollout-052"
window_1_percent = LiveWindow(
"1%", candidate_id, rollout_id, NOW - timedelta(minutes=15), NOW,
request_count=1_200, support_checked_count=1_200, unsupported_serve_count=0,
latency_sample_count=1_200, p95_latency_ms=481,
transport_checked_count=1_200, error_count=2, shadow_drop_rate=0.001,
supported_request_count=800, useful_supported_serve_count=752,
rollback_supported_request_count=100, rollback_useful_serve_count=92,
)
window_10_percent = replace(
window_1_percent, name="10%", request_count=8_000, support_checked_count=8_000,
unsupported_serve_count=112, latency_sample_count=8_000, p95_latency_ms=493,
transport_checked_count=8_000, error_count=24,
supported_request_count=6_000, useful_supported_serve_count=5_640,
rollback_supported_request_count=500, rollback_useful_serve_count=465,
)
print(f"one_percent={live_gate(window_1_percent, candidate_id, rollout_id, NOW)}")
print(f"ten_percent={live_gate(window_10_percent, candidate_id, rollout_id, NOW)}")
print(f"ten_percent_unsupported_count={window_10_percent.unsupported_serve_count}")one_percent=LiveDecision(status='PASS', reason='live window passed')
ten_percent=LiveDecision(status='ABORT', reason='unsupported serve count must remain zero')
ten_percent_unsupported_count=112The two windows are alternative supplied summaries, not sequential measurements collected here. Transport errors are separate from unsupported claims: the first window has two transport errors, not zero, and remains under its 1% limit. Shadow drop rate summarizes a different comparison queue. Ingestion must verify distinct request membership, actual support reviews, latency coverage including failures/timeouts, and the transport denominator; counts alone don't establish those facts.
A parsed integer can be too large for math.isfinite to convert to a float. The guard rejects that value rather than crashing the controller. Try malformed latency in both evidence paths: the offline gate rejects the report, while the live gate holds exposure pending trustworthy telemetry.
bad_latencies = (None, True, "472", float("nan"), float("inf"), 10 ** 1000, -1)
offline_rejected = 0
live_held = 0
for latency in bad_latencies:
offline = offline_gate(candidate, replace(candidate_offline, p95_latency_ms=latency), NOW)
live = live_gate(replace(window_1_percent, p95_latency_ms=latency), candidate_id, rollout_id, NOW)
assert offline == Decision(False, "invalid metric")
assert live == LiveDecision("HOLD", "invalid metric")
offline_rejected += 1
live_held += 1
print(f"malformed_metrics={len(bad_latencies)} offline_rejected={offline_rejected} live_held={live_held}")malformed_metrics=7 offline_rejected=7 live_held=7Try 600 requests with 600 reviews, latency samples, and transport checks, keeping usefulness counts within those totals: HOLD. Then add one attributed unsupported serve: ABORT, even below the sample minimum or with stale/missing measurement telemetry. Invalid attribution or malformed support counters instead require investigation. For zero known failures, missing support or measurement coverage must HOLD rather than become an assumed passing zero.
Zero observed failures isn't proof of zero risk. At a predeclared sample size of 1,200 independent requests with a fixed failure probability and correct failure classification, zero failures gives a one-sided 95% upper bound of 1 - 0.05 ** (1 / 1200) ≈ 0.00249, or 0.249%. Choosing when to stop based on observed results, correlated conversations, or imperfect evaluators requires a different analysis. The minimum volumes and usefulness floors are operational policy, not confidence guarantees. Retain per-case results and assess relevant production slices before promotion.
Use the failed signal to choose the next inspection. A request that can't parse points toward schema or serving-image compatibility. A parsed request with an unsupported answer belongs in trace replay against the pinned evidence snapshot. A safe but slow candidate points toward runtime capacity or decoding. Missing telemetry prevents widening because the gate lacks evidence, not because it has measured a quality regression.
Abort a canary; roll back a promotion
The right recovery verb depends on which alias moved:
- If candidate fails while only
canaryreceives traffic, abort the rollout.productionnever moved. - If candidate was promoted and fails later, roll back by repointing
productionto the retained stable release.
Check the alias state before naming the incident. That distinction keeps a report from calling an unpromoted canary failure a production rollback.
In the next drill, canary failure should set exposure to zero and restore the canary pointer. The separate promoted-release drill should move production to candidate, detect failure, then return it to stable.
canary_percent = 10
failed_window = live_gate(window_10_percent, candidate_id, rollout_id, NOW)
if failed_window.status == "ABORT":
aborted_releases.add(candidate_id)
canary_percent = 0
# Zero traffic isn't enough: any consumer of registry.resolve("canary") must leave the failed ID.
registry.move_alias("canary", stable_id, expected=candidate_id)
print(f"canary_percent_after_abort={canary_percent}")
print(f"canary_alias_after_abort={registry.resolve('canary')}")
print(f"production_after_abort={registry.resolve('production')}")
print(f"pinned_canary_thread_restored_stable={assigned_release(canary_thread, canary_percent) == stable_id}")
assert registry.resolve("canary") == stable_id
assert registry.resolve("production") == stable_id
# Independent hypothetical: production had moved before the bad window arrived.
# This isn't authorization to retry the candidate rejected above.
rollback_drill = deepcopy(registry)
previous_production = rollback_drill.resolve("production")
rollback_drill.move_alias("production", candidate_id, expected=previous_production)
post_promotion_incident = replace(window_10_percent, name="100%")
if live_gate(post_promotion_incident, candidate_id, rollout_id, NOW).status == "ABORT":
rollback_drill.move_alias("production", previous_production, expected=candidate_id)
print(f"drill_production_after_rollback={rollback_drill.resolve('production')}")
print(f"drill_restored_stable={rollback_drill.resolve('production') == stable_id}")
print(f"actual_production_unchanged={registry.resolve('production') == stable_id}")canary_percent_after_abort=0
canary_alias_after_abort=deploy-answerer@sha256:32b8ed409b8e6e47313599b0b26d8481b57c0343bc55f6bcb89ba6b0fc9b8127
production_after_abort=deploy-answerer@sha256:32b8ed409b8e6e47313599b0b26d8481b57c0343bc55f6bcb89ba6b0fc9b8127
pinned_canary_thread_restored_stable=True
drill_production_after_rollback=deploy-answerer@sha256:32b8ed409b8e6e47313599b0b26d8481b57c0343bc55f6bcb89ba6b0fc9b8127
drill_restored_stable=True
actual_production_unchanged=TrueArgo Rollouts can use analysis results to advance, pause, or abort a rollout; failed background analysis can return canary weight to zero.[8] It doesn't supply our support evaluator or conversation store. Basic canary without traffic management approximates weights through replica counts; traffic-router integration provides finer control.[9] Verify the actual routing unit and abort behavior of your chosen configuration.
Recovery needs sufficient ready capacity, not just a retained ID. This fixture declares a hosted model, so our application isn't loading its weights into our GPUs. Recovery still depends on endpoint availability, quota, ready application workers, and compatible retrieval/tool services. For self-hosted models, cold startup may also involve weight loading, compilation, cache allocation, and GPU scheduling. Measure those stages in your deployment; there's no universal container or LLM startup time.
Keeping stable capacity ready can meet a short recovery objective, but the capacity plan must handle returning traffic. Stable handling 90% of demand may lack headroom for 100% after a 10% canary abort. If both prompt releases share an unchanged model backend, they needn't duplicate all model weights. Choose retention, standby, preloading, and scale-up plans against a tested recovery-time objective, costs, state compatibility, and provider availability. Completion of a soak period alone doesn't guarantee rollback readiness.
Neither recovery undoes completed side effects, database migrations, messages, or answers users already saw. Retain backward-compatible schemas, plan compensating actions where possible, and decide how to handle in-flight requests. Verify the restored release ID in serving traces: an alias update can succeed while router caches, unhealthy replicas, or incompatible state still prevent recovery.
An alias is restored, but the previous release can't handle the returning traffic. Has recovery finished?
Answer
No. Check the served release ID, compatible state, endpoint availability, and capacity under the returning load. Ready standby can reduce delay, but its value depends on a measured recovery objective. A hosted-model application doesn't control the provider's GPU weight loading.
Record decisions so a later engineer can reconstruct them
A release process needs durable records of release IDs, evidence digests, gate verdicts, window metrics, actors, timestamps, and alias revisions. Protect the audit store and retain its referenced artifacts; a list labeled append-only isn't an integrity control by itself.
The intended trail links each user-facing inference trace to its upstream provenance:
Trace ID -> Release Digest -> Composite Manifest -> Evaluation Suite -> Controller Audit Log.
These links support reconstruction when the records are trustworthy. Trace metadata can be wrong, sampled, dropped, or edited; a digest identifies recorded bytes, not proof that a claimed runtime executed. Verify actual serving identity and collection coverage.
The hand-written list below tells a hypothetical pass-then-abort story using our supplied values. It isn't a chronological measurement record or an audit log emitted by the transitions. The failed release remains available in the local registry while production stays stable. In a real controller, emit events when transitions succeed, with actor, timestamp, alias revision, expected previous target, and evidence digest. Reconstructing a list afterward doesn't prove those transitions occurred.
@dataclass(frozen=True)
class ReleaseEvent:
stage: str
release_id: str
decision: str
evidence: str
events = (
ReleaseEvent("register", candidate_id, "RECORDED", "manifest_sha"),
ReleaseEvent("offline_gate", candidate_id, "PASSED", candidate_offline.evaluation_report),
ReleaseEvent("canary_1_percent", candidate_id, "PASSED", "live:window-001"),
ReleaseEvent("canary_10_percent", candidate_id, "ABORTED", "live:window-010"),
ReleaseEvent("production", stable_id, "UNCHANGED", "rollback:not-needed"),
)
for event in events:
print(f"{event.stage}:{event.decision}:{event.evidence}")
print("release_decision=REJECT_CANDIDATE_AFTER_CANARY_REGRESSION")
print(f"active_production={registry.resolve('production')}")register:RECORDED:manifest_sha
offline_gate:PASSED:reports/candidate-suite-7-redacted.json
canary_1_percent:PASSED:live:window-001
canary_10_percent:ABORTED:live:window-010
production:UNCHANGED:rollback:not-needed
release_decision=REJECT_CANDIDATE_AFTER_CANARY_REGRESSION
active_production=deploy-answerer@sha256:32b8ed409b8e6e47313599b0b26d8481b57c0343bc55f6bcb89ba6b0fc9b8127Where the lab maps onto a live stack
The lab uses clean in-memory Python so state transitions and invariants remain completely transparent. A production LLMOps architecture partitions these responsibilities across specialized control-plane components:
| Responsibility | Production tooling | Concrete mechanism |
|---|---|---|
| Retained artifacts and manifest | Artifact store, versioned repository, or OCI registry | Store verified byte digests and enforce retention; a filename or movable branch isn't a byte pin |
| Environment pointers | Owned release registry; MLflow alias for a component | Resolve versions into the release manifest; implement required atomic revision checks and authorization in the control plane |
| Offline evidence | CI plus a reviewed task evaluator | Retain case-level results, scorer identity, coverage, and runtime evidence; generic CI doesn't know whether a claim is supported |
| Exposure and analysis | Argo Rollouts plus a compatible traffic router | Configure steps and AnalysisTemplates; application routing separately persists conversation assignments |
| Ready serving capacity | Application workers and a managed or self-hosted model backend | Test returning load, dependency readiness, state compatibility, and retirement constraints |
| Observation and audit | Tracing/metrics collectors plus retained evidence and audit stores | Record resolved identities and measured outcomes; neither spans nor exporters are automatically complete or immutable |
A feature flag can select a registered release. Resolve and record its target when serving rather than relying on the flag's later value. A floating model string alone leaves reconstruction incomplete; provider request metadata can help, but doesn't replace the documented version contract.
Before adapting this architecture, try four boundary cases: a report older than 24 hours, NaN latency with no known unsupported serve, an under-sampled window containing one unsupported serve, and an alias update with the wrong previous target. Expect rejection, HOLD, ABORT, and a stale-update exception, respectively. The local alias check is sequential; production needs an atomic revision check. None should silently promote a candidate.
Record the resolved release at each request and transition, retain its evidence, and drill recovery under returning load. Identity supports diagnosis; trustworthy evaluation, compatible state, ready capacity, and verified routing determine whether recovery works.