Skip to content
Skip to lesson
Back to Topics
LearnApplied LLM EngineeringModel Versioning & Deployment
⚙️MediumMLOps & Deployment

Model Versioning & Deployment

Turn an evaluated LLM change into an immutable release bundle, promote it through measured traffic, and roll back without losing lineage.

32 min read
Learning path
Step 81 of 197 in the full curriculum

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

Imagine an assistant claiming a deployment is approved when the ReleaseOps database has no approval record. An engineer reverts the prompt template in Git, but the serving image still loads a newer compiled prompt artifact. The retrieved evidence has changed too. Which combination is actually answering users?

Reverting one component while others remain new can create an untested combination. Software and classical ML releases also depend on code, data, configuration, and infrastructure. An LLM application makes those dependencies especially visible: model identity, prompt state, tool contracts, retrieval data, policy, and runtime all affect its behavior. A release manifest records the intended combination. Hashing that manifest identifies its bytes; retention, artifact verification, and serving controls must make the declared combination real.

Our fictional deploy-answerer handles ReleaseOps questions. A supplied incident fixture, req_205, represents an unsupported approval claim for payment-tests and SHIP-4412. Recovery should activate the evaluated, retained combination and check that it works under current conditions. An older release isn't automatically safe forever.

The candidate stands in for a compiled DSPy program paired with an updated serving image. This lab doesn't generate that artifact, run DSPy search, query a hosted model, or deploy infrastructure. Its identifiers, support labels, and metric summaries are supplied fixtures; the Python exercises local state transitions. A frozen September 2 clock makes freshness checks reproducible.

Three of thirteen manifest fields change: prompt version, compile-run provenance, and serving image. The full JSON hashes to stable prefix 32b8ed409b8e and candidate prefix e31f2b6d7a2c. Model and corpus names match; artifact bytes still need verification.
Two serving components change, together with compile-run provenance. The digest identifies all 13 declared fields. It doesn't verify bytes behind their fixture names.

A release is more than model weights

deploy-answerer declares a hosted answer model, a releaseops-evidence-classifier-v2 gate, and prompt state. In a real system, admission rules and authorization determine which records may support an answer. A classifier or prompt can make mistakes; the host must check the resulting claims and routes. A partial can leave a combination that was never evaluated.

Call that complete set a release bundle. An immutable manifest identifies the code, artifacts, and configuration intended to serve together. Attach evaluation reports to its identity rather than inserting report contents into the hashed manifest: a report must name the release it evaluated without creating a circular hash dependency. Data, code, configuration, and serving infrastructure otherwise drift along separate paths, leaving reviewers to guess which change caused a behavior shift.[1][2]

Bundle coordinateRole in serving stackRisk if decoupled
Model artifact or provider snapshotIdentifies the declared modelA floating identifier can resolve to a newer model
Prompt state & compile provenanceIdentifies instructions, demos, and their buildReverting a source template may leave a different saved JSON loaded
Tool & input/output contractsDeclares compatible interfacesSchema drift can reject arguments or misinterpret state
Retrieval corpus & indexIdentifies evidence and retrieval settingsChanged chunks or indexes can alter admitted evidence
Admission & safety policyDeclares serving and refusal rulesA changed rule can admit unsupported claims or reject useful answers
Runtime & generation configIdentifies code, tokenizer, and effective settingsRuntime changes can affect parsing, latency, and outputs; temperature zero doesn't guarantee determinism
Evaluation suite & scorerIdentifies the evaluation contractA changed scorer makes old thresholds or comparisons inapplicable

Treat the manifest as a version tuple, not as one model number. Its coordinates cover prompt state, evidence, policy, schema, evaluator, runtime, and decoding. Change one coordinate and the release identity must change, even when weights stay put.

Compatibility sits inside that tuple. A candidate can pass answer-quality checks and still fail when its input, output, or tool schema no longer matches the router. Pin those contracts, test them with the same release ID, and diagnose an interface failure separately from a model-quality regression.

We'll use two bundles. stable uses the hand-edited deploy-answerer-v1 template. candidate swaps in the compiled program and the image that loads it. Other fields stay fixed. This narrows the intended change to two interacting components; it doesn't isolate the prompt's causal effect from the image's effect or eliminate model-output variation.

Names and short digest-like strings below are fixture labels, not downloadable artifacts or valid OCI digests. In a real manifest, pin artifact bytes with full digests or immutable object versions, then verify them before use.[3] Hashing a filename doesn't freeze the file behind it: overwriting v1.json would change behavior without changing this manifest. Also include relevant retrieval/embedding settings, output and tool contracts, and effective runtime options; this compact schema doesn't enumerate every production dependency.

define-release-bundles.py
from dataclasses import asdict, dataclass, fields, replace
import hashlib
import json

@dataclass(frozen=True)
class ReleaseBundle:
    service: str
    answer_model: str
    evidence_gate: str
    prompt_compile_run: str
    tokenizer: str
    prompt_version: str
    policy_version: str
    corpus_version: str
    serving_image: str
    input_schema: str
    eval_suite: str
    evaluator_version: str
    decoding: str

    def __post_init__(self) -> None:
        if any(type(getattr(self, field.name)) is not str or not getattr(self, field.name).strip() for field in fields(self)):
            raise ValueError("release fields must be nonempty strings")

stable = ReleaseBundle(
    service="deploy-answerer",
    answer_model="hosted-model@2026-04-01",
    evidence_gate="releaseops-evidence-classifier-v2@sha256:41d8",
    prompt_compile_run="manual-template-v1",
    tokenizer="deploy-tokenizer@sha256:91aa",
    prompt_version="deploy-answerer-v1.json",
    policy_version="[email protected]",
    corpus_version="releaseops-records@sha256:corpus-5",
    serving_image="registry.example/deploy-answerer@sha256:image-a",
    input_schema="deploy-answer.v2",
    eval_suite="deploy-grounding-suite@sha256:suite-7",
    evaluator_version="claim-evidence-eval-v2",
    decoding="temperature=0",
)

candidate = replace(
    stable,
    prompt_compile_run="dspy-mipro-v2-run-052",
    prompt_version="compiled_a_hosted-2026-04-01.json",
    serving_image="registry.example/deploy-answerer@sha256:image-b",
)

print(f"stable_prompt={stable.prompt_version}")
print(f"candidate_prompt={candidate.prompt_version}")
print(f"candidate_compile_run={candidate.prompt_compile_run}")
Output
stable_prompt=deploy-answerer-v1.json
candidate_prompt=compiled_a_hosted-2026-04-01.json
candidate_compile_run=dspy-mipro-v2-run-052

A human-readable name such as v3 helps people find a release, but it can't prove what went into it. Before running the next cell, predict the result of changing only prompt_version: the release ID should change, because the canonical manifest includes that field.

content-address-the-release.py
def release_id(bundle: ReleaseBundle) -> str:
    payload = json.dumps(asdict(bundle), sort_keys=True, separators=(",", ":"))
    digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()
    return f"{bundle.service}@sha256:{digest}"

stable_id = release_id(stable)
candidate_id = release_id(candidate)
prompt_patch_id = release_id(
    replace(candidate, prompt_version="compiled_a_hosted-2026-04-01.b.json")
)

print(f"stable={stable_id}")
print(f"candidate={candidate_id}")
print(f"prompt_patch={prompt_patch_id}")
print(f"prompt_patch_is_new_release={prompt_patch_id != candidate_id}")

# A name in a manifest is not a digest of the file behind that name.
from pathlib import Path
from tempfile import TemporaryDirectory

with TemporaryDirectory() as directory:
    prompt_file = Path(directory) / stable.prompt_version
    prompt_file.write_text('{"instructions":"first fixture"}')
    first_bytes_digest = hashlib.sha256(prompt_file.read_bytes()).hexdigest()
    prompt_file.write_text('{"instructions":"changed fixture"}')
    second_bytes_digest = hashlib.sha256(prompt_file.read_bytes()).hexdigest()
    print("same_manifest_id_after_overwrite=", release_id(stable) == stable_id, sep="")
    print("prompt_bytes_changed=", first_bytes_digest != second_bytes_digest, sep="")
Output
stable=deploy-answerer@sha256:32b8ed409b8e6e47313599b0b26d8481b57c0343bc55f6bcb89ba6b0fc9b8127
candidate=deploy-answerer@sha256:e31f2b6d7a2cec362416c2725a8ea4bce109b6e649cff9caabbdc181dfe03738
prompt_patch=deploy-answerer@sha256:f94cb4d1892f93eb15b91d508281d6e3603800da98428dd88c9a49df4cb4e310
prompt_patch_is_new_release=True
same_manifest_id_after_overwrite=True
prompt_bytes_changed=True

The registry uses the full digest; figures abbreviate it for display. The overwrite demonstration shows the boundary: a manifest naming the same file can retain its ID while the prompt bytes change. Pin those bytes too. A hash doesn't establish truthful metadata or trusted build provenance. Equivalent JSON from another language needs the same serialization convention to produce the same digest.

Artifacts stay fixed; aliases move

A registry record stores an immutable bundle. An alias such as production or canary is a mutable pointer used by traffic. Retaining the record lets recovery select an older bundle without rebuilding it. Actual recovery also needs ready serving capacity and routers that observe the new pointer.

MLflow's Model Registry workflow uses aliases and tags to organize model versions, and its documentation marks fixed Model Stages as deprecated.[4] The distinction matters: a model version preserves history, while an alias names the version a loader should resolve now.

An model alias still points to one registered model version. It doesn't automatically include the prompt, corpus, policy, schema, or serving image in this release bundle. Treat it as one component pointer, or register a wrapper artifact whose manifest resolves the complete bundle. A movable model alias isn't complete release identity.

An alias update changes metadata. Serving processes may cache its old target, retain a loaded artifact, or have requests already in flight. Recovery also needs retained artifacts, compatible state, ready capacity, and verified routing. Keeping the previous evaluated release available can shorten that path; neither an alias nor a registry promises a universal recovery time.

Alias step chart over two fixed release-bundle rails: canary points to candidate release e31f2b6d7a2c before promotion, production starts on stable release 32b8ed409b8e, moves to candidate at promotion, and returns to the retained stable release after an incident; both immutable bundle records remain available throughout.
The fixed rails are stored release IDs; the colored lines are aliases. Canary reaches the candidate first, production moves only after promotion, and rollback returns production to the retained stable bundle without rewriting either manifest.

The small registry below enforces that rule. register() keeps a deep copy of each bundle, and move_alias() accepts only a registered ID. Before running it, predict the alias state: registering candidate should increase the record count while production still resolves to stable.

registry-and-aliases.py
from copy import deepcopy

class ReleaseRegistry:
    def __init__(self) -> None:
        self._bundles: dict[str, ReleaseBundle] = {}
        self._aliases: dict[str, str] = {}

    def register(self, bundle: ReleaseBundle) -> str:
        bundle_id = release_id(bundle)
        existing = self._bundles.get(bundle_id)
        if existing is not None and existing != bundle:
            raise ValueError("release digest collision")
        self._bundles[bundle_id] = deepcopy(bundle)
        return bundle_id

    def move_alias(self, alias: str, bundle_id: str, *, expected: str | None = None) -> None:
        if type(alias) is not str or not alias.strip():
            raise ValueError("alias must be a nonempty string")
        if bundle_id not in self._bundles:
            raise KeyError(f"unregistered release: {bundle_id}")
        if self._aliases.get(alias) != expected:
            raise ValueError("stale alias update")
        self._aliases[alias] = bundle_id

    def resolve(self, alias: str) -> str:
        return self._aliases[alias]

    def bundle(self, bundle_id: str) -> ReleaseBundle:
        return deepcopy(self._bundles[bundle_id])

registry = ReleaseRegistry()
assert registry.register(stable) == stable_id
assert registry.register(candidate) == candidate_id
registry.move_alias("production", stable_id)

print(f"registered={len(registry._bundles)}")
print(f"production={registry.resolve('production')}")
print(f"candidate_registered={candidate_id in registry._bundles}")
Output
registered=2
production=deploy-answerer@sha256:32b8ed409b8e6e47313599b0b26d8481b57c0343bc55f6bcb89ba6b0fc9b8127
candidate_registered=True

Try moving production to candidate without supplying expected=stable_id: the registry rejects the stale update instead of overwriting an existing target. This check is sequential Python, not a distributed lock. A production must perform the compare-and-swap write atomically, authenticate the actor, and record an audit event. Tracking a monotonically increasing alias revision alongside its target catches ABA problems where an alias moves away and returns to the same release during a deployment race.

Promotion begins with controlled evidence

Registration makes a bundle addressable, not approved. Continuous delivery for machine learning adds and monitoring to ordinary build-and-deploy practices because a valid artifact can still produce unacceptable behavior.[5]

The gate should speak the same language as the , evaluation, and experiment lessons that came before it. Reuse the grounded evidence contract instead of inventing one generic quality score:

  • supported_evidence_f1 scores the fixture's binary route decision against reference labels. With SERVE as the positive class, F1=2TP/(2TP+FP+FN)F_1 = 2TP / (2TP + FP + FN): TP is a supported request served, FP is an unsupported request served, and FN is a supported request abstained. A real evaluator must define invalid outputs and undefined-denominator behavior too.
  • unsupported_serve_count must remain zero in the frozen high-risk suite. That's the req_205 failure: serving a deploy-approval claim ReleaseOps didn't support.
  • rollback_slice_f1 checks the rollback questions separately. Naming a slice in the suite doesn't enforce its floor; the gate must inspect the result.
  • p95_latency_ms () keeps a behaviorally acceptable release inside its response budget.
  • Schema, evaluation-suite, and evaluator identifiers check the declared contract. Timestamps, case counts, support-review coverage, and latency coverage check supplied report metadata; they don't prove the checks actually ran.
  • evaluation_report preserves the report artifact for a reviewer or incident responder to inspect later.

The supplied candidate summary has 0.93 F1, 0.91 rollback-slice F1, zero unsupported serves, and 472 ms p95. Our exercise requires 200 evaluated cases with complete support and latency checks, plus a report completed within 24 hours. These values aren't measured model results or universal thresholds. Route F1 isn't a score for generated answer truthfulness; the unsupported-claim counter requires a separate claim-support review in a real system. Predict why a strong report from evaluator v3 must still fail.

offline-promotion-gate.py
from datetime import datetime, timedelta, timezone
import math

NOW = datetime(2026, 9, 2, 12, 0, tzinfo=timezone.utc)

@dataclass(frozen=True)
class OfflineEvidence:
    release_id: str
    eval_suite: str
    evaluator_version: str
    input_schema: str
    evaluation_report: str
    supported_evidence_f1: float
    unsupported_serve_count: int
    p95_latency_ms: int
    rollback_slice_f1: float
    evaluated_count: int
    support_checked_count: int
    latency_sample_count: int
    completed_at: datetime

@dataclass(frozen=True)
class Decision:
    allowed: bool
    reason: str

def finite_between(value: object, low: float, high: float) -> bool:
    if type(value) not in (int, float):
        return False
    try:
        return math.isfinite(value) and low <= value <= high
    except OverflowError:
        return False

def fresh(completed_at: datetime, as_of: datetime, max_age: timedelta) -> bool:
    if not isinstance(completed_at, datetime) or not isinstance(as_of, datetime):
        return False
    if completed_at.utcoffset() is None or as_of.utcoffset() is None:
        return False
    return timedelta(0) <= as_of - completed_at <= max_age

def offline_gate(
    bundle: ReleaseBundle, evidence: OfflineEvidence, as_of: datetime
) -> Decision:
    if type(bundle) is not ReleaseBundle or type(evidence) is not OfflineEvidence:
        return Decision(False, "invalid release or evidence record")
    if evidence.release_id != release_id(bundle):
        return Decision(False, "evidence belongs to another release")
    if evidence.eval_suite != bundle.eval_suite:
        return Decision(False, "evaluation suite changed")
    if evidence.evaluator_version != bundle.evaluator_version:
        return Decision(False, "evaluator version changed")
    if evidence.input_schema != bundle.input_schema:
        return Decision(False, "schema mismatch")
    if type(evidence.evaluation_report) is not str or not evidence.evaluation_report.strip():
        return Decision(False, "evaluation report missing")
    if not fresh(evidence.completed_at, as_of, timedelta(hours=24)):
        return Decision(False, "report is stale, future-dated, or lacks a timezone")
    if type(evidence.evaluated_count) is not int or evidence.evaluated_count != 200:
        return Decision(False, "suite requires 200 evaluated cases")
    if any(type(count) is not int or count != evidence.evaluated_count for count in (
        evidence.support_checked_count, evidence.latency_sample_count
    )):
        return Decision(False, "support or latency coverage incomplete")
    if not all(finite_between(score, 0, 1) for score in (
        evidence.supported_evidence_f1, evidence.rollback_slice_f1
    )) or not finite_between(evidence.p95_latency_ms, 0, math.inf):
        return Decision(False, "invalid metric")
    if type(evidence.unsupported_serve_count) is not int or not 0 <= evidence.unsupported_serve_count <= evidence.support_checked_count:
        return Decision(False, "unsupported count must be a valid integer")
    if evidence.supported_evidence_f1 < 0.92:
        return Decision(False, "supported_evidence_f1 below 0.92")
    if evidence.rollback_slice_f1 < 0.90:
        return Decision(False, "rollback slice below 0.90")
    if evidence.unsupported_serve_count != 0:
        return Decision(False, "unsupported answer was served")
    if evidence.p95_latency_ms > 500:
        return Decision(False, "p95 latency exceeds 500 ms")
    return Decision(True, "offline gate passed")

candidate_offline = OfflineEvidence(
    release_id=candidate_id,
    eval_suite=candidate.eval_suite,
    evaluator_version=candidate.evaluator_version,
    input_schema=candidate.input_schema,
    evaluation_report="reports/candidate-suite-7-redacted.json",
    supported_evidence_f1=0.93,
    unsupported_serve_count=0,
    p95_latency_ms=472,
    rollback_slice_f1=0.91,
    evaluated_count=200,
    support_checked_count=200,
    latency_sample_count=200,
    completed_at=NOW - timedelta(hours=2),
)
weaker_candidate = replace(candidate_offline, supported_evidence_f1=0.89)
changed_evaluator = replace(candidate_offline, evaluator_version="claim-evidence-eval-v3")

print(f"candidate={offline_gate(candidate, candidate_offline, NOW)}")
print(f"weak_metric={offline_gate(candidate, weaker_candidate, NOW)}")
print(f"changed_evaluator={offline_gate(candidate, changed_evaluator, NOW)}")
Output
candidate=Decision(allowed=True, reason='offline gate passed')
weak_metric=Decision(allowed=False, reason='supported_evidence_f1 below 0.92')
changed_evaluator=Decision(allowed=False, reason='evaluator version changed')

Passing the offline gate permits the next experiment. It doesn't replace production yet. Create a alias while production still points to the known-good bundle. Creating that pointer alone sends no requests; traffic routing comes later.

The gate assumes an authenticated evaluator supplied the parsed report. It doesn't fetch the artifact, verify its digest, or prove that 200 distinct cases received valid support reviews and latency measurements. Ingestion must verify the retained report, case membership, review semantics, and actual runtime identity before trusting its summary. Latency coverage here includes terminal failures and timeouts, not only successful responses. The 24-hour rule bounds reliance on a report while the operating environment may change; it doesn't invalidate mathematics overnight.

open-canary-only-after-gate.py
def open_canary(
    bundle: ReleaseBundle, evidence: OfflineEvidence, as_of: datetime
) -> Decision:
    decision = offline_gate(bundle, evidence, as_of)
    if decision.allowed:
        registry.move_alias("canary", release_id(bundle))
    return decision

canary_decision = open_canary(candidate, candidate_offline, NOW)

print(f"canary_opened={canary_decision.allowed}")
print(f"canary={registry.resolve('canary')}")
print(f"production_still_stable={registry.resolve('production') == stable_id}")
Output
canary_opened=True
canary=deploy-answerer@sha256:e31f2b6d7a2cec362416c2725a8ea4bce109b6e649cff9caabbdc181dfe03738
production_still_stable=True

Managed models need a documented pin

For self-hosted weights, verifying a content hash identifies checkpoint bytes; inference code, configuration, and hardware can still affect outputs. For a managed API, use the strongest version identifier the provider documents and capture the effective request settings.[6] The fixture hosted-model@2026-04-01 is an invented stand-in, not an available model.

A floating alias can change without a deployment in your application. A dated identifier reduces that source of drift according to the provider's contract; its spelling isn't proof of immutable service behavior, deterministic output, or permanent availability. OpenAI explicitly recommends pinned versions plus evaluations, and publishes snapshot retirements.[6][7] Monitor regression cases and retirement notices, and evaluate identifier/configuration changes as new candidates. A retired or unavailable stable endpoint cannot be your usable rollback target.

Replay a failed production trace against the candidate

Frozen fixtures test failures you thought to write down. Trace replay starts with a recorded failure: reuse its request and recorded tool responses against another release. Holding those inputs fixed removes two sources of variation. It doesn't guarantee deterministic model output, even with temperature zero.

The regression question is narrow: does the candidate repeat a recorded failure when relevant inputs are held fixed? Replay can complement offline and shadow evaluation; it isn't required to occupy one universal funnel stage. Substitute recorded tool results rather than executing production writes. Replay still consumes inference compute when a real model runs.

Apply the question to fixture req_205. Its encoded deploy-answerer-v1.1-regression branch serves an unsupported approval; the encoded stable and candidate branches honor the supplied support label. Compare those branches while checking the declared model, corpus, and evidence-gate names.

The next cell is a deterministic policy-branch simulation. A supplied boolean represents whether the claim is supported; no recorded request body, evidence bytes, tool results, or generated answer are replayed. The seed and temperature are returned as metadata, not applied to inference. The checks compare declared corpus, model, and evidence-gate names, then distinguish an encoded regressed branch from an encoded corrected branch.

simulate-trace-replay.py
@dataclass(frozen=True)
class RecordedTrace:
    request_id: str
    origin_release: str
    prompt_version: str
    evidence_version: str
    evidence_gate: str
    tool_sequence: tuple[str, ...]
    evidence_supports_claim: bool
    served_unsupported_claim: bool
    temperature: float = 0.0
    seed: int = 0
    model_snapshot: str = ""

def prompt_would_serve(bundle: ReleaseBundle, evidence_supports_claim: bool) -> bool:
    if type(evidence_supports_claim) is not bool:
        raise ValueError("fixture support label must be a boolean")
    # Historical v1.1 bypassed the gate; known-good v1 and the candidate don't.
    if bundle.prompt_version == "deploy-answerer-v1.1-regression.json":
        return True
    if bundle.prompt_version in {stable.prompt_version, candidate.prompt_version}:
        return evidence_supports_claim
    raise ValueError("no simulated policy for this prompt")

def replay_against(trace: RecordedTrace, bundle: ReleaseBundle) -> dict[str, object]:
    if trace.evidence_version != bundle.corpus_version:
        raise ValueError("replay evidence snapshot changed")
    if trace.model_snapshot != bundle.answer_model:
        raise ValueError("replay model pin changed")
    if trace.evidence_gate != bundle.evidence_gate:
        raise ValueError("replay evidence gate changed")
    would_serve = prompt_would_serve(bundle, trace.evidence_supports_claim)
    reproduces = would_serve and not trace.evidence_supports_claim
    return {
        "candidate_release": release_id(bundle),
        "prompt_version": bundle.prompt_version,
        "tool_sequence": trace.tool_sequence,
        "pins": (trace.temperature, trace.seed, trace.model_snapshot),
        "candidate_reproduces_failure": reproduces,
    }

historical_incident = replace(
    stable,
    prompt_compile_run="regressed-template-v1.1",
    prompt_version="deploy-answerer-v1.1-regression.json",
    serving_image="registry.example/deploy-answerer@sha256:image-regressed",
)
failed_trace = RecordedTrace(
    request_id="req_205",
    origin_release=release_id(historical_incident),
    prompt_version=historical_incident.prompt_version,
    evidence_version="releaseops-records@sha256:corpus-5",
    evidence_gate=stable.evidence_gate,
    tool_sequence=("lookup_release", "fetch_record", "answer"),
    evidence_supports_claim=False,
    served_unsupported_claim=True,
    temperature=0.0,
    seed=7,
    model_snapshot=stable.answer_model,
)
historical_replay = replay_against(failed_trace, historical_incident)
stable_replay = replay_against(failed_trace, stable)
candidate_replay = replay_against(failed_trace, candidate)

print(f"origin_release={failed_trace.origin_release}")
print(f"historical_reproduces_failure={historical_replay['candidate_reproduces_failure']}")
print(f"stable_reproduces_failure={stable_replay['candidate_reproduces_failure']}")
print(f"replayed_against={candidate_replay['candidate_release']}")
print(f"candidate_prompt={candidate_replay['prompt_version']}")
print(f"recorded_step_names={candidate_replay['tool_sequence']}")
print(f"pins={candidate_replay['pins']}")
print(f"candidate_reproduces_failure={candidate_replay['candidate_reproduces_failure']}")

assert historical_replay["candidate_reproduces_failure"] is True
assert stable_replay["candidate_reproduces_failure"] is False
assert candidate_replay["candidate_reproduces_failure"] is False
Output
origin_release=deploy-answerer@sha256:efe48fbcb5a2fb27baf7d386061cbf9383afd3f9dfade71e4ba0a27b3140c966
historical_reproduces_failure=True
stable_reproduces_failure=False
replayed_against=deploy-answerer@sha256:e31f2b6d7a2cec362416c2725a8ea4bce109b6e649cff9caabbdc181dfe03738
candidate_prompt=compiled_a_hosted-2026-04-01.json
recorded_step_names=('lookup_release', 'fetch_record', 'answer')
pins=(0.0, 7, 'hosted-model@2026-04-01')
candidate_reproduces_failure=False

The output verifies our simulated policy branch. It says nothing about the real compiled program's quality. A production replay harness must load that program, substitute recorded tool inputs and outputs, check unexpected tool calls, and score the generated answer. A candidate can choose a different tool sequence; don't silently supply an unrelated recorded result just to finish the replay.

Retain the request, evidence snapshot, actual runtime identity, and supported sampling controls. OpenAI's compatibility guidance explicitly describes model outputs as variable and recommends pinned versions plus evaluations.[6] Run repeated comparisons when outputs vary, and judge the distribution of failures rather than demanding byte-for-byte reproduction. Keep past incidents in the regression suite, while recognizing that passing known cases can't rule out new failures.

Shadow evaluation must be read-only

Replay gives us one known failure. Offline still can't cover every real request shape. A shadow sends a sanitized copy of a production request to the candidate while the stable release alone supplies the user-visible answer. It reveals latency or evidence-support problems without exposing candidate text to customers.

The safety boundary is strict: can't send messages or perform production writes. Use recorded tool results or approved read-only operations; its output is evaluation data. The envelope below redacts a ship identifier, but doesn't implement a real queue or authorization boundary.

Before running it, inspect the flags that must stay false: side effects must be disabled and candidate output must stay hidden. Here the recorded stable response abstained, and the simulated candidate should abstain too. Agreement is one comparison, not a correctness score: two releases can agree on a bad answer.

shadow-envelope.py
import re

@dataclass(frozen=True)
class ShadowEnvelope:
    candidate_release: str
    stable_release: str
    sanitized_text: str
    side_effects_enabled: bool
    response_visible_to_user: bool
    evidence_supports_claim: bool
    observed_stable_route: str

@dataclass(frozen=True)
class ShadowComparison:
    route_agreement: bool
    candidate_route_supported: bool
    candidate_would_serve: bool
    stable_route: str
    candidate_route: str
    dropped: bool

def make_shadow(
    text: str, evidence_supports_claim: bool, observed_stable_route: str
) -> ShadowEnvelope:
    if type(text) is not str or type(evidence_supports_claim) is not bool:
        raise ValueError("need text and a boolean fixture support label")
    if observed_stable_route not in {"serve", "abstain"}:
        raise ValueError("unknown stable route")
    sanitized = re.sub(r"SHIP-\d+", "[SHIP_ID]", text)
    return ShadowEnvelope(
        candidate_release=registry.resolve("canary"),
        stable_release=registry.resolve("production"),
        sanitized_text=sanitized,
        side_effects_enabled=False,
        response_visible_to_user=False,
        evidence_supports_claim=evidence_supports_claim,
        observed_stable_route=observed_stable_route,
    )

def compare_shadow(envelope: ShadowEnvelope, queue_full: bool = False) -> ShadowComparison:
    if any(type(value) is not bool for value in (
        queue_full, envelope.side_effects_enabled, envelope.response_visible_to_user,
        envelope.evidence_supports_claim,
    )) or envelope.observed_stable_route not in {"serve", "abstain"}:
        raise ValueError("invalid shadow metadata or route")
    if queue_full or envelope.side_effects_enabled or envelope.response_visible_to_user:
        return ShadowComparison(False, False, False, "dropped", "dropped", True)
    # Stable already answered the user; shadow only scores the candidate copy.
    stable_route = envelope.observed_stable_route
    candidate_bundle = registry.bundle(envelope.candidate_release)
    candidate_would_serve = prompt_would_serve(
        candidate_bundle, envelope.evidence_supports_claim
    )
    candidate_route = "serve" if candidate_would_serve else "abstain"
    route_agreement = candidate_route == stable_route
    candidate_route_supported = candidate_route == "abstain" or envelope.evidence_supports_claim
    return ShadowComparison(
        route_agreement=route_agreement,
        candidate_route_supported=candidate_route_supported,
        candidate_would_serve=candidate_would_serve,
        stable_route=stable_route,
        candidate_route=candidate_route,
        dropped=False,
    )

shadow = make_shadow(
    "Is payment-tests approved for SHIP-4412 without a ReleaseOps record?",
    evidence_supports_claim=False,
    observed_stable_route="abstain",
)
comparison = compare_shadow(shadow)
dropped = compare_shadow(shadow, queue_full=True)

print(f"shadow_text={shadow.sanitized_text}")
print(f"candidate={shadow.candidate_release}")
print(f"side_effects_enabled={shadow.side_effects_enabled}")
print(f"response_visible={shadow.response_visible_to_user}")
print(f"candidate_route={comparison.candidate_route}")
print(f"route_agreement={comparison.route_agreement}")
print(f"candidate_route_supported={comparison.candidate_route_supported}")
print(f"dropped_when_queue_full={dropped.dropped}")

assert comparison.candidate_would_serve is False
assert comparison.route_agreement is True
assert comparison.candidate_route_supported is True
assert dropped.dropped is True
Output
shadow_text=Is payment-tests approved for [SHIP_ID] without a ReleaseOps record?
candidate=deploy-answerer@sha256:e31f2b6d7a2cec362416c2725a8ea4bce109b6e649cff9caabbdc181dfe03738
side_effects_enabled=False
response_visible=False
candidate_route=abstain
route_agreement=True
candidate_route_supported=True
dropped_when_queue_full=True

The shadow comparison uses the same boolean policy simulation, not sanitized text or a generated answer. candidate_route_supported means the encoded route doesn't serve against a false fixture support label; it doesn't establish answer truthfulness or overall safety. The one-pattern redaction is also a fixture. A real path needs schema-aware data minimization for identifiers and before queues, logs, or candidate services. Redaction can change retrieval or routing; compare releases on equivalent sanitized inputs or account for that difference. Isolate shadow capacity and bound its cost.

At the service boundary, put the envelope on a bounded worker queue and count dropped or failed comparisons. The compare step is the other half of shadow: after the candidate answers the sanitized copy, record route agreement, claim safety, and latency against the stable path. Feed shadow_drop_rate from dropped or timed-out comparisons. Don't start an untracked background task in request scope and assume its evaluation record will survive a process restart.

side_effects_enabled=False in this teaching envelope is metadata, not an authorization boundary. The worker still needs a read-only tool allowlist and credentials that can't write production state. Reject a shadow request that asks for a side effect.

Canary traffic is visible and sticky

Shadow results can justify limited exposure, not automatic promotion. A canary returns candidate responses to a limited share of users. For this multi-turn product, our rollout policy pins each conversation to one release during normal widening. Switching releases can alter behavior or make stored state incompatible; products with independent requests may use a different assignment unit.

Use deterministic hashing of a conversation ID to choose new conversations consistently across workers. Python's built-in hash() is process-dependent, so it can't provide that bucket. Hashing alone isn't enough: when exposure widens from 1% to 10%, a higher threshold could move an existing conversation from stable to candidate. Persist the first resolved release ID for the conversation lifetime.

Predict the widening case before running the router. A thread assigned to stable at 10% should still resolve to stable after the threshold becomes 100%; an abort must be able to override that assignment.

sticky-canary-routing.py
def bucket(conversation_id: str) -> int:
    if type(conversation_id) is not str or not conversation_id.strip():
        raise ValueError("conversation ID must be a nonempty string")
    digest = hashlib.sha256(conversation_id.encode("utf-8")).hexdigest()
    return int(digest[:8], 16) % 100

conversation_assignments: dict[str, str] = {}
aborted_releases: set[str] = set()

def assigned_release(conversation_id: str, canary_percent: int) -> str:
    if type(conversation_id) is not str or not conversation_id.strip():
        raise ValueError("conversation ID is required")
    if type(canary_percent) is not int or not 0 <= canary_percent <= 100:
        raise ValueError("canary percentage must be an integer from 0 to 100")
    existing = conversation_assignments.get(conversation_id)
    if existing is not None and existing not in aborted_releases:
        return existing
    alias = "canary" if bucket(conversation_id) < canary_percent else "production"
    bundle_id = registry.resolve(alias)
    if bundle_id in aborted_releases:
        bundle_id = registry.resolve("production")
    if bundle_id in aborted_releases:
        raise RuntimeError("no healthy release is available")
    conversation_assignments[conversation_id] = bundle_id
    return bundle_id

canary_thread = next(
    f"thread-{index}" for index in range(1000) if bucket(f"thread-{index}") < 10
)
assignments = [assigned_release(canary_thread, canary_percent=10) for _ in range(3)]
stable_thread = next(
    f"thread-{index}" for index in range(1000) if bucket(f"thread-{index}") >= 10
)
stable_before_widen = assigned_release(stable_thread, canary_percent=10)
stable_after_widen = assigned_release(stable_thread, canary_percent=100)

print(f"canary_thread={canary_thread}")
print(f"bucket={bucket(canary_thread)}")
print(f"same_release_each_turn={len(set(assignments)) == 1}")
print(f"assigned_to_candidate={assignments[0] == candidate_id}")
print(f"existing_stable_thread_pinned_after_widen={stable_before_widen == stable_after_widen == stable_id}")
Output
canary_thread=thread-6
bucket=6
same_release_each_turn=True
assigned_to_candidate=True
existing_stable_thread_pinned_after_widen=True

The in-memory dictionary is a local fixture. For this policy, a production router needs shared persisted assignments, atomic first assignment, a defined expiry, and resolved release IDs in traces. Hashing chooses a bucket; persistence preserves the release when thresholds or aliases change. Incompatible conversation/tool-state schemas can make a release switch fail, so test compatibility rather than assuming any switch necessarily crashes.

A 10% threshold assigns roughly 10% of newly initiated conversations, not 10% of raw requests. Active conversations dominate request volume until they conclude. When exposure widens from 10% to 25% or 100%, existing stable conversations finish their lifecycles on stable.

An abort must override failed-release assignments. Applying that change across routers takes propagation and cache handling; it isn't instantaneous. If state is incompatible with fallback, pause, reset, or migrate it under a reviewed policy. Don't claim recovery until traces show the restored release handling requests.

Alternative supplied windows. At 1%, 1,200 requests have zero unsupported serves and two transport errors; the gate passes. At 10%, 8,000 requests have 112 unsupported serves and 24 transport errors; the gate aborts despite passing latency and transport thresholds. Recovery changes routing metadata and requires verification at serving.
These summaries are alternative fixtures, not measured sequential traffic. One confirmed unsupported serve is enough to abort this attempt. Hashing needs persisted assignments, and recovery needs verified routing to ready capacity.

A canary window answers a new question

Offline evidence covers frozen examples. A live window asks how candidate behaves under the current traffic mix. Bind it to both the release and rollout attempt; yesterday's attempt isn't evidence for today's exposure. Our exercise requires an interval ending within five minutes and at least 1,000 requests with complete support, latency, and transport checks. It also requires 100 answerable requests, including 50 answerable rollback requests, and at least 90% useful answers in each of those groups. These are declared teaching policies, not statistical guarantees or universal thresholds.

An answerable request has admitted records that support a useful answer under the reviewed task rubric. The useful-serve counts require acceptable delivered answers, not just SERVE routes. Errors, empty responses, and unnecessary abstentions don't count as useful. A rollout that refuses everything may avoid unsupported claims while failing the product's purpose.

The controller needs three outcomes: PASS permits advancing to the next ramp tier, HOLD pauses to collect more trustworthy evidence, and ABORT trips the circuit breaker to remove a defective candidate. Missing telemetry isn't a passing zero. Conversely, one confirmed unsupported serve triggers an immediate abort, even before reaching the sample minimum. In a deployed system, missing telemetry also requires a strict escalation timeout so a canary can't linger in production without active monitoring.

Read the two windows in the figure as an operational verdict. The 1% sample clears this contract; the 10% sample fails even though its latency, error rate, and usefulness floors pass. A positive unsupported counter overrides those passing metrics. Zero unsupported serves alone still can't establish acceptable usefulness.

live-canary-window.py
@dataclass(frozen=True)
class LiveWindow:
    name: str
    release_id: str
    rollout_id: str
    started_at: datetime
    ended_at: datetime
    request_count: int
    support_checked_count: int
    unsupported_serve_count: int
    latency_sample_count: int
    p95_latency_ms: int
    transport_checked_count: int
    error_count: int
    shadow_drop_rate: float
    supported_request_count: int
    useful_supported_serve_count: int
    rollback_supported_request_count: int
    rollback_useful_serve_count: int

    @property
    def error_rate(self) -> float:
        return self.error_count / self.request_count

@dataclass(frozen=True)
class LiveDecision:
    status: str
    reason: str

def live_gate(
    window: LiveWindow, expected_release: str, expected_rollout: str, as_of: datetime
) -> LiveDecision:
    if type(window) is not LiveWindow:
        return LiveDecision("HOLD", "invalid window record")
    if (window.release_id, window.rollout_id) != (expected_release, expected_rollout):
        return LiveDecision("HOLD", "window belongs to another release or rollout")
    counts = (window.request_count, window.support_checked_count, window.unsupported_serve_count)
    if any(type(n) is not int or n < 0 for n in counts):
        return LiveDecision("HOLD", "invalid count")
    if not window.unsupported_serve_count <= window.support_checked_count <= window.request_count:
        return LiveDecision("HOLD", "inconsistent counts")
    if window.unsupported_serve_count > 0:
        return LiveDecision("ABORT", "unsupported serve count must remain zero")
    if (
        not fresh(window.ended_at, as_of, timedelta(minutes=5))
        or not isinstance(window.started_at, datetime)
        or window.started_at.utcoffset() is None
        or window.started_at >= window.ended_at
    ):
        return LiveDecision("HOLD", "invalid or stale observation interval")
    if any(type(count) is not int or count < 0 for count in (
        window.latency_sample_count, window.transport_checked_count, window.error_count
    )):
        return LiveDecision("HOLD", "invalid measurement count")
    if not (window.error_count <= window.transport_checked_count <= window.request_count
            and window.latency_sample_count <= window.request_count):
        return LiveDecision("HOLD", "inconsistent measurement counts")
    if window.request_count == 0:
        return LiveDecision("HOLD", "fewer than 1000 requests observed")
    if not (
        finite_between(window.p95_latency_ms, 0, math.inf)
        and finite_between(window.shadow_drop_rate, 0, 1)
    ):
        return LiveDecision("HOLD", "invalid metric")
    if window.p95_latency_ms > 550:
        return LiveDecision("ABORT", "live latency exceeded 550 ms")
    if window.error_rate > 0.01:
        return LiveDecision("ABORT", "error rate exceeded 1%")
    if window.shadow_drop_rate > 0.02:
        return LiveDecision("HOLD", "shadow telemetry incomplete")
    if window.support_checked_count != window.request_count:
        return LiveDecision("HOLD", "support evaluation incomplete")
    if window.latency_sample_count != window.request_count or window.transport_checked_count != window.request_count:
        return LiveDecision("HOLD", "latency or transport coverage incomplete")
    if window.request_count < 1_000:
        return LiveDecision("HOLD", "fewer than 1000 requests observed")
    usefulness_counts = (
        window.supported_request_count, window.useful_supported_serve_count,
        window.rollback_supported_request_count, window.rollback_useful_serve_count,
    )
    if any(type(count) is not int or count < 0 for count in usefulness_counts):
        return LiveDecision("HOLD", "invalid usefulness count")
    if not (
        window.useful_supported_serve_count <= window.supported_request_count <= window.request_count
        and window.rollback_useful_serve_count <= window.rollback_supported_request_count <= window.supported_request_count
        and window.rollback_useful_serve_count <= window.useful_supported_serve_count
        and window.useful_supported_serve_count - window.rollback_useful_serve_count <= window.supported_request_count - window.rollback_supported_request_count
    ):
        return LiveDecision("HOLD", "inconsistent usefulness counts")
    if window.supported_request_count < 100 or window.rollback_supported_request_count < 50:
        return LiveDecision("HOLD", "too few answerable or rollback cases")
    if (10 * window.useful_supported_serve_count < 9 * window.supported_request_count
            or 10 * window.rollback_useful_serve_count < 9 * window.rollback_supported_request_count):
        return LiveDecision("ABORT", "useful-answer floor below 90%")
    return LiveDecision("PASS", "live window passed")

rollout_id = "rollout-052"
window_1_percent = LiveWindow(
    "1%", candidate_id, rollout_id, NOW - timedelta(minutes=15), NOW,
    request_count=1_200, support_checked_count=1_200, unsupported_serve_count=0,
    latency_sample_count=1_200, p95_latency_ms=481,
    transport_checked_count=1_200, error_count=2, shadow_drop_rate=0.001,
    supported_request_count=800, useful_supported_serve_count=752,
    rollback_supported_request_count=100, rollback_useful_serve_count=92,
)
window_10_percent = replace(
    window_1_percent, name="10%", request_count=8_000, support_checked_count=8_000,
    unsupported_serve_count=112, latency_sample_count=8_000, p95_latency_ms=493,
    transport_checked_count=8_000, error_count=24,
    supported_request_count=6_000, useful_supported_serve_count=5_640,
    rollback_supported_request_count=500, rollback_useful_serve_count=465,
)

print(f"one_percent={live_gate(window_1_percent, candidate_id, rollout_id, NOW)}")
print(f"ten_percent={live_gate(window_10_percent, candidate_id, rollout_id, NOW)}")
print(f"ten_percent_unsupported_count={window_10_percent.unsupported_serve_count}")
Output
one_percent=LiveDecision(status='PASS', reason='live window passed')
ten_percent=LiveDecision(status='ABORT', reason='unsupported serve count must remain zero')
ten_percent_unsupported_count=112

The two windows are alternative supplied summaries, not sequential measurements collected here. Transport errors are separate from unsupported claims: the first window has two transport errors, not zero, and 2/1200≈0.167%2/1200 \approx 0.167\% remains under its 1% limit. Shadow drop rate summarizes a different comparison queue. Ingestion must verify distinct request membership, actual support reviews, latency coverage including failures/timeouts, and the transport denominator; counts alone don't establish those facts.

A parsed integer can be too large for math.isfinite to convert to a float. The guard rejects that value rather than crashing the controller. Try malformed latency in both evidence paths: the offline gate rejects the report, while the live gate holds exposure pending trustworthy telemetry.

malformed-release-metrics.py
bad_latencies = (None, True, "472", float("nan"), float("inf"), 10 ** 1000, -1)
offline_rejected = 0
live_held = 0
for latency in bad_latencies:
    offline = offline_gate(candidate, replace(candidate_offline, p95_latency_ms=latency), NOW)
    live = live_gate(replace(window_1_percent, p95_latency_ms=latency), candidate_id, rollout_id, NOW)
    assert offline == Decision(False, "invalid metric")
    assert live == LiveDecision("HOLD", "invalid metric")
    offline_rejected += 1
    live_held += 1
print(f"malformed_metrics={len(bad_latencies)} offline_rejected={offline_rejected} live_held={live_held}")
Output
malformed_metrics=7 offline_rejected=7 live_held=7

Try 600 requests with 600 reviews, latency samples, and transport checks, keeping usefulness counts within those totals: HOLD. Then add one attributed unsupported serve: ABORT, even below the sample minimum or with stale/missing measurement telemetry. Invalid attribution or malformed support counters instead require investigation. For zero known failures, missing support or measurement coverage must HOLD rather than become an assumed passing zero.

Zero observed failures isn't proof of zero risk. At a predeclared sample size of 1,200 independent requests with a fixed failure probability and correct failure classification, zero failures gives a one-sided 95% upper bound of 1 - 0.05 ** (1 / 1200) ≈ 0.00249, or 0.249%. Choosing when to stop based on observed results, correlated conversations, or imperfect evaluators requires a different analysis. The minimum volumes and usefulness floors are operational policy, not confidence guarantees. Retain per-case results and assess relevant production slices before promotion.

Use the failed signal to choose the next inspection. A request that can't parse points toward schema or serving-image compatibility. A parsed request with an unsupported answer belongs in trace replay against the pinned evidence snapshot. A safe but slow candidate points toward runtime capacity or decoding. Missing telemetry prevents widening because the gate lacks evidence, not because it has measured a quality regression.

Abort a canary; roll back a promotion

The right recovery verb depends on which alias moved:

  • If candidate fails while only canary receives traffic, abort the rollout. production never moved.
  • If candidate was promoted and fails later, roll back by repointing production to the retained stable release.

Check the alias state before naming the incident. That distinction keeps a report from calling an unpromoted canary failure a production rollback.

In the next drill, canary failure should set exposure to zero and restore the canary pointer. The separate promoted-release drill should move production to candidate, detect failure, then return it to stable.

abort-and-rollback.py
canary_percent = 10
failed_window = live_gate(window_10_percent, candidate_id, rollout_id, NOW)
if failed_window.status == "ABORT":
    aborted_releases.add(candidate_id)
    canary_percent = 0
    # Zero traffic isn't enough: any consumer of registry.resolve("canary") must leave the failed ID.
    registry.move_alias("canary", stable_id, expected=candidate_id)

print(f"canary_percent_after_abort={canary_percent}")
print(f"canary_alias_after_abort={registry.resolve('canary')}")
print(f"production_after_abort={registry.resolve('production')}")
print(f"pinned_canary_thread_restored_stable={assigned_release(canary_thread, canary_percent) == stable_id}")

assert registry.resolve("canary") == stable_id
assert registry.resolve("production") == stable_id

# Independent hypothetical: production had moved before the bad window arrived.
# This isn't authorization to retry the candidate rejected above.
rollback_drill = deepcopy(registry)
previous_production = rollback_drill.resolve("production")
rollback_drill.move_alias("production", candidate_id, expected=previous_production)
post_promotion_incident = replace(window_10_percent, name="100%")

if live_gate(post_promotion_incident, candidate_id, rollout_id, NOW).status == "ABORT":
    rollback_drill.move_alias("production", previous_production, expected=candidate_id)

print(f"drill_production_after_rollback={rollback_drill.resolve('production')}")
print(f"drill_restored_stable={rollback_drill.resolve('production') == stable_id}")
print(f"actual_production_unchanged={registry.resolve('production') == stable_id}")
Output
canary_percent_after_abort=0
canary_alias_after_abort=deploy-answerer@sha256:32b8ed409b8e6e47313599b0b26d8481b57c0343bc55f6bcb89ba6b0fc9b8127
production_after_abort=deploy-answerer@sha256:32b8ed409b8e6e47313599b0b26d8481b57c0343bc55f6bcb89ba6b0fc9b8127
pinned_canary_thread_restored_stable=True
drill_production_after_rollback=deploy-answerer@sha256:32b8ed409b8e6e47313599b0b26d8481b57c0343bc55f6bcb89ba6b0fc9b8127
drill_restored_stable=True
actual_production_unchanged=True

Argo Rollouts can use analysis results to advance, pause, or abort a rollout; failed background analysis can return canary weight to zero.[8] It doesn't supply our support evaluator or conversation store. Basic canary without traffic management approximates weights through replica counts; traffic-router integration provides finer control.[9] Verify the actual routing unit and abort behavior of your chosen configuration.

Recovery needs sufficient ready capacity, not just a retained ID. This fixture declares a hosted model, so our application isn't loading its weights into our GPUs. Recovery still depends on endpoint availability, quota, ready application workers, and compatible retrieval/tool services. For self-hosted models, cold startup may also involve weight loading, compilation, cache allocation, and GPU scheduling. Measure those stages in your deployment; there's no universal container or LLM startup time.

Keeping stable capacity ready can meet a short recovery objective, but the capacity plan must handle returning traffic. Stable handling 90% of demand may lack headroom for 100% after a 10% canary abort. If both prompt releases share an unchanged model backend, they needn't duplicate all model weights. Choose retention, standby, preloading, and scale-up plans against a tested recovery-time objective, costs, state compatibility, and provider availability. Completion of a soak period alone doesn't guarantee rollback readiness.

Neither recovery undoes completed side effects, database migrations, messages, or answers users already saw. Retain backward-compatible schemas, plan compensating actions where possible, and decide how to handle in-flight requests. Verify the restored release ID in serving traces: an alias update can succeed while router caches, unhealthy replicas, or incompatible state still prevent recovery.

Record decisions so a later engineer can reconstruct them

A release process needs durable records of release IDs, evidence digests, gate verdicts, window metrics, actors, timestamps, and alias revisions. Protect the audit store and retain its referenced artifacts; a list labeled append-only isn't an integrity control by itself.

The intended trail links each user-facing inference trace to its upstream provenance: Trace ID -> Release Digest -> Composite Manifest -> Evaluation Suite -> Controller Audit Log. These links support reconstruction when the records are trustworthy. Trace metadata can be wrong, sampled, dropped, or edited; a digest identifies recorded bytes, not proof that a claimed runtime executed. Verify actual serving identity and collection coverage.

The hand-written list below tells a hypothetical pass-then-abort story using our supplied values. It isn't a chronological measurement record or an audit log emitted by the transitions. The failed release remains available in the local registry while production stays stable. In a real controller, emit events when transitions succeed, with actor, timestamp, alias revision, expected previous target, and evidence digest. Reconstructing a list afterward doesn't prove those transitions occurred.

release-decision-record.py
@dataclass(frozen=True)
class ReleaseEvent:
    stage: str
    release_id: str
    decision: str
    evidence: str

events = (
    ReleaseEvent("register", candidate_id, "RECORDED", "manifest_sha"),
    ReleaseEvent("offline_gate", candidate_id, "PASSED", candidate_offline.evaluation_report),
    ReleaseEvent("canary_1_percent", candidate_id, "PASSED", "live:window-001"),
    ReleaseEvent("canary_10_percent", candidate_id, "ABORTED", "live:window-010"),
    ReleaseEvent("production", stable_id, "UNCHANGED", "rollback:not-needed"),
)

for event in events:
    print(f"{event.stage}:{event.decision}:{event.evidence}")
print("release_decision=REJECT_CANDIDATE_AFTER_CANARY_REGRESSION")
print(f"active_production={registry.resolve('production')}")
Output
register:RECORDED:manifest_sha
offline_gate:PASSED:reports/candidate-suite-7-redacted.json
canary_1_percent:PASSED:live:window-001
canary_10_percent:ABORTED:live:window-010
production:UNCHANGED:rollback:not-needed
release_decision=REJECT_CANDIDATE_AFTER_CANARY_REGRESSION
active_production=deploy-answerer@sha256:32b8ed409b8e6e47313599b0b26d8481b57c0343bc55f6bcb89ba6b0fc9b8127

Where the lab maps onto a live stack

The lab uses clean in-memory Python so state transitions and invariants remain completely transparent. A production LLMOps architecture partitions these responsibilities across specialized control-plane components:

ResponsibilityProduction toolingConcrete mechanism
Retained artifacts and manifestArtifact store, versioned repository, or OCI registryStore verified byte digests and enforce retention; a filename or movable branch isn't a byte pin
Environment pointersOwned release registry; MLflow alias for a componentResolve versions into the release manifest; implement required atomic revision checks and authorization in the control plane
Offline evidenceCI plus a reviewed task evaluatorRetain case-level results, scorer identity, coverage, and runtime evidence; generic CI doesn't know whether a claim is supported
Exposure and analysisArgo Rollouts plus a compatible traffic routerConfigure steps and AnalysisTemplates; application routing separately persists conversation assignments
Ready serving capacityApplication workers and a managed or self-hosted model backendTest returning load, dependency readiness, state compatibility, and retirement constraints
Observation and auditTracing/metrics collectors plus retained evidence and audit storesRecord resolved identities and measured outcomes; neither spans nor exporters are automatically complete or immutable

A feature flag can select a registered release. Resolve and record its target when serving rather than relying on the flag's later value. A floating model string alone leaves reconstruction incomplete; provider request metadata can help, but doesn't replace the documented version contract.

Before adapting this architecture, try four boundary cases: a report older than 24 hours, NaN latency with no known unsupported serve, an under-sampled window containing one unsupported serve, and an alias update with the wrong previous target. Expect rejection, HOLD, ABORT, and a stale-update exception, respectively. The local alias check is sequential; production needs an atomic revision check. None should silently promote a candidate.

Record the resolved release at each request and transition, retain its evidence, and drill recovery under returning load. Identity supports diagnosis; trustworthy evaluation, compatible state, ready capacity, and verified routing determine whether recovery works.

Complete the lesson

Mastery Check

Answer every question, then check your score. Score 75% or higher to mark this lesson complete.

1.Stable and candidate have identical answer-model and evidence-gate identifiers. The candidate changes prompt_version from deploy-answerer-v1.json to compiled_a_hosted-2026-04-01.json and serving_image from image-a to image-b. How should the release registry treat it before any traffic moves?
2.A release bundle calls a hosted model using a provider's floating model name rather than a documented fixed snapshot. What risk does that introduce?
3.A candidate bundle declares eval_suite=suite-7, evaluator_version=claim-evidence-eval-v2, and input_schema=deploy-answer.v2. Its evidence has the same release_id and metrics supported_evidence_f1=0.93, unsupported_serve_count=0, and p95_latency_ms=472, but evaluator_version=claim-evidence-eval-v3. What should the offline gate do?
4.A shadow worker receives a sanitized copy of a production request. Which configuration satisfies the shadow safety invariant?
5.An active conversation was assigned to stable at a 10% canary. The rollout widens to 100% before that conversation ends. How should the router handle its next turn?
6.A fully reviewed window has 10,000 requests, p95 latency 540 ms, 80 transport errors (0.8%), 10 unsupported serves, and shadow drop rate 1%. The limits are 550 ms, 1%, zero unsupported serves, and 2%. What happens?
7.A fresh window has 1,200 fully checked requests, passing latency/error/drop metrics, and zero unsupported serves. Of 800 answerable requests (100 rollback requests), it answers none usefully. Both useful-answer floors are 90%. What happens?
8.Production still points at the stable release while a candidate receives 10% of conversations through a canary alias. The live window fails because the candidate serves unsupported deploy-approval claims. What should be recorded?
9.A candidate passes offline gating and a 1% canary, production is moved from stable_id to candidate_id, and a later 100% live window fails the latency threshold. What should the release record show?
10.A self-hosted controller restores the stable alias, but its replicas were scaled to zero and this deployment takes several minutes to load their weights. What hasn't recovered yet?

10 questions remaining.

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.