LearnApplied LLM EngineeringModel Versioning & Deployment

⚙️MediumMLOps & Deployment

Model Versioning & Deployment

Turn an evaluated LLM change into an immutable release bundle, promote it through measured traffic, and roll back without losing lineage.

22 min read

Learning path

Step 81 of 196 in the full curriculum

Prompt Optimization with DSPy Semantic Caching & Cost Optimization

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

In our worked incident, trace req_205 shows deploy-answerer claiming that payment-tests are approved for SHIP-4412 even though the admitted ReleaseOps evidence doesn't support that claim. The on-call points production back to the known-good template, and new requests return to expected behavior. A review question remains: which prompt, hosted-model snapshot, evidence corpus, and serving image produced the bad answer?

The DSPy compile passed its frozen holdout, but that compiled JSON still isn't a production release. A live answer can change through the prompt program, hosted-model pin, ReleaseOps corpus, serving image, decoding settings, or schema. A label such as v1 can't tell you which combination was active.

The operational question is blunt: which complete, evaluated release produced this response, and how quickly can traffic return to the last known-good release?

Release identity changes when any behavior-producing field changes. The stable and candidate deploy-answerer bundles keep the hosted-model pin, evidence gate, ReleaseOps corpus, and evaluation contract pinned, but change the compiled prompt program, compile run, and serving image, producing different release IDs. — Pinned fields stay shared and reviewable. Changing compiled prompt state, its source run, or the serving image creates a new release bundle and a new release ID.

A release is more than model weights

deploy-answerer combines a hosted answer model, a releaseops-evidence-classifier-v2 gate, and prompt state. The gate decides which ReleaseOps excerpts are trusted enough to admit. The prompt program must constrain the answer to those excerpts and abstain when admitted evidence doesn't support the requested claim. If you roll back only one component while a newer prompt, corpus, or image remains active, you can land on a mixed release that was never evaluated.

Call that complete set a release bundle. Store every input that can change visible output or operational safety in one immutable manifest. Store the evaluation contract and resulting report with it, too. Data, code, configuration, and serving infrastructure otherwise drift along separate paths, leaving reviewers to guess which change caused a behavior shift. The bundle keeps those dependencies reviewable.^{[1]Reference 1Hidden Technical Debt in Machine Learning Systems.https://research.google/pubs/hidden-technical-debt-in-machine-learning-systems/}^{[2]Reference 2Challenges in Deploying Machine Learning: a Survey of Case Studies.https://arxiv.org/abs/2011.09926}

Bundle field	Why it belongs in the release
Answer-model and evidence-gate identifiers	Determine model behavior
Prompt-program artifact and compile run	Tie optimized instructions and demonstrations to their source experiment
Tokenizer, prompt version, and decoding	Change the text the model sees and how it samples
Policy and corpus versions	Decide which evidence is available and when it's sufficient to serve
Serving image and schema	Change runtime behavior and API compatibility
Evaluation-suite hash and evaluator version	Declare the comparable scoring contract required before promotion

Treat the manifest as a version tuple, not as one model number. Its coordinates cover prompt state, evidence, policy, schema, evaluator, runtime, and decoding. Change one coordinate and the release identity must change, even when weights stay put.

Compatibility sits inside that tuple. A candidate can pass answer-quality checks and still fail when its input, output, or tool schema no longer matches the router. Pin those contracts, test them with the same release ID, and diagnose an interface failure separately from a model-quality regression.

We'll use two full bundles. stable is what's serving today: the hand-edited deploy-answerer-v1 template. candidate swaps in the DSPy-compiled program after development selection and an independent holdout gate. The hosted-model pin, evidence gate, ReleaseOps corpus, and eval suite stay fixed, giving us a clean comparison: if behavior changes later, the compiled prompt or the image that loads it changed.

Read the two constructors before running the cell. Which fields should differ? Candidate changes only its compile run, prompt version, and serving image; every other behavior-producing field stays pinned.

define-release-bundles.py

from dataclasses import asdict, dataclass, replace
import hashlib
import json

@dataclass(frozen=True)
class ReleaseBundle:
    service: str
    answer_model: str
    evidence_gate: str
    prompt_compile_run: str
    tokenizer: str
    prompt_version: str
    policy_version: str
    corpus_version: str
    serving_image: str
    input_schema: str
    eval_suite: str
    evaluator_version: str
    decoding: str

stable = ReleaseBundle(
    service="deploy-answerer",
    answer_model="hosted-model@2026-04-01",
    evidence_gate="releaseops-evidence-classifier-v2@sha256:41d8",
    prompt_compile_run="manual-template-v1",
    tokenizer="deploy-tokenizer@sha256:91aa",
    prompt_version="deploy-answerer-v1.json",
    policy_version="[email protected]",
    corpus_version="releaseops-records@sha256:corpus-5",
    serving_image="registry.example/deploy-answerer@sha256:image-a",
    input_schema="deploy-answer.v2",
    eval_suite="deploy-grounding-suite@sha256:suite-7",
    evaluator_version="claim-evidence-eval-v2",
    decoding="temperature=0",
)

candidate = replace(
    stable,
    prompt_compile_run="dspy-mipro-v2-run-052",
    prompt_version="compiled_a_hosted-2026-04-01.json",
    serving_image="registry.example/deploy-answerer@sha256:image-b",
)

print(f"stable_prompt={stable.prompt_version}")
print(f"candidate_prompt={candidate.prompt_version}")
print(f"candidate_compile_run={candidate.prompt_compile_run}")

Output

stable_prompt=deploy-answerer-v1.json
candidate_prompt=compiled_a_hosted-2026-04-01.json
candidate_compile_run=dspy-mipro-v2-run-052

A human-readable name such as v3 helps people find a release, but it can't prove what went into it. Before running the next cell, predict the result of changing only prompt_version: the release ID should change, because the canonical manifest includes that field.

content-address-the-release.py

def release_id(bundle: ReleaseBundle) -> str:
    payload = json.dumps(asdict(bundle), sort_keys=True, separators=(",", ":"))
    # Short prefix keeps the teaching output readable. Retain the full digest in production.
    digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()[:12]
    return f"{bundle.service}@sha256:{digest}"

stable_id = release_id(stable)
candidate_id = release_id(candidate)
prompt_patch_id = release_id(
    replace(candidate, prompt_version="compiled_a_hosted-2026-04-01.b.json")
)

print(f"stable={stable_id}")
print(f"candidate={candidate_id}")
print(f"prompt_patch={prompt_patch_id}")
print(f"prompt_patch_is_new_release={prompt_patch_id != candidate_id}")

Output

stable=deploy-answerer@sha256:32b8ed409b8e
candidate=deploy-answerer@sha256:e31f2b6d7a2c
prompt_patch=deploy-answerer@sha256:f94cb4d1892f
prompt_patch_is_new_release=True

The lab abbreviates each SHA-256 digest to 12 hexadecimal characters so the state transitions stay readable. A production registry should retain the full digest as identity and use short prefixes only for display.

Artifacts stay fixed; aliases move

A registry record stores an immutable bundle. An alias such as production or canary is a mutable pointer used by traffic. Because the record never changes, rollback only needs to move the pointer back; it doesn't need to rebuild or rewrite the bundle.

MLflow's current Model Registry workflow uses aliases and tags to organize model versions, and its documentation marks fixed Model Stages as deprecated. The distinction matters: a model version preserves history, while an alias names the version a loader should resolve now.^{[3]Reference 3Model Registry Workflows | MLflow AI Platformhttps://mlflow.org/docs/latest/ml/model-registry/workflow/}

An MLflow model alias still points to one registered model version. It doesn't automatically include the prompt, corpus, policy, schema, or serving image in this release bundle. Treat it as one component pointer, or register a wrapper artifact whose manifest resolves the complete bundle. A movable model alias isn't complete release identity.

Alias step chart over two fixed release-bundle rails: canary points to candidate release e31f2b6d7a2c before promotion, production starts on stable release 32b8ed409b8e, moves to candidate at promotion, and returns to the retained stable release after an incident; both immutable bundle records remain available throughout. — The fixed rails are stored release IDs; the colored lines are aliases. Canary reaches the candidate first, production moves only after promotion, and rollback returns production to the retained stable bundle without rewriting either manifest.

The small registry below enforces that rule. register() keeps a deep copy of each bundle, and move_alias() accepts only a registered ID. Before running it, predict the alias state: registering candidate should increase the record count while production still resolves to stable.

registry-and-aliases.py

from copy import deepcopy

class ReleaseRegistry:
    def __init__(self) -> None:
        self._bundles: dict[str, ReleaseBundle] = {}
        self._aliases: dict[str, str] = {}

    def register(self, bundle: ReleaseBundle) -> str:
        bundle_id = release_id(bundle)
        existing = self._bundles.get(bundle_id)
        if existing is not None and existing != bundle:
            raise ValueError("release digest collision")
        self._bundles[bundle_id] = deepcopy(bundle)
        return bundle_id

    def move_alias(self, alias: str, bundle_id: str) -> None:
        if bundle_id not in self._bundles:
            raise KeyError(f"unregistered release: {bundle_id}")
        self._aliases[alias] = bundle_id

    def resolve(self, alias: str) -> str:
        return self._aliases[alias]

registry = ReleaseRegistry()
assert registry.register(stable) == stable_id
assert registry.register(candidate) == candidate_id
registry.move_alias("production", stable_id)

print(f"registered={len(registry._bundles)}")
print(f"production={registry.resolve('production')}")
print(f"candidate_registered={candidate_id in registry._bundles}")

Output

registered=2
production=deploy-answerer@sha256:32b8ed409b8e
candidate_registered=True

A teaching registry can make move_alias() one assignment. A production control plane needs an atomic compare-and-set, the expected previous target, an authorization check, and an audit event. Those checks stop a stale pipeline from overwriting a newer rollout decision.

Promotion begins with controlled evidence

Registration makes a bundle addressable, not approved. Continuous delivery for machine learning adds evaluation gates and monitoring to ordinary build-and-deploy practices because a valid artifact can still produce unacceptable behavior.^{[4]Reference 4Continuous Delivery for Machine Learning.https://martinfowler.com/articles/cd4ml.html}

The gate should speak the same language as the observability, evaluation, and experiment lessons that came before it. Reuse the grounded evidence contract instead of inventing one generic quality score:

supported_evidence_f1 asks whether supported deploy claims are served correctly.
unsupported_serve_count must remain zero in the frozen high-risk suite. That's the req_205 failure: serving a deploy-approval claim ReleaseOps didn't support.
The suite hash includes the rollback slice from the DSPy holdout. A candidate that lifts the mean while missing that floor is a different experiment, not a comparable pass.
p95_latency_ms (p95 latency) keeps a behaviorally acceptable release inside its response budget.
Schema hash, evaluation-suite hash, and evaluator version keep incomparable evidence out of the decision.
evaluation_report preserves the report artifact for a reviewer or incident responder to inspect later.

Offline promotion gate matches one candidate deploy-answerer bundle to a pinned contract, passes release metrics, opens canary, and leaves production pinned. — Offline evidence counts only when release identity and evaluation contract match. Passing that gate can open canary traffic, while production stays pinned to the stable bundle.

Pause at the gate boundary. Candidate's 0.93 F1, zero unsupported serves, and 472 ms p95 all clear. Predict the two failures in the next cell: a weaker F1 should fail its threshold, and an otherwise identical report from evaluator v3 should fail comparability.

offline-promotion-gate.py

@dataclass(frozen=True)
class OfflineEvidence:
    release_id: str
    eval_suite: str
    evaluator_version: str
    input_schema: str
    evaluation_report: str
    supported_evidence_f1: float
    unsupported_serve_count: int
    p95_latency_ms: int

@dataclass(frozen=True)
class Decision:
    allowed: bool
    reason: str

def offline_gate(bundle: ReleaseBundle, evidence: OfflineEvidence) -> Decision:
    if evidence.release_id != release_id(bundle):
        return Decision(False, "evidence belongs to another release")
    if evidence.eval_suite != bundle.eval_suite:
        return Decision(False, "evaluation suite changed")
    if evidence.evaluator_version != bundle.evaluator_version:
        return Decision(False, "evaluator version changed")
    if evidence.input_schema != bundle.input_schema:
        return Decision(False, "schema mismatch")
    if not evidence.evaluation_report:
        return Decision(False, "evaluation report missing")
    if evidence.supported_evidence_f1 < 0.92:
        return Decision(False, "supported_evidence_f1 below 0.92")
    if evidence.unsupported_serve_count != 0:
        return Decision(False, "unsupported answer was served")
    if evidence.p95_latency_ms > 500:
        return Decision(False, "p95 latency exceeds 500 ms")
    return Decision(True, "offline gate passed")

candidate_offline = OfflineEvidence(
    release_id=candidate_id,
    eval_suite=candidate.eval_suite,
    evaluator_version=candidate.evaluator_version,
    input_schema=candidate.input_schema,
    evaluation_report="reports/candidate-suite-7-redacted.json",
    supported_evidence_f1=0.93,
    unsupported_serve_count=0,
    p95_latency_ms=472,
)
weaker_candidate = replace(candidate_offline, supported_evidence_f1=0.89)
changed_evaluator = replace(candidate_offline, evaluator_version="claim-evidence-eval-v3")

print(f"candidate={offline_gate(candidate, candidate_offline)}")
print(f"weak_metric={offline_gate(candidate, weaker_candidate)}")
print(f"changed_evaluator={offline_gate(candidate, changed_evaluator)}")

Output

candidate=Decision(allowed=True, reason='offline gate passed')
weak_metric=Decision(allowed=False, reason='supported_evidence_f1 below 0.92')
changed_evaluator=Decision(allowed=False, reason='evaluator version changed')

Passing the offline gate permits the next experiment. It doesn't replace production yet. Open a canary alias while production still points to the known-good bundle.

open-canary-only-after-gate.py

def open_canary(bundle: ReleaseBundle, evidence: OfflineEvidence) -> Decision:
    decision = offline_gate(bundle, evidence)
    if decision.allowed:
        registry.move_alias("canary", release_id(bundle))
    return decision

canary_decision = open_canary(candidate, candidate_offline)

print(f"canary_opened={canary_decision.allowed}")
print(f"canary={registry.resolve('canary')}")
print(f"production_still_stable={registry.resolve('production') == stable_id}")

Output

canary_opened=True
canary=deploy-answerer@sha256:e31f2b6d7a2c
production_still_stable=True

Managed models need a documented pin

When your team owns weights, a digest can identify them directly. deploy-answerer calls a hosted model, so the bundle records the strongest fixed identifier the provider documents. OpenAI's model documentation describes snapshots as a way to lock a specific version so performance and behavior remain consistent.^{[5]Reference 5Models | OpenAI APIhttps://platform.openai.com/docs/models} In this lab that's hosted-model@2026-04-01, the same pin used during the DSPy evaluation.

A provider can change the model behind a name you thought was stable, shifting outputs with no deploy on your side. Chen, Zaharia, and Zou compared March and June 2023 GPT-4 snapshots on the same tasks. With chain-of-thought prompting, accuracy on a 1,000-question prime-versus-composite set dropped from 84.0% to 51.1%. The product name stayed familiar, but the behavior didn't.^{[6]Reference 6How Is ChatGPT's Behavior Changing over Time?https://arxiv.org/abs/2307.09009} Pinning a documented snapshot reduces that risk, but it doesn't remove it. Providers can retire snapshots, so keep a golden-set monitor and treat any model-identifier change as a fresh release decision.^{[7]Reference 7Deprecations | OpenAI APIhttps://developers.openai.com/api/docs/deprecations}

That guarantee doesn't automatically transfer to every provider or alias. Check the provider's exact documentation, store the chosen identifier in the bundle, watch deprecation notices, and rerun release gates before changing it.

Replay a failed production trace against the candidate

Frozen fixtures test failures you thought to write down. A deterministic replay starts with a failure you shipped: pull its recorded trace, then run the exact prompt and tool-call sequence against the candidate while holding the evidence snapshot fixed. With the request and evidence held constant, a behavioral difference points to the release under test rather than a new request or a moved corpus.

The regression question is narrow: would the candidate have made the same mistake on this real request? Replay sits between the offline gate and shadow because it reuses recorded inputs without inventing fixtures or spending live traffic. It's read-only: recorded tool results go back in, rather than tools running again, so ReleaseOps state can't change.

Now apply that question to req_205. Historical deploy-answerer-v1.1-regression served a deploy-approval claim that admitted evidence didn't support; traffic has since returned to known-good deploy-answerer-v1. Replay the same failure against the recovered stable bundle and compiled candidate while keeping the evidence gate and recorded inputs fixed.

Before running the cell, predict the three verdicts: historical should reproduce the failure, while stable and candidate should avoid it. If stable also reproduces it, the supposed recovery isn't a valid comparison.

deterministic-replay.py

@dataclass(frozen=True)
class RecordedTrace:
    request_id: str
    origin_release: str
    prompt_version: str
    evidence_version: str
    tool_sequence: tuple[str, ...]
    evidence_supports_claim: bool
    served_unsupported_claim: bool
    temperature: float = 0.0
    seed: int = 0
    model_snapshot: str = ""

def prompt_would_serve(bundle: ReleaseBundle, evidence_supports_claim: bool) -> bool:
    # Historical v1.1 bypassed the gate; known-good v1 and the candidate don't.
    if bundle.prompt_version == "deploy-answerer-v1.1-regression.json":
        return True
    return evidence_supports_claim

def replay_against(trace: RecordedTrace, bundle: ReleaseBundle) -> dict[str, object]:
    would_serve = prompt_would_serve(bundle, trace.evidence_supports_claim)
    reproduces = would_serve and not trace.evidence_supports_claim
    return {
        "candidate_release": release_id(bundle),
        "prompt_version": bundle.prompt_version,
        "tool_sequence": trace.tool_sequence,
        "pins": (trace.temperature, trace.seed, trace.model_snapshot),
        "candidate_reproduces_failure": reproduces,
    }

historical_incident = replace(
    stable,
    prompt_compile_run="regressed-template-v1.1",
    prompt_version="deploy-answerer-v1.1-regression.json",
    serving_image="registry.example/deploy-answerer@sha256:image-regressed",
)
failed_trace = RecordedTrace(
    request_id="req_205",
    origin_release=release_id(historical_incident),
    prompt_version=historical_incident.prompt_version,
    evidence_version="releaseops-records@sha256:corpus-5",
    tool_sequence=("lookup_release", "fetch_record", "answer"),
    evidence_supports_claim=False,
    served_unsupported_claim=True,
    temperature=0.0,
    seed=7,
    model_snapshot=stable.answer_model,
)
historical_replay = replay_against(failed_trace, historical_incident)
stable_replay = replay_against(failed_trace, stable)
candidate_replay = replay_against(failed_trace, candidate)

print(f"origin_release={failed_trace.origin_release}")
print(f"historical_reproduces_failure={historical_replay['candidate_reproduces_failure']}")
print(f"stable_reproduces_failure={stable_replay['candidate_reproduces_failure']}")
print(f"replayed_against={candidate_replay['candidate_release']}")
print(f"candidate_prompt={candidate_replay['prompt_version']}")
print(f"inputs_held_fixed={candidate_replay['tool_sequence']}")
print(f"pins={candidate_replay['pins']}")
print(f"candidate_reproduces_failure={candidate_replay['candidate_reproduces_failure']}")

assert historical_replay["candidate_reproduces_failure"] is True
assert stable_replay["candidate_reproduces_failure"] is False
assert candidate_replay["candidate_reproduces_failure"] is False

Output

origin_release=deploy-answerer@sha256:efe48fbcb5a2
historical_reproduces_failure=True
stable_reproduces_failure=False
replayed_against=deploy-answerer@sha256:e31f2b6d7a2c
candidate_prompt=compiled_a_hosted-2026-04-01.json
inputs_held_fixed=('lookup_release', 'fetch_record', 'answer')
pins=(0.0, 7, 'hosted-model@2026-04-01')
candidate_reproduces_failure=False

The clean replay is evidence against this one historical failure, not a general guarantee. The toy gate reads bundle.prompt_version, so the bad historical bundle disagrees with both safe releases on the same inputs while the evidence gate stays constant. A production replay harness also needs the recorded prompt, retrieved-evidence version, tool inputs and outputs, and sampling pins on the trace (temperature, seed, and model snapshot). Without them, a "replay" quietly becomes a fresh run whose difference you can't attribute. Keep a growing library of failed traces and require future candidates to clear each past production mistake before promotion.

Shadow evaluation must be read-only

Replay gives us one known failure. Offline fixtures still can't cover every real request shape. A shadow sends a sanitized copy of a production request to the candidate while the stable release alone supplies the user-visible answer. It reveals latency or evidence-support problems without exposing candidate text to customers.

The safety boundary is strict: shadow traffic can't execute tools, send messages, change ReleaseOps state, or write production state. Its output is evaluation data only. The envelope below also redacts a ship identifier before queuing the request.

Before running it, inspect the flags you expect to stay false: side effects must be disabled and candidate output must stay hidden. The candidate may abstain where stable served, and that disagreement is useful evidence rather than a user-visible incident.

shadow-envelope.py

import re

@dataclass(frozen=True)
class ShadowEnvelope:
    candidate_release: str
    stable_release: str
    sanitized_text: str
    side_effects_enabled: bool
    response_visible_to_user: bool
    evidence_supports_claim: bool

@dataclass(frozen=True)
class ShadowComparison:
    route_agreement: bool
    candidate_claim_safe: bool
    candidate_would_serve: bool
    stable_route: str
    candidate_route: str
    dropped: bool

def make_shadow(text: str, evidence_supports_claim: bool) -> ShadowEnvelope:
    sanitized = re.sub(r"SHIP-\d+", "[SHIP_ID]", text)
    return ShadowEnvelope(
        candidate_release=registry.resolve("canary"),
        stable_release=registry.resolve("production"),
        sanitized_text=sanitized,
        side_effects_enabled=False,
        response_visible_to_user=False,
        evidence_supports_claim=evidence_supports_claim,
    )

def compare_shadow(envelope: ShadowEnvelope, queue_full: bool = False) -> ShadowComparison:
    if queue_full or envelope.side_effects_enabled:
        return ShadowComparison(False, False, False, "dropped", "dropped", True)
    # Stable already answered the user; shadow only scores the candidate copy.
    stable_route = "serve"  # production response already left the system
    candidate_bundle = (
        candidate if envelope.candidate_release == candidate_id else stable
    )
    candidate_would_serve = prompt_would_serve(
        candidate_bundle, envelope.evidence_supports_claim
    )
    candidate_route = "serve" if candidate_would_serve else "abstain"
    route_agreement = candidate_route == stable_route
    candidate_claim_safe = candidate_route == "abstain" or envelope.evidence_supports_claim
    return ShadowComparison(
        route_agreement=route_agreement,
        candidate_claim_safe=candidate_claim_safe,
        candidate_would_serve=candidate_would_serve,
        stable_route=stable_route,
        candidate_route=candidate_route,
        dropped=False,
    )

shadow = make_shadow(
    "Is payment-tests approved for SHIP-4412 without a ReleaseOps record?",
    evidence_supports_claim=False,
)
comparison = compare_shadow(shadow)
dropped = compare_shadow(shadow, queue_full=True)

print(f"shadow_text={shadow.sanitized_text}")
print(f"candidate={shadow.candidate_release}")
print(f"side_effects_enabled={shadow.side_effects_enabled}")
print(f"response_visible={shadow.response_visible_to_user}")
print(f"candidate_route={comparison.candidate_route}")
print(f"route_agreement={comparison.route_agreement}")
print(f"candidate_claim_safe={comparison.candidate_claim_safe}")
print(f"dropped_when_queue_full={dropped.dropped}")

assert comparison.candidate_would_serve is False
assert comparison.route_agreement is False
assert comparison.candidate_claim_safe is True
assert dropped.dropped is True

Output

shadow_text=Is payment-tests approved for [SHIP_ID] without a ReleaseOps record?
candidate=deploy-answerer@sha256:e31f2b6d7a2c
side_effects_enabled=False
response_visible=False
candidate_route=abstain
route_agreement=False
candidate_claim_safe=True
dropped_when_queue_full=True

The one-pattern redaction is a teaching fixture. A real shadow path needs schema-aware data minimization that covers every customer identifier and secret before the request reaches a queue, log, or candidate service.

At the service boundary, put the envelope on a bounded worker queue and count dropped or failed comparisons. The compare step is the other half of shadow: after the candidate answers the sanitized copy, record route agreement, claim safety, and latency against the stable path. Feed shadow_drop_rate from dropped or timed-out comparisons. Don't start an untracked background task in request scope and assume its evaluation record will survive a process restart.

side_effects_enabled=False in this teaching envelope is metadata, not an authorization boundary. The worker still needs a read-only tool allowlist and credentials that can't write production state. Reject a shadow request that asks for a side effect.

Canary traffic is visible and sticky

Shadow results can justify limited exposure, not automatic promotion. A canary sends a small share of real conversations to the candidate and returns those responses to users. That changes the safety problem: one conversation must remain on one bundle throughout the rollout. Otherwise adjacent turns can receive conflicting answers from stable and candidate.

Use deterministic hashing of a conversation ID to choose new conversations consistently across workers. Python's built-in hash() is process-dependent, so it can't provide that bucket. Hashing alone isn't enough: when exposure widens from 1% to 10%, a higher threshold could move an existing conversation from stable to candidate. Persist the first resolved release ID for the conversation lifetime.

Predict the widening case before running the router. A thread assigned to stable at 10% should still resolve to stable after the threshold becomes 100%; an abort must be able to override that assignment.

sticky-canary-routing.py

def bucket(conversation_id: str) -> int:
    digest = hashlib.sha256(conversation_id.encode("utf-8")).hexdigest()
    return int(digest[:8], 16) % 100

conversation_assignments: dict[str, str] = {}
aborted_releases: set[str] = set()

def assigned_release(conversation_id: str, canary_percent: int) -> str:
    existing = conversation_assignments.get(conversation_id)
    if existing is not None and existing not in aborted_releases:
        return existing
    alias = "canary" if bucket(conversation_id) < canary_percent else "production"
    bundle_id = registry.resolve(alias)
    if bundle_id in aborted_releases:
        bundle_id = registry.resolve("production")
    conversation_assignments[conversation_id] = bundle_id
    return bundle_id

canary_thread = next(
    f"thread-{index}" for index in range(1000) if bucket(f"thread-{index}") < 10
)
assignments = [assigned_release(canary_thread, canary_percent=10) for _ in range(3)]
stable_thread = next(
    f"thread-{index}" for index in range(1000) if bucket(f"thread-{index}") >= 10
)
stable_before_widen = assigned_release(stable_thread, canary_percent=10)
stable_after_widen = assigned_release(stable_thread, canary_percent=100)

print(f"canary_thread={canary_thread}")
print(f"bucket={bucket(canary_thread)}")
print(f"same_release_each_turn={len(set(assignments)) == 1}")
print(f"assigned_to_candidate={assignments[0] == candidate_id}")
print(f"existing_stable_thread_pinned_after_widen={stable_before_widen == stable_after_widen == stable_id}")

Output

canary_thread=thread-6
bucket=6
same_release_each_turn=True
assigned_to_candidate=True
existing_stable_thread_pinned_after_widen=True

The dictionary is a teaching fixture. A real router persists assignments in conversation state or a rollout store, writes the first assignment atomically so concurrent opening turns can't disagree, expires it when the conversation ends, and records the resolved release ID in traces. Stickiness governs normal routing, not recovery: an abort must override it for a failed candidate.

Bar chart of live unsupported serves for deploy-answerer canary windows: the 1 percent window has 0 of 1,200, the 10 percent window has 112 of 8,000 (1.4 percent), which trips the zero-tolerance gate and aborts while production stays on the stable release. — The 1% window records 0 unsupported serves in 1,200 requests. At 10%, 112 of 8,000 requests are unsupported (1.4% against a required 0%), so the controller aborts and leaves production on the stable bundle.

A canary window answers a new question

Offline evidence covers frozen examples. A live window asks a different question: how does candidate behave under real traffic mix, serving latency, error rate, and evidence pressure? Set thresholds and a minimum sample size before sending candidate traffic. A rate from a handful of requests isn't enough to widen exposure. For deploy-answerer, support is a hard safety invariant, so one live unsupported serve fails the window instead of disappearing inside an error budget.

Read the two windows in the figure as a decision, not a scorecard. The 1% sample clears this contract; the 10% sample fails even though its latency and error rate pass. Zero unsupported serves is the controlling condition.

live-canary-window.py

@dataclass(frozen=True)
class LiveWindow:
    name: str
    request_count: int
    p95_latency_ms: int
    error_rate: float
    unsupported_serve_rate: float
    shadow_drop_rate: float

def live_gate(window: LiveWindow) -> Decision:
    if window.request_count < 1_000:
        return Decision(False, "fewer than 1000 requests observed")
    if window.p95_latency_ms > 550:
        return Decision(False, "live latency exceeded 550 ms")
    if window.error_rate > 0.01:
        return Decision(False, "error rate exceeded 1%")
    if window.unsupported_serve_rate > 0.0:
        return Decision(False, "unsupported serve rate must remain zero")
    if window.shadow_drop_rate > 0.02:
        return Decision(False, "shadow telemetry incomplete")
    return Decision(True, "live window passed")

window_1_percent = LiveWindow("1%", 1_200, 481, 0.002, 0.000, 0.001)
window_10_percent = LiveWindow("10%", 8_000, 493, 0.003, 0.014, 0.001)

print(f"one_percent={live_gate(window_1_percent)}")
print(f"ten_percent={live_gate(window_10_percent)}")
print(
    "ten_percent_unsupported_count="
    f"{round(window_10_percent.unsupported_serve_rate * window_10_percent.request_count)}"
)

Output

one_percent=Decision(allowed=True, reason='live window passed')
ten_percent=Decision(allowed=False, reason='unsupported serve rate must remain zero')
ten_percent_unsupported_count=112

Both windows exceed the 1,000-request minimum, so this teaching contract can issue a verdict for each. A low-volume window would remain blocked even if every observed rate happened to be zero.

Use the failed signal to choose the next inspection. A request that can't parse points toward schema or serving-image compatibility. A parsed request with an unsupported answer belongs in trace replay against the pinned evidence snapshot. A safe but slow candidate points toward runtime capacity or decoding, while missing telemetry is a reason to stop exposure because the gate can't prove safety.

Abort a canary; roll back a promotion

The right recovery verb depends on which alias moved:

If candidate fails while only canary receives traffic, abort the rollout. production never moved.
If candidate was promoted and fails later, roll back by repointing production to the retained stable release.

Check the alias state before naming the incident. That distinction keeps a report from calling an unpromoted canary failure a production rollback.

In the next drill, canary failure should set exposure to zero and restore the canary pointer. The separate promoted-release drill should move production to candidate, detect failure, then return it to stable.

abort-and-rollback.py

canary_percent = 10
failed_window = live_gate(window_10_percent)
if not failed_window.allowed:
    aborted_releases.add(candidate_id)
    canary_percent = 0
    # Zero traffic isn't enough: any consumer of registry.resolve("canary") must leave the failed ID.
    registry.move_alias("canary", stable_id)

print(f"canary_percent_after_abort={canary_percent}")
print(f"canary_alias_after_abort={registry.resolve('canary')}")
print(f"production_after_abort={registry.resolve('production')}")
print(f"pinned_canary_thread_restored_stable={assigned_release(canary_thread, canary_percent) == stable_id}")

assert registry.resolve("canary") == stable_id
assert registry.resolve("production") == stable_id

# Separate rollback drill: a promoted candidate later regresses at wider traffic.
rollback_drill = deepcopy(registry)
previous_production = rollback_drill.resolve("production")
rollback_drill.move_alias("production", candidate_id)
post_promotion_incident = replace(window_10_percent, name="100%")

if not live_gate(post_promotion_incident).allowed:
    rollback_drill.move_alias("production", previous_production)

print(f"drill_production_after_rollback={rollback_drill.resolve('production')}")
print(f"drill_restored_stable={rollback_drill.resolve('production') == stable_id}")
print(f"actual_production_unchanged={registry.resolve('production') == stable_id}")

Output

canary_percent_after_abort=0
canary_alias_after_abort=deploy-answerer@sha256:32b8ed409b8e
production_after_abort=deploy-answerer@sha256:32b8ed409b8e
pinned_canary_thread_restored_stable=True
drill_production_after_rollback=deploy-answerer@sha256:32b8ed409b8e
drill_restored_stable=True
actual_production_unchanged=True

Real progressive-delivery controllers encode the same mechanics. Argo Rollouts supports weighted canary steps and pauses, and its analysis can abort an unsuccessful rollout. With traffic routing, keeping the stable replica set available lets traffic move back immediately on abort, at the cost of extra capacity during rollout.^{[8]Reference 8Argo Rollouts - Kubernetes Progressive Delivery Controllerhttps://argoproj.github.io/argo-rollouts/}

Record decisions so a later engineer can reconstruct them

A pipeline that moves aliases without recording why still creates mystery during an incident. Store release IDs, evidence references, gate verdicts, rollout windows, actor or controller identity, decision timestamp, and final alias state as append-only events.

The event stream below records candidate's rejection after its 10% window serves unsupported deploy-approval claims. The failed release remains addressable for diagnosis, while traffic stays with the known-good bundle. That pairing matters: recovery changes routing, while evidence preserves the path that led to the decision.

release-decision-record.py

@dataclass(frozen=True)
class ReleaseEvent:
    stage: str
    release_id: str
    decision: str
    evidence: str

events = (
    ReleaseEvent("register", candidate_id, "RECORDED", "manifest_sha"),
    ReleaseEvent("offline_gate", candidate_id, "PASSED", candidate_offline.evaluation_report),
    ReleaseEvent("canary_1_percent", candidate_id, "PASSED", "live:window-001"),
    ReleaseEvent("canary_10_percent", candidate_id, "ABORTED", "live:window-010"),
    ReleaseEvent("production", stable_id, "UNCHANGED", "rollback:not-needed"),
)

for event in events:
    print(f"{event.stage}:{event.decision}:{event.evidence}")
print("release_decision=REJECT_CANDIDATE_AFTER_CANARY_REGRESSION")
print(f"active_production={registry.resolve('production')}")

Output

register:RECORDED:manifest_sha
offline_gate:PASSED:reports/candidate-suite-7-redacted.json
canary_1_percent:PASSED:live:window-001
canary_10_percent:ABORTED:live:window-010
production:UNCHANGED:rollback:not-needed
release_decision=REJECT_CANDIDATE_AFTER_CANARY_REGRESSION
active_production=deploy-answerer@sha256:32b8ed409b8e

Where the lab maps onto a live stack

The lab uses plain Python so the state transitions stay visible. A deployed stack usually splits the same responsibilities:

Responsibility	Production form
Store immutable model or component version	Artifact store plus model registry
Store prompt, policy, corpus, tokenizer, decoding, schema, and evaluator pins	Release manifest in source control or deployment registry
Move candidate/production pointers	Registry aliases, deployment config, or feature flags limited to registered releases
Run offline evidence gates	CI job tied to exact manifest digest with a retained report artifact
Shift live traffic and pause on regressions	Progressive-delivery controller and metric analysis
Reconstruct impact	Request trace logs with resolved release ID and rollout event log

Feature flags remain useful, but their values must resolve to registered immutable release IDs. A flag that points at an arbitrary model name makes rapid changes easy and incident reconstruction impossible.

Keep one invariant as you ship: every request, gate, and alias event names the same immutable release ID. Then a failed answer can be replayed, a canary can be stopped, and production can return to known-good without guessing which bundle was active.

Next Step

Continue to Semantic Caching & Cost Optimization

You now know that every response must resolve to an exact release bundle. Next you'll reuse prior answers only when cache scope and invalidation include that same release, plus the corpus, tenant, and freshness constraints that produced them.

PreviousPrompt Optimization with DSPy

Share this article