LearnApplied LLM EngineeringModel Versioning & Deployment

⚙️MediumMLOps & Deployment

Model Versioning & Deployment

Turn an evaluated LLM change into an immutable release bundle, promote it through measured traffic, and roll back without losing lineage.

16 min read

Learning path

Step 71 of 155 in the full curriculum

Mixed Precision Training Semantic Caching & Cost Optimization

Model Versioning & Deployment

The previous lesson ended with a precision candidate blocked until a target-GPU profile could prove its speed and stability. Suppose that missing profile now passes. You still don't have something safe to deploy. A live answer can change because of weights, a support gate, a prompt, a tokenizer, a policy, a container image, or a schema.

For the delivery-support system you've been building, deployment means answering one exact question: which complete, evaluated release produced this response, and how quickly can traffic return to the last known good release?

Release bundle history showing pinned gate, prompt, and runtime snapshots beside a production alias that moves for promotion or rollback. — A deployable release is a fixed bundle of behavior-producing components. Production changes by moving a traffic pointer to a different bundle, not by editing a deployed bundle in place.

A release is more than model weights

The service now contains an answer model plus the carrier-evidence-classifier-v2 gate trained in the previous lesson. That gate blocks answers whose delivery claim isn't supported by a carrier scan. If you roll back only its weights but leave a newer prompt or policy active, you haven't restored previous behavior.

This is the release-bundle idea: store every input that can change visible output or operational safety in one immutable manifest. ML systems accumulate hidden dependencies between data, code, configuration, and serving infrastructure; deployment records need to make those dependencies reviewable.^[1]^[2]

Bundle field	Why it belongs in the release
Answer-model and evidence-gate identifiers	Determine model behavior
Training run and precision policy	Explain how the new gate was produced
Tokenizer and prompt version	Change the text the model sees
Policy version	Decides when evidence is sufficient to serve
Serving image and schema	Change runtime behavior and API compatibility
Evaluation-suite hash	Identifies the test set that justified promotion

Start with two full bundles. stable represents what's serving today. candidate changes the evidence gate after the profiled BF16 training run has passed outside the previous lesson's blocked fixture.

define-release-bundles.py

from dataclasses import asdict, dataclass, replace
import hashlib
import json

@dataclass(frozen=True)
class ReleaseBundle:
    service: str
    answer_model: str
    evidence_gate: str
    gate_training_run: str
    precision_policy: str
    tokenizer: str
    prompt_version: str
    policy_version: str
    serving_image: str
    input_schema: str
    eval_suite: str

stable = ReleaseBundle(
    service="delivery-evidence-answerer",
    answer_model="answer-model@sha256:3b07",
    evidence_gate="carrier-evidence-classifier-v1@sha256:0f12",
    gate_training_run="run_fp32_baseline",
    precision_policy="fp32",
    tokenizer="delivery-tokenizer@sha256:91aa",
    prompt_version="[email protected]",
    policy_version="[email protected]",
    serving_image="registry.example/answerer@sha256:image-a",
    input_schema="delivery-answer.v2",
    eval_suite="delivery-grounding-suite@sha256:suite-7",
)

candidate = replace(
    stable,
    evidence_gate="carrier-evidence-classifier-v2@sha256:41d8",
    gate_training_run="run_bf16_profiled_target_gpu",
    precision_policy="bf16",
    prompt_version="[email protected]",
    serving_image="registry.example/answerer@sha256:image-b",
)

print(f"stable_gate={stable.evidence_gate}")
print(f"candidate_gate={candidate.evidence_gate}")
print(f"candidate_training_run={candidate.gate_training_run}")
print(f"candidate_precision={candidate.precision_policy}")

Output

stable_gate=carrier-evidence-classifier-v1@sha256:0f12
candidate_gate=carrier-evidence-classifier-v2@sha256:41d8
candidate_training_run=run_bf16_profiled_target_gpu
candidate_precision=bf16

A version name such as v2 is useful for people, but it doesn't prove contents stayed fixed. The next cell derives identity from the canonical manifest. Change a prompt or image digest and the release ID changes too.

content-address-the-release.py

def release_id(bundle: ReleaseBundle) -> str:
    payload = json.dumps(asdict(bundle), sort_keys=True, separators=(",", ":"))
    # Short prefix keeps the teaching output readable. Retain the full digest in production.
    digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()[:12]
    return f"{bundle.service}@sha256:{digest}"

stable_id = release_id(stable)
candidate_id = release_id(candidate)
prompt_patch_id = release_id(replace(candidate, prompt_version="[email protected]"))

print(f"stable={stable_id}")
print(f"candidate={candidate_id}")
print(f"prompt_patch={prompt_patch_id}")
print(f"prompt_patch_is_new_release={prompt_patch_id != candidate_id}")

Output

stable=delivery-evidence-answerer@sha256:df2d4fe7b0c5
candidate=delivery-evidence-answerer@sha256:d9c7ecff4a9c
prompt_patch=delivery-evidence-answerer@sha256:9a1e41a39815
prompt_patch_is_new_release=True

This lab abbreviates each SHA-256 digest to 12 hexadecimal characters so the state transitions stay readable. A production registry should retain the full digest as identity and use short prefixes only for display.

Artifacts stay fixed; aliases move

A registry record stores an immutable bundle. An alias such as production or canary is a mutable pointer used by traffic. This separation is what makes rollback simple: preserve both bundles and move the pointer back.

MLflow's current Model Registry workflow provides version aliases and tags, and its documentation marks fixed Model Stages as deprecated. That distinction matters: a model version is history; an alias expresses the current deployment decision.^[3]

Immutable release bundles on left, deployment aliases on right, and promotion or rollback moving aliases without rewriting stored manifests. — Both stable and candidate releases remain available for audit. Promotion and rollback update named pointers while the stored manifests remain unchanged.

The small registry below enforces that rule. register() keeps a deep copy of the bundle, and move_alias() only points at a registered ID.

registry-and-aliases.py

from copy import deepcopy

class ReleaseRegistry:
    def __init__(self) -> None:
        self._bundles: dict[str, ReleaseBundle] = {}
        self._aliases: dict[str, str] = {}

    def register(self, bundle: ReleaseBundle) -> str:
        bundle_id = release_id(bundle)
        existing = self._bundles.get(bundle_id)
        if existing is not None and existing != bundle:
            raise ValueError("release digest collision")
        self._bundles[bundle_id] = deepcopy(bundle)
        return bundle_id

    def move_alias(self, alias: str, bundle_id: str) -> None:
        if bundle_id not in self._bundles:
            raise KeyError(f"unregistered release: {bundle_id}")
        self._aliases[alias] = bundle_id

    def resolve(self, alias: str) -> str:
        return self._aliases[alias]

registry = ReleaseRegistry()
assert registry.register(stable) == stable_id
assert registry.register(candidate) == candidate_id
registry.move_alias("production", stable_id)

print(f"registered={len(registry._bundles)}")
print(f"production={registry.resolve('production')}")
print(f"candidate_registered={candidate_id in registry._bundles}")

Output

registered=2
production=delivery-evidence-answerer@sha256:df2d4fe7b0c5
candidate_registered=True

Promotion begins with controlled evidence

Registration isn't approval. Continuous delivery for machine learning adds evaluation and monitoring to ordinary build-and-deploy practices because a valid artifact can still produce unacceptable behavior.^[4]

In this running system, don't introduce a generic "quality score" after spending several lessons defining a grounded support contract. The release gate should use the same measurements the incident, evaluation, and experiment lessons established:

supported_evidence_f1 measures whether supported delivery claims are served correctly.
unsupported_serve_count must remain zero in the frozen high-risk suite.
p95_latency_ms prevents a behaviorally acceptable gate from breaking the response budget.
Schema and evaluation-suite hashes prevent incomparable evidence from entering the decision.

Release ladder from immutable delivery bundle through offline evidence and limited live exposure to promotion or rollback. — Build the immutable candidate once, then gather progressively more realistic evidence. A release doesn't earn live traffic until its controlled gate passes.

offline-promotion-gate.py

@dataclass(frozen=True)
class OfflineEvidence:
    release_id: str
    eval_suite: str
    input_schema: str
    supported_evidence_f1: float
    unsupported_serve_count: int
    p95_latency_ms: int

@dataclass(frozen=True)
class Decision:
    allowed: bool
    reason: str

def offline_gate(bundle: ReleaseBundle, evidence: OfflineEvidence) -> Decision:
    if evidence.release_id != release_id(bundle):
        return Decision(False, "evidence belongs to another release")
    if evidence.eval_suite != bundle.eval_suite:
        return Decision(False, "evaluation suite changed")
    if evidence.input_schema != bundle.input_schema:
        return Decision(False, "schema mismatch")
    if evidence.supported_evidence_f1 < 0.92:
        return Decision(False, "supported_evidence_f1 below 0.92")
    if evidence.unsupported_serve_count != 0:
        return Decision(False, "unsupported answer was served")
    if evidence.p95_latency_ms > 500:
        return Decision(False, "p95 latency exceeds 500 ms")
    return Decision(True, "offline gate passed")

candidate_offline = OfflineEvidence(
    release_id=candidate_id,
    eval_suite=candidate.eval_suite,
    input_schema=candidate.input_schema,
    supported_evidence_f1=0.93,
    unsupported_serve_count=0,
    p95_latency_ms=472,
)
weaker_candidate = replace(candidate_offline, supported_evidence_f1=0.89)

print(f"candidate={offline_gate(candidate, candidate_offline)}")
print(f"weak_metric={offline_gate(candidate, weaker_candidate)}")

Output

candidate=Decision(allowed=True, reason='offline gate passed')
weak_metric=Decision(allowed=False, reason='supported_evidence_f1 below 0.92')

Passing the offline gate permits further evaluation; it doesn't immediately replace production. Open a canary alias while production still points to the known-good bundle.

open-canary-only-after-gate.py

def open_canary(bundle: ReleaseBundle, evidence: OfflineEvidence) -> Decision:
    decision = offline_gate(bundle, evidence)
    if decision.allowed:
        registry.move_alias("canary", release_id(bundle))
    return decision

canary_decision = open_canary(candidate, candidate_offline)

print(f"canary_opened={canary_decision.allowed}")
print(f"canary={registry.resolve('canary')}")
print(f"production_still_stable={registry.resolve('production') == stable_id}")

Output

canary_opened=True
canary=delivery-evidence-answerer@sha256:d9c7ecff4a9c
production_still_stable=True

Managed models need a documented pin

When your team owns weights, a digest can identify them directly. If a service calls a hosted model, the bundle must instead record the strongest fixed identifier that provider documents. For example, OpenAI model pages that expose snapshots describe snapshots as locking a particular model version for consistent behavior.^[5]

This matters because a provider can change the model behind a name you thought was stable, shifting your outputs with no deploy on your side. A controlled study comparing the March and June 2023 releases of GPT-4 and GPT-3.5 reported large behavior swings on the same tasks between snapshots, so the "same" service was not always the same service.^[6] Pinning a documented snapshot reduces that risk but doesn't remove it: snapshots get deprecated, and a golden-set monitor against production still earns its keep.^[7]

Don't generalize that guarantee to every provider or every alias. Verify the exact provider documentation, store the chosen model identifier in the bundle, monitor deprecation notices, and rerun release gates before changing it.

Shadow evaluation must be read-only

Offline fixtures can't cover every real request shape. A shadow sends a sanitized copy of a production request to the candidate while the stable release alone supplies the user-visible answer. It can reveal latency or evidence-support problems without exposing the candidate's text to customers.

The critical safety rule is easy to miss: a shadow isn't permitted to execute tools, send messages, change orders, or write customer state. Its output is evaluation data only. The envelope below also redacts an order identifier before queuing the shadow request.

shadow-envelope.py

import re

@dataclass(frozen=True)
class ShadowEnvelope:
    candidate_release: str
    sanitized_text: str
    side_effects_enabled: bool
    response_visible_to_user: bool

def make_shadow(text: str) -> ShadowEnvelope:
    sanitized = re.sub(r"ORD-\d+", "[ORDER_ID]", text)
    return ShadowEnvelope(
        candidate_release=registry.resolve("canary"),
        sanitized_text=sanitized,
        side_effects_enabled=False,
        response_visible_to_user=False,
    )

shadow = make_shadow("Will ORD-48192 arrive Friday based on scan S3?")

print(f"shadow_text={shadow.sanitized_text}")
print(f"candidate={shadow.candidate_release}")
print(f"side_effects_enabled={shadow.side_effects_enabled}")
print(f"response_visible={shadow.response_visible_to_user}")

Output

shadow_text=Will [ORDER_ID] arrive Friday based on scan S3?
candidate=delivery-evidence-answerer@sha256:d9c7ecff4a9c
side_effects_enabled=False
response_visible=False

In a deployed service, enqueue this envelope to a bounded worker queue and count dropped or failed comparisons. Don't start an untracked background task in request scope and assume its evaluation record will survive process restarts.

The boolean in this teaching envelope is metadata, not an authorization boundary. Its worker still needs a read-only tool allowlist and credentials that can't write production state. Reject a shadow request if it asks for a side effect.

Canary traffic is visible and sticky

Shadow results can justify limited exposure, not automatic promotion. A canary sends a small share of real conversations to the candidate and returns those candidate responses to users. For conversational systems, one thread must remain on one bundle throughout the rollout. Otherwise a user can receive conflicting answers from stable and candidate releases in adjacent turns.

Use deterministic hashing of a conversation ID to choose new conversations reproducibly across workers. Python's built-in hash() is intentionally process-dependent, so it's the wrong bucketing function for that job. Hashing alone isn't enough, though: when canary traffic widens from 1% to 10%, the higher threshold could move an existing conversation from stable to candidate. Persist the first resolved release ID for the conversation lifetime.

sticky-canary-routing.py

def bucket(conversation_id: str) -> int:
    digest = hashlib.sha256(conversation_id.encode("utf-8")).hexdigest()
    return int(digest[:8], 16) % 100

conversation_assignments: dict[str, str] = {}
aborted_releases: set[str] = set()

def assigned_release(conversation_id: str, canary_percent: int) -> str:
    existing = conversation_assignments.get(conversation_id)
    if existing is not None and existing not in aborted_releases:
        return existing
    alias = "canary" if bucket(conversation_id) < canary_percent else "production"
    bundle_id = registry.resolve(alias)
    if bundle_id in aborted_releases:
        bundle_id = registry.resolve("production")
    conversation_assignments[conversation_id] = bundle_id
    return bundle_id

canary_thread = next(
    f"thread-{index}" for index in range(1000) if bucket(f"thread-{index}") < 10
)
assignments = [assigned_release(canary_thread, canary_percent=10) for _ in range(3)]
stable_thread = next(
    f"thread-{index}" for index in range(1000) if bucket(f"thread-{index}") >= 10
)
stable_before_widen = assigned_release(stable_thread, canary_percent=10)
stable_after_widen = assigned_release(stable_thread, canary_percent=100)

print(f"canary_thread={canary_thread}")
print(f"bucket={bucket(canary_thread)}")
print(f"same_release_each_turn={len(set(assignments)) == 1}")
print(f"assigned_to_candidate={assignments[0] == candidate_id}")
print(f"existing_stable_thread_pinned_after_widen={stable_before_widen == stable_after_widen == stable_id}")

Output

canary_thread=thread-6
bucket=6
same_release_each_turn=True
assigned_to_candidate=True
existing_stable_thread_pinned_after_widen=True

The dictionary is a teaching fixture. A real router persists assignments in conversation state or a rollout store, expires them when each conversation ends, and records the resolved release ID in traces. Stickiness is normal-routing behavior, not permission to keep serving a failed candidate: an abort must override it.

Offline evidence gate feeding read-only shadow, sticky canary, production promotion, and rollback when support metrics regress. — Canary traffic is real exposure. Widen traffic only while monitored windows pass, and keep the stable release available for an immediate abort or rollback.

A canary window answers a new question

Offline evidence proves behavior on frozen examples. A live window tests traffic mix, serving latency, error rate, and evidence failures under actual request volume. Define those thresholds before sending any candidate traffic.

live-canary-window.py

@dataclass(frozen=True)
class LiveWindow:
    name: str
    p95_latency_ms: int
    error_rate: float
    unsupported_serve_rate: float
    shadow_drop_rate: float

def live_gate(window: LiveWindow) -> Decision:
    if window.p95_latency_ms > 550:
        return Decision(False, "live latency exceeded 550 ms")
    if window.error_rate > 0.01:
        return Decision(False, "error rate exceeded 1%")
    if window.unsupported_serve_rate > 0.005:
        return Decision(False, "unsupported serve rate exceeded 0.5%")
    if window.shadow_drop_rate > 0.02:
        return Decision(False, "shadow telemetry incomplete")
    return Decision(True, "live window passed")

window_1_percent = LiveWindow("1%", 481, 0.002, 0.000, 0.001)
window_10_percent = LiveWindow("10%", 493, 0.003, 0.014, 0.001)

print(f"one_percent={live_gate(window_1_percent)}")
print(f"ten_percent={live_gate(window_10_percent)}")

Output

one_percent=Decision(allowed=True, reason='live window passed')
ten_percent=Decision(allowed=False, reason='unsupported serve rate exceeded 0.5%')

Abort a canary; roll back a promotion

These actions sound similar but refer to different alias states:

If the candidate fails while only the canary alias receives traffic, abort the rollout. production never moved.
If a candidate was already promoted and later fails, roll back by repointing production to the retained stable release.

Keeping the distinction explicit prevents an incident report from claiming production was rolled back when the candidate was never production.

abort-and-rollback.py

canary_percent = 10
failed_window = live_gate(window_10_percent)
if not failed_window.allowed:
    aborted_releases.add(candidate_id)
    canary_percent = 0

print(f"canary_percent_after_abort={canary_percent}")
print(f"production_after_abort={registry.resolve('production')}")
print(f"pinned_canary_thread_restored_stable={assigned_release(canary_thread, canary_percent) == stable_id}")

# Separate rollback drill: a promoted candidate later regresses at wider traffic.
rollback_drill = deepcopy(registry)
previous_production = rollback_drill.resolve("production")
rollback_drill.move_alias("production", candidate_id)
post_promotion_incident = replace(window_10_percent, name="100%")

if not live_gate(post_promotion_incident).allowed:
    rollback_drill.move_alias("production", previous_production)

print(f"drill_production_after_rollback={rollback_drill.resolve('production')}")
print(f"drill_restored_stable={rollback_drill.resolve('production') == stable_id}")
print(f"actual_production_unchanged={registry.resolve('production') == stable_id}")

Output

canary_percent_after_abort=0
production_after_abort=delivery-evidence-answerer@sha256:df2d4fe7b0c5
pinned_canary_thread_restored_stable=True
drill_production_after_rollback=delivery-evidence-answerer@sha256:df2d4fe7b0c5
drill_restored_stable=True
actual_production_unchanged=True

Real progressive-delivery controllers encode the same operational mechanics. Argo Rollouts supports weighted canary steps and pauses; its background analysis can abort an unsuccessful rollout. With traffic routing, keeping the stable replica set available allows traffic to move back immediately on abort, at the cost of additional capacity during rollout.^[8]

Record the decision a later engineer can reconstruct

A pipeline that moves aliases but doesn't persist why it moved them still creates mystery during an incident. Store the release IDs, evidence references, gate verdicts, rollout windows, and final alias state as append-only events.

The candidate below is correctly rejected for production after its 10% canary window serves unsupported delivery claims. The release isn't deleted. It remains available for diagnosis, while traffic stays with the known-good bundle.

release-decision-record.py

@dataclass(frozen=True)
class ReleaseEvent:
    stage: str
    release_id: str
    decision: str
    evidence: str

events = (
    ReleaseEvent("register", candidate_id, "RECORDED", "manifest_sha"),
    ReleaseEvent("offline_gate", candidate_id, "PASSED", "offline:suite-7"),
    ReleaseEvent("canary_1_percent", candidate_id, "PASSED", "live:window-001"),
    ReleaseEvent("canary_10_percent", candidate_id, "ABORTED", "live:window-010"),
    ReleaseEvent("production", stable_id, "UNCHANGED", "rollback:not-needed"),
)

for event in events:
    print(f"{event.stage}:{event.decision}:{event.evidence}")
print("release_decision=REJECT_CANDIDATE_AFTER_CANARY_REGRESSION")
print(f"active_production={registry.resolve('production')}")

Output

register:RECORDED:manifest_sha
offline_gate:PASSED:offline:suite-7
canary_1_percent:PASSED:live:window-001
canary_10_percent:ABORTED:live:window-010
production:UNCHANGED:rollback:not-needed
release_decision=REJECT_CANDIDATE_AFTER_CANARY_REGRESSION
active_production=delivery-evidence-answerer@sha256:df2d4fe7b0c5

Where this maps to production systems

The lab deliberately uses plain Python so the state transitions are visible. A deployed stack usually splits the same responsibilities:

Responsibility	Production form
Store immutable model or component version	Artifact store plus model registry
Store prompt, policy, tokenizer, and schema pins	Release manifest in source control or deployment registry
Move candidate/production pointers	Registry aliases, deployment config, or feature flags limited to registered releases
Run offline evidence gates	CI job tied to exact manifest digest
Shift live traffic and pause on regressions	Progressive-delivery controller and metric analysis
Reconstruct impact	Request trace logs with resolved release ID and rollout event log

Feature flags remain useful, but their values must resolve to registered immutable release IDs. A flag that points at an arbitrary model name makes rapid changes easy and incident reconstruction impossible.

Mastery check

Key concepts

Release bundle: all pinned components that determine served behavior and operational compatibility.
Release ID: a content-derived identity for one immutable bundle.
Alias: a movable traffic pointer such as production or canary.
Offline gate: comparable controlled evidence required before any user exposure.
Shadow: read-only candidate evaluation on sanitized production-shaped traffic.
Canary: limited user-visible candidate traffic with predefined live thresholds.
Abort versus rollback: stop a not-yet-promoted candidate versus restore production after promotion.

Practice tasks

Add corpus_version to ReleaseBundle. Show that a new policy-document snapshot creates a different release ID even if weights and prompts remain fixed.
Add a gate that blocks candidates with a missing gate_training_run. Explain why this prevents a registered model from escaping experiment lineage.
Add a rollback_ready check that refuses canary exposure unless the stable serving target is healthy and warm.
Write a release trace for a candidate that passes at 1% and 10%, is promoted, then rolls back after a latency incident at 100%.

Evaluation rubric

Foundational: Explains why weights alone don't identify served behavior and names the prompt, policy, tokenizer, runtime, schema, and evidence suite as bundle dependencies.
Foundational: Distinguishes an immutable release ID from mutable production and canary aliases.
Intermediate: Implements a controlled offline gate that compares the candidate against the declared support and latency contract.
Intermediate: Explains why shadow execution is read-only and why canary routing must be sticky by conversation.
Advanced: Distinguishes a canary abort from a production rollback using alias state and retained evidence.
Advanced: Produces an append-only decision record from which an incident reviewer can reconstruct what was exposed and why traffic stayed or moved.

Self-check questions

Common pitfalls

Only weights are versioned

Symptom: A rollback restores model files, but answers still differ from the last known good release. Cause: Prompt, policy, tokenizer, schema, or serving image continued to float. Fix: Content-address the complete release bundle and log its resolved ID per request.

Offline metrics change between candidates

Symptom: Candidate appears better, but its score came from a newer or easier evaluation suite. Cause: Promotion compared different evidence contracts. Fix: Pin eval_suite in the bundle and reject evidence produced for another suite.

Shadow evaluation performs actions

Symptom: A candidate that users never saw still duplicates a workflow action or touches customer state. Cause: Shadow execution reused the live tool path instead of a read-only evaluation path. Fix: Disable side effects in the shadow envelope, sanitize payloads, and monitor dropped comparisons.

Conversation routing flickers

Symptom: One customer thread switches between answers during canary rollout. Cause: Request-level randomness, process-local hashing, or a widening threshold reassigns an existing thread. Fix: Use deterministic bucketing for new conversations, then persist the first resolved release ID for each conversation lifetime.

Rollback target is cold

Symptom: The pointer moves back immediately, but recovery still produces timeouts. Cause: The stable deployment was scaled down or unloaded too early. Fix: Keep stable capacity ready until the candidate passes burn-in, then test rollback in a drill.

Next Step

Continue to Semantic Caching & Cost Optimization

You now know that every response must resolve to an exact release bundle. Next you will reuse prior responses safely by making cache scope and invalidation depend on the model, prompt, policy, corpus, and tenant context that produced them.

PreviousMixed Precision Training

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Hidden Technical Debt in Machine Learning Systems.

Sculley et al. · 2015

Challenges in Deploying Machine Learning: a Survey of Case Studies.

Paleyes, A., Urma, R. G., & Lawrence, N. D. · 2022 · ACM Computing Surveys

Model Registry Workflows | MLflow AI Platform

MLflow · 2026

Continuous Delivery for Machine Learning.

Sato, D., Wider, A., & Windheuser, C. · 2019

Models | OpenAI API

OpenAI · 2026

How Is ChatGPT's Behavior Changing over Time?

Chen, L., Zaharia, M., & Zou, J. · 2023

Deprecations | OpenAI API

OpenAI · 2026

Argo Rollouts - Kubernetes Progressive Delivery Controller

Argo Project · 2026

Back to Topics

LearnApplied LLM EngineeringModel Versioning & Deployment

⚙️MediumMLOps & Deployment

Model Versioning & Deployment

Turn an evaluated LLM change into an immutable release bundle, promote it through measured traffic, and roll back without losing lineage.

16 min read

Learning path

Step 71 of 155 in the full curriculum

Mixed Precision Training Semantic Caching & Cost Optimization

Model Versioning & Deployment

A release is more than model weights

Bundle field	Why it belongs in the release
Answer-model and evidence-gate identifiers	Determine model behavior
Training run and precision policy	Explain how the new gate was produced
Tokenizer and prompt version	Change the text the model sees
Policy version	Decides when evidence is sufficient to serve
Serving image and schema	Change runtime behavior and API compatibility
Evaluation-suite hash	Identifies the test set that justified promotion

define-release-bundles.py

from dataclasses import asdict, dataclass, replace
import hashlib
import json

@dataclass(frozen=True)
class ReleaseBundle:
    service: str
    answer_model: str
    evidence_gate: str
    gate_training_run: str
    precision_policy: str
    tokenizer: str
    prompt_version: str
    policy_version: str
    serving_image: str
    input_schema: str
    eval_suite: str

stable = ReleaseBundle(
    service="delivery-evidence-answerer",
    answer_model="answer-model@sha256:3b07",
    evidence_gate="carrier-evidence-classifier-v1@sha256:0f12",
    gate_training_run="run_fp32_baseline",
    precision_policy="fp32",
    tokenizer="delivery-tokenizer@sha256:91aa",
    prompt_version="[email protected]",
    policy_version="[email protected]",
    serving_image="registry.example/answerer@sha256:image-a",
    input_schema="delivery-answer.v2",
    eval_suite="delivery-grounding-suite@sha256:suite-7",
)

candidate = replace(
    stable,
    evidence_gate="carrier-evidence-classifier-v2@sha256:41d8",
    gate_training_run="run_bf16_profiled_target_gpu",
    precision_policy="bf16",
    prompt_version="[email protected]",
    serving_image="registry.example/answerer@sha256:image-b",
)

print(f"stable_gate={stable.evidence_gate}")
print(f"candidate_gate={candidate.evidence_gate}")
print(f"candidate_training_run={candidate.gate_training_run}")
print(f"candidate_precision={candidate.precision_policy}")

Output

stable_gate=carrier-evidence-classifier-v1@sha256:0f12
candidate_gate=carrier-evidence-classifier-v2@sha256:41d8
candidate_training_run=run_bf16_profiled_target_gpu
candidate_precision=bf16

content-address-the-release.py

def release_id(bundle: ReleaseBundle) -> str:
    payload = json.dumps(asdict(bundle), sort_keys=True, separators=(",", ":"))
    # Short prefix keeps the teaching output readable. Retain the full digest in production.
    digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()[:12]
    return f"{bundle.service}@sha256:{digest}"

stable_id = release_id(stable)
candidate_id = release_id(candidate)
prompt_patch_id = release_id(replace(candidate, prompt_version="[email protected]"))

print(f"stable={stable_id}")
print(f"candidate={candidate_id}")
print(f"prompt_patch={prompt_patch_id}")
print(f"prompt_patch_is_new_release={prompt_patch_id != candidate_id}")

Output

stable=delivery-evidence-answerer@sha256:df2d4fe7b0c5
candidate=delivery-evidence-answerer@sha256:d9c7ecff4a9c
prompt_patch=delivery-evidence-answerer@sha256:9a1e41a39815
prompt_patch_is_new_release=True

Artifacts stay fixed; aliases move

The small registry below enforces that rule. register() keeps a deep copy of the bundle, and move_alias() only points at a registered ID.

registry-and-aliases.py

from copy import deepcopy

class ReleaseRegistry:
    def __init__(self) -> None:
        self._bundles: dict[str, ReleaseBundle] = {}
        self._aliases: dict[str, str] = {}

    def register(self, bundle: ReleaseBundle) -> str:
        bundle_id = release_id(bundle)
        existing = self._bundles.get(bundle_id)
        if existing is not None and existing != bundle:
            raise ValueError("release digest collision")
        self._bundles[bundle_id] = deepcopy(bundle)
        return bundle_id

    def move_alias(self, alias: str, bundle_id: str) -> None:
        if bundle_id not in self._bundles:
            raise KeyError(f"unregistered release: {bundle_id}")
        self._aliases[alias] = bundle_id

    def resolve(self, alias: str) -> str:
        return self._aliases[alias]

registry = ReleaseRegistry()
assert registry.register(stable) == stable_id
assert registry.register(candidate) == candidate_id
registry.move_alias("production", stable_id)

print(f"registered={len(registry._bundles)}")
print(f"production={registry.resolve('production')}")
print(f"candidate_registered={candidate_id in registry._bundles}")

Output

registered=2
production=delivery-evidence-answerer@sha256:df2d4fe7b0c5
candidate_registered=True

Promotion begins with controlled evidence

supported_evidence_f1 measures whether supported delivery claims are served correctly.
unsupported_serve_count must remain zero in the frozen high-risk suite.
p95_latency_ms prevents a behaviorally acceptable gate from breaking the response budget.
Schema and evaluation-suite hashes prevent incomparable evidence from entering the decision.

offline-promotion-gate.py

@dataclass(frozen=True)
class OfflineEvidence:
    release_id: str
    eval_suite: str
    input_schema: str
    supported_evidence_f1: float
    unsupported_serve_count: int
    p95_latency_ms: int

@dataclass(frozen=True)
class Decision:
    allowed: bool
    reason: str

def offline_gate(bundle: ReleaseBundle, evidence: OfflineEvidence) -> Decision:
    if evidence.release_id != release_id(bundle):
        return Decision(False, "evidence belongs to another release")
    if evidence.eval_suite != bundle.eval_suite:
        return Decision(False, "evaluation suite changed")
    if evidence.input_schema != bundle.input_schema:
        return Decision(False, "schema mismatch")
    if evidence.supported_evidence_f1 < 0.92:
        return Decision(False, "supported_evidence_f1 below 0.92")
    if evidence.unsupported_serve_count != 0:
        return Decision(False, "unsupported answer was served")
    if evidence.p95_latency_ms > 500:
        return Decision(False, "p95 latency exceeds 500 ms")
    return Decision(True, "offline gate passed")

candidate_offline = OfflineEvidence(
    release_id=candidate_id,
    eval_suite=candidate.eval_suite,
    input_schema=candidate.input_schema,
    supported_evidence_f1=0.93,
    unsupported_serve_count=0,
    p95_latency_ms=472,
)
weaker_candidate = replace(candidate_offline, supported_evidence_f1=0.89)

print(f"candidate={offline_gate(candidate, candidate_offline)}")
print(f"weak_metric={offline_gate(candidate, weaker_candidate)}")

Output

candidate=Decision(allowed=True, reason='offline gate passed')
weak_metric=Decision(allowed=False, reason='supported_evidence_f1 below 0.92')

Passing the offline gate permits further evaluation; it doesn't immediately replace production. Open a canary alias while production still points to the known-good bundle.

open-canary-only-after-gate.py

def open_canary(bundle: ReleaseBundle, evidence: OfflineEvidence) -> Decision:
    decision = offline_gate(bundle, evidence)
    if decision.allowed:
        registry.move_alias("canary", release_id(bundle))
    return decision

canary_decision = open_canary(candidate, candidate_offline)

print(f"canary_opened={canary_decision.allowed}")
print(f"canary={registry.resolve('canary')}")
print(f"production_still_stable={registry.resolve('production') == stable_id}")

Output

canary_opened=True
canary=delivery-evidence-answerer@sha256:d9c7ecff4a9c
production_still_stable=True

Managed models need a documented pin

Shadow evaluation must be read-only

shadow-envelope.py

import re

@dataclass(frozen=True)
class ShadowEnvelope:
    candidate_release: str
    sanitized_text: str
    side_effects_enabled: bool
    response_visible_to_user: bool

def make_shadow(text: str) -> ShadowEnvelope:
    sanitized = re.sub(r"ORD-\d+", "[ORDER_ID]", text)
    return ShadowEnvelope(
        candidate_release=registry.resolve("canary"),
        sanitized_text=sanitized,
        side_effects_enabled=False,
        response_visible_to_user=False,
    )

shadow = make_shadow("Will ORD-48192 arrive Friday based on scan S3?")

print(f"shadow_text={shadow.sanitized_text}")
print(f"candidate={shadow.candidate_release}")
print(f"side_effects_enabled={shadow.side_effects_enabled}")
print(f"response_visible={shadow.response_visible_to_user}")

Output

shadow_text=Will [ORDER_ID] arrive Friday based on scan S3?
candidate=delivery-evidence-answerer@sha256:d9c7ecff4a9c
side_effects_enabled=False
response_visible=False

Canary traffic is visible and sticky

sticky-canary-routing.py

def bucket(conversation_id: str) -> int:
    digest = hashlib.sha256(conversation_id.encode("utf-8")).hexdigest()
    return int(digest[:8], 16) % 100

conversation_assignments: dict[str, str] = {}
aborted_releases: set[str] = set()

def assigned_release(conversation_id: str, canary_percent: int) -> str:
    existing = conversation_assignments.get(conversation_id)
    if existing is not None and existing not in aborted_releases:
        return existing
    alias = "canary" if bucket(conversation_id) < canary_percent else "production"
    bundle_id = registry.resolve(alias)
    if bundle_id in aborted_releases:
        bundle_id = registry.resolve("production")
    conversation_assignments[conversation_id] = bundle_id
    return bundle_id

canary_thread = next(
    f"thread-{index}" for index in range(1000) if bucket(f"thread-{index}") < 10
)
assignments = [assigned_release(canary_thread, canary_percent=10) for _ in range(3)]
stable_thread = next(
    f"thread-{index}" for index in range(1000) if bucket(f"thread-{index}") >= 10
)
stable_before_widen = assigned_release(stable_thread, canary_percent=10)
stable_after_widen = assigned_release(stable_thread, canary_percent=100)

print(f"canary_thread={canary_thread}")
print(f"bucket={bucket(canary_thread)}")
print(f"same_release_each_turn={len(set(assignments)) == 1}")
print(f"assigned_to_candidate={assignments[0] == candidate_id}")
print(f"existing_stable_thread_pinned_after_widen={stable_before_widen == stable_after_widen == stable_id}")

Output

canary_thread=thread-6
bucket=6
same_release_each_turn=True
assigned_to_candidate=True
existing_stable_thread_pinned_after_widen=True

A canary window answers a new question

live-canary-window.py

@dataclass(frozen=True)
class LiveWindow:
    name: str
    p95_latency_ms: int
    error_rate: float
    unsupported_serve_rate: float
    shadow_drop_rate: float

def live_gate(window: LiveWindow) -> Decision:
    if window.p95_latency_ms > 550:
        return Decision(False, "live latency exceeded 550 ms")
    if window.error_rate > 0.01:
        return Decision(False, "error rate exceeded 1%")
    if window.unsupported_serve_rate > 0.005:
        return Decision(False, "unsupported serve rate exceeded 0.5%")
    if window.shadow_drop_rate > 0.02:
        return Decision(False, "shadow telemetry incomplete")
    return Decision(True, "live window passed")

window_1_percent = LiveWindow("1%", 481, 0.002, 0.000, 0.001)
window_10_percent = LiveWindow("10%", 493, 0.003, 0.014, 0.001)

print(f"one_percent={live_gate(window_1_percent)}")
print(f"ten_percent={live_gate(window_10_percent)}")

Output

one_percent=Decision(allowed=True, reason='live window passed')
ten_percent=Decision(allowed=False, reason='unsupported serve rate exceeded 0.5%')

Abort a canary; roll back a promotion

These actions sound similar but refer to different alias states:

If the candidate fails while only the canary alias receives traffic, abort the rollout. production never moved.
If a candidate was already promoted and later fails, roll back by repointing production to the retained stable release.

Keeping the distinction explicit prevents an incident report from claiming production was rolled back when the candidate was never production.

abort-and-rollback.py

canary_percent = 10
failed_window = live_gate(window_10_percent)
if not failed_window.allowed:
    aborted_releases.add(candidate_id)
    canary_percent = 0

print(f"canary_percent_after_abort={canary_percent}")
print(f"production_after_abort={registry.resolve('production')}")
print(f"pinned_canary_thread_restored_stable={assigned_release(canary_thread, canary_percent) == stable_id}")

# Separate rollback drill: a promoted candidate later regresses at wider traffic.
rollback_drill = deepcopy(registry)
previous_production = rollback_drill.resolve("production")
rollback_drill.move_alias("production", candidate_id)
post_promotion_incident = replace(window_10_percent, name="100%")

if not live_gate(post_promotion_incident).allowed:
    rollback_drill.move_alias("production", previous_production)

print(f"drill_production_after_rollback={rollback_drill.resolve('production')}")
print(f"drill_restored_stable={rollback_drill.resolve('production') == stable_id}")
print(f"actual_production_unchanged={registry.resolve('production') == stable_id}")

Output

canary_percent_after_abort=0
production_after_abort=delivery-evidence-answerer@sha256:df2d4fe7b0c5
pinned_canary_thread_restored_stable=True
drill_production_after_rollback=delivery-evidence-answerer@sha256:df2d4fe7b0c5
drill_restored_stable=True
actual_production_unchanged=True

Record the decision a later engineer can reconstruct

release-decision-record.py

@dataclass(frozen=True)
class ReleaseEvent:
    stage: str
    release_id: str
    decision: str
    evidence: str

events = (
    ReleaseEvent("register", candidate_id, "RECORDED", "manifest_sha"),
    ReleaseEvent("offline_gate", candidate_id, "PASSED", "offline:suite-7"),
    ReleaseEvent("canary_1_percent", candidate_id, "PASSED", "live:window-001"),
    ReleaseEvent("canary_10_percent", candidate_id, "ABORTED", "live:window-010"),
    ReleaseEvent("production", stable_id, "UNCHANGED", "rollback:not-needed"),
)

for event in events:
    print(f"{event.stage}:{event.decision}:{event.evidence}")
print("release_decision=REJECT_CANDIDATE_AFTER_CANARY_REGRESSION")
print(f"active_production={registry.resolve('production')}")

Output

register:RECORDED:manifest_sha
offline_gate:PASSED:offline:suite-7
canary_1_percent:PASSED:live:window-001
canary_10_percent:ABORTED:live:window-010
production:UNCHANGED:rollback:not-needed
release_decision=REJECT_CANDIDATE_AFTER_CANARY_REGRESSION
active_production=delivery-evidence-answerer@sha256:df2d4fe7b0c5

Where this maps to production systems

The lab deliberately uses plain Python so the state transitions are visible. A deployed stack usually splits the same responsibilities:

Responsibility	Production form
Store immutable model or component version	Artifact store plus model registry
Store prompt, policy, tokenizer, and schema pins	Release manifest in source control or deployment registry
Move candidate/production pointers	Registry aliases, deployment config, or feature flags limited to registered releases
Run offline evidence gates	CI job tied to exact manifest digest
Shift live traffic and pause on regressions	Progressive-delivery controller and metric analysis
Reconstruct impact	Request trace logs with resolved release ID and rollout event log

Mastery check

Key concepts

Release bundle: all pinned components that determine served behavior and operational compatibility.
Release ID: a content-derived identity for one immutable bundle.
Alias: a movable traffic pointer such as production or canary.
Offline gate: comparable controlled evidence required before any user exposure.
Shadow: read-only candidate evaluation on sanitized production-shaped traffic.
Canary: limited user-visible candidate traffic with predefined live thresholds.
Abort versus rollback: stop a not-yet-promoted candidate versus restore production after promotion.

Practice tasks

Add corpus_version to ReleaseBundle. Show that a new policy-document snapshot creates a different release ID even if weights and prompts remain fixed.
Add a gate that blocks candidates with a missing gate_training_run. Explain why this prevents a registered model from escaping experiment lineage.
Add a rollback_ready check that refuses canary exposure unless the stable serving target is healthy and warm.
Write a release trace for a candidate that passes at 1% and 10%, is promoted, then rolls back after a latency incident at 100%.

Evaluation rubric

Foundational: Explains why weights alone don't identify served behavior and names the prompt, policy, tokenizer, runtime, schema, and evidence suite as bundle dependencies.
Foundational: Distinguishes an immutable release ID from mutable production and canary aliases.
Intermediate: Implements a controlled offline gate that compares the candidate against the declared support and latency contract.
Intermediate: Explains why shadow execution is read-only and why canary routing must be sticky by conversation.
Advanced: Distinguishes a canary abort from a production rollback using alias state and retained evidence.
Advanced: Produces an append-only decision record from which an incident reviewer can reconstruct what was exposed and why traffic stayed or moved.

Self-check questions

Common pitfalls

Only weights are versioned

Offline metrics change between candidates

Shadow evaluation performs actions

Conversation routing flickers

Rollback target is cold

Next Step

Continue to Semantic Caching & Cost Optimization

PreviousMixed Precision Training

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Hidden Technical Debt in Machine Learning Systems.

Sculley et al. · 2015

Challenges in Deploying Machine Learning: a Survey of Case Studies.

Paleyes, A., Urma, R. G., & Lawrence, N. D. · 2022 · ACM Computing Surveys

Model Registry Workflows | MLflow AI Platform

MLflow · 2026

Continuous Delivery for Machine Learning.

Sato, D., Wider, A., & Windheuser, C. · 2019

Models | OpenAI API

OpenAI · 2026

How Is ChatGPT's Behavior Changing over Time?

Chen, L., Zaharia, M., & Zou, J. · 2023

Deprecations | OpenAI API

OpenAI · 2026

Argo Rollouts - Kubernetes Progressive Delivery Controller

Argo Project · 2026

Model Versioning & Deployment

Model Versioning & Deployment

A release is more than model weights

Why should a one-line prompt correction produce a new release ID?

Artifacts stay fixed; aliases move

Promotion begins with controlled evidence

Managed models need a documented pin

Shadow evaluation must be read-only

Canary traffic is visible and sticky

A canary window answers a new question

Abort a canary; roll back a promotion

Why might keeping the stable deployment warm during a canary be worth its GPU cost?

Record the decision a later engineer can reconstruct

Where this maps to production systems

Mastery check

Key concepts

Practice tasks

Evaluation rubric

Self-check questions

The model weights are unchanged, but a serving-image digest changes. Is that a new release?

An offline suite passes, then the 10% canary serves unsupported delivery claims. Should production be marked rolled back?

Why is a floating hosted-model alias risky inside an otherwise immutable release bundle?

Common pitfalls

Only weights are versioned

Offline metrics change between candidates

Shadow evaluation performs actions

Conversation routing flickers

Rollback target is cold

Model Versioning & Deployment

Model Versioning & Deployment

A release is more than model weights

Why should a one-line prompt correction produce a new release ID?

Artifacts stay fixed; aliases move

Promotion begins with controlled evidence

Managed models need a documented pin

Shadow evaluation must be read-only

Canary traffic is visible and sticky

A canary window answers a new question

Abort a canary; roll back a promotion

Why might keeping the stable deployment warm during a canary be worth its GPU cost?

Record the decision a later engineer can reconstruct

Where this maps to production systems

Mastery check

Key concepts

Practice tasks

Evaluation rubric

Self-check questions

The model weights are unchanged, but a serving-image digest changes. Is that a new release?

An offline suite passes, then the 10% canary serves unsupported delivery claims. Should production be marked rolled back?

Why is a floating hosted-model alias risky inside an otherwise immutable release bundle?

Common pitfalls

Only weights are versioned

Offline metrics change between candidates

Shadow evaluation performs actions

Conversation routing flickers

Rollback target is cold