Model Versioning & Deployment

A precision candidate can stay blocked until a target-GPU profile proves speed and stability. Suppose that missing profile now passes. You still don't have something safe to deploy. A live answer can change because of weights, an evidence gate, a prompt, a tokenizer, a policy corpus, a container image, or a schema.

For the incident-response assistant you've been building, deployment means answering one exact question: which complete, evaluated release produced this response, and how quickly can traffic return to the last known good release?

A release is more than model weights

The service now contains an answer model plus the runbook-evidence-classifier-v2 gate trained in the previous lesson. That gate blocks answers whose incident claim isn't supported by a runbook excerpt. If you roll back only its weights but leave a newer prompt, policy, or corpus active, you haven't restored previous behavior.

This is the release-bundle idea: store every input that can change visible output or operational safety in one immutable manifest. Keep the declared evaluation contract there too, then attach the resulting report artifact to the promotion decision. ML systems accumulate hidden dependencies between data, code, configuration, and serving infrastructure; deployment records need to make those dependencies reviewable.^{[1]Reference 1Hidden Technical Debt in Machine Learning Systems.https://research.google/pubs/hidden-technical-debt-in-machine-learning-systems/[2]Reference 2Challenges in Deploying Machine Learning: a Survey of Case Studies.https://arxiv.org/abs/2011.09926}

Start with two full bundles. stable represents what's serving today. candidate changes the evidence gate after the profiled BF16 training run has passed outside the previous lesson's blocked fixture.

A version name such as v2 is useful for people, but it doesn't prove contents stayed fixed. The next cell derives identity from the canonical manifest. Change a prompt or image digest and the release ID changes too.

This lab abbreviates each SHA-256 digest to 12 hexadecimal characters so the state transitions stay readable. A production registry should retain the full digest as identity and use short prefixes only for display.

Artifacts stay fixed; aliases move

A registry record stores an immutable bundle. An alias such as production or canary is a mutable pointer used by traffic. This separation is what makes rollback simple: preserve both bundles and move the pointer back.

MLflow's current Model Registry workflow provides version aliases and tags, and its documentation marks fixed Model Stages as deprecated. That distinction matters: a model version is history; an alias expresses the current deployment decision.^{[3]Reference 3Model Registry Workflows | MLflow AI Platformhttps://mlflow.org/docs/latest/ml/model-registry/workflow/}

An MLflow model alias still points to one registered model version, not automatically to the prompt, corpus, policy, schema, and serving image in the release bundle. Treat it as one component pointer, or register a wrapper artifact whose manifest resolves the complete bundle. Don't mistake a movable model alias for complete release identity.

The small registry below enforces that rule. register() keeps a deep copy of the bundle, and move_alias() only points at a registered ID.

The teaching registry keeps move_alias() deliberately small. A production control plane should make that update atomic, record the expected previous target, and reject a stale promotion if another rollout moved the alias first.

Promotion begins with controlled evidence

Registration isn't approval. Continuous delivery for machine learning adds evaluation gates and monitoring to ordinary build-and-deploy practices because a valid artifact can still produce unacceptable behavior.^{[4]Reference 4Continuous Delivery for Machine Learning.https://martinfowler.com/articles/cd4ml.html}

In this running system, don't introduce a generic "quality score" after spending several lessons defining a grounded evidence contract. The release gate should use the same measurements the incident, evaluation, and experiment lessons established:

• supported_evidence_f1 measures whether supported incident claims are served correctly.
• unsupported_serve_count must remain zero in the frozen high-risk suite.
• p95_latency_ms (p95 latency) prevents a behaviorally acceptable gate from breaking the response budget.
• Schema hash, evaluation-suite hash, and evaluator version prevent incomparable evidence from entering the decision.
• evaluation_report preserves the report artifact a reviewer or incident responder can inspect later.

Passing the offline gate permits further evaluation; it doesn't immediately replace production. Open a canary alias while production still points to the known-good bundle.

Managed models need a documented pin

When your team owns weights, a digest can identify them directly. If a service calls a hosted model, the bundle must instead record the strongest fixed identifier that provider documents. For example, OpenAI model pages that expose snapshots describe snapshots as locking a particular model version for consistent behavior.^{[5]Reference 5Models | OpenAI APIhttps://platform.openai.com/docs/models}

A provider can change the model behind a name you thought was stable, shifting your outputs with no deploy on your side. A controlled study comparing the March and June 2023 releases of GPT-4 and GPT-3.5 reported large behavior swings on the same tasks between snapshots, so the "same" service was not always the same service.^{[6]Reference 6How Is ChatGPT's Behavior Changing over Time?https://arxiv.org/abs/2307.09009} Pinning a documented snapshot reduces that risk but doesn't remove it: snapshots get deprecated, and a golden-set monitor against production still earns its keep.^{[7]Reference 7Deprecations | OpenAI APIhttps://developers.openai.com/api/docs/deprecations}

Don't generalize that guarantee to every provider or every alias. Verify the exact provider documentation, store the chosen model identifier in the bundle, monitor deprecation notices, and rerun release gates before changing it.

Replay a failed production trace against the candidate

Frozen fixtures test the failures you already imagined. A deterministic replay tests a failure you actually shipped: pull the recorded trace for a request that went wrong under the current release, then re-run its exact prompt and tool-call sequence against the candidate release while holding the recorded evidence snapshot fixed. The only thing that varies is the release under test, so a difference in behavior is attributable to the candidate, not to a new request or a moved corpus.

This answers a precise regression question: would the candidate have made the same mistake on this real request? It sits between the offline gate and the shadow because it reuses recorded inputs rather than inventing fixtures or spending live traffic. The replay is read-only: it feeds recorded tool results back in rather than re-executing tools, so no incident state changes.

The stable release served a claim unsupported by admitted evidence. Replay that trace against the candidate's evidence gate:

A clean replay is real evidence that the candidate fixes this specific incident, but it isn't a general guarantee. A production replay harness needs the recorded prompt, retrieved evidence version, tool inputs and outputs, and any non-deterministic settings pinned (temperature, seed, and model snapshot), or the "replay" quietly becomes a fresh run whose difference you can't attribute. Keep a growing library of failed traces and add each new incident to it, so a future candidate must clear every past production mistake before promotion.

Shadow evaluation must be read-only

Offline fixtures can't cover every real request shape. A shadow sends a sanitized copy of a production request to the candidate while the stable release alone supplies the user-visible answer. It can reveal latency or evidence-support problems without exposing the candidate's text to customers.

The safety rule is easy to miss: a shadow isn't permitted to execute tools, send messages, change incident state, or write production state. Its output is evaluation data only. The envelope below also redacts an incident identifier before queuing the shadow request.

The one-pattern redaction is only a teaching fixture. A production shadow path needs schema-aware data minimization that covers every customer identifier and secret before the request reaches a queue, log, or candidate service.

In a deployed service, enqueue this envelope to a bounded worker queue and count dropped or failed comparisons. Don't start an untracked background task in request scope and assume its evaluation record will survive process restarts.

The boolean in this teaching envelope is metadata, not an authorization boundary. Its worker still needs a read-only tool allowlist and credentials that can't write production state. Reject a shadow request if it asks for a side effect.

Canary traffic is visible and sticky

Shadow results can justify limited exposure, not automatic promotion. A canary sends a small share of real conversations to the candidate and returns those candidate responses to users. For conversational systems, one thread must remain on one bundle throughout the rollout. Otherwise a user can receive conflicting answers from stable and candidate releases in adjacent turns.

Use deterministic hashing of a conversation ID to choose new conversations reproducibly across workers. Python's built-in hash() is intentionally process-dependent, so it's the wrong bucketing function for that job. Hashing alone isn't enough, though: when canary traffic widens from 1% to 10%, the higher threshold could move an existing conversation from stable to candidate. Persist the first resolved release ID for the conversation lifetime.

The dictionary is a teaching fixture. A real router persists assignments in conversation state or a rollout store, writes the first assignment atomically so concurrent opening turns can't disagree, expires it when the conversation ends, and records the resolved release ID in traces. Stickiness is normal-routing behavior, not permission to keep serving a failed candidate: an abort must override it.

A canary window answers a new question

Offline evidence proves behavior on frozen examples. A live window tests traffic mix, serving latency, error rate, and evidence failures under actual request volume. Define those thresholds and a minimum sample size before sending any candidate traffic. A rate computed from a handful of requests isn't enough evidence to widen exposure. For this incident assistant, support is a hard safety invariant: one live unsupported serve fails the window rather than being absorbed into an error budget.

Both windows exceed the 1,000-request minimum, so the metric verdicts are meaningful under this teaching contract. A low-volume window would remain blocked even if every observed rate happened to be zero.

Abort a canary; roll back a promotion

These actions sound similar but refer to different alias states:

• If the candidate fails while only the canary alias receives traffic, abort the rollout. production never moved.
• If a candidate was already promoted and later fails, roll back by repointing production to the retained stable release.

Keeping the distinction explicit prevents an incident report from claiming production was rolled back when the candidate was never production.

Real progressive-delivery controllers encode the same operational mechanics. Argo Rollouts supports weighted canary steps and pauses; its background analysis can abort an unsuccessful rollout. With traffic routing, keeping the stable replica set available allows traffic to move back immediately on abort, at the cost of additional capacity during rollout.^{[8]Reference 8Argo Rollouts - Kubernetes Progressive Delivery Controllerhttps://argoproj.github.io/argo-rollouts/}

Record the decision a later engineer can reconstruct

A pipeline that moves aliases but doesn't persist why it moved them still creates mystery during an incident. Store the release IDs, evidence references, gate verdicts, rollout windows, actor or controller identity, decision timestamp, and final alias state as append-only events.

The candidate below is correctly rejected for production after its 10% canary window serves unsupported incident claims. The release isn't deleted. It remains available for diagnosis, while traffic stays with the known-good bundle.

Production mapping

The lab deliberately uses plain Python so the state transitions are visible. A deployed stack usually splits the same responsibilities:

Feature flags remain useful, but their values must resolve to registered immutable release IDs. A flag that points at an arbitrary model name makes rapid changes easy and incident reconstruction impossible.

Mastery check

Key concepts

• Release bundle: all pinned components that determine served behavior and operational compatibility, plus the declared evaluation contract.
• Release ID: a content-derived identity for one immutable bundle.
• Alias: a movable traffic pointer such as production or canary.
• Offline gate: comparable controlled evidence required before any user exposure.
• Shadow: read-only candidate evaluation on sanitized production-shaped traffic.
• Canary: limited user-visible candidate traffic with predefined live thresholds.
• Abort versus rollback: stop a not-yet-promoted candidate versus restore production after promotion.

Practice tasks

1. Change corpus_version in ReleaseBundle. Show that a new policy-document snapshot creates a different release ID even if weights and prompts remain fixed.
2. Add a gate that blocks candidates with a missing gate_training_run. Explain why this prevents a registered model from escaping experiment lineage.
3. Add a rollback_ready check that refuses canary exposure unless the stable serving target is healthy, warm, and compatible with the active request schema.
4. Write a release trace for a candidate that passes at 1% and 10%, is promoted, then rolls back after a latency incident at 100%.
5. Build a small library of recorded failed traces and require a candidate to pass each replay as part of the offline gate before shadow or canary exposure. Explain why pinning temperature, seed, and model snapshot is what makes the replay a replay.

Evaluation rubric

• Foundational: Explains why weights alone don't identify served behavior and names the prompt, policy, corpus, tokenizer, runtime, schema, evaluation suite, and evaluator version as bundle dependencies.
• Foundational: Distinguishes an immutable release ID from mutable production and canary aliases.
• Intermediate: Implements controlled offline and live gates that compare the candidate against declared support, latency, telemetry, and minimum-volume requirements.
• Intermediate: Explains why shadow execution is read-only and why canary routing must be sticky by conversation.
• Advanced: Distinguishes a canary abort from a production rollback using alias state and retained evidence.
• Advanced: Produces an append-only decision record from which an incident reviewer can reconstruct what was exposed and why traffic stayed or moved.

Self-check questions

Common pitfalls

Only weights are versioned

• Symptom: A rollback restores model files, but answers still differ from the last known good release.
• Cause: Prompt, policy, corpus, tokenizer, schema, or serving image continued to float.
• Fix: Content-address the complete release bundle and log its resolved ID per request.

Offline evidence changes between candidates

• Symptom: Candidate appears better, but its score came from a newer suite, a changed evaluator, or a missing report artifact.
• Cause: Promotion compared different evidence contracts or discarded the evidence a reviewer needs to inspect.
• Fix: Pin eval_suite and evaluator_version in the bundle, then retain the exact evaluation_report artifact used for promotion.

Shadow evaluation performs actions

• Symptom: A candidate that users never saw still duplicates a workflow action or touches customer state.
• Cause: Shadow execution reused the live tool path instead of a read-only evaluation path.
• Fix: Disable side effects in the shadow envelope, sanitize payloads, and monitor dropped comparisons.

Conversation routing flickers

• Symptom: One customer thread switches between answers during canary rollout.
• Cause: Request-level randomness, process-local hashing, or a widening threshold reassigns an existing thread.
• Fix: Use deterministic bucketing for new conversations, then persist the first resolved release ID for each conversation lifetime.

Rollback target is cold

• Symptom: The pointer moves back immediately, but recovery still produces timeouts.
• Cause: The stable deployment was scaled down or unloaded too early.
• Fix: Keep schema-compatible stable capacity ready until the candidate passes burn-in, then test rollback in a drill.

Canary evidence is too sparse

• Symptom: A tiny canary window reports zero errors and is promoted before it has seen enough traffic to reveal rare failures.
• Cause: The live gate checked rates but didn't require a minimum request count or complete observation window.
• Fix: Declare minimum volume and duration before exposure, then keep the rollout paused until both requirements and every metric threshold pass.

A "replay" is actually a fresh run

• Symptom: Replaying a failed production trace against the candidate gives a different answer each time, so it can't confirm a fix.
• Cause: The prompt, retrieved evidence version, tool outputs, temperature, seed, or model snapshot weren't pinned, so the replay re-derived inputs instead of reusing them.
• Fix: Record and feed back the exact inputs and non-deterministic settings, and keep a growing trace library the candidate must clear before promotion.