Capstone: Production ML Pipeline

Training finishing isn't permission to change live behavior. A reviewer has to trace any live decision back to data, features, model, policy, and promotion evidence. That means stored receipts, not a mutable Boolean that happens to say passed.

The vision service now has a versioned bundle, a recorded rollback target, and specialist labels that arrive after the photo route. The ETA, ranking, and forecast capstones left the same kind of artifact: frozen data, a metric that can wait on delayed outcomes, and a previous release you could restore. None of those chapters owned the shared controller. This lab is that controller.

Define the shared release tuple

Models differ, but their release manifest can share a schema:

When something breaks, you shouldn't have to guess which threshold, crop, or gate policy was live. Sculley et al. treat data dependencies, configuration, and feedback loops as ML-specific debt: if those boundaries aren't versioned, the system becomes expensive to change and hard to roll back.^{[1]Reference 1Hidden Technical Debt in Machine Learning Systems.https://research.google/pubs/hidden-technical-debt-in-machine-learning-systems/} The tuple is how this lab pins them.

A shared schema still doesn't move traffic. The next question is who is allowed to change an alias (a mutable pointer such as production or canary), and on what evidence. Limited live exposure (a canary) comes only after offline gates pass. Rollback restores the previous tuple; it doesn't retrain.

Compute the failing slice by hand

The controller you'll build uses delivery-risk-v1 as the running candidate, with delivery-risk-v0 live. Before any registry code, look at the candidate that must not get a canary. Four held-out rows:

Critical-slice recall is true positives on late critical rows, divided by those late critical rows:

$\text{critical-slice recall} = \frac{1}{2} = 0.50$

The policy floor is $0.90$. Cost did improve: $96 / 100 - 1 = -0.04$, which clears max_cost_delta = 0. That cheaper cost still doesn't buy a canary. A failed slice is a stored hold_offline receipt, and production stays on delivery-risk-v0.

Register a candidate without moving production

Start by freezing the whole release tuple. The candidate names its previous release, artifact package, and promotion-policy version. The policy registry owns the immutable thresholds, so a candidate can't loosen its own gate. Fingerprints bind each evaluation to those exact objects before any traffic moves. The next cells turn the four-row fixture into an immutable package, then derive the gates from those rows instead of accepting a caller-owned Boolean.

Offline gates aren't caller-owned booleans or thresholds. evaluate_candidate loads the registered artifact package, checks schema and split membership, derives recall and cost from its held-out rows, and binds the result to fingerprints for artifact, release, and policy. Only then can open_canary fetch that evidence by ID. Canary and production monitoring use the same pattern later.

A failed offline slice leaves production unchanged. Leakage fails no_leakage even when recall and cost look good: overlapping train and eval IDs are derived from the registered package, not reported by the caller. Passing evaluation gates opens only the canary alias. Each attempt appends an immutable receipt with candidate, production base, evidence ID, and policy, release, and artifact fingerprints. Nothing in this cell can overwrite production. Caller-owned metrics, booleans, or thresholds never open canary traffic; only bound evidence derived from the registered artifact can.

Wait for canary evidence

Fast checks catch broken schemas, errors, and latency spikes. They can't prove prediction quality when labels arrive later. A canary still needs both kinds of evidence.

Don't confuse that gate with an A/B test. A canary is limited live exposure with abort rules: if errors, latency, or a later labeled cost breach the policy, you pull the candidate back. An A/B test is a powered comparison with sticky assignment and a predeclared metric, which the experiment-design chapter already covered. Google Cloud's MLOps guidance lists canary deployment and A/B testing as online validation after offline model validation, not as replacements for it.^{[2]Reference 2MLOps: Continuous Delivery and Automation Pipelines in Machine Learning.https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning} This controller only answers whether delivery-risk-v1 is safe enough to keep exposing.

The functions below store each observation window inside an immutable receipt, refuse to promote after the first hour because late-delivery outcomes aren't ready yet, and treat an empty window list as a hold rather than a crash.

late_warning_cost_delta=-0.08 means the candidate reduced late-warning cost by eight percent relative to the previous release on this local fixture. It's a teaching threshold, not a universal production policy. Real teams choose windows and limits from product risk, traffic volume, and label delay.

An incomplete window returns hold_canary: gather more evidence without widening exposure. A measured latency, error-rate, or delayed-quality regression returns abort_canary and removes the canary alias. Progressive-delivery controllers such as Argo Rollouts use the same split: a failed analysis aborts and returns traffic to the previous stable revision rather than waiting for another sample.^{[3]Reference 3Argo Rollouts - Kubernetes Progressive Delivery Controllerhttps://argoproj.github.io/argo-rollouts/} Corrupted or mismatched telemetry aborts too, because the controller can't prove safe exposure. Missing evidence and negative evidence aren't the same operational state.

Promote last, then prove rollback

Promotion is the first moment this controller is allowed to move production. It fetches a stored canary receipt by ID, follows it back to offline evidence, and recomputes the policy, release, and artifact bindings. A ready decision becomes stale if any bound object changes. Rollback follows the same rule: production metrics become a stored receipt before they can restore the previous release. Both paths recheck the live production alias immediately before movement.

Rollback restores delivery-risk-v0, not its weights alone. That distinction matters because preprocessing, features, thresholds, and policy can all change serving behavior. Each audit event names the stored receipt that authorized its alias movement, so a later reviewer can reconstruct both promotion and rollback.

Production monitoring mirrors the canary hold-vs-abort rule. A measured error-rate, latency, or labeled cost regression returns rollback_required and can restore the previous alias. Incomplete delayed labels return hold_monitor: keep production on the live release and wait for outcomes. Missing evidence isn't negative evidence, so it must not look like a quality failure.

💡 Key insight: Unmeasured quality holds. A labeled regression aborts a canary or rolls back production. A Boolean named passed isn't a receipt.

Join fast and delayed monitoring

Live checks differ by product, but the promotion controller handles the same categories:

For scoring systems with delayed labels, canary monitoring should pause wider promotion until enough outcomes arrive. After promotion, incomplete delayed labels hold the production monitor rather than rolling back; only measured regressions restore the previous alias. A model that hasn't failed yet isn't the same as a model that has passed.

⚠️ Common mistake: Treating missing delayed labels as a rollback reason, or treating a canary as an A/B test. The first confuses "not yet measured" with "measured worse." The second confuses a safety abort with a powered lift estimate.

Continuous training is appropriate when a schedule or monitored condition creates a candidate run. Google Cloud's MLOps guidance puts automated data validation, model validation, metadata, serving, monitoring, and continuous-training triggers around that promotion, and still expects an online canary or experiment before a candidate serves all traffic.^{[2]Reference 2MLOps: Continuous Delivery and Automation Pipelines in Machine Learning.https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning} Retraining on a clock isn't a skip-the-gate button. The pipeline's value isn't automation alone; it's refusing untraceable changes.

Practice: break the release controller

Run the runnable examples again after each mutation. Predict which receipt or alias changes before reading output, then revert the mutation before trying the next one.

1. Change the third argument in the delivery-risk-v1 release_from_artifact(...) call from "delivery-risk-v0" to "delivery-risk-v-missing".
2. In recall_fail_artifact, change the second critical row's predicted_late value from 0 to 1. Rerun and confirm recall is derived as 1.0 from rows rather than accepted as an argument.
3. Pass [day_seven, first_hour] to one evaluate_canary call. Confirm that the receipt records window_order_invalid.
4. Change the day_seven constructor's delayed_labels_ready argument from True to False.
5. Change the day_seven constructor's late_warning_cost_delta argument from -0.08 to 0.05.
6. Replace ready_receipt.receipt_id with "canary-receipt-missing" in the approved promotion call.
7. Before approved promotion, set aliases["production"] = "delivery-risk-v1" to simulate an out-of-band alias move. Confirm that promotion holds, then reset it to "delivery-risk-v0".
8. After ready_receipt is stored but before promotion, set POLICIES["eta-promotion-v1"] = replace(POLICIES["eta-promotion-v1"], max_p95_latency_ms=500). The ID stays the same, but promotion must hold on fingerprint mismatch. Rerun from the start to restore the policy.
9. After ready_receipt is stored but before promotion, set artifacts["eta-artifact-v1"] = replace(passing_artifact, model_artifact="delay-model-v1-repacked"). Promotion must hold because the registered package no longer matches the artifact fingerprint. Rerun from the start to restore it.
10. Change the degraded constructor's late_warning_cost_delta argument from 0.14 to -0.01.
11. Change the degraded constructor's release ID from "delivery-risk-v1" to "delivery-risk-v0". Confirm that monitoring evidence for a different release can't move the current alias.
12. Replace degraded_receipt.receipt_id with "production-receipt-missing" in the rollback call. Which authorization check holds the alias?
13. Change the degraded constructor so delayed_labels_ready=False and late_warning_cost_delta=None (reset the cost delta and release ID to the original passing-service values first). Confirm the decision is hold_monitor, rollback_if_needed leaves production on delivery-risk-v1, and no rollback audit event is appended.

Practice answer sketches

Why does a mismatched previous_release block canary traffic?

Answer

previous_release_mismatch fails before traffic moves. The candidate points at a rollback target that isn't the live production release, so restoring it wouldn't prove a return to current behavior.

What happens when the critical offline slice fails?

Answer

The offline receipt returns hold_offline. production stays on delivery-risk-v0, and no canary alias opens.

Why do missing delayed labels and a positive cost delta produce different canary decisions?

Answer

Missing labels mean quality hasn't been measured yet, so the receipt returns hold_canary. A positive delta means measured late-warning cost regressed, so the receipt returns abort_canary and removes the limited-exposure alias. Both block promotion, but only one proves the candidate failed.

Why can't promotion accept any dictionary whose decision field says ready_for_promotion?

Answer

A mutable caller-generated dictionary doesn't prove who evaluated the candidate, which windows were measured, or whether the evidence belongs to the current release. Promotion fetches a stored immutable receipt by ID, verifies its full evidence chain, and rechecks the production alias before moving traffic.

Why do fingerprints matter when policy and artifact IDs haven't changed?

Answer

An ID can be reused while thresholds or artifact contents change. The policy, release, and artifact fingerprints bind the receipt to exact immutable values. A same-ID replacement makes old evidence stale and blocks promotion until the new objects are evaluated.

Why does rollback fetch a stored production receipt instead of accepting caller-supplied reasons?

Answer

Caller-owned reasons can be fabricated or changed. A registered receipt binds the monitored release, window metrics, gate failures, and decision before rollback rechecks the live alias.

What happens when the degraded production delta becomes -0.01?

Answer

No rollback reason remains in this fixture. rollback_if_needed returns keep_production, so delivery-risk-v1 stays live. Production policies should restore a prior alias only when an explicit rollback condition fires.

What happens when a production window has healthy service metrics but delayed_labels_ready=False?

Answer

The monitor records delayed_quality_not_ready and returns hold_monitor. rollback_if_needed keeps delivery-risk-v1 live and appends no rollback event. Incomplete labels mean quality hasn't been measured yet, so they must not restore the previous alias the way a labeled cost regression would.

Submission checklist

The lab you submit is a small local platform surface, not a copy of a cloud vendor's stack. Prove each boundary with a deterministic fixture and test:

The promotion controller refuses an alias move unless it's bound to stored evidence. Document QA needs the same receipt habit: an answer is only as auditable as the source it cites.