Capstone: Production ML Pipeline

Four products now need the same operational discipline: a late-delivery warning model, a product ranker, a warehouse demand forecast, and a damaged-package photo classifier. Each uses different metrics, but each relies on immutable data evidence, validated candidates, controlled promotion, monitoring, and rollback.

This capstone assembles that discipline into one ML platform workflow. It isn't tied to a particular orchestrator or cloud vendor. A reviewer must be able to trace any live decision back to data, feature, model, policy, and promotion evidence. That requires stored receipts, not a mutable Boolean that happens to say passed.

Define the Shared Release Tuple

Models differ, but their release manifest can share a schema:

The release tuple prevents an incident review from asking which threshold, feature transform, or gate policy happened to be active. Sculley et al. warn that ML systems accumulate debt through data dependencies, configuration, and feedback loops unless those boundaries are managed explicitly.^{[1]Reference 1Hidden Technical Debt in Machine Learning Systems.https://research.google/pubs/hidden-technical-debt-in-machine-learning-systems/}

Build the Portfolio Repository

Submit a small but inspectable platform surface:

Google Cloud's MLOps architecture separates automated data/model validation, metadata, serving, monitoring, and continuous-training triggers around promotion.^{[2]Reference 2MLOps: Continuous Delivery and Automation Pipelines in Machine Learning.https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning} Your repository needn't copy that platform, but it should prove each boundary through a deterministic local fixture and test.

Register a Candidate Without Moving Production

Training completion is evidence, not permission to change live behavior. Start by freezing the whole release tuple. The candidate records both its previous release and its promotion policy, so a reviewer can replay the rollback target and gate thresholds before any traffic moves.

A failed offline slice leaves production unchanged. Passing evaluation gates opens only the canary alias. Each attempt appends an immutable receipt with candidate, production base, policy version, verdict, and reasons. Nothing in this cell can overwrite production.

Wait for Canary Evidence

Fast checks catch broken schemas, errors, and latency spikes. They can't prove prediction quality when labels arrive later. A canary rollout needs both kinds of evidence. The controller below stores each observation window inside an immutable receipt, refuses to promote after the first hour because late-delivery outcomes aren't ready yet, and handles an empty window list as a hold rather than crashing.

late_warning_cost_delta=-0.08 means the candidate reduced late-warning cost by eight percent relative to the previous release on this local fixture. It's a teaching threshold, not a universal production policy. Real teams choose windows and limits from product risk, traffic volume, and label delay.

An incomplete window returns hold_canary: gather more evidence without widening exposure. A measured latency, error-rate, or delayed-quality regression returns abort_canary and removes the canary alias. Corrupted or mismatched telemetry aborts too because the controller can't prove safe exposure. Missing evidence and negative evidence aren't the same operational state.

Promote Last, Then Prove Rollback

The final cell makes alias movement explicit. Promotion fetches a stored canary receipt by ID rather than trusting a caller-supplied decision dictionary. Rollback follows the same rule: production metrics become a stored receipt before they can restore the previous release. Both paths recheck the live production alias immediately before movement.

Rollback restores delivery-risk-v0, not its weights alone. That distinction matters because preprocessing, features, thresholds, and policy can all change serving behavior. Each audit event names the stored receipt that authorized its alias movement, so a later reviewer can reconstruct both promotion and rollback.

Join Fast and Delayed Monitoring

Live checks differ by product, but the promotion controller handles the same categories:

For scoring systems with delayed labels, canary monitoring should pause wider promotion until enough outcomes arrive. A model that hasn't failed yet isn't the same as a model that has passed.

Continuous training is appropriate when a schedule or monitored condition creates a candidate run. It should never skip data validation, offline comparisons, or a promotion record. The pipeline's value isn't automation alone; it's refusing untraceable changes.

Practice: break the release controller

Run the runnable examples again after each mutation. Predict which receipt or alias changes before reading output.

1. Change the delivery-risk-v1 constructor's previous_release from "delivery-risk-v0" to "delivery-risk-v-missing".
2. Set good_gates["critical_slice_pass"] back to False.
3. Pass [day_seven, first_hour] to one evaluate_canary call. Confirm that the receipt records window_order_invalid.
4. Change the day_seven constructor's delayed_labels_ready argument from True to False.
5. Change the day_seven constructor's late_warning_cost_delta argument from -0.08 to 0.05.
6. Replace ready_receipt.receipt_id with "canary-receipt-missing" in the approved promotion call.
7. Before approved promotion, set aliases["production"] = "delivery-risk-v1" to simulate an out-of-band alias move. Confirm that promotion holds, then reset it to "delivery-risk-v0".
8. Change the degraded constructor's late_warning_cost_delta argument from 0.14 to -0.01.
9. Change the degraded constructor's release ID from "delivery-risk-v1" to "delivery-risk-v0". Confirm that monitoring evidence for a different release can't move the current alias.
10. Replace degraded_receipt.receipt_id with "production-receipt-missing" in the rollback call. Which authorization check holds the alias?

Practice answer sketches

Submission checklist

This completes the conventional production ML portfolio. The next capstone returns to LLM products: document QA must apply the same lineage and release discipline to retrieved evidence and generated answers.

Mastery check

Evaluation rubric

Common failures