Capstone: Image Damage Classifier

Shipment rows, ranked items, and warehouse time series all fit into tables. A customer return adds a new input type: a photo of a package that may be crushed, torn, blurred, dark, or unrelated to the order.

Earlier, you traced a convolutional neural network (CNN) over a damaged-package image patch. This capstone turns that spatial reasoning into a product: an image triage endpoint that flags likely visible damage, rejects unusable photos, preserves evidence for human review, and never turns an uncertain image score directly into a refund.

Define the photo decision first

ShopFlow receives return photos from customers and warehouse intake stations. The useful product question isn't "does the model recognize every defect?" It's: which photo should a specialist inspect first, and when is the photo too weak to support any decision?

Use three operational outcomes:

The classifier isn't a refund policy. Product eligibility still depends on order ownership, item type, return window, and specialist judgment. This separation prevents a shadow or reflection in a photo from issuing a costly action.

A model card should state the intended use, input constraints, decision threshold, evaluated slices, and known failure cases. Model cards were proposed as structured reports for exactly this type of deployed-model context: users need more than a metric without its operating conditions.^{[1]Reference 1Model Cards for Model Reportinghttps://arxiv.org/abs/1810.03993}

Build a dataset that can't leak

For tabular models, leakage may be a future delivery timestamp. For photos, leakage often hides in nearly identical pixels. A customer may upload three bursts of the same crushed box. A warehouse may photograph one parcel from four angles. If related images land in both train and test sets, the model can memorize one package rather than generalize to new damage.

Your manifest should contain:

Evaluate at least daylight versus dark uploads, customer versus warehouse source, packaging type, and visible-defect size. A global score can hide the exact failure that matters: small tears disappearing in dark phone images.

Use the CNN learned earlier as a baseline, then fine-tune a pretrained image encoder only if you record its preprocessing and measure it under the same split. A later deep-dive explains Vision Transformer image encoders; this capstone doesn't require that architecture.^{[2]Reference 2An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

Freeze a grouped photo manifest

Start with the dataset boundary. Multiple photos from one physical package belong in one split even when filenames differ. Keep time direction intact too: later return cases should test a model trained on earlier cases.

The local manifest below is small enough to inspect line by line. This historical evaluation fixture contains frozen specialist labels, reviewer IDs, and guideline versions for dataset audit. Damage labels appear only when a specialist judged the image usable. Unusable photos keep confirmed_damage=None because poor evidence shouldn't become supervision. Candidate scores arrive in a separate object, so route code can't quietly read labels as model output.

The fixture uses explicit splits so the invariant stays visible. A larger pipeline can use a group-aware splitter, then freeze and audit the resulting manifest. The important claim isn't that one splitter solves every dataset: no physical case may cross evaluation boundaries, and later cases remain later.

Encode quality-first review traces

The model endpoint should receive a preprocessing result and a damage score, then choose a review route. Quality checks run before the damage threshold. Otherwise a confidently scored blur, dark frame, or unrelated object can create an unsupported escalation.

The score fixture below is separate from the manifest. Labels stay available for offline evaluation, but the router never reads them. A production system may run its cheap quality checks before an expensive damage model; this compact fixture keeps both outputs so you can test that an unsupported high damage score still abstains.

Cases R-406-a and R-409-a are the important failure tests. Both have high damage scores. Neither score counts as usable evidence because quality checks fail first. The endpoint asks for another photo instead of escalating an unsupported claim.

Package the Vision Service

Submit an inspectable repository, not a notebook screenshot:

The serving bundle must pin image resize and crop behavior, color normalization, model weights, label version, route policy, damage threshold, quality-gate thresholds, and previous-bundle pointer. A change from center crop to full-frame resize may change whether a torn corner remains visible; it's a model behavior change even when weights remain constant.

The routing cells emit that trace shape directly. They let a reviewer reconstruct the route:

Join delayed outcomes before promotion

Photo models drift when the image source changes. A new warehouse camera, winter lighting, a mobile upload compressor, or new packaging graphics can alter pixels before a confirmed-damage label exists.

Separate immediate checks from delayed quality:

Google Cloud's MLOps guidance treats serving, monitoring, validation, metadata, and continuous training as connected stages rather than a one-time deploy step.^{[3]Reference 3MLOps: Continuous Delivery and Automation Pipelines in Machine Learning.https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning} Apply that same discipline here: a change in image quality creates an investigation or candidate run, never an automatic production replacement.

The final local receipt appends specialist outcomes only after routing, then joins each outcome to a physical case. Each delayed outcome also carries its reviewer and label guideline, so precision and recall can't silently mix definitions. It measures review precision and recall once per usable test case, records immediate request rates by source, proves unusable photos abstain, and keeps the previous bundle beside the candidate. When one package has multiple usable photos, the case escalates if any usable photo crosses the priority threshold. These tiny counts teach the contract; production promotion still needs larger slices and shadow traffic.

candidate_for_specialist_shadow_review is narrower than launch approval. The receipt says this frozen bundle deserves comparison beside current production routing. It doesn't claim that four usable local cases prove every camera, lighting condition, package type, or damage shape.

Practice: break the vision contract

Use the runnable examples as a release harness. Change one condition at a time, predict the failure, then rerun the examples.

1. Change R-401-b split from train to validation. Which dataset gate fails?
2. Change R-406-a brightness in MODEL_OUTPUTS from 0.12 to 0.30. Why does this expose a quality-detector failure rather than prove the escalation is safe?
3. Remove damage_threshold from the response trace. Which receipt gate fails?
4. Change R-408-a damage probability in MODEL_OUTPUTS from 0.63 to 0.75. Which delayed metrics improve?
5. Set BUNDLE["previous_bundle"] = "". Why is shadow evidence no longer promotion-ready?
6. Replace SPECIALIST_OUTCOMES with an empty list. Why do evidence gates fail instead of crashing or passing?
7. Add SpecialistOutcome("R-999", 12, True, "S-12", "visible-damage-v1"). Which join gate fails?
8. Change R-408 specialist outcome to guideline visible-damage-v2. Which provenance gate fails?

Practice answer sketches

Mastery check

Evaluation rubric

Common failures