Capstone: Image Damage Classifier

A damage classifier receives photos that may be crushed, torn, blurred, dark, or unrelated to the reported case. Unlike a late-delivery score or a seven-day parcel forecast, the input can fail before classification begins because the evidence itself isn't usable.

The last capstone packaged a demand forecast with shadow evidence, delayed observation joins, and a rollback pointer. Reuse that receipt habit for pixels. Ship an image-triage endpoint that flags likely visible damage, rejects unusable photos, preserves evidence for human review, and never turns an uncertain image score into a refund.

You already traced a convolutional neural network (CNN) over a cracked equipment-panel patch in CNNs from Scratch. That spatial score is the starting point, not the product: quality checks and specialist confirmation still sit in front of any costly action.

Define the photo decision first

The service receives return photos from customer uploads and intake cameras. The useful product question isn't "does the model recognize every defect?" It's: which photo should a specialist inspect first, and when is the photo too weak to support any decision?

Use three operational outcomes:

The classifier isn't a refund policy. Product eligibility still depends on order ownership, item type, return window, and specialist judgment. This separation prevents a shadow or reflection in a photo from issuing a costly action.

A model card should state intended use (and out-of-scope uses), decision thresholds, the slices you actually evaluated, and known limitations. Mitchell et al. proposed model cards as short reports so people can see those operating conditions instead of a lone metric.^{[1]Reference 1Model Cards for Model Reportinghttps://arxiv.org/abs/1810.03993}

Those three actions only stay honest if related photos of one physical package can't leak into both train and test. That's the dataset job next.

Build a dataset that can't leak

For tabular models, leakage may be a future delivery timestamp. For photos, leakage often hides in nearly identical pixels. A customer may upload three bursts of the same crushed box. A warehouse may photograph one parcel from four angles. If related images land in both train and test sets, the model can memorize one package rather than generalize to new damage.

Your manifest should contain:

Evaluate at least daylight versus dark uploads, customer versus warehouse source, packaging type, and visible-defect size. A global score can hide the exact failure that matters: small tears disappearing in dark phone images.

Use the CNN from CNNs from Scratch as a baseline. Fine-tune a pretrained image encoder only if you record its preprocessing and measure it under the same grouped, time-ordered split. A later deep-dive covers Vision Transformer image encoders; this capstone doesn't require that architecture.^{[2]Reference 2An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

Freeze a grouped photo manifest

Start with the dataset boundary. Multiple photos from one physical package belong in one split even when filenames differ. Keep time direction intact too: later return cases should test a model trained on earlier cases.

The local manifest below is small enough to inspect line by line. This historical evaluation fixture contains frozen specialist labels, reviewer IDs, and guideline versions for dataset audit. Damage labels appear only when a specialist judged the image usable. Unusable photos keep confirmed_damage=None because poor evidence shouldn't become supervision. Candidate scores arrive in a separate object, so route code can't quietly read labels as model output. Manifest confirmed_damage fields are offline audit only; the serving path must not read them. MODEL_OUTPUTS below are frozen fixtures for the routing receipt. Fitting train_cnn_baseline.py and applying the preprocessing version to pixels is a separate training lab.

The fixture uses explicit splits so the invariant stays visible. A larger pipeline can use a group-aware splitter, then freeze and audit the resulting manifest. The important claim isn't that one splitter solves every dataset: no physical case may cross evaluation boundaries, and later cases remain later.

The manifest proves the split and the label contract. It doesn't yet prove the endpoint refuses to read those labels when it chooses a route.

Encode quality-first review traces

The model endpoint should receive a preprocessing result and a damage score, then choose a review route. Quality checks run before the damage threshold. Otherwise a confidently scored blur, dark frame, or unrelated object can create an unsupported escalation.

The cheap checks in this fixture are three numbers you can inspect without opening a CNN:

• box_visible: is a parcel in the frame?
• brightness: mean intensity in [0, 1]. Below 0.20 the photo is too dark to judge a tear.
• blur_score: higher means more blur. Above 0.45 the edges that would show a crush are gone.

A production service may compute those with a blur filter or an exposure histogram. The routing invariant doesn't depend on the exact formula: quality fails first.

A high damage probability still isn't proof. Guo et al. showed that modern networks are often overconfident relative to their accuracy, so you shouldn't treat a 0.94 as a calibrated chance of visible damage, especially on a dark or empty frame.^{[3]Reference 3On Calibration of Modern Neural Networkshttps://arxiv.org/abs/1706.04599} That's why the router never lets the score override a failed quality check.

The score fixture below is separate from the manifest. Labels stay available for offline evaluation, but the router never reads them. A production system may run its cheap quality checks before an expensive damage model; this compact fixture keeps both outputs so you can test that an unsupported high damage score still abstains.

Cases R-406-a and R-409-a are the important failure tests. Both have high damage scores. Neither score counts as usable evidence because quality checks fail first. The endpoint asks for another photo instead of escalating an unsupported claim.

The scatter makes the same contract visible. Usable photos cluster in the lower-right region. R-406-a is the large dot on the dark side of the brightness gate. R-409-a is the hollow dot in that cluster: light and sharp enough, but the package isn't in the frame. Dot size is the damage score, so both rejected photos are large and still not priority routes.

Package the vision service

Submit an inspectable repository, not a notebook screenshot:

The serving bundle must pin image resize and crop behavior, color normalization, model weights, label version, route policy, damage threshold, quality-gate thresholds, and previous-bundle pointer. A change from center crop to full-frame resize may change whether a torn corner remains visible; it's a model behavior change even when weights remain constant.

The routing cells emit that trace shape directly. They let a reviewer reconstruct the route:

A replayable trace records what the bundle decided. It still isn't a confirmed-damage label. Those arrive later, and they join by case.

Join delayed outcomes before promotion

Photo models drift when the image source changes. A new warehouse camera, winter lighting, a mobile upload compressor, or new packaging graphics can alter pixels before a confirmed-damage label exists.

Separate immediate checks from delayed quality:

Google Cloud's MLOps guidance treats data validation, model validation, serving, monitoring, metadata, and continuous training as connected stages rather than a one-time deploy.^{[4]Reference 4MLOps: Continuous Delivery and Automation Pipelines in Machine Learning.https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning} Use that here: a jump in dark warehouse-camera photos starts an investigation or a candidate run. Promotion still waits for the delayed specialist join, not for an automatic weight swap.

The final local receipt appends specialist outcomes only after routing, then joins each outcome to a physical case. Each delayed outcome also carries its reviewer and label guideline, so precision and recall can't silently mix definitions. It measures review precision and recall once per usable test case, records immediate request rates by source, proves unusable photos abstain, and keeps the previous bundle beside the candidate. When one package has multiple usable photos, the case escalates if any usable photo crosses the priority threshold. These tiny counts teach the contract; production promotion still needs larger slices and shadow traffic.

candidate_for_specialist_shadow_review is narrower than launch approval. The receipt says this frozen bundle deserves comparison beside current production routing. It doesn't claim that four usable local cases prove every camera, lighting condition, package type, or damage shape.

Those delayed counts are case-level, not photo-level. A true positive (TP) is a priority route the specialist later confirmed; a false positive (FP) is a priority route they rejected; a false negative (FN) is a damaged case left on the normal path; a true negative (TN) is a correctly quiet case. The four joined usable test cases are:

Priority precision is $1 / (1 + 1) = 0.50$ and recall is $1 / (1 + 1) = 0.50$. R-404-a and R-404-b collapse to one case. Unusable photos never enter this table.

Practice: break the vision contract

Use the runnable examples as a release harness. Change one condition at a time, predict the failure, then rerun the examples.

1. Change R-401-b split from train to validation. Which dataset gate fails?
2. Change R-406-a brightness in MODEL_OUTPUTS from 0.12 to 0.30. Why does this expose a quality-detector failure rather than prove the escalation is safe?
3. Remove damage_threshold from the response trace. Which receipt gate fails?
4. Change R-408-a damage probability in MODEL_OUTPUTS from 0.63 to 0.75. Which delayed metrics improve?
5. Set BUNDLE["previous_bundle"] = "". Why is shadow evidence no longer promotion-ready?
6. Replace SPECIALIST_OUTCOMES with an empty list. Why do evidence gates fail instead of crashing or passing?
7. Add SpecialistOutcome("R-999", 12, True, "S-12", "visible-damage-v1"). Which join gate fails?
8. Change R-408 specialist outcome to guideline visible-damage-v2. Which provenance gate fails?

Practice answer sketches

Which gate fails when R-401-b moves to validation?

Answer

case_groups_do_not_cross_splits fails because two photos of physical return R-401 now cross training and validation. Evaluation can reward memorization of one package.

Why isn't raising R-406-a brightness a safe escalation?

Answer

The router now sees an apparently usable high-score image, but specialist label still says the evidence is unusable. Both unusable_photos_abstain and unsafe_priority_routes_absent fail. That mismatch belongs in quality-model evaluation before promotion.

Which gate fails when route traces omit damage_threshold?

Answer

route_traces_replayable fails. A reviewer can't reconstruct whether a policy threshold change altered the route.

What changes when R-408-a score rises from 0.63 to 0.75?

Answer

It becomes a true-positive priority route. Case-level recall rises from 1 / 2 to 2 / 2 = 1.0, and precision rises from 1 / 2 to 2 / 3 = 0.667.

Why keep previous_bundle beside candidate?

Answer

Shadow evidence may justify a later promotion, but promotion still needs a known rollback target. A candidate without previous alias doesn't prove rollback can restore earlier behavior.

What happens when no specialist outcomes have arrived?

Answer

Precision and recall become None, then outcome-completeness and metric-evidence gates fail. No denominator means no evidence. It isn't a crash, measured success, or measured failure.

Which gate fails when outcome R-999 appears?

Answer

specialist_outcomes_join_usable_test_cases fails. A delayed specialist label can't become evaluation evidence unless it joins a usable test case routed by this bundle.

Which gate fails when R-408 uses visible-damage-v2?

Answer

specialist_outcome_label_contract_matches_bundle fails. Delayed metrics only support this bundle when every outcome records a reviewer and uses the bundle's visible-damage-v1 label contract.