Capstone: Image Damage Classifier

Consider a damage score of 0.94 attached to a photo that's too dark to inspect. Should the service prioritize it because the score is high, or ask for a clearer photo because the evidence is poor? A useful classifier needs an answer to that question before its accuracy matters.

Imagine the specialist opening that dark photo first, ahead of a clearly photographed torn box. The score has consumed attention without supplying inspectable evidence. Asking for another photo needs a reason the uploader can act on: improve the light, steady the camera, or include the whole parcel. Neither a high score nor a request for a clearer image settles the return claim.

The project makes one bounded triage decision: request a clearer photo, keep an accepted low-score image in normal review, or send an accepted high-score image to priority review. Every route records its reason, thresholds, and bundle version. A later specialist outcome joins by physical return, so the score helps choose a queue without becoming an unverified refund.

The runnable code uses invented records and scores. There are no package photographs, trained weights, or measured model results supplied with this lesson. The local exercise tests routing and evaluation logic. Completing the image-classifier project also requires the real-data and training work described below; passing the fixture doesn't qualify a model for deployment or shadow traffic.

The preceding capstone examined a demand forecast and delayed observations. Here the input changes to pixels, but predictions and later outcomes still need separate records. CNNs from Scratch introduced how a convolutional neural network (CNN) extracts spatial features. This project adds the data split, decision policy, and evaluation needed around an image model.

Start with the route, not the model

A return photo can support a queue decision even when it can't support a final claim. The useful product question isn't "does the model recognize every defect?" It's: which photo should a specialist inspect first, and when should the service ask for a new one?

A score of 0.88 doesn't authorize a refund. Define the review actions first, then decide what evidence each action needs. The following diagram shows the synthetic policy implemented in this lesson; it makes no timing or model-performance claim.

The fixture implements three actions. Its thresholds are authored examples, not settings validated on package photographs:

A classifier isn't a refund policy. Order ownership, item type, return window, and specialist judgment still determine product eligibility. This prototype implements no refund, replacement, or restocking action.

A model card records intended use, out-of-scope uses, decision thresholds, evaluated slices, and known limitations. Mitchell et al. proposed this kind of short report so people can see operating conditions instead of treating one metric as a promise.^{[1]Reference 1Model Cards for Model Reportinghttps://arxiv.org/abs/1810.03993}

Those triage actions only stay honest if related photos of one physical package can't leak into both train and test. Before choosing a model, the dataset must make that boundary real.

Keep one physical package in one split

For tabular models, leakage may be a future delivery timestamp. For photos, it often hides in nearly identical pixels: a customer uploads three burst photos of the same crushed box, or a warehouse photographs one parcel from four camera angles on an automated conveyor. If related images land in both train and test, the model can memorize one package instead of generalizing to new damage.

Consider what happens when a customer returns a damaged parcel. They pull out their smartphone and snap four rapid burst shots from slightly different angles. The background carpet, the merchant shipping barcode, the handwritten Sharpie note, and the specific printed carton logo appear identically across all four frames.

Suppose three photos enter training and the fourth enters test. A model could exploit the repeated barcode, carton logo, or floor texture rather than learn damage features that transfer to a new parcel. The test then measures recognition of a previously seen case as well as damage detection. We have no measured accuracy for this example. The problem is the evaluation boundary, even if a particular model doesn't exploit it.

Your manifest should contain:

Cryptographic hashes like SHA-256 identify exact byte duplicates. Recompression, resizing, or metadata removal can change those bytes while preserving a recognizable view of the same parcel. Group re-uploads and alternate views by physical case. A correct case ID still needs auditing: the manifest alone cannot discover one parcel mistakenly assigned two different IDs.

Define the label before collecting images. For example: "visible external packaging damage" includes a torn seam, crushed corner, or punctured box wall, but doesn't establish that the product inside is broken. Record an uncertain label when specialists disagree. Don't quietly turn uncertain or unusable images into negative damage examples. Obtain permission to use uploads, restrict access, and remove shipping labels or other personal information from images used in a portfolio.

Create groups before splitting, including re-uploads and alternate views. Choose chronological boundaries, then quarantine cases whose photos span a boundary instead of splitting that case across datasets. A group-aware splitter such as GroupKFold prevents group overlap but doesn't enforce time ordering.^{[2]Reference 2GroupKFoldhttps://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html} Holding out later cases tests transfer across the changes that actually occurred in that period; it doesn't guarantee coverage of future cameras or packaging. Record when labels became available too. A model trained at a historical cutoff may use only labels known then, even if the photos were captured earlier.

Make the split inspectable

Look at R-401-a and R-401-b: different photos, one physical return. They belong in one split even though their filenames differ. Time has a similar rule: later return cases should test a model trained on earlier cases, not leak backward into training.

The local manifest is small enough to inspect line by line. This synthetic evaluation fixture assigns invented specialist labels, reviewer IDs, and guideline versions. It tests the bookkeeping contract, not label quality or image identity. A real manifest also needs the image URI and content hash described above.

Damage labels appear only on images marked usable. Unusable photos keep confirmed_damage=None: they can supervise a quality detector, but they aren't evidence for "undamaged." Candidate scores live in a separate object from these offline labels. The router uses identity and capture metadata from the manifest, never its labels. MODEL_OUTPUTS contains hand-written scores, not predictions produced by a CNN.

from collections import Counter, defaultdict
from dataclasses import dataclass
import json
import math

LABEL_GUIDELINE = "visible-damage-v1"

def nonblank(value, name):
    if type(value) is not str or not value.strip() or value != value.strip():
        raise ValueError(f"{name} must be a nonblank string without outer whitespace")

def unit_score(value, name):
    # Check the range before converting an arbitrarily large integer to float.
    if type(value) not in (int, float) or not 0 <= value <= 1 or not math.isfinite(value):
        raise ValueError(f"{name} must be finite and in [0, 1]")

@dataclass(frozen=True)
class PhotoManifestRow:
    case_id: str
    photo_id: str
    capture_day: int
    split: str
    source: str
    packaging: str
    quality_label: str
    confirmed_damage: bool | None
    reviewer_id: str
    guideline_version: str

    def __post_init__(self):
        for name in ("case_id", "photo_id", "source", "packaging", "reviewer_id", "guideline_version"):
            nonblank(getattr(self, name), name)
        if type(self.capture_day) is not int or self.capture_day < 1:
            raise ValueError("capture_day must be a positive integer")
        if self.split not in ("train", "validation", "test"):
            raise ValueError("unknown split")
        if self.quality_label not in ("usable", "unusable"):
            raise ValueError("unknown quality_label")
        if self.quality_label == "usable" and type(self.confirmed_damage) is not bool:
            raise ValueError("usable photos require a boolean damage label")
        if self.quality_label == "unusable" and self.confirmed_damage is not None:
            raise ValueError("unusable photos must abstain from damage labeling")

PHOTOS = [
    PhotoManifestRow("R-401", "R-401-a", 1, "train", "customer_phone", "corrugated", "usable", True, "S-12", "visible-damage-v1"),
    PhotoManifestRow("R-401", "R-401-b", 2, "train", "customer_phone", "corrugated", "usable", True, "S-12", "visible-damage-v1"),
    PhotoManifestRow("R-402", "R-402-a", 3, "validation", "customer_phone", "corrugated", "unusable", None, "S-08", "visible-damage-v1"),
    PhotoManifestRow("R-403", "R-403-a", 4, "validation", "warehouse_camera", "mailer", "usable", False, "S-08", "visible-damage-v1"),
    PhotoManifestRow("R-404", "R-404-a", 5, "test", "customer_phone", "mailer", "usable", True, "S-12", "visible-damage-v1"),
    PhotoManifestRow("R-404", "R-404-b", 6, "test", "customer_phone", "mailer", "usable", True, "S-12", "visible-damage-v1"),
    PhotoManifestRow("R-405", "R-405-a", 7, "test", "warehouse_camera", "corrugated", "usable", False, "S-08", "visible-damage-v1"),
    PhotoManifestRow("R-406", "R-406-a", 8, "test", "customer_phone", "corrugated", "unusable", None, "S-12", "visible-damage-v1"),
    PhotoManifestRow("R-407", "R-407-a", 9, "test", "customer_phone", "corrugated", "usable", False, "S-12", "visible-damage-v1"),
    PhotoManifestRow("R-408", "R-408-a", 10, "test", "warehouse_camera", "corrugated", "usable", True, "S-08", "visible-damage-v1"),
    PhotoManifestRow("R-409", "R-409-a", 11, "test", "warehouse_camera", "corrugated", "unusable", None, "S-08", "visible-damage-v1"),
]

print("photos:", len(PHOTOS))
print("cases:", len({photo.case_id for photo in PHOTOS}))

photos: 11
cases: 9

splits_by_case = defaultdict(set)
for photo in PHOTOS:
    splits_by_case[photo.case_id].add(photo.split)

split_days = {
    split: [photo.capture_day for photo in PHOTOS if photo.split == split]
    for split in ("train", "validation", "test")
}
manifest_checks = {
    "photo_ids_unique": len({photo.photo_id for photo in PHOTOS}) == len(PHOTOS),
    "known_split_and_quality_labels": all(
        photo.split in {"train", "validation", "test"}
        and photo.quality_label in {"usable", "unusable"}
        for photo in PHOTOS
    ),
    "case_groups_do_not_cross_splits": all(len(splits) == 1 for splits in splits_by_case.values()),
    "time_ordered_splits": (
        all(split_days.values())
        and max(split_days["train"]) < min(split_days["validation"])
        and max(split_days["validation"]) < min(split_days["test"])
    ),
    "usable_labels_complete": all(type(photo.confirmed_damage) is bool for photo in PHOTOS if photo.quality_label == "usable"),
    "unusable_labels_abstain": all(photo.confirmed_damage is None for photo in PHOTOS if photo.quality_label == "unusable"),
    "label_lineage_recorded": all(photo.guideline_version == LABEL_GUIDELINE for photo in PHOTOS),
}

print("split photo counts:", dict(Counter(photo.split for photo in PHOTOS)))
print("R-401 splits:", sorted(splits_by_case["R-401"]))
print(json.dumps(manifest_checks, indent=2))
assert all(manifest_checks.values())

split photo counts: {'train': 2, 'validation': 2, 'test': 7}
R-401 splits: ['train']
{
  "photo_ids_unique": true,
  "known_split_and_quality_labels": true,
  "case_groups_do_not_cross_splits": true,
  "time_ordered_splits": true,
  "usable_labels_complete": true,
  "unusable_labels_abstain": true,
  "label_lineage_recorded": true
}

Two checks matter before a model appears: no physical case crosses an evaluation boundary, and later cases remain later. A larger pipeline can use a group-aware splitter, then freeze and audit its output. The splitter can change; those invariants can't.

These checks validate the small manifest's structure. They can't detect a wrongly assigned case ID or two visually identical files because this fixture contains no pixels. The next question is independent: will the router reject a failed quality check even when the damage score is high?

Let quality veto the damage score

The endpoint receives two kinds of evidence: a quality result and a damage score. Which one wins when they disagree? Quality checks run first. Otherwise a confidently scored blur, dark frame, or unrelated object can create an unsupported escalation.

A real service can check quality before invoking the damage model. Whether this saves time depends on the checks, models, hardware, and rejected-input rate. A learned parcel detector is itself neural inference; running it on a CPU doesn't make it a deterministic edge heuristic. Measure the whole path on the intended device rather than assign it a latency budget from this fixture.

Three kinds of check deserve separate validation:

1. Focus statistic. One discrete approximation to the Laplacian uses the four-neighbor stencil $K = \begin{bmatrix} 0 & 1 & 0 \\ 1 & -4 & 1 \\ 0 & 1 & 0 \end{bmatrix}.$ Compute it in signed or floating-point arithmetic so negative responses survive. The population variance over $N$ retained response pixels is $v = \frac{1}{N}\sum_{i=1}^{N}(L_i-\bar L)^2.$ Edges can raise this statistic, but so can texture, contrast, and noise. The OpenCV Laplacian tutorial smooths its example before differentiation to reduce noise.^{[3]Reference 3Laplace Operatorhttps://docs.opencv.org/4.x/d5/db5/tutorial_laplace_operator.html} Choose a region, resizing policy, and boundary convention before comparing values. A large variance alone doesn't establish an inspectable parcel.
2. Exposure. Mean pixel brightness can hide dark parcel regions beside a bright label. Histograms and clipped-pixel fractions within a parcel region can expose that problem. Pixel brightness is not a measurement of illumination in lux. Choose and validate thresholds on the actual capture path; the fixture checks only a supplied scalar and implements no histogram or overexposure rule.
3. Presence and detail. Check that the relevant parcel is in frame and retains enough detail after preprocessing. As a geometry example, resizing a whole 1,024-pixel-wide image to 224 pixels reduces a four-pixel-wide feature to about $4(224/1024)=0.875$ pixels. Its physical width in centimeters cannot determine its pixel width without scale information. Compare full-frame, parcel-crop, and higher-resolution inputs on real small-defect examples.

Try a narrower question: can the focus statistic fall without blurring the edge? This cell changes contrast, keeps the edge in the same place, and drops outer pixels instead of padding them.

from statistics import pvariance

def laplacian_variance(image):
    height, width = len(image), len(image[0])
    responses = [
        image[y - 1][x] + image[y + 1][x]
        + image[y][x - 1] + image[y][x + 1] - 4 * image[y][x]
        for y in range(1, height - 1)
        for x in range(1, width - 1)
    ]
    return pvariance(responses)

edge = [[0.0, 0.0, 1.0, 1.0, 1.0] for _ in range(5)]
lower_contrast = [[value * 0.5 for value in row] for row in edge]
original = laplacian_variance(edge)
changed = laplacian_variance(lower_contrast)
print(f"original: {original:.6f}; lower contrast: {changed:.6f}")
print(f"variance ratio: {changed / original:.2f}")
assert changed == original / 4

original: 0.666667; lower contrast: 0.166667
variance ratio: 0.25

The operator is linear, so multiplying pixel values by $c$ multiplies its responses by $c$ and their variance by $c^2$. This is a counterexample to interpreting the statistic as focus alone, not a package-quality benchmark. In a real quality evaluation, include sharp low-contrast parcels and noisy blurred images alongside obvious successes.

A high damage score still isn't proof. Guo et al. studied miscalibration and temperature scaling on classification benchmarks.^{[4]Reference 4On Calibration of Modern Neural Networkshttps://arxiv.org/abs/1706.04599} Their result doesn't measure this fictional parcel model or establish its behavior on dark frames. Fit and assess a calibrator on held-out, relevant data; good calibration on accepted images doesn't guarantee reliability on unfamiliar inputs. The routing policy independently prevents a score from overriding a failed quality check.

The score fixture below stays separate from the manifest. Labels remain available for offline evaluation, but the router never reads them. Its hand-written blur_score ranges from zero to one, with larger values meaning more blur. It is not raw Laplacian variance, whose direction is different, and no conversion is implemented. The supplied brightness and box-visibility values are also invented. Keeping all outputs, including damage scores for rejected photos, lets you test the veto without claiming that image processing occurred.

Before running the route, find the test rows with high damage scores and a failed quality check. Predict their actions. The answer should follow from the ordering above, not from the score alone.

@dataclass(frozen=True)
class ModelOutput:
    damage_probability: float
    blur_score: float
    brightness: float
    box_visible: bool

    def __post_init__(self):
        for name in ("damage_probability", "blur_score", "brightness"):
            unit_score(getattr(self, name), name)
        if type(self.box_visible) is not bool:
            raise ValueError("box_visible must be a boolean")

MODEL_OUTPUTS = {
    "R-401-a": ModelOutput(0.91, 0.12, 0.66, True),
    "R-401-b": ModelOutput(0.88, 0.10, 0.70, True),
    "R-402-a": ModelOutput(0.93, 0.71, 0.51, True),
    "R-403-a": ModelOutput(0.18, 0.08, 0.75, True),
    "R-404-a": ModelOutput(0.83, 0.10, 0.64, True),
    "R-404-b": ModelOutput(0.79, 0.14, 0.61, True),
    "R-405-a": ModelOutput(0.32, 0.11, 0.68, True),
    "R-406-a": ModelOutput(0.94, 0.12, 0.12, True),
    "R-407-a": ModelOutput(0.73, 0.09, 0.65, True),
    "R-408-a": ModelOutput(0.63, 0.07, 0.72, True),
    "R-409-a": ModelOutput(0.81, 0.08, 0.74, False),
}

BUNDLE = {
    "bundle_id": "synthetic-policy-v1",
    "previous_bundle": "synthetic-policy-v0",
    "preprocessing": "not-run-synthetic-scores",
    "label_guideline": "visible-damage-v1",
    "route_policy": "quality-first-review-v1",
    "damage_threshold": 0.70,
    "max_blur": 0.45,
    "min_brightness": 0.20,
}

def validate_bundle(bundle):
    required = {
        "bundle_id", "previous_bundle", "preprocessing", "label_guideline",
        "route_policy", "damage_threshold", "max_blur", "min_brightness",
    }
    if type(bundle) is not dict or set(bundle) != required:
        raise ValueError("bundle fields must match the policy schema")
    for name in ("bundle_id", "preprocessing", "label_guideline", "route_policy"):
        nonblank(bundle[name], name)
    if bundle["label_guideline"] != LABEL_GUIDELINE:
        raise ValueError("unsupported label guideline for this fixture")
    if type(bundle["previous_bundle"]) is not str:
        raise ValueError("previous_bundle must be a string; an empty target fails a later gate")
    for name in ("damage_threshold", "max_blur", "min_brightness"):
        unit_score(bundle[name], name)

validate_bundle(BUNDLE)

print("bundle:", BUNDLE["bundle_id"], "threshold=", BUNDLE["damage_threshold"])

def route(photo: PhotoManifestRow, model_output: ModelOutput, bundle=None) -> dict[str, object]:
    bundle = BUNDLE if bundle is None else bundle
    validate_bundle(bundle)
    if type(photo) is not PhotoManifestRow or type(model_output) is not ModelOutput:
        raise ValueError("route requires validated photo metadata and model outputs")
    if not model_output.box_visible:
        action, reason = "request_new_photo", "package_not_visible"
    elif model_output.blur_score > bundle["max_blur"] or model_output.brightness < bundle["min_brightness"]:
        action, reason = "request_new_photo", "image_quality_gate"
    elif model_output.damage_probability >= bundle["damage_threshold"]:
        action, reason = "priority_damage_review", "damage_threshold"
    else:
        action, reason = "normal_review", "below_threshold"

    return {
        "route_id": f"{bundle['bundle_id']}:{photo.photo_id}",
        "photo_id": photo.photo_id,
        "case_id": photo.case_id,
        "routed_day": photo.capture_day,
        "source": photo.source,
        "packaging": photo.packaging,
        "bundle_id": bundle["bundle_id"],
        "previous_bundle": bundle["previous_bundle"],
        "preprocessing": bundle["preprocessing"],
        "label_guideline": bundle["label_guideline"],
        "route_policy": bundle["route_policy"],
        "damage_threshold": bundle["damage_threshold"],
        "max_blur": bundle["max_blur"],
        "min_brightness": bundle["min_brightness"],
        "damage_probability": model_output.damage_probability,
        "blur_score": model_output.blur_score,
        "brightness": model_output.brightness,
        "box_visible": model_output.box_visible,
        "action": action,
        "reason": reason,
    }

print("route keys:", len(route(PHOTOS[0], MODEL_OUTPUTS["R-401-a"])))

test_photos = [photo for photo in PHOTOS if photo.split == "test"]
photo_by_id = {photo.photo_id: photo for photo in PHOTOS}
PHOTO_SNAPSHOT = photo_by_id.copy()
OUTPUT_SNAPSHOT = MODEL_OUTPUTS.copy()
BUNDLE_SNAPSHOT = BUNDLE.copy()
traces = [route(photo, OUTPUT_SNAPSHOT[photo.photo_id], BUNDLE_SNAPSHOT) for photo in test_photos]
trace_by_photo = {trace["photo_id"]: trace for trace in traces}
unsafe_priority_routes = [
    trace["photo_id"]
    for photo, trace in zip(test_photos, traces)
    if trace["action"] == "priority_damage_review" and photo.quality_label != "usable"
]

for photo_id in ("R-404-a", "R-406-a", "R-409-a"):
    trace = trace_by_photo[photo_id]
    print(photo_id, trace["action"], trace["reason"], f"score={trace['damage_probability']}")
print("route counts:", dict(Counter(trace["action"] for trace in traces)))
print("unsafe priority routes:", unsafe_priority_routes)
assert unsafe_priority_routes == []
assert trace_by_photo["R-406-a"]["action"] == "request_new_photo"
assert trace_by_photo["R-409-a"]["action"] == "request_new_photo"

R-404-a priority_damage_review damage_threshold score=0.83
R-406-a request_new_photo image_quality_gate score=0.94
R-409-a request_new_photo package_not_visible score=0.81
route counts: {'priority_damage_review': 3, 'normal_review': 2, 'request_new_photo': 2}
unsafe priority routes: []

R-406-a and R-409-a expose the failure the opening described. Both have high damage scores, but neither provides usable evidence: one is too dark, the other has no package in frame. The endpoint asks for another photo instead of escalating an unsupported claim.

R-406-a fails brightness even though its score is 0.94. R-409-a passes both numerical checks but fails package visibility. The router needs all three quality conditions; the two-dimensional region alone isn't a definition of usable evidence.

Make every route replayable

A correct route still needs an explanation after it leaves the notebook. The following is a proposed project layout, not a supplied download or a list of files the cells create. Implement these pieces when moving from the synthetic exercise to real photographs:

damage-vision-service/
  data/
    label_guidelines.md
    photo_manifest.parquet
    split_manifest.json
  model/
    train_cnn_baseline.py
    evaluate_slices.py
    model_card.md
  service/
    preprocess.py
    route_review.py
    trace_schema.json
  monitoring/
    input_quality_report.py
    delayed_review_outcomes.py
    specialist_shadow_receipt.py
  tests/
    test_case_groups_do_not_cross_splits.py
    test_blurry_photo_never_escalates.py
    test_later_outcomes_join_routes.py
    test_empty_review_window_holds.py
    test_route_trace_is_versioned.py
    test_previous_bundle_required.py

The serving bundle pins image resize and crop behavior, color normalization, model weights, label version, route policy, damage threshold, quality-gate thresholds, and previous-bundle pointer. Preprocessing belongs in the bundle identity: changing center crop to full-frame resize can change whether a torn corner remains visible even when weights stay constant.

For example, Torchvision's documented ResNet-18 weights transform resizes the short side and then center-crops.^{[5]Reference 5resnet18https://docs.pytorch.org/vision/stable/models/generated/torchvision.models.resnet18.html} A corner outside that crop never reaches the classifier. That can remove a useful cue, but it doesn't imply the model will output zero damage. Pin the actual preprocessing implementation and verify that training, evaluation, and serving use their documented policies; a version name alone doesn't enforce them.

The routing cells emit a dictionary with the policy inputs and decision. They retain copies of the bundle, metadata, and score table for later replay; changing the active BUNDLE does not change that comparison. These in-memory copies are still editable and provide no independent provenance. A Python dictionary isn't immutable storage: in a service, write the event to a store with enforced retention and access controls and give each attempt a unique event ID. The fixture's bundle_id:photo_id key assumes one attempt per photo per bundle, and its routed_day assumes routing on the capture day. A real event records its own timestamp. A production bundle ID should resolve to retained weights, code, preprocessing, and policy artifacts, not merely a mutable name.

A replayable trace tells us what the bundle decided. It isn't a confirmed-damage label, and it can't answer whether the route was right. Those outcomes arrive later and join by case.

Backbone selection and multi-label damage classification

The tiny training split contains two positive records for one package, with no supplied images or negative training examples. Validation has no usable positive example. These rows serve contract tests; they cannot establish discrimination between damaged and undamaged parcels. Collect a separate, sufficiently varied image dataset before attempting the steps below.

When building a real packaging damage classifier, backbone selection requires balancing inductive biases against operational constraints:

Compare representations on the same cases

Start with a pretrained ResNet-18 as an understandable baseline, replace its ImageNet classification head, and train for the agreed visible-damage label. The supplied ImageNet weights do not already produce a package-damage probability. On a real dataset, a frozen-feature linear head is one first experiment; unfreezing the backbone is a separate comparison.

Two further candidates illustrate different representations:

ConvNeXt uses $7\times7$ depthwise convolutions in its blocks; its paper evaluates ImageNet and downstream detection and segmentation, not this parcel task.^{[6]Reference 6A ConvNet for the 2020shttps://arxiv.org/abs/2201.03545} Convolution imposes locality and shared spatial weights, but padding and strided downsampling prevent a blanket guarantee of exact translation equivariance for the complete classifier.

The original ViT projects each flattened patch into a token and uses attention across tokens.^{[7]Reference 7An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929} A patch is not replaced by its average pixel: features smaller than a patch are not automatically erased. Input resizing, the learned representation, and training data still affect small-defect detection. Its paper's large-data pretraining results don't make one particular dataset a prerequisite for every pretrained ViT application.

Compare candidates using the same physical-case holdout and a documented training budget. Record checkpoint, input resolution, preprocessing, precision, batch size, device, and measured latency. There is no evidence here that one architecture is fastest or best for package damage.

Formulating multi-label defect classification

A binary visible-damage label fits this lesson's queue decision. If specialists also need defect categories, define an optional multi-label task: one parcel can have both a crushed corner and a torn seam. Each category needs its own annotation rule and examples.

Use $K$ sigmoid outputs, each modeling a category's marginal probability:

$\hat{y}_k = \sigma(z_k) = \frac{1}{1 + e^{-z_k}}, \quad k \in \{1, \dots, K\}$

where a fully observed target vector $\mathbf{y} \in \{0, 1\}^K$ could cover:

• crushed_corner_or_wall
• torn_tape_or_seam
• puncture_hole
• visible_stain

Separate outputs do not imply statistically independent defects; categories can co-occur and share a backbone. Keep unreadable labels in a separately defined readability task rather than quietly equate them with external damage. An image can show a stain without establishing its substance or the condition of the contents.

The model trains against multi-label binary cross-entropy:

$\mathcal{L}_{\text{BCE}} = -\frac{1}{K} \sum_{k=1}^K \left[ y_k \log \hat{y}_k + (1 - y_k) \log (1 - \hat{y}_k) \right]$

For logits, use BCEWithLogitsLoss instead of applying sigmoid and then computing logarithms yourself; PyTorch combines these operations for numerical stability.^{[8]Reference 8BCEWithLogitsLosshttps://docs.pytorch.org/docs/2.14/generated/torch.nn.BCEWithLogitsLoss.html} The equation above averages $K$ observed categories for one example with equal weights. If some annotations are unknown, mask their loss terms and normalize over observed entries. Unknown is not negative.

Evaluate precision, recall, and review workload per category before defining category-specific queues. A low crush score doesn't certify that a product can be restocked. This optional head, its annotations, and any new policy are outside the runnable binary-score fixture.

Preprocessing and threshold calibration

Preprocessing must be explicit. Torchvision's ResNet18_Weights.IMAGENET1K_V1 provides a transform that resizes, center-crops to 224 pixels, rescales, and normalizes channels.^{[5]Reference 5resnet18https://docs.pytorch.org/vision/stable/models/generated/torchvision.models.resnet18.html} A center crop can discard a damaged corner of a parcel. Compare that transform with a documented full-parcel crop or resize policy on validation images, then freeze the chosen policy for test and serving. Don't change preprocessing only at inference time.

Train on training cases only. Use validation cases for checkpoint selection, quality thresholds, and the damage threshold. If fitting a probability calibrator (such as Platt scaling or temperature scaling, as detailed by Guo et al.^{[4]Reference 4On Calibration of Modern Neural Networkshttps://arxiv.org/abs/1706.04599}), reserve held-out validation cases rather than fitting it on the final test set. Apply image augmentation only after case splitting so transformed copies can't cross the boundary.

Threshold selection is a workload decision. On validation cases, tabulate missed damaged cases, unnecessary priority reviews, and the total priority queue size for each candidate threshold. A lower threshold may recover more damage while exceeding specialist capacity. Choose a rule against an agreed recall target and review budget, freeze it, and evaluate the final holdout once. The fixture's 0.70 wasn't selected by this process and has no empirical justification.

Keep the quality stage in that evaluation. A classifier can look excellent on accepted images while the quality gate rejects many legitimate damaged photos. Report the rejection rate, the fraction of human-usable photos rejected, damage recall among usable cases, and performance by source, lighting, packaging, and defect size. Include counts; a slice with one positive case can't establish dependable recall.

For a portfolio submission, include permitted example images, the immutable split manifest, training command and checkpoint, preprocessing specification, validation threshold table, and held-out error analysis. Until those artifacts exist, describe the result as a tested routing prototype, not a trained damage classifier.

Wait for specialist outcomes before promotion

The next evidence arrives on a different clock. A new warehouse camera, winter lighting, a mobile upload compressor, or new packaging graphics can change incoming pixels before a confirmed-damage label exists. Split immediate signals from delayed quality so one can't masquerade as the other.

Separate immediate checks from delayed quality:

Google Cloud's MLOps guidance treats data validation, model validation, serving, monitoring, metadata, and continuous training as connected stages rather than a one-time deploy.^{[9]Reference 9MLOps: Continuous Delivery and Automation Pipelines in Machine Learning.https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning} Apply that separation here: a jump in dark warehouse-camera photos starts an investigation or candidate run. Promotion still waits for the delayed specialist join, not an automatic weight swap.

The final local report simulates outcomes arriving after routing, then joins them to physical cases. Each outcome carries its reviewer, label guideline, review day, and availability day. The report's cutoff is day 12; it uses only outcomes available by then, with the matching guideline and a review after the case's last capture. Duplicate case outcomes are rejected rather than silently overwrite one another. A real correction history needs versioned outcomes and a documented selection rule, not this one-outcome fixture. Review normal and rejected cases as well as priority cases; labels collected only from the priority queue can't measure false negatives.

The calculation measures review precision and recall once per test case with at least one human-usable image. This is a conditional evaluation cohort, not the whole incoming population. Case actions come from actual routes: any priority route takes precedence, then any normal route, otherwise the case requests a new photo. Human quality labels select the evaluation cohort but never rewrite its served actions.

@dataclass(frozen=True)
class SpecialistOutcome:
    case_id: str
    reviewed_day: int
    confirmed_damage: bool
    reviewer_id: str
    guideline_version: str
    available_day: int

    def __post_init__(self):
        for name in ("case_id", "reviewer_id", "guideline_version"):
            nonblank(getattr(self, name), name)
        if type(self.confirmed_damage) is not bool:
            raise ValueError("confirmed_damage must be a boolean")
        if type(self.reviewed_day) is not int or self.reviewed_day < 1:
            raise ValueError("reviewed_day must be a positive integer")
        if type(self.available_day) is not int or self.available_day < self.reviewed_day:
            raise ValueError("available_day must be an integer at or after reviewed_day")

SPECIALIST_OUTCOMES = [
    SpecialistOutcome("R-404", 12, True, "S-12", "visible-damage-v1", 12),
    SpecialistOutcome("R-405", 12, False, "S-08", "visible-damage-v1", 12),
    SpecialistOutcome("R-407", 12, False, "S-12", "visible-damage-v1", 12),
    SpecialistOutcome("R-408", 12, True, "S-08", "visible-damage-v1", 12),
]
EVALUATION_DAY = 12

usable_test_case_ids = {
    photo.case_id
    for photo in test_photos
    if photo.quality_label == "usable"
}
traces_by_case = defaultdict(list)
for trace in traces:
    traces_by_case[trace["case_id"]].append(trace)

print("usable test cases:", sorted(usable_test_case_ids))
print("specialist outcomes:", len(SPECIALIST_OUTCOMES))

def aggregate_case_action(case_id: str) -> str:
    case_traces = traces_by_case.get(case_id, [])
    if not case_traces:
        raise ValueError(f"no traces for {case_id}")
    actions = [trace.get("action") for trace in case_traces]
    if any(action not in ("priority_damage_review", "normal_review", "request_new_photo") for action in actions):
        raise ValueError("unknown review action")
    if any(trace["action"] == "priority_damage_review" for trace in case_traces):
        return "priority_damage_review"
    if any(trace["action"] == "normal_review" for trace in case_traces):
        return "normal_review"
    return "request_new_photo"

case_actions = {
    case_id: aggregate_case_action(case_id)
    for case_id in sorted(usable_test_case_ids)
}
def eligible_outcomes(outcomes, as_of_day):
    if type(as_of_day) is not int or as_of_day < 1:
        raise ValueError("as_of_day must be a positive integer")
    if as_of_day < max((photo.capture_day for photo in test_photos), default=0):
        raise ValueError("evaluation cutoff cannot precede the recorded routes")
    if type(outcomes) is not list:
        raise ValueError("outcomes must be a list of SpecialistOutcome records")
    if any(type(outcome) is not SpecialistOutcome for outcome in outcomes):
        raise ValueError("outcomes must be validated SpecialistOutcome records")
    case_ids = [outcome.case_id for outcome in outcomes]
    if len(set(case_ids)) != len(case_ids):
        raise ValueError("duplicate specialist case outcomes")
    last_capture = {
        case_id: max(photo.capture_day for photo in test_photos if photo.case_id == case_id)
        for case_id in usable_test_case_ids
    }
    return {
        outcome.case_id: outcome
        for outcome in outcomes
        if outcome.case_id in usable_test_case_ids
        and outcome.available_day <= as_of_day
        and outcome.guideline_version == BUNDLE_SNAPSHOT["label_guideline"]
        and outcome.reviewed_day > last_capture[outcome.case_id]
    }

outcome_by_case = eligible_outcomes(SPECIALIST_OUTCOMES, EVALUATION_DAY)
joined_outcomes = [
    (case_id, case_actions[case_id], outcome_by_case[case_id].confirmed_damage)
    for case_id in sorted(case_actions.keys() & outcome_by_case.keys())
]

print("case_actions:", case_actions)
print("joined usable cases:", len(joined_outcomes))

def rate_or_none(numerator: int, denominator: int) -> float | None:
    return round(numerator / denominator, 3) if denominator else None

true_positives = sum(
    action == "priority_damage_review" and confirmed_damage
    for _, action, confirmed_damage in joined_outcomes
)
false_positives = sum(
    action == "priority_damage_review" and not confirmed_damage
    for _, action, confirmed_damage in joined_outcomes
)
false_negatives = sum(
    action != "priority_damage_review" and confirmed_damage
    for _, action, confirmed_damage in joined_outcomes
)
delayed_quality = {
    "priority_precision": rate_or_none(true_positives, true_positives + false_positives),
    "priority_recall": rate_or_none(true_positives, true_positives + false_negatives),
    "reviewed_usable_cases": len(joined_outcomes),
}

source_slices = {
    source: {
        "photos": len(rows),
        "request_new_photo_rate": round(
            sum(trace["action"] == "request_new_photo" for trace in rows) / len(rows),
            3,
        ),
    }
    for source in sorted({trace["source"] for trace in traces})
    for rows in [[trace for trace in traces if trace["source"] == source]]
}

packaging_slices = {
    packaging: {
        "photos": len(rows),
        "priority_rate": round(
            sum(trace["action"] == "priority_damage_review" for trace in rows) / len(rows),
            3,
        ),
    }
    for packaging in sorted({trace["packaging"] for trace in traces})
    for rows in [[trace for trace in traces if trace["packaging"] == packaging]]
}

# No sample-count cutoff turns synthetic data into measured model evidence.
delayed_quality["evidence_type"] = "synthetic_policy_fixture"

source_quality = {}
for source in sorted(source_slices):
    source_case_ids = {photo.case_id for photo in test_photos if photo.source == source}
    rows = [row for row in joined_outcomes if row[0] in source_case_ids]
    tp = sum(action == "priority_damage_review" and label for _, action, label in rows)
    fp = sum(action == "priority_damage_review" and not label for _, action, label in rows)
    fn = sum(action != "priority_damage_review" and label for _, action, label in rows)
    source_quality[source] = {
        "cases": len(rows), "tp": tp, "fp": fp, "fn": fn,
        "precision": rate_or_none(tp, tp + fp),
        "recall": rate_or_none(tp, tp + fn),
    }

print("delayed_quality:", delayed_quality)
print("source_slices:", source_slices)
print("packaging_slices:", packaging_slices)
print("source_quality:", source_quality)

required_trace_fields = {
    "route_id", "photo_id", "case_id", "routed_day", "source", "packaging",
    "bundle_id", "previous_bundle", "preprocessing", "label_guideline",
    "route_policy", "damage_threshold", "max_blur", "min_brightness",
    "damage_probability", "blur_score", "brightness", "box_visible",
    "action", "reason",
}
def route_replays(trace):
    if type(trace) is not dict or set(trace) != required_trace_fields:
        return False
    numeric_fields = {
        "damage_threshold", "max_blur", "min_brightness",
        "damage_probability", "blur_score", "brightness",
    }
    string_fields = required_trace_fields - numeric_fields - {"routed_day", "box_visible"}
    if any(type(trace[name]) is not str for name in string_fields):
        return False
    if type(trace["routed_day"]) is not int or type(trace["box_visible"]) is not bool:
        return False
    photo_id = trace["photo_id"]
    if photo_id not in PHOTO_SNAPSHOT or photo_id not in OUTPUT_SNAPSHOT:
        return False
    try:
        for name in numeric_fields:
            unit_score(trace[name], name)
        expected = route(PHOTO_SNAPSHOT[photo_id], OUTPUT_SNAPSHOT[photo_id], BUNDLE_SNAPSHOT)
    except (ValueError, TypeError, KeyError):
        return False
    return trace == expected

outcome_case_ids = [outcome.case_id for outcome in SPECIALIST_OUTCOMES]
release_gates = {
    **manifest_checks,
    "route_ids_unique": len({trace["route_id"] for trace in traces}) == len(traces),
    "all_test_photos_routed": (
        Counter(trace["photo_id"] for trace in traces)
        == Counter(photo.photo_id for photo in test_photos)
    ),
    "unusable_photos_abstain": all(
        trace_by_photo[photo.photo_id]["action"] == "request_new_photo"
        for photo in test_photos
        if photo.quality_label == "unusable"
    ),
    "unsafe_priority_routes_absent": not unsafe_priority_routes,
    "route_traces_replayable": all(route_replays(trace) for trace in traces),
    "specialist_outcome_case_ids_unique": len(set(outcome_case_ids)) == len(outcome_case_ids),
    "specialist_outcomes_join_usable_test_cases": set(outcome_case_ids) <= usable_test_case_ids,
    "usable_test_cases_have_specialist_outcomes": usable_test_case_ids <= set(outcome_by_case),
    "specialist_outcomes_arrive_after_capture": all(
        outcome.reviewed_day > max(
            photo.capture_day for photo in test_photos if photo.case_id == outcome.case_id
        )
        for outcome in SPECIALIST_OUTCOMES
        if outcome.case_id in usable_test_case_ids
    ),
    "specialist_outcome_label_contract_matches_bundle": all(
        outcome.guideline_version == BUNDLE_SNAPSHOT["label_guideline"]
        for outcome in SPECIALIST_OUTCOMES
    ),
    "specialist_outcomes_available_at_cutoff": all(
        outcome.available_day <= EVALUATION_DAY for outcome in SPECIALIST_OUTCOMES
    ),
    "priority_metrics_defined": (
        delayed_quality["priority_precision"] is not None
        and delayed_quality["priority_recall"] is not None
    ),
    "synthetic_evidence_disclosed": delayed_quality["evidence_type"] == "synthetic_policy_fixture",
    "source_slices_recorded": set(source_slices) == {"customer_phone", "warehouse_camera"},
    "packaging_slices_recorded": set(packaging_slices) >= {"corrugated", "mailer"},
    "rollback_target_named": bool(BUNDLE_SNAPSHOT["previous_bundle"].strip()),
}

print("release_gates_pass:", all(release_gates.values()))

receipt = {
    "candidate_bundle": BUNDLE_SNAPSHOT["bundle_id"],
    "previous_bundle": BUNDLE_SNAPSHOT["previous_bundle"],
    "preprocessing": BUNDLE_SNAPSHOT["preprocessing"],
    "label_guideline": BUNDLE_SNAPSHOT["label_guideline"],
    "route_policy": BUNDLE_SNAPSHOT["route_policy"],
    "evaluation_day": EVALUATION_DAY,
    "route_traces": len(traces),
    "later_specialist_outcomes": len(SPECIALIST_OUTCOMES),
    "joined_usable_cases": len(joined_outcomes),
    "case_actions": case_actions,
    "delayed_quality": delayed_quality,
    "source_slices": source_slices,
    "packaging_slices": packaging_slices,
    "source_quality": source_quality,
    "release_gates": release_gates,
    "candidate_decision": "hold_synthetic_fixture",
}

print("bundle:", receipt["candidate_bundle"], "rollback:", receipt["previous_bundle"])
print("routes/outcomes:", receipt["route_traces"], receipt["later_specialist_outcomes"], receipt["joined_usable_cases"])
print("delayed_quality:", receipt["delayed_quality"])
print("source_slices:", receipt["source_slices"])
print("source_quality:", source_quality)
print("release_gates_pass:", all(receipt["release_gates"].values()))
print("candidate_decision:", receipt["candidate_decision"])
assert all(receipt["release_gates"].values())
assert receipt["candidate_decision"] == "hold_synthetic_fixture"

bundle: synthetic-policy-v1 rollback: synthetic-policy-v0
routes/outcomes: 7 4 4
delayed_quality: {'priority_precision': 0.5, 'priority_recall': 0.5, 'reviewed_usable_cases': 4, 'evidence_type': 'synthetic_policy_fixture'}
source_slices: {'customer_phone': {'photos': 4, 'request_new_photo_rate': 0.25}, 'warehouse_camera': {'photos': 3, 'request_new_photo_rate': 0.333}}
source_quality: {'customer_phone': {'cases': 2, 'tp': 1, 'fp': 1, 'fn': 0, 'precision': 0.5, 'recall': 1.0}, 'warehouse_camera': {'cases': 2, 'tp': 0, 'fp': 0, 'fn': 1, 'precision': None, 'recall': 0.0}}
release_gates_pass: True
candidate_decision: hold_synthetic_fixture

The variable release_gates checks the synthetic harness's contract. Defined precision and recall are not acceptance criteria for model quality. Even when every check passes, the decision stays hold_synthetic_fixture: there's no trained model or real image evidence to promote. rollback_target_named checks only a nonblank name, not whether an earlier artifact exists, loads, or restores service. A real shadow run needs permission to process images, a pinned runnable model, representative cases, acceptance criteria chosen before examining results, and a tested recovery path.

The delayed counts are case-level, not photo-level. A true positive (TP) is a priority route the specialist later confirmed; a false positive (FP) is a priority route they rejected; a false negative (FN) is a damaged case not sent to priority review, including one incorrectly asked for a new photo. A true negative (TN) is an undamaged case not sent to priority review. The four joined cases with usable images are:

Priority precision is $1 / (1 + 1) = 0.50$ and recall is $1 / (1 + 1) = 0.50$. R-404-a and R-404-b collapse to one case. Cases without any human-usable photos are excluded from this conditional damage table; their quality-routing errors need their own report.

Subgroup slice analysis: detecting hidden failure modes

An aggregate can hide different error patterns. Test a hypothesis with labeled examples instead of assigning a failure rate to a material or defect without evidence:

Define slices before final evaluation. Include denominators, uncertainty, and examples of errors. A source name doesn't establish brightness, focal distance, or camera quality.

The source slices in the code output expose this exact divergence: customer-phone cases show one true positive and one false positive, while warehouse cases have one true negative and one false negative. Warehouse recall is zero; its precision is undefined because it has no priority cases. These two-case slices demonstrate arithmetic, not a final source-performance conclusion. If a real case has photos from both sources, this slice convention includes it in both, so the slice counts won't sum to the overall total.

Practice: break the vision contract

Change one condition at a time, predict which check should reject it, then rerun the examples and inspect the first failure. These mutations test the harness. Improving a hand-written test score isn't model tuning; choosing thresholds after viewing real test outcomes contaminates the holdout.

1. Change R-401-b split from train to validation. Which dataset gate fails?
2. Change R-406-a brightness in MODEL_OUTPUTS from 0.12 to 0.30. Why does this expose a quality-detector failure rather than prove the escalation is safe?
3. Remove damage_threshold from the response trace. Which receipt gate fails?
4. Change R-408-a damage probability in MODEL_OUTPUTS from 0.63 to 0.75. Which delayed metrics improve?
5. Set BUNDLE["previous_bundle"] = "" or " " before rebuilding routes. Which rollback check fails?
6. Replace SPECIALIST_OUTCOMES with an empty list. Why do evidence gates fail instead of crashing or passing?
7. Add SpecialistOutcome("R-999", 12, True, "S-12", "visible-damage-v1", 12). Which join gate fails?
8. Change R-408 specialist outcome to guideline visible-damage-v2. Which provenance gate fails?
9. Keep R-408 reviewed on day 12, but change its availability to day 13. Which metric denominator changes at the day-12 cutoff?
10. Append a duplicate R-404 outcome or put issue_refund in a case's action list. Why should neither silently become a normal evaluation record?

For mutation 9, only three cases join at day 12: TP=1, FP=1, FN=0, so precision remains 0.5 and the observed recall becomes 1.0. The missing damaged case has not become a successful route. Availability and completeness gates fail, and the receipt remains a hold. Mutation 10 raises a clear error before ambiguous outcomes or unsupported actions can enter the calculation.

Practice answer sketches