Dataset Pipelines and Data Quality

A classifier can train on prepared tensors, but the harder production question comes earlier: what must be true before a ticket is allowed to become one of those tensors?

A routing model can appear impressive for the wrong reason. If a build message, a paraphrase of it, or another message from the same conversation is present in both training and evaluation, the score measures memory or case overlap instead of useful generalization. A dataset pipeline is the code and evidence trail that turns raw records into rows a model may learn from and held-out rows a reviewer may trust.

That idea scales past this small classifier. Pretraining corpora, fine-tuning examples, retrieval evaluations, and agent benchmarks all need provenance, cleaning rules, frozen splits, and contamination checks. A model result is only as credible as the data path beneath it.

Raw records aren't training examples yet

Build a dataset for the ticket-escalation classifier from the previous chapter. Each raw row was exported from a developer support conversation:

Some rows aren't safe:

Start with a schema gate. It catches absent fields, blank messages, and labels outside the reviewed vocabulary. Schema validation can't prove that a label is correct, but it prevents malformed rows from silently becoming model supervision.

Normalize conservatively, then fingerprint

Tickets often differ only because one export added spaces or changed letter case. For this routing fixture, Unicode compatibility normalization, case-folding, and whitespace collapse are a reviewable starting rule for revealing obvious copies. Normalization is task-specific: confirm that it doesn't erase distinctions your label depends on. It doesn't justify deleting IDs, dates, environment names, or negation: build passed and build not passed must remain different examples.

Once text is normalized, a deterministic hash becomes an exact fingerprint. It isn't a semantic similarity score. It says only that two normalized strings match byte for byte.

The shortened hashes make the output readable. Store the full digest in a released artifact so display truncation doesn't weaken the identity check.

Duplicate labels can expose a deeper problem

Dropping a repeated row is safe only when its supervision agrees. Suppose two identical ticket texts receive different labels. One of three things may be true: an annotation is wrong, the route changed with hidden context, or the feature set omitted a decisive field such as deployment target or policy status.

A careful pipeline quarantines that fingerprint instead of picking the first label and pretending the conflict didn't happen.

This is a small but important shift in thinking. Data cleaning isn't only deletion. It's investigation: the pipeline should expose cases that could teach you the current label definition or feature schema is incomplete.

Split identities, not individual rows

The earlier validation chapter introduced a train, validation, and test split. Data pipelines must decide the split unit: the entity that isn't allowed to cross those boundaries.

For developer tickets, splitting by row is too weak. Multiple turns from the same conversation share incident details and issue history. If one turn trains the model while another appears in test, evaluation becomes easier than a truly new conversation. Split on conversation_id, not ticket_id.

A stable hash assignment gives every conversation the same bucket on every rerun. The cryptographic hash here isn't protecting a secret; it produces repeatable buckets from a frozen rule.

A hash rule keeps assignments stable, but it doesn't guarantee exact split sizes or label balance. Audit the resulting counts. If a release needs better coverage, freeze a documented group-aware assignment rule and version the change.

From leakage to LLM benchmark contamination

If a training record duplicates a held-out record, you already know the metric is compromised. In language-model work this problem appears at a larger boundary: web-scale pretraining data can contain benchmark prompts, answers, or close variants. The evaluation set then no longer measures behavior on genuinely unseen tasks.

Brown et al. performed n-gram overlap analysis between GPT-3 training data and evaluation sets. Their paper reports that a filtering bug left near-complete overlap for several language-modeling benchmarks and the Children's Book Test, so those results were omitted from aggregate reporting.^{[1]Reference 1Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165} This is what scientific hygiene looks like: measure contamination, document it, and withhold claims when the test is no longer clean.

Our first guard is exact overlap between prepared training text and locked evaluation text.

An exact guard is necessary but not sufficient. A benchmark row may be lightly edited while preserving the task and answer.

Measure near-duplicate candidates

Represent a message as a set of adjacent word pairs, called shingles. The Jaccard similarity between two sets is:

$$J(A, B) = \frac{|A \cap B|}{|A \cup B|}$$

The numerator counts shared shingles; the denominator counts all unique shingles across both messages. A score near 1.0 means surface text is highly similar. Broder defined shingle-set resemblance and showed how compact per-document sketches can estimate it without comparing full documents, so this idea could scale past exhaustive pairwise comparison.^{[2]Reference 2On the Resemblance and Containment of Documents.https://www.cs.princeton.edu/courses/archive/spring13/cos598C/broder97resemblance.pdf}

For build has not passed and my build has not passed, the first set has three word pairs and the second has four. Three are shared, so similarity is 3 / 4 = 0.75.

Short messages need explicit handling because they may contain fewer words than the requested shingle width. Blank messages should already have failed the schema gate. Don't automatically delete every high-similarity example. Templates may be legitimate in training, and false matches can erase rare issues. Near-duplicate screening should produce a review queue or a documented rule tuned against real examples.

Build a reproducible artifact

Now put the mechanics together. This compact artifact build repeats the conflict check inside the release path so the earlier quarantine rule can't be skipped. It:

1. validates required fields and route labels,
2. normalizes messages,
3. groups exact fingerprints, drops agreeing copies, and quarantines conflicting labels,
4. splits by conversation rather than row,
5. writes cleaned JSONL plus a rejection log and manifest.

The manifest is a small dataset receipt. Gebru et al. propose datasheets that document a dataset's motivation, composition, collection process, and recommended uses so consumers can judge whether it fits their task.^{[3]Reference 3Datasheets for Datasetshttps://arxiv.org/abs/1803.09010} A production dataset deserves a full datasheet; this lab begins with the fields your next model run needs immediately.

The printed digest prefix is for human scanning; the manifest stores the full SHA-256 digest. The build groups fingerprints before choosing representatives, so agreeing exact copies are dropped while every row in a conflicting fingerprint group is quarantined. Preserve all rejection records for review.

One edge case remains: this lab assigns the conversation split while building the fingerprint groups, then keeps only the first agreeing duplicate. If the same normalized text appears in two conversations with the same label, the second conversation's group can vanish from its split. That is acceptable for this compact fixture, but production should usually deduplicate (or apply a within-group policy) and then split, so whole conversation groups are not silently dropped across buckets.

Reload the receipt and enforce invariants

An artifact isn't trustworthy because it exists. Load it as a downstream training job would and fail early when its contract is broken.

These four checks protect different truths: a split invariant, an artifact identity, a supervised-label contract, and an exact-deduplication invariant. A training script should reject data that fails any one of them.

Audit coverage as well as correctness

A cleaned dataset can still be unhelpful. If validation contains no escalation cases, its accuracy won't reveal whether the model misses urgent builds. Inspect label coverage by split before launching training.

With a tiny sample, the right response isn't to keep resampling until validation looks convenient. Collect additional reviewed groups or adopt a documented group-aware assignment rule, record the change as a new dataset version, and rerun evaluation.

Protect sensitive content before export

Developer support text can contain emails, phone numbers, incident IDs, access tokens, or private incident notes. A beginner pipeline shouldn't promise that one regular expression safely anonymizes all personal data. It should detect obvious sensitive patterns, quarantine rows for a reviewed redaction path, and record that decision in the manifest.

This is a deliberately narrow detector. Ticket 501 still contains an incident ID and private timeline reference, so "no email match" isn't the same as "safe to export." A production export needs privacy review, access controls, retention rules, and tested redaction coverage for its actual data sources. A pipeline that hides what it can't detect is less useful than one that exposes its limits.

Version changes should be visible

Data changes are model changes in disguise. Adding reviewed examples, adjusting normalization, repairing labels, or changing split rules can all move a metric. Give each released artifact a content digest and pipeline version so a comparison can name what changed.

Hugging Face Datasets applies a related idea in its cache: a fingerprint tracks dataset state and is updated when transformations such as map() or shuffle() change the data processing history.^{[4]Reference 4Datasets Documentation.https://huggingface.co/docs/datasets/index} You still need human-readable documentation alongside machine fingerprints. Record source snapshots, license and privacy decisions, label guidelines, exclusions, transformations, split policy, known gaps, and intended uses in a datasheet or equivalent review artifact.^{[3]Reference 3Datasheets for Datasetshttps://arxiv.org/abs/1803.09010}

What scales from this small lab

The developer-ticket example contains only a handful of rows, but its controls map directly to larger AI systems:

The general habit is simple: never quote a model score without being able to identify the exact data artifact, the split policy, and the contamination check that made the score interpretable.

Practice: try to break the receipt

Use the runnable labs above as a controlled failure exercise. Make one change at a time, predict which check should fail, then run the example.

1. In 08-verify-artifact.py, append {**rows[0], "ticket_id": 999, "split": "test"} to rows before building groups.
2. In 08-verify-artifact.py, append a newline to rows_path after loading saved_manifest and before computing digest_matches.
3. In 06-shingle-overlap.py, print jaccard("build", "build") and jaccard("", "").
4. In 04-stable-grouped-split.py, assign buckets from str(row["ticket_id"]) instead of row["conversation_id"]. Explain what can happen to two turns from one conversation.

Expected observations

1. no_group_leakage becomes False because one conversation now crosses splits.
2. digest_matches becomes False because artifact bytes no longer match released receipt.
3. One-word match returns 1.0; two blank strings return 0.0. Blank rows belong at schema gate, not similarity queue.
4. Turns from same conversation can land in different splits, making evaluation reward familiarity with same case.

Carry the receipt into feature engineering

A trustworthy dataset row can be traced from raw export to released artifact. A training loop tells you how weights changed; a validation plan tells you how evidence was measured; a dataset receipt tells you whether those measurements deserve trust.

The next chapter turns timestamped incident and deploy records into model inputs. The same discipline still applies: each feature needs a definition, a time boundary, and a version so offline training and live serving mean the same thing.