Dataset Pipelines and Data Quality

The last chapter trained a classifier from prepared tensors. This chapter asks what must be true before a ticket is allowed to become one of those tensors.

A support model can appear impressive for the wrong reason. If a refund message, a paraphrase of it, or another message from the same conversation is present in both training and evaluation, the score measures memory or customer overlap instead of useful generalization. A dataset pipeline is the code and evidence trail that turns raw records into rows a model may learn from and held-out rows a reviewer may trust.

That idea scales past this small classifier. Pretraining corpora, fine-tuning examples, retrieval evaluations, and agent benchmarks all need provenance, cleaning rules, frozen splits, and contamination checks. A model result is only as credible as the data path beneath it.

Raw records aren't training examples yet

We'll build a dataset for the ticket-escalation classifier from the previous chapter. Each raw row was exported from a support conversation:

Some rows aren't safe:

Start with a schema gate. It catches absent fields, blank messages, and labels outside the reviewed vocabulary. Schema validation can't prove that a label is correct, but it prevents malformed rows from silently becoming model supervision.

Normalize conservatively, then fingerprint

Tickets often differ only because one export added spaces or changed letter case. For this routing fixture, Unicode compatibility normalization, case-folding, and whitespace collapse are a reviewable starting rule for revealing obvious copies. Normalization is task-specific: confirm that it doesn't erase distinctions your label depends on. It doesn't justify deleting prices, order identifiers, dates, or negation: refund received and refund not received must remain different examples.

Once text is normalized, a deterministic hash becomes an exact fingerprint. It isn't a semantic similarity score. It says only that two normalized strings match byte for byte.

The shortened hashes make the output readable. Store the full digest in a released artifact so display truncation doesn't weaken the identity check.

Duplicate labels can expose a deeper problem

Dropping a repeated row is safe only when its supervision agrees. Suppose two identical ticket texts receive different labels. One of three things may be true: an annotation is wrong, the route changed with hidden context, or the feature set omitted a decisive field such as order value or policy status.

A careful pipeline quarantines that fingerprint instead of picking the first label and pretending the conflict didn't happen.

This is a small but important shift in thinking. Data cleaning isn't only deletion. It's investigation: the pipeline should expose cases that could teach you the current label definition or feature schema is incomplete.

Split identities, not individual rows

The earlier validation chapter introduced a train, validation, and test split. Data pipelines must decide the split unit: the entity that isn't allowed to cross those boundaries.

For support tickets, splitting by row is too weak. Multiple turns from the same conversation share customer details and issue history. If one turn trains the model while another appears in test, evaluation becomes easier than a truly new conversation. Split on conversation_id, not ticket_id.

A stable hash assignment gives every conversation the same bucket on every rerun. The cryptographic hash here isn't protecting a secret; it produces repeatable buckets from a frozen rule.

A hash rule keeps assignments stable, but it doesn't guarantee exact split sizes or label balance. Audit the resulting counts. If a release needs better coverage, freeze a documented group-aware assignment rule and version the change.

From leakage to LLM benchmark contamination

If a training record duplicates a held-out record, you already know the metric is compromised. In language-model work this problem appears at a larger boundary: web-scale pretraining data can contain benchmark prompts, answers, or close variants. The evaluation set then no longer measures behavior on genuinely unseen tasks.

Brown et al. performed n-gram overlap analysis between GPT-3 training data and evaluation sets. Their paper reports that a filtering bug left near-complete overlap for several language-modeling benchmarks and the Children's Book Test, so those results were omitted from aggregate reporting.^[1] This is what scientific hygiene looks like: measure contamination, document it, and withhold claims when the test is no longer clean.

Our first guard is exact overlap between prepared training text and locked evaluation text.

An exact guard is necessary but not sufficient. A benchmark row may be lightly edited while preserving the task and answer.

Measure near-duplicate candidates

Represent a message as a set of adjacent word pairs, called shingles. The Jaccard similarity between two sets is:

$$J(A, B) = \frac{|A \cap B|}{|A \cup B|}$$

The numerator counts shared shingles; the denominator counts all unique shingles across both messages. A score near 1.0 means surface text is highly similar. Broder defined shingle-set resemblance and showed how compact per-document sketches can estimate it without comparing full documents, so this idea could scale past exhaustive pairwise comparison.^[2]

For refund has not arrived and my refund has not arrived, the first set has three word pairs and the second has four. Three are shared, so similarity is 3 / 4 = 0.75.

Short messages need explicit handling because they may contain fewer words than the requested shingle width. Blank messages should already have failed the schema gate. Don't automatically delete every high-similarity example. Templates may be legitimate in training, and false matches can erase rare issues. Near-duplicate screening should produce a review queue or a documented rule tuned against real examples.

Build a reproducible artifact

Now put the mechanics together. This compact artifact build repeats the conflict check inside the release path so the earlier quarantine rule can't be skipped. It:

1. validates required fields and route labels,
2. normalizes messages,
3. groups exact fingerprints, drops agreeing copies, and quarantines conflicting labels,
4. splits by conversation rather than row,
5. writes cleaned JSONL plus a rejection log and manifest.

The manifest is a small dataset receipt. Gebru et al. propose datasheets that document a dataset's motivation, composition, collection process, and recommended uses so consumers can judge whether it fits their task.^[3] A production dataset deserves a full datasheet; this lab begins with the fields your next model run needs immediately.

The printed digest prefix is for human scanning; the manifest stores the full SHA-256 digest. The build groups fingerprints before choosing representatives, so agreeing exact copies are dropped while every row in a conflicting fingerprint group is quarantined. Preserve all rejection records for review.

Reload the receipt and enforce invariants

An artifact isn't trustworthy because it exists. Load it as a downstream training job would and fail early when its contract is broken.

These four checks protect different truths: a split invariant, an artifact identity, a supervised-label contract, and an exact-deduplication invariant. A training script should reject data that fails any one of them.

Audit coverage, not only correctness

A cleaned dataset can still be unhelpful. If validation contains no escalation cases, its accuracy won't reveal whether the model misses urgent refunds. Inspect label coverage by split before launching training.

With a tiny sample, the right response isn't to keep resampling until validation looks convenient. Collect additional reviewed groups or adopt a documented group-aware assignment rule, record the change as a new dataset version, and rerun evaluation.

Protect sensitive content before export

Customer support text can contain emails, phone numbers, order IDs, addresses, or payment details. A beginner pipeline shouldn't promise that one regular expression safely anonymizes all personal data. It should detect obvious sensitive patterns, quarantine rows for a reviewed redaction path, and record that decision in the manifest.

This is a deliberately narrow detector. Ticket 501 still contains an order ID, so "no email match" isn't the same as "safe to export." A production export needs privacy review, access controls, retention rules, and tested redaction coverage for its actual data sources. A pipeline that hides what it can't detect is less useful than one that exposes its limits.

Version changes should be visible

Data changes are model changes in disguise. Adding reviewed examples, adjusting normalization, repairing labels, or changing split rules can all move a metric. Give each released artifact a content digest and pipeline version so a comparison can name what changed.

Hugging Face Datasets applies a related idea in its cache: a fingerprint tracks dataset state and is updated when transformations such as map() or shuffle() change the data processing history.^[4] You still need human-readable documentation alongside machine fingerprints. Record source snapshots, license and privacy decisions, label guidelines, exclusions, transformations, split policy, known gaps, and intended uses in a datasheet or equivalent review artifact.^[3]

What scales from this small lab

The support example contains only a handful of rows, but its controls map directly to larger AI systems:

The general habit is simple: never quote a model score without being able to identify the exact data artifact, the split policy, and the contamination check that made the score interpretable.

Practice: Try to break the receipt

Use the runnable labs above as a controlled failure exercise. Make one change at a time, predict which check should fail, then run the example.

1. In 08-verify-artifact.py, append {**rows[0], "ticket_id": 999, "split": "test"} to rows before building groups.
2. In 08-verify-artifact.py, append a newline to rows_path after loading saved_manifest and before computing digest_matches.
3. In 06-shingle-overlap.py, print jaccard("refund", "refund") and jaccard("", "").
4. In 04-stable-grouped-split.py, assign buckets from str(row["ticket_id"]) instead of row["conversation_id"]. Explain what can happen to two turns from one conversation.

Expected observations

1. no_group_leakage becomes False because one conversation now crosses splits.
2. digest_matches becomes False because artifact bytes no longer match released receipt.
3. One-word match returns 1.0; two blank strings return 0.0. Blank rows belong at schema gate, not similarity queue.
4. Turns from same conversation can land in different splits, making evaluation reward familiarity with same case.

Carry the receipt into feature engineering

You can now trace a dataset row from raw export to released artifact. A training loop tells you how weights changed; a validation plan tells you how evidence was measured; a dataset receipt tells you whether those measurements deserve trust.

The next chapter turns timestamped shipment records into model inputs. The same discipline still applies: each feature needs a definition, a time boundary, and a version so offline training and live serving mean the same thing.

Explain why the score deserves trust

Before moving on, explain why a model trained on duplicated or contaminated data can show a stronger score while becoming no more useful for new customers. Then answer these checks.

Evaluation rubric

• Foundational: Explains schema validation, conservative normalization, exact fingerprints, and why duplicates must be resolved before evaluation.
• Intermediate: Builds a deterministic pipeline that writes rows and a manifest, keeps conversations in one split, and verifies downstream invariants.
• Advanced: Designs a contamination and documentation gate for an LLM training or evaluation release, including near-match review, coverage limits, and sensitive-data handling.

Common Pitfalls

• Splitting rows before grouping by conversation or document source, which makes evaluation easier through correlated examples.
• Dropping duplicate text without checking label conflicts, which hides annotation errors or missing features.
• Publishing a benchmark result before exact and near-duplicate contamination checks are recorded.
• Saving cleaned rows without a versioned manifest, split rule, limitations, and privacy review receipt.

Reuse this dataset contract

• A cleaned support-ticket JSONL artifact with stable grouped splits and reject logs.
• A contamination gate that blocks exact overlap and queues near matches for review.
• A dataset receipt that lets a later training run identify its inputs and limitations.

Key Terms

• Schema gate: Validation that rows have required fields and admissible values.
• Fingerprint: Deterministic identifier for exact normalized content.
• Split unit: Entity kept entirely inside one dataset partition.
• Contamination: Training exposure to material intended to measure unseen behavior.
• Datasheet: Human-readable documentation of dataset creation, contents, uses, and limits.