Dataset Pipelines and Data Quality

Tickets 101 and 102 look like two separate training rows in an incident export. They differ only in leading spaces and letter case, yet both belong to conversation c-51. If 101 lands in the training partition while 102 lands in evaluation, the test score won't measure generalization to new incidents. It'll measure memorization of incident c-51's specific tokens.

The previous chapter trained a ticket router from pre-computed numerical tensors. Now trace those tensors backward: which raw records can become supervision, and which held-out records can an evaluation engineer actually trust? A dataset pipeline turns raw records into model inputs and records how it did so. That record supplies provenance: where each example came from, which transformations changed it, and which gates approved it. Cleaning, splitting, and contamination checks protect different parts of the same training claim.

Ingestion contracts: schema enforcement and type validation

The router consumes features derived from support tickets. Start one step earlier with exported messages and labels from a developer-support conversation dump:

Read these four exported rows as four distinct pipeline decisions:

Ticket 104 presents the first breakdown: a message exists, but no reviewed label does. What should happen before normalization or splitting?

Start with an ingestion contract gate. In modern data architectures, validation operates across three distinct tiers:

1. Structural gates: Enforce column presence, non-nullity, and strict data types. A float ticket ID or a string where an integer belongs gets rejected immediately.
2. Domain gates: Enforce value bounds, regex patterns, and categorical membership. Here, labels must belong strictly to {"standard", "escalate"}.
3. Distributional gates: Detect batch-level anomalies, such as unexpected null rates or drastic shifts in label ratios before data reaches disk.

Declarative validation libraries formalize these tiers. Pydantic models with strict typing (pydanticStrict2026, pydanticModels2026) reject silent type coercions, such as string "101" turning into int 101, or truthy strings turning into booleans. Pandera provides DataFrame-level schemas with column checks and statistical hypothesis gates for batch tabular data. Great Expectations runs declarative assertion suites in ingestion pipelines to intercept corrupted records before model ingestion.

Schema evolution requires constant vigilance. Upstream services frequently add fields (additive changes) or deprecate columns, widen types, and rename identifiers (breaking changes). A strict ingestion gate fails fast on breaking changes, isolating the malformed batch rather than letting corrupt records silently train your model.

The gate catches ticket 104 at the intake boundary. Schema validation can't prove that a label is correct, but it stops unreviewed or malformed rows from becoming supervision.

Deterministic normalization and exact fingerprinting

Rows 101 and 102 differ only because an export system injected spaces and changed letter casing. Text sanitization strips control characters, surrogate pairs, and non-printable bytes before text reaches downstream tokenizers.

Next comes Unicode normalization. The Unicode standard defines multiple canonical and compatibility forms: NFC, NFD, NFKC, and NFKD. NFKC (Compatibility Decomposition followed by Canonical Composition) replaces compatibility characters with standard equivalents. It collapses typographic ligatures like fi into fi, turns superscript ² into 2, and normalizes full-width characters into standard ASCII. Following NFKC with str.casefold() (which handles tricky multilingual casing like German ß mapping to ss) and whitespace collapsing exposes duplicate text reliably.

Conservative normalization has boundaries. Case-folding merges case-sensitive identifiers like Prod and prod, while compatibility normalization can alter code syntax. Never strip negation or environment names: build passed and build not passed must remain distinct records.

Once text is normalized, a deterministic SHA-256 hash becomes an exact fingerprint. It isn't a semantic similarity score or proof that two records describe the same event. Equal digests provide an efficient lookup key for duplicate detection; compare normalized strings directly when exact identity matters, since cryptographic hashes aren't mathematically collision-free.

The printed hashes are shortened for display; identity comparisons always use the full 64-character hex digest. Record the Python runtime and Unicode database versions alongside the normalization rule when reproducible bytes matter across architectures.^{[1]Reference 1unicodedata: Unicode Databasehttps://docs.python.org/3/library/unicodedata.html[2]Reference 2hashlib: Secure Hashes and Message Digestshttps://docs.python.org/3/library/hashlib.html}

Why normalize text before computing its exact fingerprint?

Answer

Normalization ensures harmless formatting differences like capitalization and extra spacing map to identical byte representations before hashing. It must stay conservative so semantic distinctions like negation stay intact.

Applying schema filtering and normalization to tickets 101 through 104 produces a clean separation: ticket 104 is rejected, 102 merges into 101, and 101 plus 103 proceed.

Duplicate labels and the quarantine boundary

Rows 201, 202, and 203 collapse to the same normalized text, but row 203 carries label standard while rows 201 and 202 carry escalate. Should the pipeline keep the first row, or quarantine the entire group?

A naive deduplicator picks whichever row arrived first ("first-in-wins"). That's an anti-pattern. Silent resolution sweeps critical flaws under the rug: annotation error, guideline drift, or missing features (like customer SLA tier or staging flag) that justified different routing decisions.

A defensive pipeline isolates the entire fingerprint group into an anomaly quarantine. Label conflict is evidence to investigate, not noise to discard.

Grouping runs directly on the normalized string, so hash collisions can't fabricate a false label conflict. In this build, agreeing copies collapse only within the same conversation. Identical text appearing across separate conversations routes to human review, ensuring common response templates don't skew training weights.

Grouped splits and target leakage prevention

Passing schema validation and deduplication doesn't make an example safe for evaluation. Standard random splitting (like an 80/20 train/test split) fails in multi-turn dialogues, medical records, or user-session feeds.

When multiple records share an underlying entity, random splitting causes entity leakage. If turns 1 and 3 of incident c-51 train the model while turn 2 tests it, the model memorizes incident-specific server names and jargon. The test score measures recall of known incidents, not generalization to new ones.

Preventing entity leakage requires choosing the correct split unit. Split on conversation_id, not ticket_id. All records sharing an entity key must atomically land in the same partition.

Using a stable cryptographic hash (sha256(conversation_id) % 100) assigns each entity deterministically without requiring a centralized state store:

Both turns of build-001 stay in train. build-005 hashes into validation and build-006 into test.

Stable hashing ensures repeatability, but doesn't guarantee balanced row counts or uniform label distributions. Large conversations contribute dozens of turns while small ones contribute one. Always audit partition sizes and class distributions after applying grouped splits.

Entity grouping addresses generalization across conversations, not across time. If records arrive chronologically, use out-of-time (OOT) validation splits where the test set is strictly bounded by a future timestamp. This prevents temporal leakage, where features computed from future events pollute training rows.

Two rows come from the same conversation. Why isn't a random row-level split enough?

Answer

Turns within a conversation share context, incident details, and entity jargon. Assigning the whole conversation_id to a single partition ensures evaluation measures generalization to completely unseen conversations.

Benchmark contamination and de-duplication audits

Grouped splits protect conversation boundaries. They don't prevent training data from containing locked benchmark questions and answers. In language model development, web crawlers ingest millions of pages that contain benchmark prompts, test suites, or discussion forums, leaving evaluation sets that no longer measure true reasoning on unseen tasks.

During GPT-3's pre-training, an 8-word to 13-word n-gram overlap filter removed benchmark test sets from training data (brown2020). However, a filtering bug left overlaps in the final dataset, and retraining wasn't economically feasible. For Wikipedia language modeling benchmarks and the Children's Book Test, contamination was nearly total, so OpenAI omitted those benchmark results entirely.

Start with an exact normalized overlap check between candidate training data and locked evaluation sets:

Exact matching is necessary, but insufficient on its own. A benchmark question might be paraphrased or formatted with minor edits while preserving the identical reasoning path.

Post-hoc black-box contamination checks (like Membership Inference Attacks or Min-K% log-probability scoring) often fail to detect contamination reliably, yielding near-chance AUC-ROC scores on frontier models. Preventative pre-training decontamination and rigorous n-gram filtering remain the only dependable defenses.

Near-duplicate detection: MinHash, shingling, and LSH

Consider two support tickets: build has not passed and my build has not passed. Their exact SHA-256 fingerprints are completely different. Yet they share three out of four adjacent word pairs.

Break text into contiguous n-grams of words or characters called shingles. For word pairs ($n=2$):

• Document $A$: {"build has", "has not", "not passed"} (size 3)
• Document $B$: {"my build", "build has", "has not", "not passed"} (size 4)

Measure their set overlap with Jaccard similarity:

$$J(A, B) = \frac{|A \cap B|}{|A \cup B|}$$

For this pair, the intersection contains 3 shared shingles and the union contains 4 distinct shingles:

$$J(A, B) = \frac{3}{4} = 0.75$$

A similarity of 0.75 crosses a 0.70 inspection threshold:

Computing all-pairs Jaccard similarity across millions of documents requires $O(N^2)$ comparisons, which becomes impossible at scale. Broder's MinHash algorithm (broder1997) solves this scaling bottleneck.

MinHash uses $K$ independent hash functions applied to each shingle. For each hash function $h_k$, compute:

$$s_k(A) = \min_{w \in A} h_k(w)$$

The probability that two documents produce the same minimum hash value equals their Jaccard similarity:

$$P\left(s_k(A) = s_k(B)\right) = J(A, B)$$

Concatenating $K$ minimum hashes creates a compact signature vector. Locality-Sensitive Hashing (LSH) groups these $K$ signatures into $b$ bands of $r$ rows ($K = b \times r$). Two documents sharing identical hashes in any single band become candidate pairs. This finds near-duplicates in sub-quadratic time.

Lee et al. (lee2022) showed that removing near-duplicates from web corpora reduced memorization by roughly 10x without harming downstream accuracy. Modern foundation model corpora like FineWeb (penedo2024, penedo2025fineweb2), Dolma (soldaini2024), and Llama 3 (touvron2024) run 5-gram MinHash LSH across billions of tokens to eliminate duplicated web content.

Two messages have Jaccard shingle similarity 0.92. Should the pipeline delete one automatically?

Answer

No. The score flags a candidate near-duplicate. A human or policy gate must verify whether the examples share equivalent answers and labels before merging, discarding, or declaring contamination.

Privacy, PII detection, and provenance

Support tickets frequently contain Personally Identifiable Information (PII): developer emails, phone numbers, server IP addresses, authentication tokens, and internal ticket references. Training models on unredacted text creates data leak vulnerabilities where models reproduce private keys or employee contacts during generation.

PII detection employs a layered defense:

1. Deterministic regex: Scans for structured patterns with fixed formats: email addresses, UUIDs, IPv4/IPv6 addresses, and credit card numbers.
2. Named Entity Recognition (NER): Detects unstructured entities: human names, company names, and physical addresses.
3. Dedicated de-identification SDKs: Microsoft Presidio (presidio2023) combines pattern recognizers, contextual rule engines, and spaCy NER to detect and de-identify sensitive spans across text streams.

Once detected, pipelines apply two redaction strategies:

• Document-level dropping: If PII density is high (e.g. Dolma drops documents with more than 5 PII matches), the entire document gets removed (soldaini2024).
• Span masking: For sparse PII, sensitive spans get replaced with typed surrogate tokens (<EMAIL>, <PHONE_NUMBER>, <PERSON>).

A regex non-match doesn't prove safety: row 501 still contains an internal incident identifier (INC-10234). Production pipelines enforce privacy reviews, access tiers, and verified redaction models before public release.

Data systems also distinguish provenance from lineage:

• Provenance: Captures origin, custody, collection guidelines, and legal licensing.
• Lineage: Maps the computational DAG: exact source snapshots, transform parameters, code commit SHAs, and output checksums (often emitted as OpenLineage events).

Dataset versioning bridges the two. Tools like DVC store lightweight pointer files in Git while syncing large binary chunks to content-addressable storage (CAS) like S3. Table formats like Delta Lake and Apache Iceberg provide ACID transactions, schema enforcement, and immutable snapshot time-travel over Parquet files.

Modern foundation model data pipelines: web filtering and synthetic gates

When building pre-training datasets from raw Common Crawl dumps, filtering is the single largest factor determining model quality (raffel2020, penedo2024, touvron2024). Raw web text is flooded with HTML boilerplate, navigation menus, SEO keyword soup, and machine-translated spam.

Modern pipelines apply a multi-tier filtering battery:

1. Heuristic text filtering

Rules based on document structure eliminate obvious junk without running expensive models:

• Length bounds: Discard documents with fewer than 50 or more than 100,000 words.
• Symbol-to-word ratio: Reject text where symbols or punctuation exceed 15% to 20% of all tokens (catches broken code dumps or terminal dumps).
• Stopword thresholds: Natural language exhibits stable stopword distributions. A page lacking common stopwords (e.g. the, and, is) is usually a product list, keyword cluster, or scraped catalog.
• Repetition filters: Remove lines or documents with repeating n-grams or duplicated paragraphs (catches looping web scrapers).

2. Model-based perplexity filtering

Heuristics catch structural garbage, but can't evaluate prose quality. High-scale pipelines train a lightweight reference language model (like a 5-gram KenLM or a compact transformer) on high-quality curated sources such as Wikipedia, textbooks, and vetted documentation (penedo2024).

The reference model scores the perplexity of each candidate document:

$$\text{PPL}(X) = \exp\left( -\frac{1}{N} \sum_{i=1}^N \log P(x_i \mid x_{<i}) \right)$$

Documents with extreme perplexity are discarded: high perplexity indicates incoherent babble or OCR noise, while abnormally low perplexity indicates repetitive boilerplate or templated spam.

3. Synthetic data quality gating

Modern instruction-tuning (SFT) and preference modeling (DPO/RLHF) rely heavily on synthetic data generated by frontier teacher models. Automated quality gates must vet synthetic records before they hit training pipelines:

• Format and schema validation: Enforce JSON schema adherence, tag balance (<think>...</think>), and response length bounds.
• Verifiable execution: For coding and math pipelines, execute generated code in sandboxed containers; reject solutions failing unit tests.
• LLM-as-a-judge scoring: Prompt evaluator models to grade response completeness, truthfulness, and reasoning coherence, dropping outputs falling below strict quality thresholds.

Building a reproducible candidate artifact

Now assemble all verification mechanics into one deterministic pipeline. It validates raw records, normalizes text, resolves duplicate groups, assigns grouped splits, and writes JSONL outputs alongside a rejection log and signed manifest.

A candidate artifact means these mechanics executed successfully. It doesn't mean privacy or contamination review has signed off on release.

The manifest acts as a structured dataset datasheet (gebru2021). Gebru et al. propose datasheets that document motivation, composition, collection, and recommended uses so consumers can evaluate fit.

Notice the manifest explicitly tracks pending reviews: split coverage, benchmark overlap, near-duplicates, and privacy. Don't mark a dataset ready for release until those checks pass.

Verifying the candidate contract like downstream code

Downstream training code shouldn't blindly ingest files because they exist on disk. A production data loader executes pre-flight assertions before spending GPU compute:

1. Zero group leakage: No conversation ID spans multiple splits.
2. Deterministic split adherence: Every record matches the stable hash assignment.
3. Cryptographic integrity: File bytes match artifact_sha256 and rejects_sha256 in the manifest.
4. Structural validity: 100% of rows pass schema requirements.
5. Fingerprint consistency: Recomputed SHA-256 fingerprints match row metadata.
6. Accounting balance: kept + rejected == total raw rows.

Each assertion verifies an independent invariant: group isolation, byte identity, record structure, content fingerprints, uniqueness, and accounting. A loader that fails fast prevents silent corruption from invalidating an expensive training run.

Auditing distribution drift and split coverage

Even if every row passes structural checks, the dataset might still fail downstream needs. If the validation split contains zero escalate cases, validation accuracy can't evaluate whether your model detects high-priority incidents.

Always audit label coverage and feature distributions across splits:

This fixture fails label coverage: train contains only escalate examples while validation and test contain only standard examples.

In production, test distributions across partitions using statistical tests:

• Categorical features: Chi-square test of independence or Population Stability Index (PSI).
• Continuous features (e.g. sequence lengths): Two-sample Kolmogorov-Smirnov (KS) test or Wasserstein distance.

When a partition fails coverage, collect additional reviewed conversation groups or adopt stratified grouped splitting. Never cherry-pick individual records across group boundaries to artificially balance a split.

Tracking dataset versions across pipeline evolutions

Suppose pipeline v2 adds an annotation or updates the normalization regex. If model accuracy shifts, did the architecture improve, or did the underlying data change?

Assigning each dataset release an immutable SHA-256 digest and pipeline version makes performance shifts attributable:

Hugging Face Datasets computes an Arrow caching fingerprint over dataset state and each transformation step (huggingface_datasets). While Arrow caching manages local reuse, an immutable manifest digest binds a published dataset release to Git commit tags and experiment runs.

Mapping pipeline gates to enterprise AI systems

The mechanisms applied to our developer ticket fixture scale directly to foundation model pipelines:

Never quote an evaluation metric without naming the exact dataset artifact, split policy, and decontamination audit that produced it.

Controlled failure labs: break the pipeline on purpose

Run these targeted experiments against the runnable code blocks above to observe how specific assertions catch mutations:

1. Simulate entity leakage: In 08-verify-artifact.py, append {**rows[0], "ticket_id": 999, "split": "test"} to rows before computing groups.
2. Simulate bit rot or file tampering: In 08-verify-artifact.py, append a newline to rows_path after loading saved_manifest and before evaluating digest_matches.
3. Evaluate boundary conditions in shingling: In 06-shingle-overlap.py, print jaccard("build", "build") and jaccard("", "").
4. Violate grouped split integrity: In 04-stable-grouped-split.py, assign partitions by hashing str(row["ticket_id"]) rather than row["conversation_id"].

What each failure reveals

1. no_group_leakage, identities_unique, and manifest counts fail simultaneously. Multiple assertions catch the corruption from different perspectives.
2. digest_matches becomes False. Even an invisible trailing byte breaks cryptographic identity.
3. A single matching word returns 1.0; two empty strings return 0.0. Blank strings must be caught by schema gates, not similarity estimators.
4. Turns from conversation build-001 separate into different splits, causing entity leakage and inflating evaluation scores.

Carrying clean data into feature engineering

A dataset manifest gives reviewers verifiable lineage: raw sources enter, transformations and split boundaries act on them, and cryptographic digests lock the resulting bytes. Passing structural assertions is a prerequisite, not a guarantee that every evaluation requirement is met.

Clean partitions answer one core question: which records can meet during training and evaluation? Feature engineering introduces the next operational challenge: was each feature knowable at prediction time, and does it mean the same thing during live inference as it did during batch training?