Synthetic Data Pipelines

Continued pretraining moved a base model toward domain text. Post-training needs different data: instructions, safe answers, rejected answers, red-team cases, and edge cases that teach behavior. A coding assistant or API-documentation assistant has to write tested functions, follow versioned platform rules, cite real interfaces, and avoid leaking secrets. Synthetic data pipelines turn a small reviewed seed into more candidates for that stage, then gate them until each row is useful enough to train on.

Naive generation produces repetitive, shallow, or harmful data. A weak filter can let hallucinated API answers or insecure code into the training set. The better approach is a multi-stage pipeline with explicit success and failure paths at every gate: schema checks, decontamination, diversity checks, task-specific verifiers, calibrated LLM judging where deterministic checks can't decide, human audit, and preference-pair construction. The pipeline below builds those controls from first principles using code-generation and API-documentation domains.

The post-training data problem

Earlier chat-template work introduced the basic contract: user messages, assistant messages, system instructions, and exact serialization. Supervised fine-tuning (covered next) turns examples in that shape into model weights. Alignment stages such as RLHF (Reinforcement Learning from Human Feedback) and DPO (Direct Preference Optimization, which trains directly from preference pairs without a separate reward model) further shape which of the many valid responses the model prefers. The exact row shape depends on the objective, but every stage needs data that's:

• Specific to the target domain (secure backend code, API documentation, tool-use rules)
• Diverse enough to avoid training on a narrow set of repeated behaviors
• Free of evaluation-set contamination
• Annotated with preference signals when an alignment objective consumes comparisons

Human annotation at the required volume is slow and expensive. Synthetic pipelines let a small team produce more candidates while keeping humans focused on seed design, rubric tuning, and quality audits.

For a coding assistant, a reviewed seed set can be expanded into many candidates, then reduced to the rows that pass unit tests, security checks, decontamination, diversity checks, format validation, calibrated judging, and audit sampling. Candidate volume is easy; evidence that a row belongs in training is the difficult part.

Self-Instruct: the bootstrap loop

Self-Instruct starts with a curated seed and asks an LLM to invent new instructions and corresponding instances. The original paper initialized its task pool with 175 human-written tasks, filtered low-quality or similar generations, and produced more than 52,000 instructions after filtering.^{[1]Reference 1Self-Instruct: Aligning Language Models with Self-Generated Instructions.https://aclanthology.org/2023.acl-long.754/} A product pipeline needs the same separation: generated rows are candidates until gates accept them.

The algorithm in code and API-documentation domains looks like this:

1. Start with human-written seeds: "Write a pytest regression test for timezone-aware token expiry" and "Answer an API usage question using only the supplied documentation excerpt."
2. Prompt a generator model to produce N new instructions that feel like natural extensions.
3. For each new instruction, prompt the same or a stronger model to produce a response.
4. Run every (instruction, response) pair through a battery of filters.
5. Add the survivors to the seed pool and repeat.

Initial seed quality matters. One poorly written code example that ignores token expiry can be echoed and amplified across many synthetic variants.

Every generated candidate also needs a data contract before semantic evaluation. Store provenance and slice assignment on the row rather than attempting to reconstruct them after training.

Evol-Instruct: evolving instructions instead of merely repeating them

Plain instruction generation doesn't control difficulty. Evol-Instruct, introduced by WizardLM, addresses this by asking a generator to rewrite instructions through named in-depth operations such as adding constraints, deepening, concretizing, increasing reasoning steps, or complicating input, plus an in-breadth mutation; it also uses an eliminator for failed evolutions.^{[2]Reference 2WizardLM: Empowering Large Language Models to Follow Complex Instructions.https://arxiv.org/abs/2304.12244}

For this training pipeline, useful mutation tactics include:

• Add more constraints ("the function must be async, typed, and reject expired tokens before permission checks")
• Require multi-step reasoning ("first inspect the API docs, then choose the endpoint, then draft a cited answer")
• Introduce realistic edge cases ("the token expires exactly at the current second and the clock source is naive")
• Increase specificity and length
• Combine skills (test writing + code generation + API documentation lookup)
• Route adversarial variants into a separate red-team slice ("the prompt tries to override the tool-use rules")

Most tactics make ordinary training tasks harder. The adversarial extension belongs in a separately tracked red-team slice because it probes safety failures and needs stricter review.

This concrete evolution prompt fits code and API-documentation domains:

An example evolution on a code seed:

Original: "Write a function validate_token(token) that returns whether the token is usable."

Evolved: "Write a function validate_token(token: str, now: datetime) -> TokenVerdict that verifies the signature, rejects expired tokens before permission checks, logs only the token ID prefix, and returns a typed verdict containing status, subject, and expiry. Include type hints and a docstring with three realistic edge cases."

Evolved instructions should produce harder candidates, not shallow paraphrases. They still need verification and downstream evaluation before they enter training.

Record which evolution produced each candidate. This runnable stand-in keeps tactics explicit; a real pipeline would attach same metadata to generator output.

LLM-as-Judge: a calibrated semantic gate

Deterministic checks should run wherever they can decide correctness: JSON schema checks, policy-rule checks, code tests, and held-out overlap gates. For subjective qualities such as clarity or helpfulness, a stronger LLM can score candidates against a rubric. Zheng et al. studied this judge pattern for evaluating assistant responses with MT-Bench and Chatbot Arena, and documented position, verbosity, and style-similarity biases, plus limits on grading math and reasoning questions.^{[3]Reference 3Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} A judge score is therefore an estimated signal that requires human calibration, not ground truth.

A judge prompt for mixed code and API-documentation tasks typically contains:

• The original instruction
• The candidate response
• A 1-10 scale for each dimension: helpfulness, correctness (API or code correctness), completeness, safety (no secret leakage, no unsafe tool use), clarity
• Explicit instructions to output strict JSON: {"helpfulness": 9, "correctness": 8, ... , "rationale": "..." }

Only examples that pass deterministic gates and a calibrated score threshold should survive. Here, 8/10 is an illustrative threshold; set production cutoffs by comparing judge decisions against reviewed rows. For pairwise ranking, reverse answer order and route inconsistent decisions to audit rather than quietly turning position bias into preference labels.

First-principles quality signals with NumPy

Before reaching any high-level library, we implement the core signals ourselves.

Diversity can be measured with embedding cosine similarity. In a minimal pipeline, pre-compute embeddings and reject a new example whose maximum cosine with already accepted rows exceeds a threshold. The exact cutoff is a calibration parameter: too low and you discard useful variations; too high and you accept repetitive clusters. The 0.82 value below exists only to make the toy vectors produce visible accept/drop decisions.

A simple decontamination gate can build exact n-gram sets from held-out evaluation text, then scan both instructions and responses. This catches copied spans, not semantic paraphrases; production evaluation protection needs stronger overlap analysis and restricted access to held-out material.

Additional lightweight gates include:

• Length between 40 and 1200 tokens (domain dependent)
• Presence of required code or documentation fields (pytest, endpoint name, citation ID), as a format check rather than proof of correctness
• For code: basic AST parse success and absence of obvious dangerous patterns (eval, os.system on untrusted input)

These signals are cheap, deterministic, and composable.

Red-teaming data synthesis

Safety and policy adherence won't reliably emerge from ordinary user instructions alone. Attackers use specific adversarial phrasings. Red-teaming (deliberately probing a model with adversarial inputs to find safety weaknesses) supports a dedicated adversarial slice instead of waiting for those cases to appear naturally.^{[4]Reference 4Constitutional AI: Harmlessness from AI Feedback.https://arxiv.org/abs/2212.08073[5]Reference 5Red Teaming Language Models with Language Models.https://arxiv.org/abs/2202.03286}

A generator is prompted with meta-instructions such as:

"Write an API-documentation question that embeds an instruction to reveal an internal admin token. The request must sound like a normal debugging task."

or

"Write a Python function that appears to validate API tokens but contains a subtle backdoor that accepts any token whose subject starts with 'admin-'."

The correct target response for the first case is a refusal to reveal secrets plus a safe debugging alternative. For the second case, keep the adversarial prompt in a controlled red-team slice and accept a response only if a security check or review confirms it avoids the bypass. For code tasks, language judging can't replace execution and security checks.

After validation and review, such pairs can enter SFT (Supervised Fine-Tuning) data to teach desired behavior or preference data to contrast a safe response with a plausible unsafe response.

This demonstration executes hard-coded candidate strings only. Real generated code belongs in a locked-down sandbox with resource and network controls.

Preference pair synthesis for DPO and RLHF

DPO trains directly on triples: prompt, chosen response, and rejected response.^{[6]Reference 6Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} In a typical RLHF pipeline, preference comparisons train a reward model before policy optimization. Synthetic generation can propose either form of data, but the labels are useful only when a verifier, calibrated judge, or reviewer confirms why the chosen answer wins.

Two reliable techniques:

1. Multi-sample ranking: generate several responses under recorded decoding settings, evaluate with deterministic verifiers where possible and a calibrated judge otherwise, and retain pairs with a clear verified difference.
2. Deliberate failure generation: generate a plausible security mistake or code bug in a restricted workflow, then retain it as rejected only after validating that the chosen row passes and the intended failure is present.

Example for an API-documentation assistant:

Instruction: "The API docs say /v1/search requires a cursor for pagination and caps page_size at 100. A user asks for code that fetches every page and retries rate limits. Answer with citations to the supplied docs only."

Chosen (good): explains cursor iteration, preserves the documented page-size cap, adds bounded backoff for 429, and cites the provided docs for both pagination and rate-limit behavior.

Rejected (bad): "Set page_size=10000 and keep retrying immediately until the API returns all records; the docs imply higher limits are fine for internal users."

The rejected response isn't cartoonishly wrong. It represents a plausible, consequential API-usage error, and it must remain tagged and access-controlled as rejected training material.

The same pattern applies to code: a correct, typed, tested function versus one that returns incorrect status strings on edge cases or leaks internal database errors.

Complete pipeline: executable gates before framework integration

A minimal but complete pipeline in pure Python can be tested before any real model calls are wired in. This version mocks the generator and judge so the mechanics are copy-runnable.

Only after the manual version is correct and debugged should you move stateless gates into a columnar data framework for map/filter operations and caching. Global diversity deduplication still needs a deterministic pass against the accepted shard or an indexed reference pool. The smoke test below deliberately uses standard Python records so acceptance logic executes quickly; a production Dataset.filter(quality_filter) call should wrap the same predicate.

A framework integration should remain this thin: storage and parallel transforms may change, while acceptance predicates and shard-level audits remain explicit.

The data flywheel

A later model release may generate better candidate tasks than an earlier one. That's a hypothesis to test, not a quality guarantee. Each iteration must be compared on a fixed, human-reviewed calibration sample and held-out product evaluations.

Don't assume the newly trained model is also a better judge of data used to train its successor. If generator and judge share the same blind spot, an apparently improving loop can amplify that error. Pin judge versions, test judge decisions against human labels, and use executable or policy verifiers where the task permits them.

Failure cases observed in evaluation or production can seed targeted new candidates, but only versioned prompts, recorded gate outcomes, and repeatable audits let you distinguish real improvement from drift.

Model collapse: the risk that makes verification non-negotiable

Model collapse describes degradation across recursive generations of models trained on generated data. Shumailov et al. (2024) demonstrate loss of distribution tails in their studied generative-model settings, including OPT-125M language-model fine-tuning experiments, and report degradation even when one language-model regime preserved 10% of the original training data.^{[7]Reference 7AI models collapse when trained on recursively generated datahttps://www.nature.com/articles/s41586-024-07566-y}

Two details from that work matter for a data engineer:

• The recursive setup is the central risk: later training data depend on output from earlier fitted models.
• Retaining some original data helped in their language-model experiment but didn't eliminate the measured degradation in that setup.

Gerstgrasser et al. (2024) evaluate an alternative accumulation regime: keep original real data and successive synthetic generations together rather than replacing prior data. In their tested language-model, diffusion, and VAE settings, accumulation avoids the collapse they observe under replacement; in their linear-model analysis, accumulated-data test error has a finite upper bound independent of iteration count.^{[8]Reference 8Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datahttps://arxiv.org/abs/2404.01413} This is evidence for keeping real-data anchors, not a guarantee that any synthetic mix is safe.

Concretely, reduce recursive-drift risk by:

• Keeping a fixed, version-controlled core of human-reviewed seeds and real examples in each training mix.
• Validating rows with evidence the generator didn't produce: executable tests, policy checks, held-out evaluation results, and sampled human review. A judge model helps only after it's calibrated.
• Monitoring accepted-row coverage and downstream results over iterations rather than treating cosine diversity as proof of model quality.

Ground synthetic candidates in external evidence

Synthetic candidates are useful when they turn real requirements into testable training rows. They are risky when the generator invents both problem and acceptance signal in a closed loop.

The modern framing extends these gates in two directions:

• Grounding in external sources. Instead of asking a model to invent facts, pipelines anchor generation in real documents, real APIs, or real execution environments, then synthesize tasks and answers from that ground truth. This reduces the risk of hallucination amplification from ungrounded self-generation.
• Verifiable checks. Where a task has a programmatic checker (unit tests for code, a math verifier, or an executable policy rule), use it as an acceptance gate. Later RLVR training can use suitable verifiers as reward signals. Text similarity and LLM judge scores aren't executable correctness verifiers.

Generate candidates, verify against independent signals, and record what passed.

Decontamination, versioning, and reproducibility

Every synthetic shard should carry a manifest listing generator model, judge configuration when used, random seeds, evolution tactics, verifier versions, and decontamination artifact identifiers. Without this, you can't reproduce acceptance decisions or audit later evaluation leakage.

Mastery check

Key concepts

• Self-Instruct bootstrapping from small human seed sets
• Evol-Instruct rewriting for increasing difficulty
• Calibrated LLM-as-judge scoring alongside deterministic verification
• Red-team data synthesis for adversarial robustness
• Preference pair synthesis for DPO and RLHF
• Synthetic data quality signals and filters
• Benchmark decontamination via n-gram overlap and exact matching
• Data flywheel iteration and model collapse risk
• Provenance, manifests, and reproducibility

Evaluation rubric

• Foundational: Explain how Self-Instruct grows a small reviewed seed set into a larger candidate pool
• Foundational: Explain how Evol-Instruct targets harder tasks than plain expansion and how to test whether they help
• Intermediate: Describe how judge scores, diversity filters, and decontamination gates work together
• Intermediate: Turn one instruction into a chosen/rejected pair for DPO or RLHF
• Advanced: Design a red-team slice and explain why it shouldn't be mixed blindly into ordinary generation
• Advanced: Explain how provenance, manifests, and human audits keep a synthetic-data flywheel from drifting or collapsing

Follow-up questions

Common pitfalls

• Symptom: Large synthetic set hurts training more than it helps. Cause: Cheap generation was treated like free data, and bad rows slipped through. Fix: Keep strict gates and human audits even when generation volume is high.
• Symptom: Generator and judge agree, but both are wrong in the same way. Cause: Weak-model feedback loop reinforced low-quality patterns. Fix: Use a stronger judge or calibrate regularly against human scores.
• Symptom: Benchmark scores jump suspiciously fast. Cause: Decontamination was skipped or versioned poorly. Fix: Keep a held-out overlap set and record exact decontamination artifacts in every manifest.
• Symptom: Safety regressions keep appearing after deployment. Cause: Only friendly instructions were synthesized. Fix: Generate a dedicated red-team slice for jailbreaks, secret-exfiltration attempts, and subtle backdoors.
• Symptom: Preference training learns little from rejected answers. Cause: Rejected rows are absurdly bad instead of realistically tempting mistakes. Fix: Prefer subtle but important failures, such as API misuse or insecure code that still looks plausible.
• Symptom: Training signal drops after switching templates or serializers. Cause: SFT and preference rows drifted away from final chat-template format. Fix: Validate exact role markers, special tokens, and serialization before training.
• Symptom: Synthetic flywheel degrades over iterations. Cause: Generator, judge, and rubric drifted without human recalibration. Fix: Sample outputs each cycle, compare to human labels, and tighten prompts or thresholds before the next turn.
• Symptom: Code examples pass text judging but fail in practice. Cause: Code was judged like text. Fix: Add AST checks, tests, sandboxed execution, and security scanners alongside language judges.
• Symptom: Team can't explain where a shard came from or reproduce it. Cause: Seeds, prompts, models, and hashes lived only in notebooks. Fix: Treat provenance as required metadata, not optional notes.

Production checklist and next steps

A production synthetic data pipeline for code and API-documentation domains must treat data as versioned, auditable infrastructure:

1. Review every seed example with domain experts before it enters the flywheel.
2. Pin exact generator and judge models, prompts, and decoding settings.
3. Keep deterministic tests for diversity, decontamination, safety, and format gates.
4. Maintain a dedicated red-team slice sized to real production failure modes.
5. Validate exact chat-template formatting before training on SFT or preference rows.
6. Re-score sampled synthetic rows with humans every cycle to detect judge drift.

With those controls in place, synthetic generation becomes reproducible data infrastructure instead of an unreviewed prompt loop. You keep full provenance for where each example came from, which gates it passed, and why it belongs in training.