Pre-training Data at Scale

Scaling laws gave us a target: if you want a capable dense model, you may need hundreds of billions or trillions of useful training tokens. Pre-training data pipelines decide what those tokens are before any fine-tuning or alignment happens. This chapter covers sourcing, cleaning, deduplication, filtering, and mixing at the scale where data quality becomes model behavior.

A coding assistant for infrastructure teams needs public code, API docs, runbooks, incident writeups, and design notes, but you can't dump raw repositories into training. Web-scale text includes broken HTML, duplicate files, spam, private data, and benchmark questions. A useful pipeline turns that messy source lake into documented training data: extract text, filter low-signal records, remove duplicates, scrub sensitive material, and convert the retained corpus into packed token sequences.

From token budget to training inventory

Start the pipeline with three ideas from earlier in the curriculum:

• Tokens: A language model doesn't read words. It reads tokens: reusable pieces such as ship, ping, punctuation marks, or byte sequences. The tokenizer converts text into integers, and the model learns to predict the next integer in a sequence.
• Next-token prediction: During pre-training, the model sees a partial sentence and learns to guess what comes next. The more diverse and high-quality the sequences it sees, the broader its knowledge becomes.
• Scaling laws: The previous chapter turned model size into a token target. Data pipelines turn that target into an engineering system.

If tokens or next-token prediction feel fuzzy, review Language Modeling & Next Tokens before continuing. Everything here assumes you understand why the model needs large amounts of varied text.

How much data is enough?

Start with a concrete number. Suppose you want to train a 70 billion parameter model. How many tokens should you feed it?

Hoffmann et al. studied this tradeoff in the Chinchilla scaling laws. Under their fixed-compute dense-transformer setup, model size and training-token count grew together. A useful planning rule from that study is about 20 training tokens per parameter^{[1]Reference 1Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}.

For a 70B model:

• 70 billion parameters x 20 tokens per parameter = 1.4 trillion tokens

That's roughly the entire text content of millions of books, web pages, and code repositories combined. That scale explains why the pipeline needs industrial machinery.

A companion formula for back-of-the-envelope planning is training FLOPs of about $6ND$, where $N$ is a consistently counted dense model size and $D$ is the number of training tokens. It's an approximation, and details such as whether output-layer parameters are counted can move fitted optima, but it's useful for quick capacity estimates.^{[1]Reference 1Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556[2]Reference 2Resolving Discrepancies in Compute-Optimal Scaling of Language Modelshttps://arxiv.org/abs/2406.19146}

Raw token counts are misleading. A smaller corpus of textbook-quality tokens can beat a much larger pile of noisy crawl text. Scaling data only helps when the extra tokens still add real information.

Meta reports pre-training Llama 3's 405B model on 15.6T tokens, about 38.5 tokens per parameter, and says that flagship configuration is approximately compute-optimal under its own fitted laws and training budget. Separately, Meta reports training its smaller models longer than compute-optimal because those models performed better at the same inference budget.^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} Treat the 20x ratio as a planning baseline, not a hard law.

Why is a token target not enough to define a pre-training corpus?

Answer

Token count measures volume, not signal. You still need source mix, filtering, deduplication, privacy scrubbing, benchmark decontamination, and tokenizer fit before those tokens become useful training inventory.

The raw corpus and the clean training mix

The raw internet is a noisy source lake. It contains useful records, but they're mixed with duplicate pages, broken markup, spam, and unsafe material. A pre-training data pipeline is the repeatable processing system that extracts useful records and turns them into a training-ready mix.

Your job as the AI engineer isn't to write the text. It's to design a processing system that handles large partitions repeatably, records each decision, and avoids throwing out high-value records.

The modern pipeline has three major phases:

1. Ingestion: Pull raw data from web crawls, code hosts, and curated archives.
2. Cleaning: Filter, deduplicate (remove duplicate documents), decontaminate (strip out leaked evaluation-benchmark examples), and scrub for safety.
3. Preparation: Tokenize, shuffle, pack, and split into training shards.

Where the text comes from

Different data sources serve different roles in the training mixture. Web data gives breadth and freshness. Code is dense in syntax, structure, and exact correctness. Books and papers provide long-form coherent explanations. Carefully curated synthetic or textbook-style data can raise signal density further.

Phi-1 is a good example: its authors report strong coding benchmark performance from a 1.3B model trained on textbook-quality and synthetic data.^{[4]Reference 4Textbooks Are All You Needhttps://arxiv.org/abs/2306.11644}

Note: Some papers disclose approximate mixtures while many don't. Llama 3 reports a final pre-training mix of roughly 50% general knowledge, 25% mathematical and reasoning data, 17% code, and 8% multilingual data.^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} The broader pattern is that mixture weights are decisions, not raw-byte proportions.

Mixture weights, curriculum, and data scheduling

Source quality is only half the decision. You also have to decide how often each source appears and whether that weighting stays fixed through the whole run.

Three knobs matter:

1. Source weighting: web, code, books, papers, synthetic data
2. Schedule over time: fixed mix vs later-stage reweighting
3. Within-source weighting: whether some slices replay more often than others

The wrong default is "sample in proportion to raw bytes." Raw crawl volume isn't the same thing as training value. The Pile is still useful as reference because it made mixture design explicit instead of treating web text as one giant bucket.^{[5]Reference 5The Pile: An 800GB Dataset of Diverse Text for Language Modeling.https://arxiv.org/abs/2101.00027} Llama 3 shows a later-stage version of the same idea: even after assembling a huge corpus, training shifted toward higher-value slices during annealing.^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

This is practical curriculum learning in pretraining. It usually isn't "easy lessons first, hard lessons later." More often it means changing source weights over time so the model sees a different mix at different stages.

Two common patterns:

• Static high-quality mix: choose one weighted mixture and keep it stable through most of training
• Late-stage curriculum: keep broad coverage early, then upweight cleaner or more target-relevant slices later^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

Don't confuse curriculum with annealing alone. Annealing usually means late-stage learning-rate decay plus a higher-quality mix. Curriculum is broader: any deliberate schedule that changes what the model sees as training progresses.

It also helps to know the public datasets that anchored a lot of later recipes:

First quality check: heuristic filtering

Before using expensive neural models to classify text, basic heuristic filters drop the lowest-signal material from web crawls. As seen in foundational datasets like C4 (Colossal Clean Crawled Corpus)^{[6]Reference 6Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.https://arxiv.org/abs/1910.10683}, these simple rules act as a highly efficient first line of defense.

The snippet below implements those basic rules. It takes a single document as input and returns True only if the document passes all checks.

What this code does, step by step

• Length check: Very short documents often carry little context; extremely long extractions may be concatenated pages or indexes.
• Repetition check: High repeated n-gram fractions are a useful spam or boilerplate signal, but legitimate templates and structured documents can also repeat.
• Policy-list check: A reviewed list can identify content a specific corpus policy excludes; its threshold is a policy choice, not a universal quality label.
• Character check: A general-text recipe can discard symbol-heavy extractions, while a code corpus needs a different recipe.
• Sentence check: Few sentence terminators can signal a malformed text extraction, but are unsuitable as a universal language or format rule.

The code intentionally uses simple thresholds so the gates are visible. C4 and FineWeb use related heuristic filtering ideas, but a real recipe must measure retention and downstream model quality on its own crawl slices before adopting thresholds.^{[6]Reference 6Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.https://arxiv.org/abs/1910.10683[7]Reference 7The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557}

These heuristics usually run on general web text before the final data mix is assembled. Code corpora often use a parallel recipe, because symbol-heavy files that look low-quality to a web-text filter can still be high-signal pre-training data.

Second quality check: classifier-based filtering

After heuristic filtering, modern pipelines use classifiers trained to distinguish high-quality educational text from low-quality web chatter:

1. Language ID: A fastText classifier screens whether the document matches the intended training languages.
2. Quality Classifier: A classifier scores whether a document looks like reference-quality material rather than random crawl text. FineWeb-Edu uses an educational-quality classifier trained on LLM-judged scores, DCLM trains a fastText classifier on instruction-formatted and ELI5 text and keeps only the top ~10% by score, and Meta reports separate Llama 3 classifiers for general quality, code, and reasoning signals.^{[7]Reference 7The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557[8]Reference 8DataComp-LM: In Search of the Next Generation of Training Sets for Language Modelshttps://arxiv.org/abs/2406.11794[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}
3. Perplexity filtering: A language model such as an n-gram model can score a document against a reference distribution. Unusually high or low scores can flag text for corpus-specific retention rules; CCNet is a published example of perplexity-based filtering.^{[9]Reference 9CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datahttps://arxiv.org/abs/1911.00359}

Filtering decisions trade yield for the type of data retained. DCLM selected a fastText filter that retained its top 10% of documents after evaluating trained models, rather than assuming a score threshold was automatically better.^{[8]Reference 8DataComp-LM: In Search of the Next Generation of Training Sets for Language Modelshttps://arxiv.org/abs/2406.11794}

Removing duplicates

Deduplication is a core quality step. The internet contains large amounts of duplicated content: boilerplate headers, repeated SEO text, mirrored sites, copied documentation, and reposted articles. Lee et al. found that, on the corpora and model sizes they tested, deduplication reduced emitted memorized text by about 10x and reached the same or better validation accuracy in fewer training steps.^{[11]Reference 11Deduplicating Training Data Makes Language Models Better.https://arxiv.org/abs/2107.06499}

Exact deduplication (hash-based)

This involves taking a cryptographic hash, such as SHA-256, of each document and removing exact matches. It's fast but misses near-duplicates, such as a news article where only the timestamp or a single word was changed.

MinHash + LSH (near-deduplication)

MinHash with Locality-Sensitive Hashing (LSH) is one widely used approach for fuzzy deduplication at scale.^{[12]Reference 12On the Resemblance and Containment of Documents.https://www.cs.princeton.edu/courses/archive/spring13/cos598C/broder97resemblance.pdf[11]Reference 11Deduplicating Training Data Makes Language Models Better.https://arxiv.org/abs/2107.06499} FineWeb, for example, applied MinHash independently per crawl using word 5-grams and parameters intended to target documents with at least 75% similarity; its experiments favored per-crawl rather than global deduplication.^{[7]Reference 7The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557}

The idea, by hand: Imagine you have three short documents:

• Doc A: "The key policy says stale accounts need security review."
• Doc B: "The key policy says stale accounts require security review."
• Doc C: "Audit logs record deployment rollback events."

Doc A and Doc B share almost every word. If you break them into overlapping 2-word chunks (shingles), they look like this:

• Doc A shingles: ["The key", "key policy", "policy says", "says stale", ...]
• Doc B shingles: ["The key", "key policy", "policy says", "says stale", ...]

The overlap is high. Doc C shares zero shingles with A or B. MinHash turns each document into a small numeric "sketch" that estimates this overlap relationship. LSH then routes similar sketches into the same candidate set so we inspect likely duplicates rather than comparing every pair. Because LSH is approximate, a candidate still needs a final similarity decision.

One candidate-then-verify pipeline looks like this:

This example shows how to build a basic MinHash and LSH pipeline using the datasketch library. For a tiny demo corpus, it builds signatures from bigram shingles, uses LSH only for candidate lookup, and verifies candidates with exact Jaccard overlap before dropping a document.

Why this works: doc1 is inserted first. When doc2 arrives, LSH returns doc1 as a candidate; an exact Jaccard check then confirms sufficient overlap before the drop. doc3 has no confirmed neighbor, so it's kept. Published recipes make different final decisions after MinHash collisions. Lee et al. verified candidate pairs with actual Jaccard and edit similarity, while FineWeb used matching MinHash buckets and transitive clustering. Production thresholds, shingle sizes, verification rules, and whether deduplication operates per crawl or globally must be evaluated for the corpus.^{[11]Reference 11Deduplicating Training Data Makes Language Models Better.https://arxiv.org/abs/2107.06499[7]Reference 7The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557}

Why does near-deduplication matter even after exact hashing?

Answer

Exact hashes catch only byte-identical copies. Web corpora contain mirrored pages, lightly edited boilerplate, and reposts, so MinHash/LSH catches duplicates that waste training compute and increase memorization risk.

Keeping benchmarks clean: decontamination

An important and often overlooked step is decontamination: keeping evaluation benchmarks out of the pre-training data. If a model sees MMLU, HumanEval, or GSM8K examples during pre-training, its reported performance becomes misleading because it may be memorizing instead of generalizing.

The risk is substantial. Web crawls contain GitHub repositories with coding interview questions, educational websites with standardized test questions, and forums where users paste benchmark prompts. Because training corpora and evaluation sets often draw from the same public web, contamination is a recurring risk rather than a hypothetical edge case.

N-gram overlap filtering

One common approach is n-gram overlap detection. For each document in the training corpus, check whether it contains spans from an evaluation dataset. The n-gram size, normalization, removal threshold, and code-specific rules must be recorded with the released corpus. GPT-3 attempted pre-training filtering with 13-gram overlaps, then reported that a filtering bug left some overlaps in place. Its post-hoc benchmark analysis used variable n-gram lengths capped at 13. Llama 3 reports excluding benchmark training sets from its annealing data.^{[13]Reference 13Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

Exact-match filtering isn't enough. A coding problem with renamed variables or a question with shuffled answer choices can still leak. Pipelines may add fuzzy or task-specific checks, then document what they removed so benchmark-clean claims are auditable.

What makes benchmark contamination different from ordinary duplicate text?

Answer

Contamination corrupts evaluation. If benchmark examples leak into pre-training, reported scores can reflect memorization rather than generalization, so the training run may look stronger than it really is.

Safety and privacy scrub

After deduplication and decontamination, a pipeline may scrub personally identifiable information (PII) and unsafe content according to its data policy and legal obligations. A model trained on scraped identifiers can reproduce sensitive strings at inference time, creating privacy and compliance risk. FineWeb reports anonymizing email and public-IP addresses; Llama 3 reports removing domains likely to contain high volumes of PII or adult content.^{[7]Reference 7The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

PII removal uses a layered approach:

• Regex patterns: Rule-based detection can catch structured patterns such as email addresses or public IP addresses cheaply.
• Entity or policy classifiers: Context-aware models can identify classes of content that rigid patterns miss, but require false-positive evaluation.
• Source and domain rules: Known high-risk sources can be excluded before model training, as reported for Llama 3.^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

Unsafe-content filtering is policy-specific and imperfect. FineWeb applies URL-level filtering for adult content but explicitly notes that harmful documents can remain. Llama 3 combines domain rules with dirty-word counting for adult websites that domain blocklists miss.^{[7]Reference 7The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} Whatever controls you choose, audit retained and removed samples across languages and source types instead of treating one score threshold as a universal safety boundary.

Turning text into numbers: tokenization

Tokenization converts raw text into integer IDs that the model can process. At the scale of trillions of tokens, tokenization itself requires parallel computation across many workers.

Choosing a tokenizer

Selecting the right tokenizer involves trading off between vocabulary size, sequence length, and language support. A smaller vocabulary can force text to split into more subwords, increasing sequence length and the computational cost of attention. A larger vocabulary can improve compression when it matches the corpus, but it also requires a larger embedding matrix, which consumes memory.

Common subword-tokenizer algorithms use data-driven vocabulary construction:

Training a BPE tokenizer

When you train a model family from scratch, the tokenizer should be trained on a representative sample of the pre-training data so it learns the most common subword patterns. Reusing a mature tokenizer can be a good engineering choice, but a mismatched tokenizer can waste context length, especially for multilingual or domain-heavy corpora. The tokenizer demo uses the Hugging Face tokenizers library to initialize a byte-level Byte-Pair Encoding (BPE) tokenizer. For the demo, it trains from an in-memory sample; production pipelines train from a carefully mixed corpus sample and may target vocabularies such as 128,000 tokens.

In a byte-level setup, you often don't need an <unk> token because any UTF-8 byte sequence can be represented. The requested vocab_size is a ceiling, not a guarantee: this tiny sample learns 293 entries because it runs out of repeated pairs worth merging before reaching 300.

Vocabulary size is a real design decision. A larger vocabulary can improve compression when its extra tokens match the target text, but it also increases embedding-table size and memory footprint. A smaller vocabulary is lighter, but it can produce longer sequences. Meta reports that Llama 3 moved to a 128K-token vocabulary, starting from a tiktoken base plus 28K extra tokens, and improved English compression from 3.17 to 3.94 characters per token relative to Llama 2^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}.

Running the pipeline at scale

Large corpus pipelines require distributed execution beyond demonstration scripts. DCLM reports starting with a 240T-token Common Crawl pool containing about 200B documents and 370TB of compressed text, and processing crawl data on hundreds of AWS CPU nodes.^{[8]Reference 8DataComp-LM: In Search of the Next Generation of Training Sets for Language Modelshttps://arxiv.org/abs/2406.11794}

At that scale, data is partitioned for extraction and filtering, intermediate artifacts are stored durably, and every transformation needs enough metadata to reproduce which records reached training.

Deduplication changes the system shape: signatures can be computed independently, but candidate grouping needs records with related signatures to meet. Whether that step uses a global operation or a restricted partition, such as FineWeb's per-crawl recipe, changes both resource cost and retained data quality.^{[7]Reference 7The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557}

Trillion-token scale challenges

• Ingestion and filtering: Extraction, language ID, rules, and classifier scoring must stream through many documents without losing provenance.
• Deduplication: Signatures are parallel-friendly; candidate grouping and representative selection need an explicit partitioning strategy.
• Training order: Tokenized shards need randomized, documented sampling so one source doesn't unintentionally dominate a run segment.
• Reproducibility: Engineers need to identify the source document, filter version, mixture weight, and shard for suspicious examples.

Data packing and sequence efficiency

Documents vary widely in length: an article might be thousands of tokens while a forum post is short. Padding each document to a fixed sequence length spends compute on padding tokens. Best-fit packing groups documents into fixed-length token blocks to reduce that waste.

Packing creates another design decision: if multiple documents share one sequence, should tokens in one document attend to tokens from another? Some causal-language-model recipes concatenate documents with end-of-document separators and keep the ordinary causal mask. Others isolate documents with a block-diagonal mask so one document can't read tokens from an earlier document in the same packed block. The isolated version below prevents artificial cross-document context while retaining packing efficiency.

Data annealing is a late-stage recipe adjustment where the learning rate decays and the data mix shifts toward higher-value slices. It isn't a universal "final 10%" rule. Llama 3, for example, reports annealing over the final 40M tokens while upsampling very high-quality sources and excluding benchmark training sets from the annealing pool^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}.

Quality and quantity must be measured together

Recent model and dataset papers show that data quality and token yield must be measured together once a corpus is already large.

The Phi lesson

Phi-1 reported strong coding benchmark performance for a 1.3B model trained with textbook-quality data, worked examples, and synthetic exercises.^{[4]Reference 4Textbooks Are All You Needhttps://arxiv.org/abs/2306.11644} Its result motivates controlled curation experiments; it doesn't make one data recipe universal.

FineWeb and FineWeb-Edu

FineWeb scales the same idea to web-scale curation. The paper builds a 15T-token dataset from 96 Common Crawl snapshots and then extracts FineWeb-Edu, a 1.3T educational subset filtered with a classifier trained on LLM-judged educational scores. The main lesson is that pipeline design matters: individual per-crawl MinHash deduplication and additional filtering each improved downstream results over weaker baselines^{[7]Reference 7The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557}.

Modern open corpora

Three published open-corpus reports illustrate different tradeoffs after FineWeb. Each changes a different part of the yield-versus-quality problem.

These reports don't establish one universally best filter. DCLM optimized benchmark quality under its evaluation setup with aggressive retention. Nemotron-CC explicitly targeted a larger long-horizon token supply and reported both quality and quantity comparisons. FineWeb2 addresses the separate multilingual problem: thresholds, word segmentation, language identification, and deduplication behavior don't transfer cleanly from English to every language.^{[8]Reference 8DataComp-LM: In Search of the Next Generation of Training Sets for Language Modelshttps://arxiv.org/abs/2406.11794[18]Reference 18Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasethttps://arxiv.org/abs/2412.02595[19]Reference 19FineWeb2: One Pipeline to Scale Them All - Adapting Pre-Training Data Processing to Every Languagehttps://arxiv.org/abs/2506.20920}

The data wall

This token-yield pressure is often called the data wall. Villalobos et al. estimate an effective stock of about 320T public human-text tokens after quality and repetition adjustments. Under their assumptions about continued dataset growth, they project full utilization between 2026 and 2032, with a median of 2028; an assumed 5x overtraining scenario shifts that median one year earlier.^{[20]Reference 20Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Datahttps://arxiv.org/abs/2211.04325} This is a scenario projection, not evidence that usable public text is already exhausted.

DCLM's selected filter keeps its top 10% of documents by classifier score. Why might a team designing a longer-horizon run test a Nemotron-CC-style approach as well?

Answer

Aggressive filtering can shrink unique-token supply. Nemotron-CC targets a different tradeoff through classifier ensembling and source-conditioned synthetic variants; a team should test whether that additional yield preserves quality for its own horizon and evaluation suite.

Additional crawl volume and stronger curation must be compared empirically. FineWeb is a good example: in its ablations, extraction, per-crawl deduplication, and filtering choices measurably changed downstream results.^{[7]Reference 7The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557}

Synthetic data: supplement, not substitute

Synthetic data can raise signal density, but it works best when it fills a specific gap rather than replacing the whole corpus.

Where synthetic data helps

Phi-1 is one clear example. Instead of trying to mimic the raw web distribution, the dataset deliberately emphasized textbook-style explanations, exercises, and synthetic examples^{[4]Reference 4Textbooks Are All You Needhttps://arxiv.org/abs/2306.11644}. That makes synthetic data useful when you want more reasoning traces, worked solutions, or domain-specific exemplars than the open web naturally provides.

One proposed response to token-yield pressure is synthetic expansion. Nemotron-CC generates variants from source documents, including question-answer pairs and distilled passages, and reports improvements in specific training comparisons.^{[18]Reference 18Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasethttps://arxiv.org/abs/2412.02595} Source conditioning gives the generated text provenance, but the authors state that they didn't verify factual accuracy or fidelity for rephrased data. Grounding therefore needs validation rather than assumption.

Why provenance and diversity still matter

Synthetic text inherits teacher errors, style biases, and coverage gaps. Shumailov et al. study degradation when successive model generations train recursively on generated data, a failure mode often called model collapse.^{[21]Reference 21AI models collapse when trained on recursively generated datahttps://www.nature.com/articles/s41586-024-07566-y} This doesn't prohibit synthetic supplementation; it means mixture weight, provenance, factual fidelity, and diversity need evaluation.

Don't conflate verified synthetic-data generation with post-training reinforcement learning. Pre-training pipelines more often use offline generation plus filtering or programmatic checks before tokens are admitted.

Common mistakes and how to catch them

These are the most frequent errors teams make when building or reasoning about pre-training pipelines, with symptoms, causes, and fixes.

Practice drill

Build a corpus-readiness checklist for a trillion-token run:

1. Define source mix, sampling weights, language coverage, deduplication threshold, benchmark-decontamination rules, and PII scrub policy.
2. Add retention metrics by source and language so quality filters can be audited for false positives.
3. Record tokenizer compression rates for target languages and domains before freezing the vocabulary.
4. Specify shard metadata: source, license or policy class, filter version, dedupe cluster, tokenizer version, and split assignment.

The checklist should make the training corpus reproducible, auditable, and measurable before any expensive run starts.