Pre-training Data at Scale

Scaling laws gave us a target: if you want a capable dense model, you may need hundreds of billions or trillions of useful training tokens. Pre-training data pipelines decide what those tokens are before any fine-tuning or alignment happens. This chapter covers sourcing, cleaning, deduplication, filtering, and mixing at the scale where data quality becomes model behavior.

Imagine you are building a customer-support chatbot for an online marketplace. It needs to answer questions about return policies, shipping delays, and payment disputes. You can't write answers for every question in advance, so you decide to train an AI system on a large collection of support transcripts, product descriptions, and shipping documentation.

But where does that collection come from? You can't dump raw web pages into the model and expect useful behavior. Web-scale text includes ads, broken HTML, duplicate product listings, forum spam, private data, and benchmark questions. Training on unfiltered data would be like stocking a fulfillment center with unsorted returns, duplicate labels, and damaged boxes. Useful records are in there, but the line spends most of its time sorting waste unless the intake process is disciplined.

This article is about the engineering pipeline that turns raw data into clean training inventory. You'll learn how teams ingest large raw corpora, filter low-signal text, remove duplicates, scrub private information, and convert what's left into the integer tokens a model trains on.

From token budget to training inventory

Before we build the pipeline, recall three ideas from earlier in the curriculum:

• Tokens: A language model doesn't read words. It reads tokens (short subword fragments like "ship", "ping", or " ##ing"). The tokenizer converts text into integers, and the model learns to predict the next integer in a sequence.
• Next-token prediction: During pre-training, the model sees a partial sentence and learns to guess what comes next. The more diverse and high-quality the sequences it sees, the broader its knowledge becomes.
• Scaling laws: The previous chapter turned model size into a token target. This chapter turns that target into an engineering system.

If tokens or next-token prediction feel fuzzy, review Language Modeling & Next Tokens before continuing. Everything here assumes you understand why the model needs large amounts of varied text.

How much data is enough?

Let's start with a concrete number. Suppose you want to train a 70 billion parameter model. How many tokens should you feed it?

Researchers at DeepMind answered this with the Chinchilla scaling laws. Under a fixed compute budget, they found that model size and training-token count should grow together. A useful rule of thumb for dense decoder-only transformers is about 20 training tokens per parameter^[1].

For a 70B model:

• 70 billion parameters x 20 tokens per parameter = 1.4 trillion tokens

That's roughly the entire text content of millions of books, web pages, and code repositories combined. It gives you a sense of why the pipeline needs industrial scale.

A companion formula for back-of-the-envelope planning is training FLOPs of about $6ND$, where $N$ is a consistently counted dense model size and $D$ is the number of training tokens. It's an approximation, and details such as whether output-layer parameters are counted can move fitted optima, but it's useful for quick capacity estimates.^[1][2]

Raw token counts are misleading. A smaller corpus of textbook-quality tokens can beat a much larger pile of noisy crawl text. Scaling data only helps when the extra tokens still add real information.

Meta reports pre-training Llama 3's 405B model on 15.6T tokens, about 38.5 tokens per parameter, and says that flagship configuration is approximately compute-optimal under its own fitted laws and training budget. Separately, Meta reports training its smaller models longer than compute-optimal because those models performed better at the same inference budget.^[3] Treat the 20x ratio as a planning baseline, not a hard law.

The raw dock and the clean training mix

Think of the raw internet as an overloaded receiving dock. It contains valuable inventory, but it's mixed with duplicate labels, broken packaging, spam, and unsafe material. A pre-training data pipeline is the automated sorting line that extracts useful records and turns them into a training-ready mix.

Your job as the AI engineer isn't to write the text. It's to design a sorting system that processes large partitions repeatably, records each decision, and avoids throwing out high-value records.

The modern pipeline has three major phases:

1. Ingestion: Pull raw data from web crawls, code hosts, and curated archives.
2. Cleaning: Filter, deduplicate (remove duplicate documents), decontaminate (strip out leaked evaluation-benchmark examples), and scrub for safety.
3. Preparation: Tokenize, shuffle, pack, and split into training shards.

Where the text comes from

Different data sources serve different roles in the training mixture. Web data gives breadth and freshness. Code is dense in syntax, structure, and exact correctness. Books and papers provide long-form coherent explanations. Carefully curated synthetic or textbook-style data can raise signal density further.

Phi-1 is a good example: its authors report strong coding benchmark performance from a 1.3B model trained on textbook-quality and synthetic data.^[4]

Note: Some papers disclose approximate mixtures while many don't. Llama 3 reports a final pre-training mix of roughly 50% general knowledge, 25% mathematical and reasoning data, 17% code, and 8% multilingual data.^[3] The broader pattern is that mixture weights are decisions, not raw-byte proportions.

Mixture weights, curriculum, and data scheduling

Source quality is only half the decision. You also have to decide how often each source appears and whether that weighting stays fixed through the whole run.

Three knobs matter:

1. Source weighting: web, code, books, papers, synthetic data
2. Schedule over time: fixed mix vs later-stage reweighting
3. Within-source weighting: whether some slices replay more often than others

The wrong default is "sample in proportion to raw bytes." Raw crawl volume isn't the same thing as training value. The Pile is still useful as reference because it made mixture design explicit instead of treating web text as one giant bucket.^[5] Llama 3 shows a later-stage version of the same idea: even after assembling a huge corpus, training shifted toward higher-value slices during annealing.^[3]

This is practical curriculum learning in pretraining. It usually isn't "easy lessons first, hard lessons later." More often it means changing source weights over time so the model sees a different mix at different stages.

Two common patterns:

• Static high-quality mix: choose one weighted mixture and keep it stable through most of training
• Late-stage curriculum: keep broad coverage early, then upweight cleaner or more target-relevant slices later^[3]

Don't confuse curriculum with annealing alone. Annealing usually means late-stage learning-rate decay plus a higher-quality mix. Curriculum is broader: any deliberate schedule that changes what the model sees as training progresses.

It also helps to know the public datasets that anchored a lot of later recipes:

First quality gate: heuristic filtering

Before using expensive neural models to classify text, basic heuristic filters drop the lowest-signal material from web crawls. As seen in foundational datasets like C4 (Colossal Clean Crawled Corpus)^[6], these simple rules act as a highly efficient first line of defense.

The snippet below demonstrates how these basic rules are implemented in practice. It takes a single document as input and returns True only if the document passes all checks.

What this code does, step by step

• Length check: Very short documents often carry little context; extremely long extractions may be concatenated pages or indexes.
• Repetition check: High repeated n-gram fractions are a useful spam or boilerplate signal, but legitimate templates and structured documents can also repeat.
• Policy-list check: A reviewed list can identify content a specific corpus policy excludes; its threshold is a policy choice, not a universal quality label.
• Character check: A general-text recipe can discard symbol-heavy extractions, while a code corpus needs a different recipe.
• Sentence check: Few sentence terminators can signal a malformed prose extraction, but are unsuitable as a universal language or format rule.

The code intentionally uses simple thresholds so the gates are visible. C4 and FineWeb use related heuristic filtering ideas, but a real recipe must measure retention and downstream model quality on its own crawl slices before adopting thresholds.^[6][7]

These heuristics usually run on general web text before the final data mix is assembled. Code corpora often use a parallel recipe, because symbol-heavy files that look low-quality to a web-text filter can still be very valuable for pre-training.

Second quality gate: classifier-based filtering

After heuristic filtering, modern pipelines use classifiers trained to distinguish high-quality educational text from low-quality web chatter:

1. Language ID: A fastText classifier screens whether the document matches the intended training languages.
2. Quality Classifier: A classifier scores whether a document looks like reference-quality material rather than random crawl text. FineWeb-Edu uses an educational-quality classifier trained on LLM-judged scores, DCLM trains a fastText classifier on instruction-formatted and ELI5 text and keeps only the top ~10% by score, and Meta reports separate Llama 3 classifiers for general quality, code, and reasoning signals.^[7][8][3]
3. Perplexity filtering: A language model such as an n-gram model can score a document against a reference distribution. Unusually high or low scores can flag text for corpus-specific retention rules; CCNet is a published example of perplexity-based filtering.^[9]

Filtering decisions trade yield for the type of data retained. DCLM selected a fastText filter that retained its top 10% of documents after evaluating trained models, rather than assuming a score threshold was automatically better.^[8]

Removing duplicates

Deduplication is a core quality step. The internet contains large amounts of duplicated content: boilerplate headers, repeated SEO text, mirrored sites, copied documentation, and reposted articles. Lee et al. found that, on the corpora and model sizes they tested, deduplication reduced emitted memorized text by about 10x and reached the same or better validation accuracy in fewer training steps.^[11]

Exact deduplication (hash-based)

This involves taking a cryptographic hash, such as SHA-256, of each document and removing exact matches. It's fast but misses near-duplicates, such as a news article where only the timestamp or a single word was changed.

MinHash + LSH (near-deduplication)

MinHash with Locality-Sensitive Hashing (LSH) is one widely used approach for fuzzy deduplication at scale.^[12][11] FineWeb, for example, applied MinHash independently per crawl using word 5-grams and parameters intended to target documents with at least 75% similarity; its experiments favored per-crawl rather than global deduplication.^[7]

The idea, by hand: Imagine you have three short documents:

• Doc A: "The return policy says damaged items need a prepaid return label."
• Doc B: "The return policy says damaged items require a prepaid return label."
• Doc C: "Carrier scans update delivery promises."

Doc A and Doc B share almost every word. If you break them into overlapping 2-word chunks (shingles), they look like this:

• Doc A shingles: ["The return", "return policy", "policy says", "says damaged", ...]
• Doc B shingles: ["The return", "return policy", "policy says", "says damaged", ...]

The overlap is high. Doc C shares zero shingles with A or B. MinHash turns each document into a small numeric "sketch" that estimates this overlap relationship. LSH then routes similar sketches into the same candidate set so we inspect likely duplicates rather than comparing every pair. Because LSH is approximate, a candidate still needs a final similarity decision.

Here is the full pipeline:

The code below shows how to build a basic MinHash and LSH pipeline using the datasketch library. For a tiny demo corpus, it builds signatures from bigram shingles, uses LSH only for candidate lookup, and verifies candidates with exact Jaccard overlap before dropping a document.

Why this works: doc1 is inserted first. When doc2 arrives, LSH returns doc1 as a candidate; an exact Jaccard check then confirms sufficient overlap before the drop. doc3 has no confirmed neighbor, so it is kept. Production thresholds, shingle sizes, and whether deduplication operates per crawl or globally must be evaluated for the corpus; FineWeb's published recipe used 5-grams and a 75% target.^[7]

Keeping benchmarks clean: decontamination

An important and often overlooked step is decontamination: ensuring that evaluation benchmarks don't leak into the pre-training data. If a model sees MMLU, HumanEval, or GSM8K examples during pre-training, its reported performance becomes misleading because it may be memorizing instead of generalizing.

The risk is substantial. Web crawls contain GitHub repositories with coding interview questions, educational websites with standardized test questions, and forums where users paste benchmark prompts. Because training corpora and evaluation sets often draw from the same public web, contamination is a recurring risk rather than a hypothetical edge case.

N-gram overlap filtering

One common approach is n-gram overlap detection. For each document in the training corpus, check whether it contains spans from an evaluation dataset. The n-gram size, normalization, removal threshold, and code-specific rules must be recorded with the released corpus. GPT-3, for example, used a conservative 13-gram analysis; Llama 3 reports excluding benchmark training sets from its annealing data.^[13][3]

Exact-match filtering isn't enough. A coding problem with renamed variables or a question with shuffled answer choices can still leak. Pipelines may add fuzzy or task-specific checks, then document what they removed so benchmark-clean claims are auditable.

Safety and privacy scrub

After deduplication and decontamination, a pipeline may scrub personally identifiable information (PII) and unsafe content according to its data policy and legal obligations. A model trained on scraped identifiers can reproduce sensitive strings at inference time, creating privacy and compliance risk. FineWeb reports anonymizing email and public-IP addresses; Llama 3 reports removing domains likely to contain high volumes of PII or adult content.^[7][3]

PII removal uses a layered approach:

• Regex patterns: Rule-based detection can catch structured patterns such as email addresses or public IP addresses cheaply.
• Entity or policy classifiers: Context-aware models can identify classes of content that rigid patterns miss, but require false-positive evaluation.
• Source and domain rules: Known high-risk sources can be excluded before model training, as reported for Llama 3.^[3]

Unsafe-content filtering is policy-specific and imperfect. FineWeb applies URL-level filtering for adult content but explicitly notes that harmful documents can remain. Llama 3 combines domain rules with dirty-word counting for adult websites that domain blocklists miss.^[7][3] Whatever controls you choose, audit retained and removed samples across languages and source types instead of treating one score threshold as a universal safety boundary.

Turning text into numbers: tokenization

Tokenization converts raw text into integer IDs that the model can process. At the scale of trillions of tokens, tokenization itself requires significant parallel computation.

Choosing a tokenizer

Selecting the right tokenizer involves trading off between vocabulary size, sequence length, and language support. A smaller vocabulary can force text to split into more subwords, increasing sequence length and the computational cost of attention. A larger vocabulary can improve compression when it matches the corpus, but it also requires a larger embedding matrix, which consumes memory.

Common subword-tokenizer algorithms use data-driven vocabulary construction:

Training a BPE tokenizer

When you train a model family from scratch, the tokenizer should be trained on a representative sample of the pre-training data so it learns the most common subword patterns. Reusing a mature tokenizer can be a good engineering choice, but a mismatched tokenizer can waste context length, especially for multilingual or domain-heavy corpora. The code below uses the Hugging Face tokenizers library to initialize a byte-level Byte-Pair Encoding (BPE) tokenizer. For the demo, it trains from an in-memory sample; production pipelines train from a carefully mixed corpus sample and may target vocabularies such as 128,000 tokens.

In a byte-level setup, you often don't need an <unk> token because any UTF-8 byte sequence can be represented.

Vocabulary size is a real design decision. A larger vocabulary can improve compression when its extra tokens match the target text, but it also increases embedding-table size and memory footprint. A smaller vocabulary is lighter, but it can produce longer sequences. Meta reports that Llama 3 moved to a 128K-token vocabulary, starting from a tiktoken base plus 28K extra tokens, and improved English compression from 3.17 to 3.94 characters per token relative to Llama 2^[3].

Running the pipeline at scale

Large corpus pipelines require distributed execution beyond demonstration scripts. DCLM reports starting with a 240T-token Common Crawl pool containing about 200B documents and 370TB of compressed text, and processing crawl data on hundreds of AWS CPU nodes.^[8]

At that scale, data is partitioned for extraction and filtering, intermediate artifacts are stored durably, and every transformation needs enough metadata to reproduce which records reached training.

Deduplication changes the system shape: signatures can be computed independently, but candidate grouping needs records with related signatures to meet. Whether that step uses a global operation or a restricted partition, such as FineWeb's per-crawl recipe, changes both resource cost and retained data quality.^[7]

Trillion-token scale challenges

• Ingestion and filtering: Extraction, language ID, rules, and classifier scoring must stream through many documents without losing provenance.
• Deduplication: Signatures are parallel-friendly; candidate grouping and representative selection need an explicit partitioning strategy.
• Training order: Tokenized shards need randomized, documented sampling so one source doesn't unintentionally dominate a run segment.
• Reproducibility: Engineers need to identify the source document, filter version, mixture weight, and shard for suspicious examples.

Data packing and sequence efficiency

Documents vary widely in length: an article might be thousands of tokens while a forum post is short. Padding each document to a fixed sequence length spends compute on padding tokens. Best-fit packing groups documents into fixed-length token blocks to reduce that waste.

Packing creates another design decision: if multiple documents share one sequence, should tokens in one document attend to tokens from another? Some causal-language-model recipes concatenate documents with end-of-document separators and keep the ordinary causal mask. Others isolate documents with a block-diagonal mask so one document can't read tokens from an earlier document in the same packed block. The isolated version below prevents artificial cross-document context while retaining packing efficiency.

Data annealing is a late-stage recipe adjustment where the learning rate decays and the data mix shifts toward higher-value slices. It isn't a universal "final 10%" rule. Llama 3, for example, reports annealing over the final 40M tokens while upsampling very high-quality sources and excluding benchmark training sets from the annealing pool^[3].

Quality beats quantity

Recent model and dataset papers show that data quality and token yield must be measured together once a corpus is already large.

The Phi lesson

Phi-1 reported strong coding benchmark performance for a 1.3B model trained with textbook-quality data, worked examples, and synthetic exercises.^[4] Its result motivates controlled curation experiments; it doesn't make one data recipe universal.

FineWeb and FineWeb-Edu

FineWeb scales the same idea to web-scale curation. The paper builds a 15T-token dataset from 96 Common Crawl snapshots and then extracts FineWeb-Edu, a 1.3T educational subset filtered with a classifier trained on LLM-judged educational scores. The main lesson is that pipeline design matters: individual per-crawl MinHash deduplication and additional filtering each improved downstream results over weaker baselines^[7].

Modern open corpora

Three published open-corpus reports illustrate different tradeoffs after FineWeb. Each changes a different part of the yield-versus-quality problem.

These reports don't establish one universally best filter. DCLM optimized benchmark quality under its evaluation setup with aggressive retention. Nemotron-CC explicitly targeted a larger long-horizon token supply and reported both quality and quantity comparisons. FineWeb2 addresses the separate multilingual problem: thresholds, word segmentation, language identification, and deduplication behavior don't transfer cleanly from English to every language.^[8][18][19]

The data wall

This token-yield pressure is often called the data wall. Villalobos et al. estimate an effective stock of about 320T public human-text tokens after quality and repetition adjustments. Under their assumptions about continued dataset growth, they project full utilization between 2026 and 2032, with a median of 2028; an assumed 5x overtraining scenario shifts that projection one to two years earlier.^[20] This is a scenario projection, not evidence that usable public text is already exhausted.

Additional crawl volume and stronger curation must be compared empirically. FineWeb is a good example: in its ablations, extraction, per-crawl deduplication, and filtering choices measurably changed downstream results.^[7]

Synthetic data: supplement, not substitute

Synthetic data can raise signal density, but it works best when it fills a specific gap rather than replacing the whole corpus.

Where synthetic data helps

Phi-1 is one clear example. Instead of trying to mimic the raw web distribution, the dataset deliberately emphasized textbook-style explanations, exercises, and synthetic examples^[4]. That makes synthetic data useful when you want more reasoning traces, worked solutions, or domain-specific exemplars than the open web naturally provides.

One proposed response to token-yield pressure is synthetic expansion. Nemotron-CC generates variants from source documents, including question-answer pairs and distilled passages, and reports improvements in specific training comparisons.^[18] Source conditioning gives the generated text provenance, but the authors state that they didn't verify factual accuracy or fidelity for rephrased data. Grounding therefore needs validation rather than assumption.

Why provenance and diversity still matter

Synthetic text inherits teacher errors, style biases, and coverage gaps. Shumailov et al. study degradation when successive model generations train recursively on generated data, a failure mode often called model collapse.^[21] This doesn't prohibit synthetic supplementation; it means mixture weight, provenance, factual fidelity, and diversity need evaluation.

Don't conflate verified synthetic-data generation with post-training reinforcement learning. Pre-training pipelines more often use offline generation plus filtering or programmatic checks before tokens are admitted.

Common mistakes and how to catch them

Here are the most frequent errors teams make when building or reasoning about pre-training pipelines, with symptoms, causes, and fixes.

What you should be able to defend

What to remember

• Data sources: Filtered web text usually dominates volume, but code and curated corpora are often upsampled relative to their raw size.
• Quality filtering: Published pipelines combine heuristics, classifiers, and sometimes perplexity filters in different orders; retention and quality must be evaluated.
• Deduplication: Near-deduplication with MinHash/LSH is a common large-corpus approach. LSH produces candidates; a documented decision rule determines removals.
• Tokenization: The choice of tokenizer (BPE, SentencePiece) and vocabulary size affects compression rates and multilingual capabilities.
• Scale: DCLM reports a 240T-token source pool processed with hundreds of CPU nodes; production pipeline choices must preserve provenance and reproducibility at that scale.
• Data packing: Best-fit packing (bin packing) improves training efficiency by minimizing padding waste.
• Data annealing: Late-stage annealing or high-quality upsampling can help, but the schedule is model-specific rather than a universal last-step recipe.
• Synthetic data: Synthetic data is best used as a targeted supplement. Provenance, diversity, and validation matter as much as volume.
• Quality vs. quantity: Filtering, deduplication, and new token supply form an empirical tradeoff, not a universal winner.
• The data wall: Villalobos et al. estimate an effective public-text stock around 320T tokens and project utilization under stated scenarios; that isn't a claim of present exhaustion.

Practice questions with answer sketches

Question 1: You are building a pipeline for a 7B parameter model. Using the 20 tokens-per-parameter rule, how many tokens should you target? Answer: 7B x 20 = 140 billion tokens.

Question 2: A document in your crawl shares 85% of its 5-grams with a document already in the training set. Should you keep it? Why or why not? Answer: Treat it as a near-duplicate under a documented threshold, then retain one representative according to provenance and quality policy. Length alone isn't enough to decide which record is better.

Question 3: Your team is debating whether to spend $1 million on another 5T tokens of raw Common Crawl or on improving the classifier filter. What argument would you make? Answer: Demand a controlled comparison: measure token retention, domain coverage, and downstream performance for an improved filter against the added crawl. FineWeb and DCLM motivate the experiment, but neither proves the answer for this corpus.

Question 4: A junior engineer suggests training the tokenizer on English-only text for a model that must also handle Spanish customer support. What's the risk? Answer: Spanish text will be split into many more subword tokens, increasing sequence length, memory use, and inference cost for every Spanish query. The tokenizer should see a representative multilingual sample.

Question 5: Why is MinHash useful when exact hashing already exists? Answer: Exact hashing catches only byte-identical documents. MinHash sketches overlapping shingles, so it can find pages that are mostly the same after boilerplate, timestamps, whitespace, or a few words change.

Question 6: How would you protect a HumanEval-style benchmark from leakage? Answer: Start with exact string and n-gram checks against benchmark prompts and solutions, then add code-aware checks such as repository filtering and fuzzy matching for renamed variables or lightly edited solutions.

Question 7: What is data annealing? Answer: It's a late-stage recipe change where the learning rate falls and the data mix shifts toward higher-value slices. It isn't a universal final percentage; the right schedule depends on the model, data mix, and target capabilities.

Question 8: When does synthetic data help pre-training? Answer: It helps when it fills a concrete gap, such as worked code examples, reasoning traces, domain procedures, or rare languages. It becomes risky when one teacher model's style overwhelms broad human-written data, which can drift toward model collapse.

Question 9: What is the "data wall," and name two pipeline techniques teams use to push past it. Answer: The data wall is a projection that training demand may approach available public human text. Villalobos et al. estimate an effective stock near 320T tokens and a median full-utilization scenario around 2028. Teams study stronger filtering, deduplicated replay, and validated source-conditioned synthetic data to change the tradeoff.