Pre-training Data at Scale

The last chapter turned a FLOP budget into a token target. This one asks what those tokens actually are. A 70B dense model using Chinchilla scaling at roughly 20 tokens per parameter needs about 1.4T training tokens. That number is a volume target, not a corpus. Raw crawls, public git trees, and book dumps still contain broken HTML, mirrored pages, spam, private strings, and leaked benchmark questions.

Suppose a 2T-token run completes and its first checkpoint looks healthy. Later, a code evaluation regresses because one shard overrepresented mirrors while a language filter removed symbol-heavy files. A raw token counter won't show either failure; lineage, yield, and source mix will.

A coding-assistant mix needs public code, API docs, runbooks, and design notes. You can't dump those sources into training as-is. The pipeline's job is to turn a messy source lake into documented shards: extract text, drop low-signal records, remove duplicates, scrub sensitive material, keep evaluation sets out, then tokenize and pack what remains.

Track yield after each gate, not just raw download volume. If 2T extracted tokens lose 30% to quality filters and another 20% of the remainder to deduplication, only 1.12T tokens survive. That corpus misses a 1.4T target even though the original crawl looked comfortably large.

From token budget to training inventory

Start with the inventory, not the downloader. A token target tells you how many training events you want; it doesn't tell you which sources fill them or whether a later shard can be traced back to a policy decision.

Three ideas from earlier in the curriculum sit underneath every later choice:

• Tokens: A language model doesn't read words. It reads tokens: reusable pieces such as ship, ping, punctuation, or byte sequences. The tokenizer converts text into integers, and the model learns to predict the next integer.
• Next-token prediction: During pre-training, the model sees a partial sequence and guesses what comes next. Broader, cleaner sequences give it more to learn from.
• Scaling laws: The previous chapter turned model size into a token target. That target still needs extract, filter, dedup, pack, and shard steps before those tokens become training inventory.

If tokens or next-token prediction feel fuzzy, review Language Modeling & Next Tokens before continuing.

How much data is enough?

Ask what the 1.4T figure means before treating it as a procurement number. It estimates training-token volume under one scaling-law setup; it doesn't promise unique, clean, or equally useful tokens.

Hoffmann et al. studied that tradeoff in the Chinchilla scaling laws. Under their fixed-compute dense-transformer setup, model size and training-token count grew together. A useful planning rule from that study is about 20 training tokens per parameter.^{[1]Reference 1Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

For a 70B model, $70 \times 10^9 \times 20 = 1.4 \times 10^{12}$ tokens. That's a lot of books, pages, and repositories, which is why the pipeline needs industrial machinery.

A companion planning formula is training FLOPs of about $6ND$, where $N$ is a consistently counted dense model size and $D$ is the number of training tokens. It's an approximation. Details such as whether output-layer parameters are counted can move fitted optima, but it's useful for capacity estimates.^{[1]Reference 1Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556[2]Reference 2Resolving Discrepancies in Compute-Optimal Scaling of Language Modelshttps://arxiv.org/abs/2406.19146}

Raw token counts still mislead. A smaller corpus of textbook-quality tokens can beat a much larger pile of noisy crawl text. Extra tokens help only when they still add information.

Meta reports pre-training Llama 3's 405B model on 15.6T tokens, about 38.5 tokens per parameter, and says that flagship configuration is approximately compute-optimal under its own fitted laws and training budget. Separately, Meta reports training its smaller models longer than compute-optimal because those models performed better at the same inference budget.^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} Treat the 20x ratio as a planning baseline, not a hard law.

Why is a token target not enough to define a pre-training corpus?

Answer

Token count measures volume, not signal. You still need source mix, filtering, deduplication, privacy scrubbing, benchmark decontamination, and tokenizer fit before those tokens become useful training inventory.

The raw corpus and the clean training mix

The raw internet is a noisy source lake. Useful records sit next to duplicate pages, broken markup, spam, and unsafe material. A pre-training data pipeline is the repeatable system that extracts useful records and turns them into a training-ready mix.

You don't write the text. You design a processing system that handles large partitions repeatably, records each decision, and avoids throwing out high-value records.

Every gate makes a different claim about a record. Keep source identity and policy metadata attached, because dropping a malformed HTML page and routing a high-signal code file to a separate recipe are different decisions.

The modern pipeline has three major phases:

1. Ingestion: Pull raw data from web crawls, code hosts, and curated archives.
2. Cleaning: Filter, deduplicate, decontaminate (strip leaked evaluation-benchmark examples), and scrub for safety.
3. Preparation: Tokenize, shuffle, pack, and split into training shards.

Where the text comes from

Common Crawl doesn't ship paragraphs. It ships WARC files of HTML. Extraction is the first real gate: a parser has to keep a runbook's steps and drop the navigation chrome around them. FineWeb extracts text from WARC snapshots; Llama 3 reports a custom HTML parser tuned for boilerplate removal and for keeping math and code structure.^{[4]Reference 4The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

After extraction, sources still aren't interchangeable. Web data gives breadth and freshness. Code is dense in syntax and exact structure. Books and papers give long-form argument. Carefully curated synthetic or textbook-style data can raise signal density further. Phi-1 is a clear example: its authors report strong coding-benchmark performance from a 1.3B model trained on textbook-quality and synthetic data.^{[5]Reference 5Textbooks Are All You Needhttps://arxiv.org/abs/2306.11644}

Before choosing weights, ask what each source can lose. A web filter that removes navigation may improve prose, while the same rule can erase code context or a license header that an audit needs. Preserve source and license metadata so a yield change has an explanation.

Note: Some papers disclose approximate mixtures while many don't. Llama 3 reports a final pre-training mix of roughly 50% general knowledge, 25% mathematical and reasoning data, 17% code, and 8% multilingual data.^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} Mixture weights are decisions, not raw-byte proportions.

Mixture weights, curriculum, and data scheduling

Source quality is only half the decision. You also have to decide how often each source appears and whether that weighting stays fixed through the whole run.

If clean code is only 10% of the retained pool, should it occupy 10% of training? Maybe not. Upsampling changes the model's exposure without pretending that the source produced more unique tokens.

Three knobs matter:

1. Source weighting: web, code, books, papers, synthetic data
2. Schedule over time: fixed mix vs later-stage reweighting
3. Within-source weighting: whether some slices replay more often than others

The wrong default is "sample in proportion to raw bytes." Raw crawl volume isn't the same thing as training value. The Pile is still useful as a reference because it made mixture design explicit instead of treating web text as one giant bucket.^{[6]Reference 6The Pile: An 800GB Dataset of Diverse Text for Language Modeling.https://arxiv.org/abs/2101.00027} Llama 3 shows a later-stage version of the same idea: even after assembling a huge corpus, training shifted toward higher-value slices during annealing.^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

This is practical curriculum learning in pretraining. It usually isn't "easy lessons first, hard lessons later." More often it means changing source weights over time so the model sees a different mix at different stages.

Two common patterns:

• Static high-quality mix: choose one weighted mixture and keep it stable through most of training
• Late-stage curriculum: keep broad coverage early, then upweight cleaner or more target-relevant slices later^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

Don't confuse curriculum with annealing alone. Annealing usually means late-stage learning-rate decay plus a higher-quality mix. Curriculum is broader: any deliberate schedule that changes what the model sees as training progresses.

It also helps to know the public datasets that anchored a lot of later recipes:

First quality check: heuristic filtering

Before using expensive neural models to classify text, basic heuristic filters drop the lowest-signal material from web crawls. C4 (Colossal Clean Crawled Corpus) is the classic published example of that first line of defense.^{[7]Reference 7Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.https://arxiv.org/abs/1910.10683}

The snippet below takes a single document and returns True only if it passes every check. The thresholds are visible on purpose. A real recipe has to measure retention and downstream quality on its own crawl slices before copying them.^{[7]Reference 7Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.https://arxiv.org/abs/1910.10683[4]Reference 4The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557}

Predict two outcomes before reading the code: a long runbook should pass, while a short but symbol-dense source file may fail its general-text checks. That second result is a routing signal, not proof that the file has no training value.

Each gate is a different failure mode:

• Length: very short documents often carry little context; extremely long extractions may be concatenated pages or indexes.
• Repetition: high repeated n-gram fractions are a useful spam or boilerplate signal, but legitimate templates can also repeat.
• Policy list: a reviewed list can exclude content a specific corpus policy doesn't want. The threshold is a policy choice, not a universal quality label.
• Character ratio: a general-text recipe can discard symbol-heavy extractions. A code corpus needs a different recipe.
• Sentence terminators: few periods can signal a malformed extraction, but they're a poor universal language or format rule.

These heuristics usually run on general web text before the final mix is assembled. Code corpora often use a parallel recipe, because symbol-heavy files that look low-quality to a web-text filter can still be high-signal pre-training data.

Second quality check: classifier-based filtering

After heuristic filtering, modern pipelines use classifiers trained to distinguish high-quality educational text from low-quality web chatter:

1. Language ID: A fastText classifier screens whether the document matches the intended training languages.
2. Quality classifier: A classifier scores whether a document looks like reference-quality material rather than random crawl text. FineWeb-Edu uses an educational-quality classifier trained on LLM-judged scores, DCLM trains a fastText classifier on instruction-formatted and ELI5 text and keeps only the top ~10% by score, and Meta reports separate Llama 3 classifiers for general quality, code, and reasoning signals.^{[4]Reference 4The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557[8]Reference 8DataComp-LM: In Search of the Next Generation of Training Sets for Language Modelshttps://arxiv.org/abs/2406.11794[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}
3. Perplexity filtering: A language model such as an n-gram model can score a document against a reference distribution. Unusually high or low scores can flag text for corpus-specific retention rules; CCNet is a published example.^{[9]Reference 9CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datahttps://arxiv.org/abs/1911.00359}

A classifier score is a ranking signal, not ground truth. The threshold is a choice about which errors and how much yield the next training experiment can afford.

Filtering decisions trade yield for the type of data retained. DCLM selected a fastText filter that retained its top 10% of documents after evaluating trained models, rather than assuming a score threshold was automatically better.^{[8]Reference 8DataComp-LM: In Search of the Next Generation of Training Sets for Language Modelshttps://arxiv.org/abs/2406.11794}

The next snippet applies one threshold to a five-document toy mix. It doesn't prove the cutoff is good. It only shows how token yield collapses as the score floor rises.

Removing duplicates

Deduplication is a core quality step. The internet contains boilerplate headers, repeated SEO text, mirrored sites, copied documentation, and reposted articles. Lee et al. found that, on the corpora and model sizes they tested, deduplication reduced emitted memorized text by about 10x and reached the same or better accuracy in fewer training steps.^{[11]Reference 11Deduplicating Training Data Makes Language Models Better.https://arxiv.org/abs/2107.06499}

Exact deduplication

Take a cryptographic hash such as SHA-256 of each document and drop exact matches. It's fast, and it misses near-duplicates: a policy page where only the timestamp or one verb changed.

MinHash + LSH (near-deduplication)

MinHash with Locality-Sensitive Hashing (LSH) is one widely used approach for fuzzy deduplication at scale.^{[12]Reference 12On the Resemblance and Containment of Documents.https://www.cs.princeton.edu/courses/archive/spring13/cos598C/broder97resemblance.pdf[11]Reference 11Deduplicating Training Data Makes Language Models Better.https://arxiv.org/abs/2107.06499} FineWeb applied MinHash independently per crawl using word 5-grams and parameters intended to target documents with at least 75% similarity. Its experiments favored per-crawl rather than global deduplication.^{[4]Reference 4The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557}

The idea, by hand. Take three short documents:

• Doc A: "The key policy says stale accounts need security review."
• Doc B: "The key policy says stale accounts require security review."
• Doc C: "Audit logs record deployment rollback events."

Doc A and Doc B share almost every word. Break them into overlapping 2-word chunks (shingles):

• Doc A shingles: ["The key", "key policy", "policy says", "says stale", ...]
• Doc B shingles: ["The key", "key policy", "policy says", "says stale", ...]

The overlap is high. Doc C shares zero shingles with A or B. Jaccard similarity is $|A \cap B| / |A \cup B|$. For these two policy pages that's $6 / 10 = 0.60$. MinHash turns each document into a small numeric sketch that estimates this overlap. LSH then routes similar sketches into the same candidate set so you inspect likely duplicates rather than comparing every pair. Because LSH is approximate, a candidate still needs a final similarity decision.

Use a candidate-then-verdict boundary. hashlib and banded LSH make likely pairs cheap to find; exact Jaccard and an explicit representative rule decide whether a document gets dropped.

A is inserted first. When B arrives, two LSH bands collide, so A becomes a candidate. Exact Jaccard 0.60 clears the 0.50 threshold, so B drops. C has no band collision and no overlap, so it's kept. Published recipes make different final decisions after MinHash collisions. Lee et al. verified candidate pairs with actual Jaccard and edit similarity, while FineWeb used matching MinHash buckets and transitive clustering. Production thresholds, shingle sizes, verification rules, and whether deduplication operates per crawl or globally must be evaluated for the corpus.^{[11]Reference 11Deduplicating Training Data Makes Language Models Better.https://arxiv.org/abs/2107.06499[4]Reference 4The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557}

Why does near-deduplication matter even after exact hashing?

Answer

Exact hashes catch only byte-identical copies. Web corpora contain mirrored pages, lightly edited boilerplate, and reposts, so MinHash/LSH catches duplicates that waste training compute and increase memorization risk.

Keeping benchmarks clean: decontamination

Decontamination means keeping evaluation benchmarks out of the pre-training data. If a model sees MMLU, HumanEval, or GSM8K examples during pre-training, reported scores can reflect memorization instead of generalization.

The risk isn't hypothetical. Web crawls contain GitHub repos with interview questions, educational sites with standardized tests, and forums where people paste benchmark prompts. Training corpora and evaluation sets often draw from the same public web, so contamination is a recurring failure mode.

Deduplication and decontamination answer different questions. A benchmark prompt can appear once in the crawl, so it isn't a duplicate and can still invalidate an evaluation.

N-gram overlap filtering

One common approach is n-gram overlap detection. For each training document, check whether it contains spans from an evaluation set. The n-gram size, normalization, removal threshold, and code-specific rules must be recorded with the released corpus. GPT-3 attempted pre-training filtering with 13-gram overlaps, then reported that a filtering bug left some overlaps in place. Its post-hoc benchmark analysis used variable n-gram lengths capped at 13. Llama 3 reports excluding benchmark training sets from its annealing data.^{[13]Reference 13Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

Exact-match filtering isn't enough. A coding problem with renamed variables or a question with shuffled answer choices can still leak. Pipelines may add fuzzy or task-specific checks, then document what they removed so benchmark-clean claims are auditable.

The snippet below shows that exact 4-grams catch a copied sentence and miss a paraphrase. That's the point: one detector isn't a complete screen.

What makes benchmark contamination different from ordinary duplicate text?

Answer

Contamination corrupts evaluation. If benchmark examples leak into pre-training, reported scores can reflect memorization rather than generalization, so the training run may look stronger than it really is.

Safety and privacy scrub

After deduplication and decontamination, a pipeline may scrub personally identifiable information (PII) and unsafe content according to its data policy and legal obligations. A model trained on scraped identifiers can reproduce sensitive strings at inference time. FineWeb reports anonymizing email and public-IP addresses; Llama 3 reports removing domains likely to contain high volumes of PII or adult content.^{[4]Reference 4The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

A high-quality score doesn't make a document safe to train on. PII can sit inside a well-written runbook, so privacy controls need their own evidence and audit trail.

PII removal uses a layered approach:

• Regex patterns: Rule-based detection can catch structured patterns such as email addresses or public IP addresses cheaply.
• Entity or policy classifiers: Context-aware models can identify classes of content that rigid patterns miss, but they need false-positive evaluation.
• Source and domain rules: Known high-risk sources can be excluded before training, as reported for Llama 3.^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

Unsafe-content filtering is policy-specific and imperfect. FineWeb applies URL-level filtering for adult content but notes that harmful documents can remain. Llama 3 combines domain rules with dirty-word counting for adult websites that domain blocklists miss.^{[4]Reference 4The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} Audit retained and removed samples across languages and source types instead of treating one score threshold as a universal safety boundary.

Turning text into numbers: tokenization

Tokenization converts raw text into integer IDs the model can process. At the scale of trillions of tokens, tokenization itself needs parallel workers.

Choosing a tokenizer

Vocabulary size, sequence length, and language coverage trade off. A smaller vocabulary can force text to split into more subwords, increasing sequence length and the cost of attention. A larger vocabulary can improve compression when it matches the corpus, but it also requires a larger embedding matrix.

Measure compression by source and language before freezing the vocabulary. One global average can hide a multilingual or code shard that consumes far more context than the rest.

Common subword algorithms:

Training a BPE tokenizer

When you train a model family from scratch, the tokenizer should be trained on a representative sample of the pre-training data so it learns the common subword patterns. Reusing a mature tokenizer can be a good engineering choice, but a mismatched tokenizer wastes context length, especially for multilingual or domain-heavy corpora.

The next demo is a tiny character-level BPE. Production pipelines use byte-level BPE and much larger samples. The merge mechanic is the same: count adjacent pairs, merge the most frequent pair, repeat. After a few merges, repeated words such as after collapse into fewer tokens.

In a byte-level setup you often don't need an <unk> token, because any UTF-8 byte sequence can be represented. Vocabulary size is still a real design decision. A larger vocabulary can improve compression when its extra tokens match the target text, but it also increases embedding-table size. Meta reports that Llama 3 moved to a 128K-token vocabulary, starting from a tiktoken base plus 28K extra tokens, and improved English compression from 3.17 to 3.94 characters per token relative to Llama 2.^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

Running the pipeline at scale

Large corpus pipelines need distributed execution beyond demonstration scripts. DCLM reports starting with a 240T-token Common Crawl pool containing about 200B documents and 370TB of compressed text, and processing crawl data on hundreds of AWS CPU nodes.^{[8]Reference 8DataComp-LM: In Search of the Next Generation of Training Sets for Language Modelshttps://arxiv.org/abs/2406.11794}

At that scale, data is partitioned for extraction and filtering, intermediate artifacts are stored durably, and every transformation needs enough metadata to reproduce which records reached training.

The failure boundary changes here. A malformed shard can waste a multi-node run even when document-level filters passed, so shard manifests need the same lineage as individual records.

Deduplication changes the system shape: signatures can be computed independently, but candidate grouping needs records with related signatures to meet. Whether that step uses a global operation or a restricted partition, such as FineWeb's per-crawl recipe, changes both resource cost and retained data quality.^{[4]Reference 4The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557}

Trillion-token scale challenges

DCLM's 240T-token, 370TB crawl pool is too large to treat as one job. Split the work along the same seams the later training job will audit:

• Ingestion and filtering: stream extraction, language ID, rules, and classifier scores without dropping the source URL or filter version.
• Deduplication: compute signatures in parallel; then pick an explicit partition (per crawl vs global) before candidate grouping.
• Training order: shuffle tokenized shards with a recorded seed so one host or one dump doesn't own a whole epoch.
• Reproducibility: given one suspicious training example, recover its source document, filter version, mixture weight, and shard.

Data packing and sequence efficiency

Documents vary widely in length: an article might be thousands of tokens while a forum post is short. Padding each document to a fixed sequence length spends compute on padding tokens. Best-fit packing groups documents into fixed-length token blocks to reduce that waste.

Packing creates another design decision: if multiple documents share one sequence, should tokens in one document attend to tokens from another? Some causal-language-model recipes concatenate documents with end-of-document separators and keep the ordinary causal mask. Others isolate documents with a block-diagonal mask so one document can't read tokens from an earlier document in the same packed block. The isolated version below prevents artificial cross-document context while retaining packing efficiency.

For the packed IDs [0, 0, 0, 1, 1], predict the first row for document 1: it should contain no attention edges into document 0. The mask makes that boundary observable instead of leaving it to a delimiter token and hope.

Data annealing is a late-stage recipe adjustment where the learning rate decays and the data mix shifts toward higher-value slices. It isn't a universal "final 10%" rule. Llama 3 reports annealing over the final 40B tokens while upsampling very high-quality sources and excluding benchmark training sets from the annealing pool.^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

Quality and quantity must be measured together

Recent model and dataset papers show that data quality and token yield must be measured together once a corpus is already large.

The question now changes from “did the record pass?” to “did the retained data improve a trained model?” A filter proxy can rise while useful coverage or token supply falls.

The Phi lesson

Phi-1 reported strong coding-benchmark performance for a 1.3B model trained with textbook-quality data, worked examples, and synthetic exercises.^{[5]Reference 5Textbooks Are All You Needhttps://arxiv.org/abs/2306.11644} That result motivates controlled curation experiments. It doesn't make one data recipe universal.

FineWeb and FineWeb-Edu

FineWeb scales the same idea to web-scale curation. The paper builds a 15T-token dataset from 96 Common Crawl snapshots and then extracts FineWeb-Edu, a 1.3T educational subset filtered with a classifier trained on LLM-judged educational scores. The main lesson is that pipeline design matters: individual per-crawl MinHash deduplication and additional filtering each improved downstream results over weaker baselines.^{[4]Reference 4The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557}

Modern open corpora

Three published open-corpus reports illustrate different tradeoffs after FineWeb. Each changes a different part of the yield-versus-quality problem.

These reports don't establish one universally best filter. DCLM optimized benchmark quality under its evaluation setup with aggressive retention. Nemotron-CC explicitly targeted a larger long-horizon token supply and reported both quality and quantity comparisons. FineWeb2 addresses the separate multilingual problem: thresholds, word segmentation, language identification, and deduplication behavior don't transfer cleanly from English to every language.^{[8]Reference 8DataComp-LM: In Search of the Next Generation of Training Sets for Language Modelshttps://arxiv.org/abs/2406.11794[18]Reference 18Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasethttps://arxiv.org/abs/2412.02595[19]Reference 19FineWeb2: One Pipeline to Scale Them All - Adapting Pre-Training Data Processing to Every Languagehttps://arxiv.org/abs/2506.20920}

The data wall

This token-yield pressure is often called the data wall. Villalobos et al. estimate an effective stock of public human text at about 320T tokens after quality and multi-epoch adjustments, with a broad 95% interval from 65T to 1700T. The paper's exhaustion-point figure is about 400T ($4 \times 10^{14}$). Under their assumptions about continued dataset growth, they project full utilization between 2026 and 2032, with a median of 2028. A 5x overtraining scenario shifts that earlier by about one year.^{[20]Reference 20Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Datahttps://arxiv.org/abs/2211.04325} This is a scenario projection, not evidence that usable public text is already exhausted.

DCLM's selected filter keeps its top 10% of documents by classifier score. Why might a team designing a longer-horizon run test a Nemotron-CC-style approach as well?

Answer

Aggressive filtering can shrink unique-token supply. Nemotron-CC targets a different tradeoff through classifier ensembling and source-conditioned synthetic variants; a team should test whether that additional yield preserves quality for its own horizon and evaluation suite.

Additional crawl volume and stronger curation must be compared empirically. FineWeb is a good example: in its ablations, extraction, per-crawl deduplication, and filtering choices measurably changed downstream results.^{[4]Reference 4The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalehttps://arxiv.org/abs/2406.17557}

Synthetic data: supplement, not substitute

Synthetic data can raise signal density, but it works best when it fills a specific gap rather than replacing the whole corpus.

Where synthetic data helps

Phi-1 is one clear example. Instead of trying to mimic the raw web distribution, the dataset emphasized textbook-style explanations, exercises, and synthetic examples.^{[5]Reference 5Textbooks Are All You Needhttps://arxiv.org/abs/2306.11644} That makes synthetic data useful when you want more reasoning traces, worked solutions, or domain-specific exemplars than the open web naturally provides.

One proposed response to token-yield pressure is synthetic expansion. Nemotron-CC generates variants from source documents, including question-answer pairs and distilled passages, and reports improvements in specific training comparisons.^{[18]Reference 18Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasethttps://arxiv.org/abs/2412.02595} Source conditioning gives the generated text provenance, but the authors state that they didn't verify factual accuracy or fidelity for rephrased data. Grounding therefore needs validation rather than assumption.

Why provenance and diversity still matter

Synthetic text inherits teacher errors, style biases, and coverage gaps. Shumailov et al. study degradation when successive model generations train recursively on generated data, a failure mode often called model collapse.^{[21]Reference 21AI models collapse when trained on recursively generated datahttps://www.nature.com/articles/s41586-024-07566-y} This doesn't prohibit synthetic supplementation. It means mixture weight, provenance, factual fidelity, and diversity need evaluation.

Source conditioning gives you a trail to inspect, not a guarantee that generated text preserved the source. Keep that distinction visible when a synthetic slice enters a training shard.

Don't conflate verified synthetic-data generation with post-training reinforcement learning. Pre-training pipelines more often use offline generation plus filtering or programmatic checks before tokens are admitted.

Common mistakes and how to catch them

Read each row as a small incident: observe the symptom, name the boundary that failed, then make one controlled change and rerun the smallest faithful slice.

Before a trillion-token run

Use one suspicious shard as the lab. Trace a record back to its source, inspect each filter and dedupe decision, calculate yield, then decide whether the shard should be rerun or reweighted.

The readiness record should include:

1. Define source mix, sampling weights, language coverage, deduplication threshold, benchmark-decontamination rules, and PII scrub policy.
2. Add retention metrics by source and language so quality filters can be audited for false positives.
3. Record tokenizer compression rates for target languages and domains before freezing the vocabulary.
4. Specify shard metadata: source, license or policy class, filter version, dedupe cluster, tokenizer version, and split assignment.

The checklist should make the training corpus reproducible, auditable, and measurable before any expensive run starts.