Continued Pretraining for Domain Shift

The scratch GPT lab trained a tiny model from raw text to checkpoint. Real teams usually start from a base model instead. Continued pretraining keeps the same next-token objective, but shifts the text distribution so the model spends more compute on your domain's language.^{[1]Reference 1Don't Stop Pretraining: Adapt Language Models to Domains and Tasks.https://aclanthology.org/2020.acl-main.740/}

Teams often confuse three different tools:

• retrieval-augmented generation (RAG) leaves the weights frozen and injects knowledge at inference time by retrieving documents into the prompt. Use it when facts change often or must be cited.
• Supervised Fine-Tuning (SFT) changes behavior, format, and tone using curated examples such as prompt-response pairs. Use it when the model already knows the domain but answers in the wrong shape.
• Continued pretraining (CPT) changes the weights with the same next-token objective so the model better fits domain terminology, document structure, and statistical patterns. Use it when raw domain text still confuses the base model.

Don't choose from labels alone. In Ovadia et al.'s knowledge-injection experiments, RAG outperformed unsupervised fine-tuning on MMLU and current-events questions, while repeated paraphrases helped fine-tuning on the new-fact task.^{[2]Reference 2Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMshttps://aclanthology.org/2024.emnlp-main.15/} Use that result for factual-update failures, not as a universal ranking. CPT earns an experiment when the model poorly fits the domain text distribution, not when you only need fresher facts or a different answer format.

What changes and what stays the same

In continued pretraining:

• the objective stays the same: predict the next token
• the model architecture stays the same
• the data distribution changes
• the goal changes from general competence to domain adaptation

That differs from later SFT, where the model learns from curated prompt-response examples instead of unlabeled text.

Diagnose the shift before spending training compute

A useful direct signal is held-out raw-text loss and its exponentiated form, perplexity: establish a base-model value on domain documents, then test whether CPT lowers it while a general-text control remains inside budget. A base model scoring worse on domain than general text is only a screening clue because corpora can have different inherent predictability; it doesn't by itself prove CPT will improve product tasks.

Fragmentation during tokenization is a weaker diagnostic. A fixed tokenizer may use more tokens for unfamiliar terminology, increasing context cost, but CPT doesn't change that tokenizer unless you deliberately redesign embeddings and retrain compatible weights. Use fertility as a corpus inspection signal, not a promise that continued pretraining will shorten tokenized documents.

Don't split a validation corpus by shuffled token chunks. Near-duplicates, revisions of the same manual, or pages from the same source can land in both training and validation and make CPT look stronger than it really is. Assign a provenance or deduplication group to one split before tokenization.

The two failure dynamics: forgetting and underfitting

Resuming training on a new distribution pulls the weights in two directions, and a good run balances them.

Catastrophic forgetting is loss of previously learned ability as parameters shift to absorb new data. Push too hard on domain text and broad validation quality can regress.

Underfitting is the opposite failure: train too gently and the domain leaves no real impression.

Two major controls for this balance are the learning-rate schedule and the data mixture; run length and corpus quality matter too.

Learning rate re-warming and re-decaying

A base checkpoint often finished its original cosine schedule at a very small learning rate. If you resume at that floor, adaptation may be inefficient. If you resume too aggressively, general-text loss may regress.

Ibrahim et al. (2024) study a related decoder-only continual-pretraining setting: updating a model with large new datasets after its original cosine schedule ended.^{[3]Reference 3Simple and Scalable Strategies to Continually Pre-train Large Language Modelshttps://openreview.net/forum?id=DimPeeCxKO} For 405M models under English-to-English and English-to-German shifts, and a 10B-parameter model under the English-to-English shift, learning-rate re-warming, re-decaying, and replay matched retraining baselines on their reported losses and evaluation averages while spending less compute. Their experiment is evidence for testing this recipe, not permission to copy one peak learning rate into every domain run.

One subtlety from that work: re-warming can itself increase loss on old data. Sweep the peak and measure both lanes instead of assuming adaptation is free. The paper also explores schedules that aren't tied to one fixed token budget.

Replay: keep prior-data signal in the mix

The second knob is replay: mix a fraction of previous or representative general-purpose data back into the incoming domain corpus. It provides training signal on broad text while the domain stream shifts the model, so it's a practical candidate for limiting regression.

How much replay? Treat it as a sweep, not a standard percentage. In Ibrahim et al.'s headline comparison, the chosen mixes use 5% replay for the SlimPajama update and 25% replay for the larger English-to-German shift.^{[3]Reference 3Simple and Scalable Strategies to Continually Pre-train Large Language Modelshttps://openreview.net/forum?id=DimPeeCxKO} Those values belong to those datasets and compute budgets. With a fixed token budget, replay also replaces some new-domain tokens, so it can reduce adaptation opportunity while controlling general regression.

Your continued-pretraining run resumes from the base checkpoint at its final tiny learning rate and uses 100% domain text. Domain perplexity barely moves. After you raise the re-warm peak, domain perplexity improves but general-text loss regresses. Which two sweeps should you run?

Answer

Sweep a moderate re-warm peak followed by re-decay: inheriting the old floor can underfit, while an aggressive peak can damage broad behavior. Sweep replay ratios too, then select against held-out domain and general metrics. Replay is a regression-control candidate, not the explanation for the original underfitting.

When continued pretraining is the right tool

Reach for continued pretraining when the domain has its own language that the base model under-serves:

• internal incident policies with recurring entity patterns
• incident event streams and service jargon
• long runbook or compliance documents
• dense technical manuals
• codebases with domain-specific APIs and naming conventions

The trigger isn't "the business wants custom behavior." The model needs more exposure to the domain's text distribution before post-training behavior shaping makes sense.

Good signals

Bad signals

Domain-adaptive vs task-adaptive pretraining

The 2020 "Don't Stop Pretraining" paper made this distinction explicit in masked-language-model experiments with RoBERTa:^{[1]Reference 1Don't Stop Pretraining: Adapt Language Models to Domains and Tasks.https://aclanthology.org/2020.acl-main.740/}

• DAPT (domain-adaptive pretraining): keep training on large unlabeled domain text such as incident runbooks or service notes
• TAPT (task-adaptive pretraining): continue on the task's own unlabeled inputs, even when the corpus is smaller

The decision remains useful for decoder-only LLM projects, but don't silently transfer RoBERTa's quantitative gains to a causal base model. You still have to measure whether more exposure to the target text distribution improves your model and downstream task.

A practical decision test

Suppose you're building an incident model for service exception handling. Test raw domain-text continuation and prompt-response behavior separately, then compare failure modes. If the model can't continue the underlying service log or incident note coherently, that points to continued pretraining. If raw continuation is competent but assistant behavior is weak, that points more directly to SFT.

Data for continued pretraining

The same discipline from large-scale pretraining still applies:

• filter low-quality text
• deduplicate aggressively
• remove benchmarks and eval leakage
• scrub PII and sensitive content
• keep provenance and usage rights for every corpus slice

The corpus can be narrower and more targeted. Domain data can also be more sensitive than public pretraining text, so provenance, access control, and removal procedures are product requirements, not cleanup tasks.

Gate the corpus before tokenization

Keep a manifest that records whether a source may be trained on, whether it contains unresolved sensitive content, and whether it's reserved for evaluation. A high-quality domain document that fails one of these gates doesn't belong in the training stream.

support-notes is licensed but still contains unresolved PII, while heldout-probes is clean and licensed. Which source can enter continued-pretraining blocks?

Answer

Neither. Unresolved PII blocks support-notes, and the evaluation-only flag keeps heldout-probes out of training. Corpus quality never overrides rights, privacy, or evaluation-isolation gates.

Keep evaluation text out of training

For a small exact-overlap gate, normalize text and hash it before building token blocks. Production pipelines also need near-duplicate detection, because formatting changes and partial copies will evade exact hashes.

Mixing strategy

Don't assume 100% domain text is always optimal. In practice, teams mix:

• a high-quality domain slice
• a smaller replay slice of general text

That replay is one guardrail against forgetting. The exact ratio is empirical: define candidate ratios, hold total training tokens fixed, and select with domain-gain and broad-regression metrics. If the model forgets too much general language while specializing, the run overshot.

BloombergGPT is a useful contrast, not replay evidence: it was trained from scratch on 51.27% financial and 48.73% public tokens, and reports strong financial performance while remaining competitive on general-purpose benchmarks.^{[4]Reference 4BloombergGPT: A Large Language Model for Financehttps://arxiv.org/abs/2303.17564} It shows that corpus composition should be explicit. It doesn't identify the right CPT replay ratio for your checkpoint.

Pack blocks and preserve the mixture

CPT uses the same causal objective as base pretraining. A common loader recipe joins document token sequences with end-of-document markers and emits full blocks. The separator marks a boundary, but it doesn't prevent cross-document attention by itself. As the data-pipeline chapter explained, choose explicitly between an ordinary causal mask and a document-isolated block-diagonal mask. Small integer token sequences make separator placement inspectable.

This small example drops an incomplete final block instead of padding it. Production loaders need an explicit remainder policy.

Once domain and replay streams are packed, make mixture selection explicit and auditable. Here each twenty-block training window uses a seeded shuffle with the requested replay count.

Evaluation: domain gain without lying to yourself

Continued pretraining needs two evaluation lanes at the same time.

Lane 1: domain gain

Measure:

• domain validation perplexity
• retrieval or classification tasks in the domain
• generation quality on held-out domain documents
• downstream task lift after later SFT

Lane 2: general regression

Measure:

• a small broad-language validation slice
• a lightweight general benchmark set
• free-form generations outside the target domain

If you only watch domain gain, you can accidentally produce a model that sounds like one incident runbook and forgot how to write broadly coherent language.

Evaluate loss in comparable token units. Perplexity is exp(mean negative log-likelihood), so aggregate token-level loss before exponentiating; don't average document perplexities and call the result a corpus metric.

Runnable checkpoint ledger

The simplest useful artifact is a checkpoint ledger. It doesn't train a model; it shows how to choose between checkpoints after a continued-pretraining sweep. Domain perplexity can improve while general text gets worse, so the chosen checkpoint needs to pass both lanes. Use general regression as a hard gate. Among survivors, rank downstream probe accuracy first and use domain perplexity as a tie-breaker. That keeps the policy visible instead of hiding trade-offs inside an arbitrary weighted score.

cpt-12k has the best domain perplexity but exceeds the allowed general-regression budget. cpt-4k passes the gate and has the best downstream probe accuracy among survivors. Which checkpoint wins?

Answer

Choose cpt-4k. General regression is a hard gate, so cpt-12k is ineligible. Among eligible checkpoints, downstream probe accuracy ranks first and domain perplexity only breaks ties.

Stopping rules

Because continued pretraining keeps the same objective, it can feel deceptively safe. It isn't safe by default.

Good stopping cues:

• domain validation loss flattens
• downstream gains after a probe SFT stop improving
• general regressions start to outweigh domain benefits

Bad stopping cues:

• "we still have more domain text"
• "loss is still going down a little"

More steps aren't a free lunch once the domain shift is already absorbed.

Where it fits relative to LoRA and SFT

Compare the choices by asking what you want to change.

LoRA and QLoRA are parameter-efficient implementation choices; QLoRA also stores the frozen base model in quantized form.^{[5]Reference 5QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314} They don't determine what supervision teaches. An adapter can be trained with a next-token domain-text objective or with prompt-response SFT. First choose objective from the failure mode, then choose full-weight or parameter-efficient training from budget and deployment constraints.

A strong training stack often looks like:

1. base model
2. continued pretraining on domain corpus
3. SFT on curated prompt-response data
4. preference optimization if needed

Not every product needs every stage. Choose the stage that matches the failure you observe.

Common pitfalls

Using continued pretraining to fix assistant tone

• Symptom: the model still formats answers badly after a long domain-text run.

• Cause: the issue was interface behavior, not domain language exposure.

• Fix: move to SFT sooner.

Over-specializing on one corpus

• Symptom: domain completions improve, but the model becomes narrow or brittle elsewhere.

• Cause: no replay mixture, or too many adaptation steps.

• Fix: keep a general-text regression lane and stop earlier.

Skipping the downstream check

• Symptom: domain perplexity improves, but the final task model barely benefits.

• Cause: the adaptation run optimized text fit that did not transfer to the product task.

• Fix: probe the adapted checkpoint with a small downstream SFT instead of judging only by perplexity.