Continued Pretraining for Domain Shift

In our fictional payment system, an alert fires at 3 AM: an upstream gateway triggers AUTH_TIMEOUT on every retry, exhausting the worker connection pool. You route the incident logs to an assistant built from a standard foundation model. It replies, "Please verify your Wi-Fi settings and ensure the authentication server is reachable." It misses the runbook's two-phase commit protocol, jittered backoff, and mandatory idempotency token for /v2/vault/token-reissue.

Is the model failing because it couldn't retrieve the runbook, because it didn't know how to format an on-call response, or because its internal weights don't understand the vocabulary, syntax, and causal relationships of your internal distributed systems architecture? Those failures call for completely different interventions.

The JAX chapter treated a training step as a replayable state transition: parameters, optimizer state, random-number keys, and metrics cross an explicit boundary. Continued pretraining (CPT) resumes a pretrained model's text-learning objective. This chapter focuses on decoder-only CPT for domain adaptation; domain-adaptive pretraining (DAPT) names that domain-specific scope. Pin the starting checkpoint, domain mix, schedule, and evaluation lanes so a falling training loss can be compared with actual task gain and regression.

What each intervention changes

Return to the incident. If a correct runbook excerpt in the prompt repairs the answer, retrieval is a useful baseline. Retrieval-augmented generation (RAG) can provide unfamiliar definitions, examples, and relationships without changing weights. The resulting attention patterns depend on that context. Fragmented service names don't make in-context learning impossible.

Long retrieved contexts still cost something. For full attention, the pairwise attention component of prefill grows quadratically with token count; projections and feed-forward layers also contribute. Total latency depends on the model, kernel, cache reuse, retrieval, and hardware. Measure time to a useful answer and task accuracy before deciding that a longer prompt is too expensive.

Supervised fine-tuning (SFT) changes trainable weights using demonstrations. It can teach facts and domain behavior as well as answer format. LIMA's Superficial Alignment Hypothesis is a hypothesis supported by its 2023 study of a strong pretrained 65B model and 1,000 curated examples. It doesn't prove that SFT cannot acquire new knowledge or that domain adaptation always needs CPT.^{[1]Reference 1LIMA: Less Is More for Alignment.https://arxiv.org/abs/2305.11206}

Also distinguish masked targets from zero gradients through prompt representations. Response-only SFT omits direct losses on prompt targets, but response predictions depend on the prompt. Their gradients can reach prompt embeddings and attention weights. The current TRL trainer supports completion-only and assistant-only masks; full-sequence training is also available. The dataset format, configuration, and chat template determine the mask.^{[2]Reference 2TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer}

Predict the result of this small causal model. Its input IDs 0 and 1 are prompt tokens; it ignores the first target and scores only response ID 2. A cumulative sum lets the response prediction depend on both prompt embeddings. This is a toy dependency graph, not a transformer. Will either prompt row receive a gradient? The independent cells in this lesson were checked on Python 3.12 with Torch 2.13.0 where required; the markers pin that dependency.

Both prompt rows participate in the scored prediction. Row 2 is unused by this input embedding, and the output head is separate and fixed, so that row has no gradient here. Tied input/output weights would add another gradient path. The mask selects loss terms; it doesn't detach their causal context.

CPT continues the autoregressive next-token objective on cleaned domain text. For a sequence $w_1,\ldots,w_n$, let nonempty $S$ contain the positions actually scored after shifting, padding, and boundary masks:

$$\mathcal{L}_{\mathrm{CLM}}(\theta) = -\frac{1}{|S|}\sum_{i\in S} \log P(w_i \mid w_{<i}; \theta)$$

Here $w_{<i}$ is the available causal context, subject to the chosen context window and document boundaries. With no beginning-of-sequence (BOS) token or preceding context, an ordinary shifted block of $n$ tokens scores $n-1$ targets. Padding or additional masks can reduce that count. Perplexity exponentiates mean negative log-likelihood over the scored positions.^{[3]Reference 3Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity}

CPT usually provides denser target supervision than response-only SFT, but not a universal 100% of raw tokens. Gradients follow the dependency graph and update the parameters you left trainable. Frozen base weights don't update in adapter-only CPT. More domain exposure is an experiment in text fit and task transfer, not a guarantee of factual grounding.

Before picking a data mix, predict what CPT directly rewards. Raw runbooks reward accurate continuations of runbook text. They don't explicitly reward the assistant format your application expects. Later SFT still uses a next-token loss, but its examples are prompt-response pairs where response-only masks restrict loss to the answer.^{[2]Reference 2TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer} Same token-level math, different supervision shape.

These are candidate experiments, not exclusive diagnoses. Compare a retrieval baseline and direct SFT with CPT followed by that same SFT recipe. One bad sampled completion is too noisy to choose an expensive training stage. Don't rank those tools from labels alone. In Ovadia et al.'s knowledge-injection experiments, RAG beat unsupervised fine-tuning on MMLU and current-events questions, while repeated paraphrases of the same fact helped fine-tuning on a new-fact task.^{[4]Reference 4Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMshttps://aclanthology.org/2024.emnlp-main.15/} CPT earns an experiment when the base model poorly fits the domain text distribution, not when you only need fresher facts or a different answer format.

Diagnose the shift before spending training compute

The first question is whether the model actually fits the target text. Record held-out raw-text loss and its exponentiated form, perplexity, for the base model. Then ask whether CPT lowers domain loss while a general-text control stays inside budget. A base model scoring worse on domain than general text is only a screening clue, because corpora can have different inherent predictability. It doesn't prove CPT will improve product tasks.

Next inspect how the tokenizer represents domain identifiers. A fixed tokenizer may spend more tokens on unfamiliar strings, raising context cost, but CPT doesn't change that tokenizer unless you redesign embeddings and retrain compatible weights. Use token fertility as a corpus-inspection signal, not a promise that continued pretraining will shorten documents.

The tiny longest-match tokenizer below is not a production BPE. Predict before running it: which string consumes more pieces, and what does that imply for context cost? timeout and AUTH are in this invented vocabulary, but uppercase TIMEOUT isn't. The remaining error-code characters become separate pieces. This result says nothing about a real model's tokenizer until you run that tokenizer on the corpus.

Vocabulary extension vs. fixed tokenizer

High fertility consumes more of the token window. Measure the whole corpus before assigning a multiplier: one fragmented identifier doesn't establish a threefold increase for every document. If total sequence length really triples, the full-attention pairwise arithmetic grows about ninefold at fixed width, but the whole forward pass and measured latency needn't do so. Subwords can represent domain concepts; fragmentation alone doesn't prove “semantic dilution.”

Keeping the tokenizer preserves vocabulary coordinates and avoids adding untrained rows. The base vocabulary $V$ and embedding matrix $E\in\mathbb R^{V\times d}$ retain their shapes. It doesn't guarantee stable optimization. Code Llama retained the Llama/Llama 2 tokenizer and added four markers for its infilling variants.^{[5]Reference 5Code Llama: Open Foundation Models for Codehttps://arxiv.org/abs/2308.12950} That historical example doesn't establish a universal 5B–10B-token cutoff for vocabulary changes. Compare measured sequence savings, occurrence counts, initialization, and downstream retention.

Alternatively, you can expand the vocabulary with $K$ domain-specific tokens, such as frequent API methods or specialized identifiers. That expands embedding matrix $E \to \mathbb{R}^{(V+K) \times d}$ and output language-modeling head $W_{\mathrm{head}} \to \mathbb{R}^{(V+K) \times d}$, possibly with shared weights.

New rows change input representations and output support. A poorly scaled initialization can disrupt predictions, but a Gaussian draw doesn't inevitably cause collapse. Subword centroid initialization is one candidate: tokenize the new string using the old tokenizer and average those rows:

$$\mathbf{e}(t_{\mathrm{new}}) = \frac{1}{m}\sum_{j=1}^{m} \mathbf{e}(s_j)$$

This gives a vector built from existing coordinates. It doesn't reproduce the contextual representation of the original multi-token string, preserve the old softmax distribution, or bound every training gradient. Vocabulary-expansion studies compare several initialization strategies rather than establishing one universal fix.^{[6]Reference 6An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Modelshttps://arxiv.org/abs/2407.05841}

An August 2026 study of Hindi adaptation in Nemotron-3-Nano-30B-A3B compares subword averaging, norm calibration, and different input/output initialization schemes. Its best observed configuration is specific to that model and language; low initial loss alone isn't a sufficient selection metric.^{[7]Reference 7Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extensionhttps://arxiv.org/abs/2608.03494}

That cell initializes an input lookup table, not a complete resized language model. Save a compatible tokenizer and config, resize the output head, preserve existing IDs, and handle tied weights explicitly. Untied input and output matrices need separate treatment.

Even a mean-initialized output row changes probabilities. Predict what happens when two old logits are both zero and the new row also produces zero. The old rows remain identical, yet the denominator gains a third term:

Here the old target's NLL rises by $\ln(3/2)\approx0.405$. Initialization can reduce disruption without eliminating it. Check initial loss, training behavior, and the final domain and general tasks.

Validation can also lie before training starts. Near-duplicates, revisions of the same OpenAPI page, or two runbook copies from one source can land in both training and validation and make CPT look stronger than the split warrants. Assign a provenance or deduplication group before tokenization, then keep each group in one split.

This split keeps the supplied groups apart; it doesn't discover duplicate groups for you. Hash bucketing targets a validation fraction over many groups, not exactly 25% of a small corpus's tokens. Check split sizes and topic coverage before accepting it.

Loss and fertility checks identify candidates for adaptation. They don't tell you how hard to push the weights once training begins.

The two failure dynamics: forgetting and underfitting

Predict the two curves before touching the optimizer. Successful text adaptation lowers held-out domain loss, though individual evaluations can fluctuate. If general-text loss rises, general text fit has regressed. Measure downstream abilities separately before calling that a loss of reasoning or instruction-following capability.

Catastrophic forgetting is loss of previously learned ability as parameters shift to absorb new data. Push too hard on AUTH_TIMEOUT runbooks and broad validation quality can regress.

Underfitting means the model still poorly fits the domain. Too few effective updates are one possible cause; limited model capacity or a mismatched recipe can also matter. A run can underfit the domain and regress elsewhere at the same time.

The main controls are the learning-rate schedule and data mixture. Run length and corpus quality matter too, so a bad corpus can't be repaired by a clever schedule.

Learning rate re-warming and re-decaying

Inspect the checkpoint's actual training recipe. Not every run ends on a cosine floor, and a released weight file often doesn't include optimizer or scheduler state. Continuing from saved training state and starting a new adaptation optimizer are different experiments.

A low learning rate may adapt slowly; a high one may worsen general performance. Neither “trapped in a local minimum” nor “shattered attention circuits” follows from a loss trace alone. The SGD identity $\Delta\theta=-\eta\nabla\mathcal L$ also doesn't describe AdamW's moment-normalized update.

The cited Ibrahim study does not prescribe a CPT peak at 0.1x the original peak. Its 405M-model sweep tests 0.5x, 1x, and 2x, with re-warming and re-decaying. It tests warmup durations of 0%, 0.5%, 1%, and 2%; early behavior differs, but later losses are similar in those settings.^{[8]Reference 8Simple and Scalable Strategies to Continually Pre-train Large Language Modelshttps://openreview.net/forum?id=DimPeeCxKO} These findings motivate a sweep, not a universal schedule for a different checkpoint.

Warmup can help manage a new optimizer and distribution change. Zero-initialized moments don't by themselves imply an exploding denominator: at AdamW's first step, bias correction gives $\hat m=g$ and $\hat v=g^2$. Ignoring weight decay, each coordinate moves by $-\eta g/(|g|+\epsilon)$. Later behavior still depends on gradient history, precision, and the recipe.

The cell below is an authored schedule, not the paper's recommendation: 1,000 zero-indexed steps, 50 warmup steps (5%), peak $3\times10^{-5}$, and floor $3\times10^{-6}$. Steps 49 and 50 share the peak; step 999 reaches the floor. A cosine schedule reaches chosen endpoints, not guaranteed convergence.

Replay: keep prior-data signal in the mix

The second critical control is replay: mixing a fraction of general-purpose pretraining data directly into the incoming domain corpus.

Domain and general losses depend on shared trainable parameters. Improving one can worsen the other when their gradients conflict. A domain-only run can regress, but it needn't lose every broad capability. Test the abilities that matter instead of inferring particular damaged circuits from text loss.

To preserve general capabilities, you interleave a replay buffer $\mathcal{D}_{\mathrm{replay}}$ into the domain stream. The effective training objective becomes a weighted expectation across data distributions:

$$\mathcal{L}_{\mathrm{mix}}(\theta) = (1 - \alpha)\,\mathbb{E}_{x \sim \mathcal{D}_{\mathrm{domain}}}[\mathcal{L}_{\mathrm{CLM}}(x; \theta)] + \alpha\,\mathbb{E}_{y \sim \mathcal{D}_{\mathrm{replay}}}[\mathcal{L}_{\mathrm{CLM}}(y; \theta)]$$

Here $\alpha \in [0,1]$ is the fraction of scored target tokens assigned to replay, with each expectation representing a token-weighted loss for its source. A batch or document fraction matches this objective only if the scored-token counts match, or you weight the losses accordingly. Random sampling needn't put both sources in every batch.

To see the competing gradients directly, shrink the predictor to one shared logit $z$. It assigns probability $\sigma(z)$ to token 1 and the remaining probability to token 0. Domain examples always target 1; replay examples always target 0. This deliberately conflicting model has no context features that could separate the tasks. At $z=0$, the gradients are $-0.5$ for domain and $+0.5$ for replay. A 25% replay mix gives $0.75(-0.5)+0.25(0.5)=-0.25$.

The CPU-only PyTorch cell takes one gradient step from the same starting weight for each mixture. It demonstrates gradient weighting, not a measured LLM forgetting curve.

Both losses start at $\log2\approx0.693$. Domain-only training improves domain loss while worsening replay loss. At 50% the gradients cancel exactly. Context features can let a richer model separate some conflicts, but a large parameter count doesn't guarantee that it will. This toy doesn't identify a production replay ratio.

Ibrahim's study selects 5% replay for its weak English-to-English shift and 25% for its stronger English-to-German shift after comparing several fractions. Its 50% runs aren't simply “wasted compute.”^{[8]Reference 8Simple and Scalable Strategies to Continually Pre-train Large Language Modelshttps://openreview.net/forum?id=DimPeeCxKO} The results contradict a universal 10%–30% safe interval. Your replay source, checkpoint, shift, budget, and required abilities determine which candidates are useful.

For the incident assistant, compare several declared ratios, including a domain-only control. Hold processed tokens fixed, log scored-token fractions, and measure both downstream benefit and regression. A 25% replay window below is an accounting example, not a guarantee of preserved reasoning.

Under a fixed token budget, replay replaces some domain tokens. At 25% replay, one quarter of a fixed window becomes general text before checking the accounting below. Equal processed tokens are a compute proxy only when model, block lengths, masks, and execution are comparable; they don't prove equal measured FLOPs or runtime.

When continued pretraining is the right tool

Reach for continued pretraining when the domain has language the base model under-serves. Common examples include internal API docs and error catalogs (AUTH_TIMEOUT, AUTH_REJECTED), on-call runbooks and incident notes, SDK guides with domain-specific method names, long compliance or protocol documents, and codebases whose APIs and identifiers barely appeared in public pretraining.

The trigger isn't simply "the product team wants custom behavior." The question is whether extra domain-text exposure improves the final task enough to justify its cost compared with direct post-training or retrieval.

Good signals

Bad signals

A practical split for the AUTH_TIMEOUT assistant is to test raw domain-text continuation and prompt-response behavior separately. Weak raw continuation motivates testing CPT; competent continuation with poor assistant behavior motivates testing SFT. Neither observation rules out retrieval or establishes a unique cause.

The 2020 "Don't Stop Pretraining" paper names two useful scopes in masked-language-model experiments with RoBERTa:^{[9]Reference 9Don't Stop Pretraining: Adapt Language Models to Domains and Tasks.https://aclanthology.org/2020.acl-main.740/} DAPT (domain-adaptive pretraining) uses large unlabeled domain text such as API docs or runbooks. TAPT (task-adaptive pretraining) continues on the task's own unlabeled inputs, even when that corpus is smaller.

The distinction remains useful for decoder-only LLM projects, but don't silently transfer RoBERTa's quantitative gains to a causal base model. Measure whether more exposure to the target text distribution improves your model and downstream task.

Data for continued pretraining

The same discipline from large-scale pretraining still applies: filter low-quality text, deduplicate aggressively, remove benchmarks and eval leakage, scrub PII and secrets from runbooks and traces, and keep provenance and usage rights for every corpus slice.

The corpus can be narrower and more targeted. Domain data can also be more sensitive than public pretraining text, so provenance, access control, and removal procedures are product requirements, not cleanup tasks.

Gate the corpus before tokenization

Keep a manifest that records whether a source may be trained on, whether it contains unresolved sensitive content, and whether it's reserved for evaluation. A high-quality domain document that fails one of these gates doesn't belong in the training stream.

Screen for evaluation overlap

The next cell screens for normalized text matches. It lowercases and collapses whitespace for comparison, while retaining the original training strings. That policy can falsely merge case-sensitive identifiers or meaningful spacing; choose normalization for the source type. It also misses partial copies, paraphrases, and related facts. Add provenance grouping and appropriate near-duplicate checks rather than treating the hash as a complete contamination guarantee.

Mixing strategy

Treat a domain-only run as a candidate, not an automatic choice. Replay is one guardrail against forgetting: define candidate ratios, hold total training tokens fixed, and select with domain-gain and broad-regression metrics. If general language degrades beyond your declared tolerance while specialization improves, the run exceeds that product's budget.

BloombergGPT is a useful contrast, not replay evidence. Its assembled corpus was 51.27% financial and 48.73% public tokens; these describe the roughly 709B-token inventory, not an exact count of the training tokens consumed. The model was trained from scratch on 569B tokens and reported strong financial results while remaining competitive on the paper's general benchmarks.^{[10]Reference 10BloombergGPT: A Large Language Model for Financehttps://arxiv.org/abs/2303.17564} This doesn't identify the right CPT replay ratio for your checkpoint.

Pack blocks and preserve the mixture

CPT uses the same causal objective as base pretraining. Packing is where your data decision becomes tensors: join document token sequences with end-of-document markers, then emit full blocks. A separator marks a boundary, but it doesn't prevent cross-document attention by itself. As the data-pipeline chapter explained, choose explicitly between an ordinary causal mask and a document-isolated block-diagonal mask. Predict where the first separator lands and whether the final short document fits before inspecting the small integer sequence.

The final [41, EOS] is returned as a remainder, not a training block. Carry it into the next packing call if you want to preserve it. Silently dropping every shard's tail can disproportionately remove short or rare sources. In a six-token block without a separate beginning context, the usual one-position shift scores five targets: block[:-1] predicts block[1:]. Padding and document-boundary masking can reduce that count further.

Once domain and replay streams are packed, make mixture selection explicit and auditable. These names stand for equal-length blocks with equal scored-token counts. Under that assumption, 5 replay blocks out of 20 also means 25% replay targets. The function selects one window, not a stateful epoch loader: a real loader must advance its cursors rather than repeatedly take the first blocks.

The window rounds the desired block count to the nearest integer using Python's ties-to-even rule. For example, 25% of ten blocks is 2.5, so this function selects two replay blocks, or 20%. Log the realized token fraction. For sustained training, a fractional accumulator or probabilistic sampler can approach the target across many windows without always rounding in the same direction.

Evaluation: domain gain without lying to yourself

Evaluate domain benefit and required general abilities at the same checkpoints. Lower held-out text loss doesn't identify memorization or establish better reasoning, coding, or instruction following. A domain win without a general-capability budget leaves the trade unmeasured.

Lane 1: Domain mastery

On the domain lane, track three complementary signals:

• Held-out domain perplexity measures fit on cleaned domain text under a fixed scoring protocol.
• Entity-span NLL sums losses over aligned subword spans for critical identifiers. State the span and token weighting; it isn't automatically an entity-level correctness score.
• Downstream probes test the actual task. If the product needs SFT, compare base-plus-SFT with CPT-plus-the-same-SFT. A fixed small training probe is one experiment; raw completion pass@1 is a separate evaluation. Choose budgets and held-out cases deliberately. Lower aggregate domain perplexity isn't a logical prerequisite for better task accuracy.

Lane 2: General capability retention and safety checks

On the general lane, define required abilities and their regression budgets:

• Held-Out General Perplexity: Evaluated on an independent, frozen slice of diverse web text (such as SlimPajama or Wikipedia).
• Task and Reasoning Checks: Historical anchors include MMLU, HumanEval or MBPP, and GSM8K. State prompting and scoring, inspect contamination and ceiling effects, and supplement them with private held-out tasks your application must handle.
• Safety and Alignment Checks: Verify that refusal behaviors and safety guardrails haven't drifted during unsupervised domain exposure.

Freeze evaluation text, tokenizer, available context, boundary protocol, and target mask across checkpoints. Tokenization changes affect perplexity.^{[3]Reference 3Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity} For a vocabulary comparison, also report encoding-based bits per byte (BPB) on the same scored text: total NLL in nats divided by the scored UTF-8 byte count and $\ln2$. Nats per byte omits the $\ln2$ division. Bytes, characters, and tokens are different units; control context and boundaries as well.

Here are two invented reports over the same 30 scored bytes. Predict which wins in token PPL and which wins in BPB:

The finer tokenization has lower loss per token but higher total NLL for these same scored bytes. Token PPL rewards the changed unit; BPB exposes that difference. These are arithmetic fixtures, not measurements of a real vocabulary extension.

For a fixed tokenizer, aggregate corpus perplexity by summing NLL and scored-token counts before exponentiating. Documents with NLL 60 over 20 targets and 160 over 40 give $\exp(220/60)\approx39.1$. Averaging their PPLs gives about 37.3, a different statistic: it changes weighting and averages after a nonlinear transform.

The diagnostic below runs that exact arithmetic on two evaluation lanes:

Gate, then rank the surviving checkpoints

Declare required abilities and regression budgets before selecting a checkpoint. General-text PPL is a text-fit metric, not a complete safety certificate. Accuracy changes need an explicit unit: a move from 70% to 68% is two percentage points, not a 2% relative drop.

The authored ledger below rejects a general-PPL rise above 1.5, then ranks survivors by downstream probe accuracy and uses domain PPL to break exact ties. It doesn't enforce reasoning or safety benchmarks, calculate uncertainty, or implement an early-stopping rule.

This is constrained, lexicographic selection. A Pareto frontier contains candidates that aren't dominated on the chosen metrics; it doesn't uniquely pick the largest probe score. Here base, 1k, and 4k all trade better domain fit against worse general fit. The declared ranking chooses 4k among eligible candidates.

Stopping rules

Because continued pretraining keeps the causal language modeling objective, runs can feel deceptively safe. They aren't safe by default. You stop when downstream probe accuracy plateaus, when domain validation loss flattens, or when general regressions approach your safety threshold.

Unused documents and a tiny loss decrease aren't sufficient reasons to keep spending compute. Set a patience window, minimum worthwhile gain, and regression limits, then consider metric uncertainty and evaluation cost. Two flat checkpoints or a 0.2-percentage-point change aren't universal stopping rules. For 500 binary task cases, one changed outcome already moves accuracy by 0.2 percentage points. Reusing the same cases for many selections can overfit the evaluation; confirm the chosen recipe on a reserved test set.

At a selected checkpoint, save weights, tokenizer, config, corpus manifest, and evaluation protocol. For exact continuation also save optimizer/scheduler state, RNG states, and loader cursors. Test the intended final assistant after its actual post-training recipe before promotion.

Where it fits relative to LoRA and SFT

Compare the choices by asking what you want to change.

LoRA and QLoRA are parameter-efficient implementation choices; QLoRA also stores the frozen base model in quantized form.^{[11]Reference 11QLoRA: Efficient Finetuning of Quantized LLMshttps://arxiv.org/abs/2305.14314} They don't determine what supervision teaches. An adapter can be trained with a next-token domain-text objective or with prompt-response SFT. First choose objective from the failure mode, then choose full-weight or parameter-efficient training from budget and deployment constraints.

A training stack can run from base model to continued pretraining on a domain corpus, then SFT on curated prompt-response data, with preference optimization if needed. Not every product needs every stage. Compare stages against the failure you observe. Code Llama's Instruct variants are one shipped example: code pretraining followed by instruction tuning.^{[5]Reference 5Code Llama: Open Foundation Models for Codehttps://arxiv.org/abs/2308.12950} Llama 3's code expert combined continued pretraining with SFT and DPO, then helped produce annotations and synthetic data for the main model.^{[12]Reference 12The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

Common pitfalls

Using continued pretraining to fix assistant tone

Poor formatting motivates an SFT comparison. Also inspect the chat template, decoding, and supplied instructions; a malformed interface can make capable weights appear unhelpful.

Over-specializing on one corpus

If domain completions improve while the model becomes narrow or brittle elsewhere, suspect a missing replay mixture or too many adaptation steps. Keep a general-text regression lane and stop earlier.

Skipping the downstream check

If domain perplexity improves but the final task model barely benefits, the adaptation run optimized text fit that didn't transfer to the product task. Probe the adapted checkpoint with a small downstream SFT instead of judging only by perplexity.