Supervised Fine-Tuning Pipeline

In this lab's fictional access-policy dataset, incident thread T102 asks whether an 18-day-old service-account key is still usable. The authored policy requires keys older than 14 days to undergo security review, beginning with a rotation ticket. A candidate assistant replies, "The key is old, but rotation can wait until next month." The answer skips the required escalation. That plausible failure is the regression this pipeline exists to catch; filing the ticket doesn't itself approve access.

The chat-template foundation showed how structured conversations turn into token sequences. The synthetic-data foundation showed how candidate rows get filtered and verified. Supervised fine-tuning (SFT) uses accepted demonstrations to increase the likelihood of desired responses through gradient updates.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer[2]Reference 2Fine-Tune Your First LLM.https://meta-pytorch.org/torchtune/stable/tutorials/first_finetune_tutorial.html} InstructGPT used supervised demonstrations before reward-model training and reinforcement learning.^{[3]Reference 3Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} SFT doesn't guarantee an exact answer or procedural compliance on a new case.

Training loss can drop smoothly while policy behavior stays broken. Prompt-dominated metrics, another turn from T102 leaking into validation, unintended cross-example attention, or selection by training loss can hide the failure. Each raises a different diagnostic question. Check the split, labels, and evaluation before funding an optimizer sweep; none of these symptoms alone proves the cause.

Count supervised answer tokens before launching any job. A packed row can fill every token slot in the context window while providing zero learning signal if aggressive truncation cuts off the completion and masks every remaining token to -100. Reject that row outright instead of pretending its context tokens represent training data.

Turn behavior into a run contract

Before picking a trainer, map out which boundary might let the T102 regression slip through: the learning objective, the data split, the label mask, the token budget, the checkpoint format, or the release gate. The examples below verify those contracts on CPU. They don't train or benchmark a large language model; the TRL configuration is an integration blueprint, not an unverified GPU script.

Define the behavioral contract first: cases with keys older than 14 days require a rotation-ticket escalation under this lab's policy. Compare prompting with the policy in context, retrieval, and SFT before assuming new weights are needed. If unfamiliar domain language remains a barrier, investigate continued pretraining too. The observed failure suggests candidate interventions; it doesn't establish a required training stage.

Next, trace what each update actually sees. Which examples and tokens contribute to the loss function? What example and supervised-token budget reaches a single optimizer step? What held-out metric dictates checkpoint selection? Finally, can the validated recipe run on a single GPU, or do Fully Sharded Data Parallel (FSDP) or Zero Redundancy Optimizer (ZeRO) become necessary?

Following this order keeps the trainer where it belongs: as an execution engine applying an update. The split, mask, batch budget, resume bundle, and export gate define what that update means.

Separate objective from parameterization

Suppose T102 outputs fluent prose but persistently advises waiting until next month. Is the missing ingredient raw domain text, a demonstration of procedural compliance, or a ranking across several plausible replies? Answering that question separates the learning objective from the parameterization that absorbs the gradient updates.

An objective determines the mathematical signal guiding the model. A parameterization specifies which weights in the model architecture are allowed to change. They aren't interchangeable.

Once the objective is SFT, make a second decision: which part of the model can carry the new behavior?

LoRA and QLoRA aren't alternatives to SFT. They're parameterizations usable for an SFT run. Continued pretraining can also use adapters. Both stages can use causal next-token cross-entropy; their training distributions and supervised targets differ. Calling every adapter run SFT would hide that distinction.

Full weights or adapters

Full fine-tuning

For our response-only objective, full fine-tuning updates all eligible model parameters. Test it when weights, gradients, optimizer state, activations, and temporary buffers fit, or when an adapter baseline appears capacity-limited. More trainable parameters permit a broader update, but don't establish better held-out behavior. Full fine-tuning doesn't require response-only labels by definition; that is a separate objective choice.

LoRA

LoRA freezes selected base weights and trains low-rank adapter matrices. Target modules and rank determine the trainable count. The original paper's GPT-3 175B comparison reported up to 10,000 times fewer trainable parameters and three times lower GPU memory requirements, not universal 99% and 70% savings.^{[6]Reference 6LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} Activations and frozen weights still consume memory. Measure the actual run; adapters are useful candidates when many variants share one base or full updates are too costly.

QLoRA

QLoRA reduces frozen-base storage while retaining adapter updates. Its paper uses 4-bit NormalFloat (NF4), quantized scaling constants, and paged optimizers. The base weights are dequantized to a computation dtype such as BF16 for forward and backward operations; adapter gradients don't mean the stored 4-bit base receives weight updates.^{[7]Reference 7QLoRA: Efficient Finetuning of Quantized LLMshttps://arxiv.org/abs/2305.14314} Check your implementation's storage, compute, and adapter dtypes separately. Compare held-out behavior under measured memory and time budgets.

Reject broken demonstrations before training

For this pipeline, store a prompt, a desired completion, and a group key for leakage-resistant splitting on every SFT row. A missing completion isn't harmless metadata: it's a training example with no answer to teach.

Before you run the validator, predict its decision for each row. T102 has an answer and a group key. B550 has an empty answer. The last row has an answer but no group key. Only one row should reach tokenization.

The output makes the contract concrete: T102 is accepted, B550 is rejected before it can become a prompt-only example, and the last row is rejected before it can contaminate a grouped split. Next, keep accepted rows from the same deployment unit together.

Split by the unit that could leak

Before formatting rows, reserve evaluation examples that training can't imitate through near duplicates. For a policy assistant, several messages from one access incident or one policy document often share facts and phrasing. Randomly splitting individual messages can put one part of the same case in training and another in evaluation.

Group by the unit your evaluation claim requires to be unseen: incident thread, policy document, tenant, or time period. Low token overlap doesn't make two turns from T102 independent. If several group identifiers connect cases, keep their connected groups together. For a chronological holdout, train on earlier periods and check windows that straddle the cutoff. The check below demonstrates thread separation only.

Choose the grouping from the deployment claim. Thread grouping tests new threads, not unseen tenants or policy documents shared by those threads. Deduplicate across groups, and reserve separate validation and final-test groups. Use validation for sweeps and checkpoint selection; inspect the final test only after freezing that choice. Repeatedly tuning on the test turns it into another validation set.

Predict the three printed sets before reading the output. T102 should stay with P771 in training, R550 should be held out, and the intersection should be empty.

Data path: template, tokenize, label, pack

Follow one row all the way through the data path. Structured conversation dictionaries must pass through a Jinja2 chat template, undergo tokenization, receive target label masks, and either get packed into fixed-length blocks or padded into batches.

Use the checkpoint's actual formatting contract. ChatML-style markers such as <|im_start|> aren't universal; templates can mix special IDs with ordinary text, spaces, or newlines. Subword splitting doesn't by itself prove a broken role boundary. Inspect the rendered template, its token IDs, and assistant masks together rather than registering invented markers and assuming the pretrained model understands them.^{[8]Reference 8Transformers Documentation: Writing a chat template.https://huggingface.co/docs/transformers/main/en/chat_templating_writing}

For complete training conversations, use add_generation_prompt=False. To start a new assistant reply, add_generation_prompt=True adds a header when the template supports it; continuing a partial assistant message is a different operation. Align the template's turn terminator with serving stop settings. Missing terminators can weaken stopping supervision, but don't prove inevitable endless generation. If you format text first and tokenize later, avoid adding duplicate special tokens.^{[8]Reference 8Transformers Documentation: Writing a chat template.https://huggingface.co/docs/transformers/main/en/chat_templating_writing}

Version this entire data path together. Delimiter, whitespace, or template changes can alter token IDs or labels. Record the change and rerun batch checks; previous runs remain evidence about their recorded formatting.

Predict which positions in T102 should teach the model before inspecting the labels below. The system instruction, user request, and delimiter tokens must remain visible so the response can condition on them. However, only the target answer and its turn terminator should contribute to cross-entropy loss. We set ignored positions to -100, the standard ignore_index in PyTorch cross-entropy loss.

These seven entries are readable spans rather than real subword token IDs. A real sentence expands into multiple integer IDs. The five prefix spans provide context without generating loss; the answer and terminator spans identify what to label after tokenization.

Causal language models shift targets by one position: logits at position $t-1$ predict the label at position $t$. The hidden state at the assistant marker <assistant> predicts the first answer token. Setting a label to -100 removes that token's direct cross-entropy loss, but it doesn't remove gradients flowing through attention layers into earlier prompt representations. Response loss still updates the weights that process the prompt. Mask padding tokens by position, not by blindly masking every EOS ID when EOS doubles as padding.^{[9]Reference 9CrossEntropyLosshttps://docs.pytorch.org/docs/main/generated/torch.nn.CrossEntropyLoss.html}

Don't confuse the label mask with the attention mask. The label mask decides which positions contribute to the loss scalar. The attention mask decides which earlier positions a token can read during the forward pass. Prompt tokens must remain visible as context even when their labels are -100; packed sequences also need attention boundaries so adjacent examples can't attend across boundaries.

The CPU check below uses real integer labels and PyTorch gradients. With all logits set to zero over a four-token vocabulary, each supervised target has cross-entropy loss $\log 4 \approx 1.3863$. Only logit rows 1 and 2 predict the labeled answer and stop positions. Most training frameworks handle this target shift internally; don't shift labels twice.

This verifies shifted loss and direct logit gradients. Test those mechanisms on collated tensors before launching long GPU jobs.

Why mask prompt targets here? We want a response-only objective and a metric that measures the response. With 800 scored prompt positions and 30 response positions, prompt targets occupy about 96% of a full-sequence mean's terms. That isn't 96% of the gradient: individual losses and their parameter derivatives can differ greatly. Nor does full-sequence training inevitably cause echoing or degraded style.

Huerta-Enochian and Ko studied prompt-loss weights using LLaMA 1/2 7B and Alpaca-derived datasets. Fractional weights improved some short-completion benchmarks in that setting; the study attributed the effect to regularization rather than increased training stability.^{[10]Reference 10Instruction Fine-Tuning: Does Prompt Loss Matter?https://arxiv.org/abs/2401.13586v4} Response-only masking is this run's deliberate choice, not a universal definition of good SFT. Compare objectives on the same held-out tasks before changing it.

Predict which number exposes the metric difference below. The supplied prefix losses are small, so adding them lowers the full-sequence average without improving the supplied answer losses:

Both numbers measure token-level cross-entropy over different target sets. In this authored fixture, five low-loss prefix positions lower the combined average to $0.596$ while answer loss remains $2.000$. For our response-only objective, the loss is mean negative log-likelihood over the labeled target set $T$:

$\mathcal{L}_{\text{SFT}}(\theta) = -\frac{1}{|T|}\sum_{t \in T} \log p_\theta(y_t \mid y_{<t})$

$\theta$ represents the trainable parameters, $y_t$ the target token, and $y_{<t}$ its available prefix. Here $T$ has two positions. Default unweighted, mean-reduced PyTorch cross-entropy with ignore_index=-100 averages only active targets. Changing the mask changes the objective; compare separately reported response loss rather than treating lower full-sequence loss as evidence of better answers.

You rarely write this mask by hand. In Hugging Face TRL's SFTTrainer, prompt-completion datasets use completion-only loss by default. Conversational datasets can set assistant_only_loss=True, provided their chat template exposes assistant spans through generation markers.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer} Always inspect a tokenized batch before launching training: confirm that only answer and stop tokens carry non--100 labels.

Inspect masks after truncation and causal shifting. Keeping only the start can remove the completion. A mean-reduced PyTorch cross-entropy batch with no active shifted targets returns NaN; an individual empty row in a mixed batch instead adds context and cost without response supervision. Count dropped or empty rows so dataset loss isn't hidden.

As checked September 22, 2026, TRL's pinned preparation path already filters fully masked rows after nonpacked truncation. Its packing path needs a separate audit.^{[11]Reference 11TRL: SFT Trainer Implementation (September 22, 2026 source checkpoint)https://github.com/huggingface/trl/blob/5bb945c38f319259eddaad7bb3838e4aaf590c49/trl/trainer/sft_trainer.py} Custom collators or skipped preparation can change this behavior. Check active labels in the actual collated tensors rather than assuming every empty row reaches the loss.

When sequence packing is active, this truncation step is skipped and overflow is handled by the packing strategy instead; bfd discards overflow tokens.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer} Predict what the guard below does when five prompt tokens consume the length budget:

Pack short examples while preserving their boundaries

Short demonstrations can leave many unused token slots when padded to a large fixed length. The fraction of padded slots isn't a measured compute-waste percentage: dynamic batch padding, length grouping, and attention kernels change the work performed. Packing is another efficiency option. For this run's independent-example objective, preserve each example's attention and loss boundaries.

For independent demonstrations, a reply for case R550 shouldn't condition on T102 merely because the loader put them in one block. Reset position IDs alone don't block attention. Enforce a causal block-diagonal pattern using an explicit mask or a kernel's segment metadata. The predicate below describes the intended edges; it doesn't execute an attention kernel:

Reset position IDs provide positional coordinates, not an attention barrier. The underlying attention kernel must enforce those boundaries during computation. Resetting positions without segment masking causes cross-example contamination:

Packing can also fail when an example gets truncated before being packed. If T102 survives truncation with every response token stripped away, packing still incorporates the prefix to fill the block, but that segment contributes zero learning signal. Inspect segment boundaries and verify non-zero answer labels before starting a training run.

TRL's bfd packs examples after truncating overlong ones; bfd_split splits overlong sequences into chunks; wrapped concatenates a stream and cuts token sequences.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer} Preserving stored tokens through splitting doesn't preserve every completion's original prompt context or every trainable target. Choose the strategy from the learning objective, then inspect the resulting segments.

The pinned TRL trainer enables padding-free mode for both bfd and bfd_split, and warns when its attention implementation isn't among the supported FlashAttention variants.^{[11]Reference 11TRL: SFT Trainer Implementation (September 22, 2026 source checkpoint)https://github.com/huggingface/trl/blob/5bb945c38f319259eddaad7bb3838e4aaf590c49/trl/trainer/sft_trainer.py} An ordinary global causal kernel can read across examples. Disable packing while debugging if necessary; verify boundary reachability before enabling it. A warning or successful forward pass isn't a boundary test.

Attention isolation alone also isn't enough. A globally shifted loss must not ask the last token of one independent segment to predict the first token of the next. The pinned padding-free collator masks labels wherever segment position IDs reset to zero.^{[11]Reference 11TRL: SFT Trainer Implementation (September 22, 2026 source checkpoint)https://github.com/huggingface/trl/blob/5bb945c38f319259eddaad7bb3838e4aaf590c49/trl/trainer/sft_trainer.py} Audit that loss boundary as well as the attention boundary.

Packing changes loader-item counts and can remove or split targets. A fixed max_steps can therefore imply a different number of data passes, without proving overfitting. Log processed input tokens, active shifted targets, and retained original examples. Throughput and learning exposure use different denominators.

Hyperparameters are experiments, not promises

A learning-rate sweep asks how much change this objective and parameterization can tolerate. Check task improvement and regressions on general capabilities together. Neither a frozen base nor a small learning rate guarantees preserved behavior.

The fragment below uses 1e-4 as an adapter candidate. That isn't a universal safe range or a fixed multiplier over full fine-tuning. Compare nearby rates under the same data exposure and evaluation rules. Dataset quality, optimizer, adapter scaling, model size, and loss normalization across microbatches all affect the result.

Warmup reduces early step sizes; it doesn't make AdamW's bias correction unnecessary or guarantee stability. A useful historical counterexample is InstructGPT's standalone SFT baseline: 16 epochs, no warmup, and cosine decay to 10% of its initial LR, with learning rates and epochs selected by searches.^{[3]Reference 3Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} That recipe isn't a recommendation to copy 16 epochs. It disproves a mandatory 3%-10% warmup or 1-3 epoch rule.

A cosine schedule's floor is configurable; standard Transformers cosine warmup decays to zero, while a minimum-LR variant can use a nonzero floor.^{[12]Reference 12Transformers: Optimization and Learning-Rate Scheduleshttps://huggingface.co/docs/transformers/main_classes/optimizer_schedules} Resolve the actual scheduler instead of assuming 10%. AdamW applies decoupled decay separately from moment estimation. Decay doesn't bound distance from the pretrained weights, and clipping raw gradients doesn't directly bound AdamW's parameter update.^{[13]Reference 13PyTorch 2.14: AdamWhttps://docs.pytorch.org/docs/2.14/generated/torch.optim.AdamW.html} Inspect both gradients and update norms.

Effective batch size is also a token budget

The batch size configured on a single device isn't the batch size the optimizer sees during an update step. For synchronous training across complete accumulation windows, the number of items per update follows:

Without packing, a loader item is one demonstration. With packing, each item is a composite block containing multiple short demonstrations, so the formula counts packed blocks rather than raw examples. Track both raw examples and actual supervised tokens per update. The CPU check below models an unpacked dataset across a four-GPU setup:

Always calculate the combined impact of batch changes rather than reasoning about single knobs. Notice how doubling world size or accumulation doubles the update volume:

When sharing batch configurations, specify accumulation steps, world size, sequence packing mode, and supervised token counts.

Loss weighting across accumulation steps matters as well. Suppose one microbatch has one target loss of 4.0 and another has nine target losses of 1.0 each. Averaging batch means gives 2.5; the token mean is $(4.0+9.0)/10=1.3$. Equal example weights and equal token weights are different objectives. For the token mean, normalize loss sums by active shifted targets across the complete update window.

In ordinary distributed data parallel training, rank gradients are averaged. If each rank divides its local loss sum by a globally summed token count, that averaging introduces another factor of world_size. Compensate for it, or use a trainer that performs the normalization correctly. Don't add the factor twice when the trainer already handles it. Unequal target counts, partial accumulation windows, and actual loss gradients need checks; the scalar-loss fixture above doesn't execute distributed training.

What a resumable run must save

A model.safetensors file doesn't encode the optimizer history or next data batch. It may be part of an inference artifact, but serving also needs compatible architecture and tokenization. An adapter export additionally needs its base model and adapter configuration.

To reconstruct the continuation, retain the state your recipe actually uses:

• model weights or LoRA adapter weights
• optimizer state (AdamW first moment $m_t$, second moment $v_t$, and step counters)
• scheduler state (step counter, warmup phase, decay multiplier)
• global step and completed epoch indices
• sampler progress and random generator states, not only original seeds
• tokenizer revision and chat-template hash
• resolved training configuration and dataset manifests
• best validation metric achieved so far
• loss-scaler state if using FP16 dynamic scaling, plus any other stateful training machinery

Optimizer history determines direction and scaling. Scheduler progress determines the next LR, which needn't be the peak if reset incorrectly. Sampler and generator states determine upcoming data and randomness. Preserve the exact tokenizer/template artifacts and the adapter's base revision. Metadata labels or filenames alone don't guarantee identical contents or complete worker state.

For ordinary AdamW without AMSGrad, the update uses newly computed, bias-corrected moments:

$m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\qquad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2$

$\widehat m_t=\frac{m_t}{1-\beta_1^t},\qquad \widehat v_t=\frac{v_t}{1-\beta_2^t}$

$\theta_t=(1-\eta_t\lambda)\theta_{t-1}-\eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon}$

Here $g_t$ is the current gradient and squaring is elementwise. A fresh optimizer with nonzero $g_1$ first computes $\widehat m_1=g_1$ and $\widehat v_1=g_1^2$. Its update doesn't divide the current gradient by epsilon alone. Dropping state changes the trajectory, but doesn't imply a mandatory spike.^{[13]Reference 13PyTorch 2.14: AdamWhttps://docs.pytorch.org/docs/2.14/generated/torch.optim.AdamW.html}

Predict the direction after gradients 2.0, then -1.0. A fresh AdamW step follows the negative gradient upward, while retained positive momentum can still move the weight downward:

This controlled CPU experiment restores optimizer history while supplying the same next gradient; it doesn't replay a stochastic LLM job. Prefer saving at completed optimizer-step boundaries. Mid-accumulation continuation also needs partial gradients and microbatch progress. Compare the next batch, LR, loss, and update against an uninterrupted reference. Bitwise parity depends on deterministic operations and a compatible environment.^{[14]Reference 14Saving and Loading Modelshttps://docs.pytorch.org/tutorials/beginner/saving_loading_models.html[15]Reference 15Universal Checkpointing with DeepSpeed: A Practical Guide.https://www.deepspeed.ai/tutorials/universal-checkpointing/}

A manifest can list required continuation artifacts. This schema fixture checks key presence only; it doesn't open files, validate hashes, or demonstrate a successful resume:

The scalar SGD-with-momentum fixture below makes the dependency on velocity explicit. Its fixed next gradient isolates optimizer state; no training data or model loss is evaluated:

Choosing the best checkpoint

Choose checkpoints against the deployment objective on validation data. Teacher-forced loss measures likelihood of reference tokens given reference prefixes; it doesn't directly measure whether a freely generated reply opens the required ticket. Generate outputs for task evaluation too, and keep a final test separate from repeated selection.

Unreliable selection criteria:

• lowest training loss alone (doesn't measure unseen-case performance)
• latest training step (risks overfitting or late degradation)
• file size or raw token count

Rigorous selection criteria:

• strict syntax and format pass rate (JSON schema compliance, valid function call formatting)
• domain policy pass rate on held-out cases via deterministic rule verifiers or calibrated judges
• human evaluation on blinded, randomized completion pairs
• validation loss with a declared role in selection, rather than an assumed universal priority

For the authored selection fixture below, require format validity of at least 98%, then maximize policy compliance and break ties with validation loss, followed by the earlier step. These rates and this threshold are supplied data and policy choices, not trained-model results or a universal release standard. In a real evaluation, report denominators, uncertainty, important failure slices, and regressions on general behavior.

Step 400 shows the lowest validation loss ($1.77$), but step 600 achieves the highest policy compliance rate ($0.97$) while clearing the format threshold ($0.98$). The selection gate picks step 600. If no checkpoint meets the format threshold, the gate yields None, signaling that the recipe needs fixing before anything ships.

Single GPU first, then scale out

Don't start debugging thread T102 across an eight-GPU cluster. If prompt templating or label masking contains subtle bugs, scaling out distributes those defects across collective communication calls rather than resolving them. Prove your pipeline on a single device first:

1. single GPU execution
2. small, representative evaluation split
3. frequent checkpointing
4. verified chat templating and response-only label masking
5. clear task compliance metric

Validate the smallest feasible run before adding scale. Some required models can't fit one device at all; debug data contracts with small fixtures, then use the minimum necessary sharding. Single-device success also doesn't prove the distributed loss normalization or checkpoint path is correct.

Signs that one GPU is still fine

A single device is feasible when weights, gradients, optimizer state, activations, and temporary buffers fit at the chosen precision and the run meets its time budget. A model that fits for inference may still fail this training-memory test.

Signs that you need FSDP or ZeRO

Consider FSDP or ZeRO when persistent model weights, gradients, and AdamW optimizer states exceed available GPU memory. Measure activation memory independently: sharding model parameters doesn't automatically eliminate activation overhead. Longer contexts often require activation checkpointing, reduced micro-batch sizes, or context parallelism. Adding more GPUs introduces network communication latency and won't speed up an improperly configured small job.

FSDP and ZeRO offer different state-sharding strategies; the chosen stage determines what is sharded.^{[16]Reference 16FullyShardedDataParallelhttps://docs.pytorch.org/docs/stable/fsdp.html[17]Reference 17ZeRO: Memory Optimizations Toward Training Trillion Parameter Models.https://arxiv.org/abs/1910.02054} Specify precision explicitly. Current TRL defaults to BF16 when FP16 isn't set, but a default doesn't establish hardware support or numerical parity.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer}

Minimal operational skeleton

The overall architectural pipeline remains consistent across modern training libraries:

TRL, torchtune, and the Alignment Handbook implement variations of this workflow.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer[18]Reference 18alignment-handbook: Robust recipes to align language models with human and AI preferences.https://github.com/huggingface/alignment-handbook[19]Reference 19Welcome to the torchtune Documentation.https://meta-pytorch.org/torchtune/stable/index.html}

The configuration fragment below is an unexecuted integration blueprint for conversational prompt-completion rows. Its pilot disables packing until boundaries are verified. API fields were checked against TRL on September 22, 2026; pin a tested package version and model revision for a real run.

This fragment omits model loading, adapter initialization, and dataset construction. Convert the earlier authoring rows to message lists so the trainer can apply the checkpoint's template; raw string fields don't automatically become a chat conversation. We explicitly choose loss_type="nll"; current TRL's default chunked NLL computes the same target objective with a different projection/memory path.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer} The evaluation loop computes teacher-forced loss, not the supplied policy/format rates. Generate and grade validation replies separately before applying the selection rule.

Common pitfalls

When a training run regresses, analyze the symptoms and identify which system boundary failed. This prevents data formatting or masking bugs from turning into futile hyperparameter sweeps.

Confusing the objective with the update surface

Saying "we tried LoRA instead of SFT" mixes two distinct decisions. LoRA and full fine-tuning describe which weights are updated; SFT defines the supervised next-token cross-entropy objective. Define the behavioral target and objective first. Compare adapter tuning against full-weight tuning only when parameter capacity or memory budget is the variable under test.

Evaluating examples that share a case with training

High validation scores followed by failures on new documents can reflect leakage, distribution shift, or a mismatched metric. Check whether related cases crossed splits and whether the grouping actually tests the deployment claim. A thread-disjoint split doesn't test unseen documents shared across those threads.

Reporting micro-batch as if it were the real batch

When two experiments claim "batch size 4" but demonstrate drastically different stability, check accumulation steps and GPU count before modifying the optimizer. Always publish per-device batch size, gradient accumulation steps, world size, packing mode, and total supervised tokens per update step.

Saving weights without resume state

Changed loss or batches after resume prompts a state comparison; it doesn't uniquely diagnose an omitted checkpoint field. Check optimizer counters, LR, upcoming data, RNG states, precision/scaler state, and artifact identities against the uninterrupted reference.

Using a packing path that crosses example boundaries

Incoherent answers don't uniquely diagnose packing leakage. Test attention reachability and shifted-label boundaries directly. For independent demonstrations, use a verified segment-aware path; for intentional continuous-text training, cross-document context may be part of the chosen objective. A reset position ID or EOS token isn't itself an enforced barrier.

Selecting the best checkpoint by train loss

Training loss measures reference-token likelihood on seen examples, not memorization by itself. Use validation behavior and declared constraints for selection, then assess the frozen choice on the final test. Checks need a task-appropriate grader; fluent output and low loss don't certify policy compliance.