Supervised Fine-Tuning Pipeline

The chat-template lesson showed how a grounded stale-key row becomes a token stream. The synthetic-data lesson showed how candidate rows get accepted or rejected. Supervised fine-tuning (SFT) is the training system that turns those accepted demonstrations into gradient updates on the replies you actually want.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer[2]Reference 2Fine-Tune Your First LLM.https://meta-pytorch.org/torchtune/stable/tutorials/first_finetune_tutorial.html} InstructGPT used supervised demonstrations as that first post-training stage before preference optimization.^{[3]Reference 3Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

Thread T102 asks whether an 18-day-old service-account key is still usable. The accepted answer is: open a rotation ticket, because keys older than 14 days need review. Loss can still fall while that behavior stays broken. Train on prompt tokens, leak T102 into the eval split, pack R550 so it attends to T102, or export the lowest-train-loss checkpoint, and you'll ship a fluent skip of the ticket.

A good SFT run doesn't start with a trainer call. It starts with a behavior target, a leakage-resistant split, a loss mask, a batch budget, checkpoints that can resume, and an evaluation rule for exporting the best artifact. First choose the learning objective. Then choose whether that objective updates every weight or a small adapter. Mixing those two decisions produces expensive runs that answer the wrong question.

What an SFT run must decide

Every serious SFT job has to answer the same questions. The T102 failures above are what happens when you skip one:

1. What is the behavior target?
2. Is SFT the right objective, or is the missing ingredient still domain pretraining?
3. Are we updating all weights or only adapters?
4. What examples and tokens contribute to training loss?
5. What is the true example and supervised-token batch budget?
6. Which held-out metric decides the best checkpoint?
7. Can this stay on one GPU, or do we need Fully Sharded Data Parallel (FSDP) or Zero Redundancy Optimizer (ZeRO)?

Once you phrase the job that way, the trainer is one component: it applies the update. The split, mask, batch card, resume bundle, and export rule still sit outside it.

Separate objective from parameterization

An objective tells the model what signal to learn from. A parameterization tells the trainer which weights may move. They aren't interchangeable.

After choosing SFT, make a second decision:

LoRA and QLoRA aren't alternatives to SFT. They can be the parameterization used for an SFT run. Continued pretraining is a different objective; it can also be parameter-efficient in an appropriate setup, but it still doesn't become SFT.

Full weights or adapters

Full fine-tuning

Full fine-tuning is the simplest conceptual update: every eligible parameter may change to reduce supervised response loss. Test it when:

• the model fits with its gradients and optimizer state
• one merged checkpoint is operationally simpler than serving adapters
• an adapter baseline underfits your held-out behavior metric

LoRA

LoRA is still SFT when trained on prompt-response rows. Use it when:

• iteration speed or training memory limits matter
• you need separate task or tenant variants over one base model
• you want a strong adapter baseline before paying for full updates

LoRA freezes the base model weights and trains low-rank adapter matrices inside selected layers, which reduces trainable parameters and memory compared with full fine-tuning.^{[6]Reference 6LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685}

QLoRA

Use QLoRA when:

• LoRA is the right algorithmic choice
• the frozen base model itself is too large to keep in BF16/FP16 memory
• you still want adapter training instead of full-weight updates

QLoRA keeps the adapter-training idea but backpropagates through a frozen 4-bit quantized base model into LoRA adapters, which is why it can make larger models fit in less memory.^{[7]Reference 7QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314} The comparison that matters is held-out behavior under an honest memory and compute budget. A cheap run that fails the task isn't a win.

Reject broken demonstrations before training

For this pipeline, store a prompt, a desired completion, and a group key for leakage-resistant splitting on every SFT row. A missing completion isn't harmless missing metadata: it's a training example with no answer to teach.

Split by the unit that could leak

Before formatting rows, reserve evaluation examples that training can't imitate through near duplicates. For a policy assistant, several messages from one access incident or one policy document often share facts and phrasing. Randomly splitting individual messages can put one part of the same case in training and another in evaluation.

Group by the smallest deployment unit that should be unseen at evaluation time: incident thread, policy document, tenant, or time window. The tiny check below keeps every message from an incident on one side of the split.

Two turns from incident T102 have different wording. Can one train and the other evaluate if their token overlap is low?

Answer

No. They share the deployment unit that can leak facts and phrasing, so both turns stay in the same split. Group by incident, document, tenant, or time window before formatting and tokenization.

Data path: template, tokenize, label, pack

An SFT run usually performs these steps:

1. read structured examples
2. apply the model-specific chat template
3. run tokenization
4. label desired response tokens and mask the prompt/context tokens
5. pack short examples with sequence-boundary handling, or pad them into batches
6. feed the trainer

The instruction-tuning lesson already covered chat-template mechanics. The operational lesson here is that you version this whole path together. If you change the template or masking logic, you changed the training distribution and therefore the experiment.^{[8]Reference 8Transformers Documentation: Writing a chat template.https://huggingface.co/docs/transformers/main/en/chat_templating_writing}

The label mask is easy to get wrong because the input still contains the system instruction, user request, and formatting control tokens. For response-only SFT, the desired answer tokens (including the turn terminator when the model must learn when to stop) contribute to loss. Prefix tokens provide context, not answer targets. The tiny example uses -100, the ignored label value used by PyTorch cross-entropy.

Don't confuse the label mask with the attention mask. The label mask decides which positions contribute to loss. The attention mask decides which earlier positions a token can read. Prompt tokens should remain visible as context even when their labels are -100; packed examples also need attention boundaries so one example can't read another.

Why bother masking easy prefix tokens? If a long repeated instruction is easy to predict, its small losses can dominate the average and hide poor answer learning:

Those two numbers are the same token-level cross-entropy, averaged over different positions. For response-only SFT, the reported loss is the mean negative log-likelihood over the labeled set $T$ (here, the answer and stop tokens), not over the whole sequence:

$\mathcal{L}_{\text{SFT}}(\theta) = -\frac{1}{|T|}\sum_{t \in T} \log p_\theta(y_t \mid y_{<t})$

$\theta$ is the trainable parameters, $y_t$ is the token at position $t$, and $y_{<t}$ is the prefix the model may attend to. In the toy row, $T$ has two positions and the mean is $2.0$. PyTorch's cross-entropy with ignore_index=-100 already averages that way. Forget the mask, and the easy prefix tokens pull the average down to $0.596$.

You rarely build this mask by hand. In Hugging Face TRL's SFTTrainer, prompt-completion datasets use completion-only loss by default. Conversational datasets can set assistant_only_loss=True; their chat template must expose assistant spans through generation markers. Current TRL can substitute supported training templates for known model families, but a custom template still needs verification.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer} Inspect a tokenized example before launching a long run: it should reveal exactly which answer and stop tokens receive labels.

Mask inspection must happen after truncation as well. Current TRL SFTConfig defaults to truncation_mode="keep_start": it keeps the beginning of the concatenated sequence and drops the end. If the prompt fills max_length, the completion (the only labeled span) disappears. TRL then drops examples whose labels are all -100, so you lose the row instead of training on a prompt-only sequence. That's still silent data loss. When packing is on, this truncation step is skipped and overflow is handled by the packing strategy instead; bfd still discards overflow tokens.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer}

The toy below is that keep_start cut: five prefix tokens fill max_length, so the ticket answer never appears. Reject the row or raise max_length; don't assume every accepted demonstration survived tokenization.

Pack short examples while preserving their boundaries

Most SFT datasets are short replies. Padding each one to a fixed length wastes compute on padding tokens. Packing places several examples in a nearly full training block so more positions carry real tokens.

For independent instruction examples, a reply for case R550 shouldn't gain context from case T102 only because the loader put them in one block. A boundary-aware packed attention path records each segment, resets positions at each sequence boundary, and prevents that cross-example attention:

Reset position IDs are boundary metadata, not a universal attention mask. The attention implementation still has to honor those boundaries when it computes attention.

What a failed packed batch looks like. Walk one block of six tokens (T102 then R550) after an upgrade that keeps reset position_ids but drops segment isolation:

The same block can also fail via truncation then pack: if T102 is kept after a cut that left every answer label at -100, packing still fills the block with tokens, but that segment teaches no supervised response. Inspect both attention reachability and nonzero answer labels per segment before a long run.

Current TRL exposes packing=True and eval_packing. Its default best-fit-decreasing (bfd) strategy packs intact examples efficiently and truncates overflow sequences. bfd_split preserves overflow tokens by splitting long sequences before packing. wrapped concatenates everything into a stream and cuts mid-sequence; TRL documents that this can mix unrelated examples and break continuity. With bfd, TRL enables padding-free processing. That flattened path relies on FlashAttention 2 or 3 to honor sequence boundaries; without a compatible attention implementation, adjacent samples may contaminate each other.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer} The right check isn't "never pack." It's: choose an SFT-appropriate strategy, confirm the supported attention path, and inspect a packed batch after library upgrades.

Packing also changes what an epoch means. TRL logs num_tokens during the run. One epoch after packing is one pass over packed blocks, not one pass over original rows. If you set max_steps from an unpacked row count, you can overshoot the intended token budget and overfit without noticing.

A packed batch resets position IDs for R550, but R550's response can still attend to T102. Is the packing path correct?

Answer

No. Reset positions don't enforce attention isolation. The segment mask or padding-free attention metadata must block cross-example attention, and every packed segment must retain at least one non--100 answer label.

Hyperparameters are experiments, not promises

SFT begins from a pretrained checkpoint, so a useful configuration starts conservatively and earns expansion through held-out results. A library default isn't evidence that one value fits your dataset or parameterization.

Current TRL SFTConfig defaults learning_rate to 2e-5, turns gradient_checkpointing on, and enables bf16 when fp16 isn't set. Its PEFT guidance suggests about 1e-4 for adapters because only new parameters are being learned.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer} That bf16 default is already a precision policy. Record it with the run; the next chapter measures whether the arithmetic is safe.

Don't turn 2e-5 or 1e-4 into a rule. Record parameterization, trainable-parameter count, sequence length, effective token budget, data snapshot, precision, and evaluation result beside every candidate configuration.

Warmup, clipping, and decay aren't decoration. Warmup limits early updates while the run is most sensitive to its initial configuration; jumping immediately to a peak adapter rate of 1e-4 can make the first steps unnecessarily noisy. Gradient clipping doesn't fix bad data, but it can stop one pathological batch from damaging the checkpoint. Weight decay belongs in the recipe record even when you disable it, so the next sweep doesn't inherit a silent default.

Effective batch size is also a token budget

Training discussions get sloppy fast here. The batch that sits on one device isn't the same thing as the batch the optimizer sees before one update. For equal-length examples, the usual rule is:

Packed response examples aren't equal length. This formula reports examples per optimizer update, but the number of supervised answer tokens can still change from update to update. Track both.

Always calculate changed configurations instead of reasoning from one changed knob:

Two habits follow from this:

• when someone reports batch size, ask for accumulation, world size, packing, and supervised tokens too
• compare learning-rate sweeps with similar token budgets or explain why they differ

What a resumable run must save

A real SFT resume bundle isn't only model_state_dict.

At minimum, retain:

• model weights
• optimizer state
• scheduler state
• global step / epoch
• RNG state and sampler/data position when repeatable continuation matters
• tokenizer + chat-template version
• training config, data manifest, and evaluation manifest
• best metric so far

Why this matters:

• optimizer state controls the next update
• scheduler state controls learning rate at resume
• sampler state prevents quietly repeating or skipping shuffled training rows
• tokenizer/template version controls what the model thinks each token means

Exact bit-for-bit replay may still depend on hardware, kernels, and determinism settings. The resume bundle should at least prevent avoidable changes to data order, learning-rate state, and model selection. torchtune and DeepSpeed both document resume flows explicitly instead of treating checkpointing as a trivial file write.^{[9]Reference 9Checkpointing in torchtune.https://meta-pytorch.org/torchtune/stable/deep_dives/checkpointer.html[10]Reference 10Universal Checkpointing with DeepSpeed: A Practical Guide.https://www.deepspeed.ai/tutorials/universal-checkpointing/}

A minimal manifest should make missing resume state obvious:

A resumed run reloads model weights and global step but not optimizer, scheduler, sampler, or tokenizer-template versions. Can its next metrics be compared as uninterrupted continuation?

Answer

No. The next update may use different momentum, learning rate, data position, randomness, or token formatting. Treat it as a new run unless the complete resume manifest is restored and validated.

Choosing the best checkpoint

The best checkpoint is the one that optimizes the product metric you care about on held-out data.

Bad rules:

• lowest train loss
• latest step
• biggest file

Better evidence:

• highest exact or structured task success where the behavior is verifiable
• best human preference rate on a blinded held-out set when quality is subjective
• judge-assisted scoring only after calibrating the judge against human or verifiable decisions
• lowest validation loss only when that truly matches the task

If the task is structured access-policy replies, check policy compliance and schema validity before style preference. Loss is useful for training diagnostics, but it doesn't define deployment quality.

The export rule for this run is a gate, then a ranking, then a tie-break:

Single GPU first, then scale out

Don't start thread T102 on 8 GPUs. First prove the recipe on one GPU:

1. one GPU
2. tiny eval slice
3. frequent checkpoints
4. verified template/masking
5. one clean metric

Only then widen world size or sequence length.

Signs that one GPU is still fine

• the model and activations fit
• throughput is acceptable
• you're still debugging correctness
• labels are the main uncertainty, not compute budget

Signs that you need FSDP or ZeRO

• full-model weights and optimizer state don't fit
• activation memory collapses the run at useful context lengths
• checkpointing and accumulation still leave the job too slow
• you need larger world size to hit the schedule

That bridge matters. FSDP and ZeRO aren't decorations on top of a stable recipe. They let the same recipe continue when memory stops fitting on one device by sharding model states across workers.^{[11]Reference 11FullyShardedDataParallelhttps://docs.pytorch.org/docs/stable/fsdp.html[12]Reference 12ZeRO: Memory Optimizations Toward Training Trillion Parameter Models.https://arxiv.org/abs/1910.02054} Precision is the other silent scaler: if SFTConfig already turned on bf16, you have a mixed-precision policy before you add a second GPU.

Minimal operational skeleton

The exact library can vary, but the shape is stable:

TRL, torchtune, and the Alignment Handbook all expose variations of this same loop.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer[13]Reference 13alignment-handbook: Robust recipes to align language models with human and AI preferences.https://github.com/huggingface/alignment-handbook[14]Reference 14Welcome to the torchtune Documentation.https://meta-pytorch.org/torchtune/stable/index.html}

For a TRL prompt-completion dataset, a configuration skeleton makes the decisions visible. Configure model_with_lora_adapters with a supported FlashAttention 2 or 3 implementation before enabling bfd packing:

This fragment deliberately leaves model loading, adapter construction, and the data manifest outside the snippet. They must be versioned inputs to a real run, not invisible defaults.

Common pitfalls

Confusing the objective with the update surface

• Symptom: A team says "we tried LoRA instead of SFT," or switches to full updates without changing data or evaluation.
• Cause: LoRA and full fine-tuning describe which parameters move; SFT describes the supervised loss objective.
• Fix: Write down the failed behavior and desired objective first. Then compare adapter and full-weight SFT only if update capacity or memory is the actual uncertainty.

Evaluating examples that share a case with training

• Symptom: Held-out score is excellent, but the model fails on new tenants, policies, or access incidents.
• Cause: Rows were randomly split while near-duplicate turns from the same incident or document appeared on both sides.
• Fix: Group the split by incident thread, policy source, tenant, or later time window, then re-run selection.

Reporting micro-batch as if it were the real batch

• Symptom: Two runs both claim "batch size 4," but convergence looks different and nobody can reconcile the results.
• Cause: The reported number ignored accumulation or world size, so the optimizer was seeing different effective batches.
• Fix: Publish per_device_batch, gradient_accumulation_steps, world_size, packed/padded mode, examples per update, and supervised tokens per update.

Saving weights without resume state

• Symptom: Resume starts, but the learning rate jumps, checkpoint selection resets, or the run diverges from prior curves.
• Cause: The checkpoint kept weights but dropped optimizer state, scheduler position, or best-metric record.
• Fix: Treat resume state as part of the checkpoint contract: weights, optimizer, scheduler, global step, RNG/sampler state, template/tokenizer version, data and eval manifests, and best metric.

Using a packing path that crosses example boundaries

• Symptom: Loss falls smoothly, but the model learns transitions that never occur in real conversations.
• Cause: A packing strategy or attention implementation let tokens from one independent example condition another.
• Fix: For TRL SFT, prefer bfd with a supported FlashAttention 2 or 3 implementation and inspect packed batches. Use bfd_split deliberately when preserving overflow matters; don't silently substitute wrapped or an unsupported attention path.

Selecting the best checkpoint by train loss

• Symptom: Exported checkpoint has the prettiest curve but underperforms on held-out policy tasks.
• Cause: Train loss rewards fitting seen data, not solving the downstream task the product cares about.
• Fix: Export the checkpoint that wins on held-out verifiable metrics or calibrated human-preference evaluation that matches deployment.