Supervised Fine-Tuning Pipeline

The synthetic-data chapter built and verified candidate rows for post-training. Supervised fine-tuning (SFT) turns accepted demonstrations, whether written by humans or generated and checked, into gradient updates on desired responses. A good SFT run doesn't start with a trainer call. It starts with a behavior target, a leakage-resistant split, a loss mask, a batch budget, checkpoints that can resume, and an evaluation rule for exporting the best artifact.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer[2]Reference 2Fine-Tune Your First LLM.https://meta-pytorch.org/torchtune/stable/tutorials/first_finetune_tutorial.html}

Turn SFT into an operational training system for access-policy assistant replies. First choose the learning objective. Then choose whether that SFT objective updates every weight or a small adapter. Keeping those two decisions separate prevents expensive experiments that answer the wrong question.

What an SFT run must decide

Every serious SFT job has to answer the same questions:

1. What is the behavior target?
2. Is SFT the right objective, or is the missing ingredient still domain pretraining?
3. Are we updating all weights or only adapters?
4. What examples and tokens contribute to training loss?
5. What is the true example and supervised-token batch budget?
6. Which held-out metric decides the best checkpoint?
7. Can this stay on one GPU, or do we need Fully Sharded Data Parallel (FSDP) or Zero Redundancy Optimizer (ZeRO)?

Once you phrase the job that way, the trainer isn't the system. It's only one component inside the system.

Separate objective from parameterization

An objective tells the model what signal to learn from. A parameterization tells the trainer which weights may move. They aren't interchangeable.

After choosing SFT, make a second decision:

LoRA and QLoRA aren't alternatives to SFT. They can be the parameterization used for an SFT run. Continued pretraining is a different objective; it can also be parameter-efficient in an appropriate setup, but it still doesn't become SFT.

Full weights or adapters

Full fine-tuning

Full fine-tuning is the simplest conceptual update: every eligible parameter may change to reduce supervised response loss. Test it when:

• the model fits with its gradients and optimizer state
• one merged checkpoint is operationally simpler than serving adapters
• an adapter baseline underfits your held-out behavior metric

LoRA

LoRA is still SFT when trained on prompt-response rows. Use it when:

• iteration speed or training memory limits matter
• you need separate task or tenant variants over one base model
• you want a strong adapter baseline before paying for full updates

LoRA freezes the base model weights and trains low-rank adapter matrices inside selected layers, which reduces trainable parameters and memory compared with full fine-tuning.^{[5]Reference 5LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685}

QLoRA

Use QLoRA when:

• LoRA is the right algorithmic choice
• the frozen base model itself is too large to keep in BF16/FP16 memory
• you still want adapter training instead of full-weight updates

QLoRA keeps the adapter-training idea but backpropagates through a frozen 4-bit quantized base model into LoRA adapters, which is why it can make larger models fit in less memory.^{[6]Reference 6QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314} The comparison that matters is held-out behavior under an honest memory and compute budget. A cheap run that fails the task isn't a win.

Reject broken demonstrations before training

For this pipeline, store a prompt, a desired completion, and a group key for leakage-resistant splitting on every SFT row. A missing completion isn't harmless missing metadata: it's a training example with no answer to teach.

Split by the unit that could leak

Before formatting rows, reserve evaluation examples that training can't imitate through near duplicates. For a policy assistant, several messages from one access incident or one policy document often share facts and phrasing. Randomly splitting individual messages can put one part of the same case in training and another in evaluation.

Group by the smallest deployment unit that should be unseen at evaluation time: incident thread, policy document, tenant, or time window. The tiny check below keeps every message from an incident on one side of the split.

Data path: template, tokenize, label, pack

An SFT run usually performs these steps:

1. read structured examples
2. apply the model-specific chat template
3. run tokenization
4. label desired response tokens and mask the prompt/context tokens
5. pack short examples with sequence-boundary handling, or pad them into batches
6. feed the trainer

The instruction-tuning lesson already covered chat-template mechanics. The operational lesson here is that you version this whole path together. If you change the template or masking logic, you changed the training distribution and therefore the experiment.^{[7]Reference 7Transformers Documentation: Writing a chat template.https://huggingface.co/docs/transformers/main/en/chat_templating_writing}

The label mask is easy to get wrong because the input still contains the system instruction, user request, and formatting control tokens. For response-only SFT, the desired answer tokens (including the turn terminator when the model must learn when to stop) contribute to loss. Prefix tokens provide context, not answer targets. The tiny example uses -100, the ignored label value used by PyTorch cross-entropy.

Don't confuse the label mask with the attention mask. The label mask decides which positions contribute to loss. The attention mask decides which earlier positions a token can read. Prompt tokens should remain visible as context even when their labels are -100; packed examples also need attention boundaries so one example can't read another.

Why bother masking easy prefix tokens? If a long repeated instruction is easy to predict, its small losses can dominate the average and hide poor answer learning:

You rarely build this mask by hand. In Hugging Face TRL's SFTTrainer, prompt-completion datasets use completion-only loss by default. Conversational datasets can set assistant_only_loss=True; their chat template must expose assistant spans through generation markers. Current TRL can substitute supported training templates for known model families, but a custom template still needs verification.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer} Inspect a tokenized example before launching a long run: it should reveal exactly which answer and stop tokens receive labels.

Mask inspection must happen after truncation as well. If the prompt fills max_length, the run can keep a row whose entire answer has disappeared:

Pack short examples while preserving their boundaries

Most SFT datasets are short replies. Padding each one to a fixed length wastes compute on padding tokens. Packing places several examples in a nearly full training block so more positions carry real tokens.

For independent instruction examples, a reply for case R550 shouldn't gain context from case T102 only because the loader put them in one block. A boundary-aware packed attention path records each segment, resets positions at each sequence boundary, and prevents that cross-example attention:

Reset position IDs are boundary metadata, not a universal attention mask. The attention implementation still has to honor those boundaries when it computes attention.

Current TRL exposes packing=True and eval_packing. Its default best-fit-decreasing (bfd) strategy packs intact examples efficiently and truncates overflow sequences. bfd_split preserves overflow tokens by splitting long sequences before packing. wrapped is the aggressive stream-and-split option, and it may mix unrelated examples. With bfd, TRL enables padding-free processing. That flattened path relies on FlashAttention 2 or 3 to honor sequence boundaries; without a compatible attention implementation, adjacent samples may contaminate each other.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer} The right check isn't "never pack." It's: choose an SFT-appropriate strategy, confirm the supported attention path, and inspect a packed batch after library upgrades.

Hyperparameters are experiments, not promises

SFT begins from a pretrained checkpoint, so a useful configuration starts conservatively and earns expansion through held-out results. A library default isn't evidence that one value fits your dataset or parameterization.

Don't turn 2e-5 or 1e-4 into a rule. Record parameterization, trainable-parameter count, sequence length, effective token budget, data snapshot, and evaluation result beside every candidate configuration.

Effective batch size is also a token budget

Training discussions get sloppy fast here. The batch that sits on one device isn't the same thing as the batch the optimizer sees before one update. For equal-length examples, the usual rule is:

Packed response examples aren't equal length. This formula reports examples per optimizer update, but the number of supervised answer tokens can still change from update to update. Track both.

Always calculate changed configurations instead of reasoning from one changed knob:

Two habits follow from this:

• when someone reports batch size, ask for accumulation, world size, packing, and supervised tokens too
• compare learning-rate sweeps with similar token budgets or explain why they differ

Warmup, clipping, and weight decay

These knobs aren't decoration.

Warmup

Warmup limits the size of early updates while the run is most sensitive to its initial configuration. For LLM SFT, jumping immediately to the peak learning rate can make the first steps unnecessarily noisy.

Gradient clipping

Clip gradients when spikes would destabilize the run. Clipping doesn't fix bad data, but it can stop one pathological batch from damaging the checkpoint.

Weight decay

Weight decay regularizes the update path. It should be treated as part of the recipe, not an afterthought copy-pasted from another config.

Don't memorize one numeric recipe. Know what each control is trying to prevent.

What a resumable run must save

A real SFT resume bundle isn't only model_state_dict.

At minimum, retain:

• model weights
• optimizer state
• scheduler state
• global step / epoch
• RNG state and sampler/data position when repeatable continuation matters
• tokenizer + chat-template version
• training config, data manifest, and evaluation manifest
• best metric so far

Why this matters:

• optimizer state controls the next update
• scheduler state controls learning rate at resume
• sampler state prevents quietly repeating or skipping shuffled training rows
• tokenizer/template version controls what the model thinks each token means

Exact bit-for-bit replay may still depend on hardware, kernels, and determinism settings. The resume bundle should at least prevent avoidable changes to data order, learning-rate state, and model selection. torchtune and DeepSpeed both document resume flows explicitly instead of treating checkpointing as a trivial file write.^{[8]Reference 8Checkpointing in torchtune.https://meta-pytorch.org/torchtune/stable/deep_dives/checkpointer.html[9]Reference 9Universal Checkpointing with DeepSpeed: A Practical Guide.https://www.deepspeed.ai/tutorials/universal-checkpointing/}

A minimal manifest should make missing resume state obvious:

Choosing the best checkpoint

The best checkpoint is the one that optimizes the product metric you care about on held-out data.

Bad rules:

• lowest train loss
• latest step
• biggest file

Better evidence:

• highest exact or structured task success where the behavior is verifiable
• best human preference rate on a blinded held-out set when quality is subjective
• judge-assisted scoring only after calibrating the judge against human or verifiable decisions
• lowest validation loss only when that truly matches the task

If the task is structured access-policy replies, check policy compliance and schema validity before style preference. Loss is useful for training diagnostics, but it doesn't define deployment quality.

Single GPU first, then scale out

A good SFT rollout usually starts with the smallest honest setup:

1. one GPU
2. tiny eval slice
3. frequent checkpoints
4. verified template/masking
5. one clean metric

Only then do you widen the job.

Signs that one GPU is still fine

• the model and activations fit
• throughput is acceptable
• you're still debugging correctness
• labels are the main uncertainty, not compute budget

Signs that you need FSDP or ZeRO

• full-model weights and optimizer state don't fit
• activation memory collapses the run at useful context lengths
• checkpointing and accumulation still leave the job too slow
• you need larger world size to hit the schedule

That bridge matters. FSDP and ZeRO aren't "advanced decorations" on top of a stable recipe. They let the same recipe continue when memory stops fitting on one device by sharding model states across workers.^{[10]Reference 10FullyShardedDataParallelhttps://docs.pytorch.org/docs/stable/fsdp.html[11]Reference 11ZeRO: Memory Optimizations Toward Training Trillion Parameter Models.https://arxiv.org/abs/1910.02054}

Minimal operational skeleton

The exact library can vary, but the shape is stable:

TRL, torchtune, and the Alignment Handbook all expose variations of this same loop.^{[1]Reference 1TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer[12]Reference 12alignment-handbook: Robust recipes to align language models with human and AI preferences.https://github.com/huggingface/alignment-handbook[13]Reference 13Welcome to the torchtune Documentation.https://meta-pytorch.org/torchtune/stable/index.html}

For a TRL prompt-completion dataset, a configuration skeleton makes the decisions visible. Configure model_with_lora_adapters with a supported FlashAttention 2 or 3 implementation before enabling bfd packing:

This fragment deliberately leaves model loading, adapter construction, and the data manifest outside the snippet. They must be versioned inputs to a real run, not invisible defaults.

Common pitfalls

Confusing the objective with the update surface

• Symptom: A team says "we tried LoRA instead of SFT," or switches to full updates without changing data or evaluation.
• Cause: LoRA and full fine-tuning describe which parameters move; SFT describes the supervised loss objective.
• Fix: Write down the failed behavior and desired objective first. Then compare adapter and full-weight SFT only if update capacity or memory is the actual uncertainty.

Evaluating examples that share a case with training

• Symptom: Held-out score is excellent, but the model fails on new tenants, policies, or access incidents.
• Cause: Rows were randomly split while near-duplicate turns from the same incident or document appeared on both sides.
• Fix: Group the split by incident thread, policy source, tenant, or later time window, then re-run selection.

Reporting micro-batch as if it were the real batch

• Symptom: Two runs both claim "batch size 4," but convergence looks different and nobody can reconcile the results.
• Cause: The reported number ignored accumulation or world size, so the optimizer was seeing different effective batches.
• Fix: Publish per_device_batch, gradient_accumulation_steps, world_size, packed/padded mode, examples per update, and supervised tokens per update.

Saving weights without resume state

• Symptom: Resume starts, but the learning rate jumps, checkpoint selection resets, or the run diverges from prior curves.
• Cause: The checkpoint kept weights but dropped optimizer state, scheduler position, or best-metric record.
• Fix: Treat resume state as part of the checkpoint contract: weights, optimizer, scheduler, global step, RNG/sampler state, template/tokenizer version, data and eval manifests, and best metric.

Using a packing path that crosses example boundaries

• Symptom: Loss falls smoothly, but the model learns transitions that never occur in real conversations.
• Cause: A packing strategy or attention implementation let tokens from one independent example condition another.
• Fix: For TRL SFT, prefer bfd with a supported FlashAttention 2 or 3 implementation and inspect packed batches. Use bfd_split deliberately when preserving overflow matters; don't silently substitute wrapped or an unsupported attention path.

Selecting the best checkpoint by train loss

• Symptom: Exported checkpoint has the prettiest curve but underperforms on held-out support tasks.
• Cause: Train loss rewards fitting seen data, not solving the downstream task the product cares about.
• Fix: Export the checkpoint that wins on held-out verifiable metrics or calibrated human-preference evaluation that matches deployment.

Evaluation rubric

Check that you can explain and operate these parts of an SFT run:

• How dataset, template, tokenizer, collator, loss mask, optimizer, checkpointing, and evaluation fit into one training loop.
• Why a response-only objective masks prompt tokens and includes the answer termination behavior.
• Why a grouped held-out split is stronger than a row-random split for related support conversations.
• How TRL's bfd, bfd_split, and wrapped strategies differ, and why the padding-free attention path still has to honor example boundaries.
• How to treat documented learning-rate defaults as sweep starting points rather than guarantees.
• How to compute examples and supervised answer tokens per optimizer update.
• How to choose an objective first, then compare full-weight, LoRA, or QLoRA SFT parameterizations.
• What a resumable checkpoint has to save beyond model weights.
• Why the best checkpoint is selected by held-out verifiable or calibrated behavior metrics rather than training loss.
• When to stay on one GPU for correctness and when FSDP or ZeRO becomes necessary.

Practice checkpoints

What to remember

• SFT is a training system, not a trainer call alone.
• Choose the learning objective before choosing full updates, LoRA, or QLoRA as its parameterization.
• Hold out real groups such as incident threads or policy sources, not randomly related messages.
• For response-only SFT, supervise answer and termination tokens; in TRL that means completion_only_loss or assistant_only_loss.
• Packing saves compute only when strategy and attention behavior preserve the boundaries between independent examples.
• Library defaults begin a sweep; they don't certify your learning rate or training duration.
• Track supervised tokens per update in addition to examples, accumulation, and world size.
• Resumable checkpoints need optimizer, scheduler, sampler, formatting, data, evaluation, and best-metric state, not weights alone.
• The best checkpoint is chosen by held-out verifiable or calibrated behavior evidence, not by train loss.
• Start on one GPU for correctness, then widen to FSDP / ZeRO when memory or schedule demands it.