Skip to content
Skip to lesson
Back to Topics
LearnAdvanced Training & AdaptationSupervised Fine-Tuning Pipeline
⚡HardFine-Tuning & Training

Supervised Fine-Tuning Pipeline

The assistant sounds reasonable and still misses the policy. Supervised fine-tuning can raise the likelihood of the right reply, but the loss, split, and checkpoint have to measure that behavior.

41 min read
Learning path
Step 109 of 199 in the full curriculum

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

Incident thread T102 asks whether an 18-day-old service-account key is still usable. The policy sends keys older than 14 days to security review, starting with a rotation ticket. The assistant replies, "The key is old, but rotation can wait until next month." It sounds reasonable until you check the policy. How would you teach the model to catch that miss, and how would you tell whether training actually helped? Filing the ticket starts review. It doesn't approve access.

A chat template can already turn the T102 thread into tokens, and the synthetic-data gates can already accept or reject the row. uses an accepted demonstration to raise the likelihood of the desired reply through updates.[1][2] InstructGPT used that supervised stage before reward-model training and reinforcement learning.[3] Compare training loss with held-out escalation cases when selecting a checkpoint.

Training loss can drop smoothly while policy behavior stays broken. Prompt-dominated metrics, another turn from T102 leaking into validation, unintended cross-example attention, or selection by training loss can hide the failure. Each raises a different diagnostic question. Check the split, labels, and evaluation before funding an optimizer sweep; none of these symptoms alone proves the cause.

Count supervised answer tokens before launching any job. A packed row can fill every token slot in the context window while providing zero learning signal if aggressive truncation cuts off the completion and masks every remaining token to -100. Reject that row outright instead of pretending its context tokens represent training data.

Eight token positions from the T102 stream in a row. Positions 0 to 4 are context tokens with label minus 100. Positions 5 to 7 are open, ticket, and end, with those same tokens as labels. Arrows run from the logits at positions 4, 5, and 6 to the labels one position to their right. Only those three predictions enter the loss, and ignored prompt tokens stay in causal context.
Each target aligns with its label and the preceding logit that predicts it: open uses logits[4], ticket uses logits[5], and end uses logits[6]. Prompt labels are -100; prompt representations can still affect response loss. These are symbolic positions, not an actual tokenizer's output.

Turn behavior into a run contract

The miss can hide in more than one place: the learning objective, the data split, the label mask, the token budget, the checkpoint format, or the release gate. The examples below check those contracts on CPU. They don't train or benchmark a large language model. The TRL configuration is an integration blueprint, not an unverified GPU script.

Define the behavioral contract first: cases with keys older than 14 days require a rotation-ticket escalation under this lab's policy. Compare prompting with the policy in context, retrieval, and SFT before choosing an intervention. If unfamiliar domain language remains a barrier, investigate continued too.

Next, trace what each update actually sees. Which examples and contribute to the loss function? What example and supervised-token budget reaches a single optimizer step? What held-out metric dictates checkpoint selection? Finally, can the validated recipe run on a single , or do Fully Sharded Data Parallel (FSDP) or Zero Redundancy (ZeRO) become necessary?

Following this order keeps the trainer where it belongs: as an execution engine applying an update. The split, mask, batch budget, resume bundle, and export gate define what that update means.

Separate objective from parameterization

Suppose T102 outputs fluent prose but persistently advises waiting until next month. Is the missing ingredient raw domain text, a demonstration of procedural compliance, or a ranking across several plausible replies? Answering that question separates the learning objective from the parameterization that absorbs the gradient updates.

An objective determines the mathematical signal guiding the model. A parameterization specifies which weights in the model architecture are allowed to change. They aren't interchangeable.

Failure you observeObjective to investigateWhy
Base model has weak exposure to domain vocabulary in unlabeled corporacontinued pretrainingnext-token training on domain text adapts the language distribution before behavior training[4]
Model can read an access policy but doesn't follow the desired escalation procedureSFTprompt-completion demonstrations directly teach that response behavior[1]
Several acceptable answers need ranking by preferencepreference training; compare against the current model or an SFT baselinecomparisons express relative preference rather than a single target answer[5]

Once the objective is SFT, make a second decision: which part of the model can carry the new behavior?

SFT parameterizationWhat moves during the same supervised objectiveWhen to test it
Full fine-tuningall trainable model weightswhen memory permits and adapters may be too restrictive
small low-rank adapter matrices; base weights stay frozenwhen iteration speed, memory, or many task variants matter[6]
QLoRALoRA adapters while the frozen base is stored in 4-bit formwhen the base model doesn't fit comfortably at higher precision[7]

LoRA and aren't alternatives to SFT. They're parameterizations usable for an SFT run. Continued pretraining can also use adapters. Both stages can use causal next-token cross-entropy; their training distributions and supervised targets differ. Calling every adapter run SFT would hide that distinction.

Full weights or adapters

Full fine-tuning

For our response-only objective, full fine-tuning updates all eligible model parameters. Test it when weights, gradients, , activations, and temporary buffers fit, or when an adapter baseline appears capacity-limited. More trainable parameters permit a broader update, but don't establish better held-out behavior. Full fine-tuning doesn't require response-only labels by definition; that is a separate objective choice.

LoRA

LoRA freezes selected base weights and trains low-rank adapter . Target modules and rank determine the trainable count. The original paper's GPT-3 175B comparison reported up to 10,000 times fewer trainable parameters and three times lower GPU memory requirements, not universal 99% and 70% savings.[6] Activations and frozen weights still consume memory. Measure the actual run; adapters are useful candidates when many variants share one base or full updates are too costly.

QLoRA

QLoRA reduces frozen-base storage while retaining adapter updates. Its paper uses 4-bit NormalFloat (NF4), quantized scaling constants, and paged optimizers. The base weights are dequantized to a computation dtype such as for forward and backward operations; adapter gradients don't mean the stored 4-bit base receives weight updates.[7] Check your implementation's storage, compute, and adapter dtypes separately. Compare held-out behavior under measured memory and time budgets.

Reject broken demonstrations before training

For this pipeline, store a prompt, a desired completion, and a group key for -resistant splitting on every SFT row. A missing completion isn't harmless metadata: it's a training example with no answer to teach.

Before you run the validator, predict its decision for each row. T102 has an answer and a group key. B550 has an empty answer. The last row has an answer but no group key. Only one row should reach tokenization.

validate_sft_rows.py
rows = [
    {"thread_id": "T102", "prompt": "Stale key", "completion": "Open a rotation ticket; keys older than 14 days need review."},
    {"thread_id": "B550", "prompt": "Expired key", "completion": ""},
    {"prompt": "Privileged role change", "completion": "Escalate privileged-role changes."},
]

required = {"thread_id", "prompt", "completion"}
accepted, rejected = [], []
for index, row in enumerate(rows):
    missing = sorted(required - row.keys())
    if missing:
        rejected.append((index, f"missing {missing}"))
    elif any(not isinstance(row[key], str) or not row[key].strip() for key in required):
        rejected.append((index, "required fields must be nonempty strings"))
    else:
        accepted.append(row)

print("accepted_threads=", [row["thread_id"] for row in accepted])
print("rejected=", rejected)
Row contract output
accepted_threads= ['T102']
rejected= [(1, 'required fields must be nonempty strings'), (2, "missing ['thread_id']")]

The output makes the contract concrete: T102 is accepted, B550 is rejected before it can become a prompt-only example, and the last row is rejected before it can contaminate a grouped split. Next, keep accepted rows from the same deployment unit together.

Split by the unit that could leak

Before formatting rows, reserve evaluation examples that training can't imitate through near duplicates. For a policy assistant, several messages from one access incident or one policy document often share facts and phrasing. Randomly splitting individual messages can put one part of the same case in training and another in evaluation.

Group by the unit your evaluation claim requires to be unseen: incident thread, policy document, tenant, or time period. Low token overlap doesn't make two turns from T102 independent. If several group identifiers connect cases, keep their connected groups together. For a chronological holdout, train on earlier periods and check windows that straddle the cutoff. The check below demonstrates thread separation only.

Choose the grouping from the deployment claim. Thread grouping tests new threads, not unseen tenants or policy documents shared by those threads. Deduplicate across groups, and reserve separate validation and final-test groups. Use validation for sweeps and checkpoint selection; inspect the final test only after freezing that choice. Repeatedly tuning on the test turns it into another validation set.

Predict the three printed sets before reading the output. T102 should stay with P771 in training, R550 should be held out, and the intersection should be empty.

grouped_sft_split.py
rows = [
    {"case_id": "T102", "turn": 1, "answer": "Open a rotation ticket; keys older than 14 days need review."},
    {"case_id": "T102", "turn": 2, "answer": "Keep the review open until security approves the rotation."},
    {"case_id": "R550", "turn": 1, "answer": "Escalate privileged-role changes."},
    {"case_id": "P771", "turn": 1, "answer": "Cite the session-timeout policy."},
]

eval_cases = {"R550"}
train = [row for row in rows if row["case_id"] not in eval_cases]
evaluation = [row for row in rows if row["case_id"] in eval_cases]

train_cases = {row["case_id"] for row in train}
held_out_cases = {row["case_id"] for row in evaluation}
assert train_cases.isdisjoint(held_out_cases)

print("train_cases=", sorted(train_cases))
print("eval_cases=", sorted(held_out_cases))
print("case_overlap=", train_cases & held_out_cases)
Grouped split output
train_cases= ['P771', 'T102']
eval_cases= ['R550']
case_overlap= set()

Data path: template, tokenize, label, pack

Follow one row all the way through the data path. Structured conversation dictionaries must pass through a Jinja2 , undergo , receive target label masks, and either get packed into fixed-length blocks or padded into batches.

Use the checkpoint's actual formatting contract. ChatML-style markers such as <|im_start|> aren't universal; templates can mix special IDs with ordinary text, spaces, or newlines. Subword splitting doesn't by itself prove a broken role boundary. Inspect the rendered template, its token IDs, and assistant masks together rather than registering invented markers and assuming the pretrained model understands them.[8]

For complete training conversations, use add_generation_prompt=False. To start a new assistant reply, add_generation_prompt=True adds a header when the template supports it; continuing a partial assistant message is a different operation. Align the template's turn terminator with serving stop settings. Missing terminators can weaken stopping supervision, but don't prove inevitable endless generation. If you format text first and tokenize later, avoid adding duplicate special tokens.[9]

Version this entire data path together. Delimiter, whitespace, or template changes can alter token IDs or labels. Record the change and rerun batch checks; previous runs remain evidence about their recorded formatting.

Predict which positions in T102 should teach the model before inspecting the labels below. The system instruction, user request, and delimiter tokens must remain visible so the response can condition on them. However, only the target answer and its turn terminator should contribute to cross-entropy loss. We set ignored positions to -100, the standard ignore_index in loss.

sft_loss_mask.py
IGNORE_INDEX = -100

tokens = [
    ("prefix", "<system>"),
    ("prefix", "Follow access policy."),
    ("prefix", "<user>"),
    ("prefix", "Key T102 is 18 days old."),
    ("prefix", "<assistant>"),
    ("target", "Open a rotation ticket; keys older than 14 days need review."),
    ("target", "<end_of_turn>"),
]

labels = [
    token if span == "target" else IGNORE_INDEX
    for span, token in tokens
]

supervised = [
    token
    for (_, token), label in zip(tokens, labels)
    if label != IGNORE_INDEX
]
print("supervised_tokens=", supervised)
print("masked_positions=", sum(label == IGNORE_INDEX for label in labels))
Loss mask output
supervised_tokens= ['Open a rotation ticket; keys older than 14 days need review.', '<end_of_turn>']
masked_positions= 5

These seven entries are readable spans rather than real subword token IDs. A real sentence expands into multiple integer IDs. The five prefix spans provide context without generating loss; the answer and terminator spans identify what to label after tokenization.

Causal language models shift targets by one position: logits at position t−1t-1 predict the label at position tt. The hidden state at the assistant marker <assistant> predicts the first answer token. Setting a label to -100 removes that token's direct cross-entropy loss, but it doesn't remove gradients flowing through attention layers into earlier prompt representations. Response loss still updates the weights that process the prompt. Mask padding tokens by position, not by blindly masking every EOS ID when EOS doubles as padding.[10]

Don't confuse the label mask with the attention mask. The label mask decides which positions contribute to the loss scalar. The attention mask decides which earlier positions a token can read during the forward pass. Prompt tokens must remain visible as context even when their labels are -100; packed sequences also need attention boundaries so adjacent examples can't attend across boundaries.

The CPU check below uses real integer labels and PyTorch gradients. With all logits set to zero over a four-token vocabulary, each supervised target has cross-entropy loss log⁡4≈1.3863\log 4 \approx 1.3863. Only logit rows 1 and 2 predict the labeled answer and stop positions. Most training frameworks handle this target shift internally; don't shift labels twice.

shifted_sft_targets.py
import math
import torch
import torch.nn.functional as F

# Positions: prompt, assistant marker, answer, stop.
labels = torch.tensor([-100, -100, 2, 3])
logits = torch.zeros(4, 4, dtype=torch.float64, requires_grad=True)
targets = labels[1:]
assert (targets != -100).any(), "Reject an all-ignored target batch"
loss = F.cross_entropy(logits[:-1], targets, ignore_index=-100)
loss.backward()
active_rows = torch.nonzero(logits.grad.abs().sum(dim=1)).flatten().tolist()
assert active_rows == [1, 2]
assert math.isclose(loss.item(), math.log(4))
assert (torch.full_like(targets, -100) != -100).sum().item() == 0
print("logit_rows_with_direct_loss=", active_rows)
print(f"mean_target_loss={loss.item():.6f}")
Shifted target check
logit_rows_with_direct_loss= [1, 2]
mean_target_loss=1.386294

This verifies shifted loss and direct logit gradients. Test those mechanisms on collated tensors before launching long GPU jobs.

Why mask prompt targets here? We want a response-only objective and a metric that measures the response. With 800 scored prompt positions and 30 response positions, prompt targets occupy about 96% of a full-sequence mean's terms. That isn't 96% of the gradient: individual losses and their parameter derivatives can differ greatly. Nor does full-sequence training inevitably cause echoing or degraded style.

Huerta-Enochian and Ko studied prompt-loss weights using LLaMA 1/2 7B and Alpaca-derived datasets. Fractional weights improved some short-completion benchmarks in that setting; the study attributed the effect to regularization.[11] Response-only masking is this run's deliberate choice. Compare objectives on the same held-out tasks before changing it.

Predict which number exposes the metric difference below. The supplied prefix losses are small, so adding them lowers the full-sequence average without improving the supplied answer losses:

response_only_loss.py
prefix_losses = [0.03, 0.04, 0.02, 0.05, 0.03]
answer_losses = [2.20, 1.80]

full_sequence_loss = sum(prefix_losses + answer_losses) / len(prefix_losses + answer_losses)
response_only_loss = sum(answer_losses) / len(answer_losses)

print(f"full_sequence_loss={full_sequence_loss:.3f}")
print(f"response_only_loss={response_only_loss:.3f}")
assert response_only_loss > full_sequence_loss
Response-only loss output
full_sequence_loss=0.596
response_only_loss=2.000

Both numbers measure token-level cross-entropy over different target sets. In this authored fixture, five low-loss prefix positions lower the combined average to 0.5960.596 while answer loss remains 2.0002.000. For our response-only objective, the loss is mean negative log-likelihood over the labeled target set TT:

LSFT(θ)=−1∣T∣∑t∈Tlog⁡pθ(yt∣y<t)\mathcal{L}_{\text{SFT}}(\theta) = -\frac{1}{|T|}\sum_{t \in T} \log p_\theta(y_t \mid y_{<t})

θ\theta represents the trainable parameters, yty_t the target token, and y<ty_{<t} its available prefix. Here TT has two positions. Default unweighted, mean-reduced PyTorch cross-entropy with ignore_index=-100 averages only active targets. Changing the mask changes the objective; compare separately reported response loss rather than treating lower full-sequence loss as evidence of better answers.

You rarely write this mask by hand, but check which format you loaded. As of TRL 1.17, SFTTrainer uses completion-only loss by default for prompt-completion datasets. A plain messages dataset scores the full sequence unless you set assistant_only_loss=True, which needs generation markers in the chat template. TRL can patch those in for known model families.[1] Always inspect a tokenized batch before launching training: confirm that only answer and stop tokens carry non--100 labels.

Inspect masks after truncation and causal shifting. Keeping only the start can remove the completion. A mean-reduced PyTorch cross-entropy batch with no active shifted targets returns NaN; an individual empty row in a mixed batch instead adds context and cost without response supervision. Count dropped or empty rows so dataset loss isn't hidden.

As of TRL 1.17, TRL's preparation path filters fully masked rows after nonpacked truncation. Its packing path needs a separate audit.[12] Custom collators or skipped preparation can change this behavior. Check active labels in the actual collated tensors rather than assuming every empty row reaches the loss.

When sequence packing is active, this truncation step is skipped and overflow is handled by the packing strategy instead; bfd discards overflow tokens.[13] Predict what the guard below does when five prompt tokens consume the length budget:

reject_truncated_completion.py
IGNORE_INDEX = -100

prompt_tokens = ["<system>", "policy", "<user>", "stale_key", "<assistant>"]
answer_tokens = ["open_ticket", "<end_of_turn>"]
max_length = 5

kept_tokens = (prompt_tokens + answer_tokens)[:max_length]
labels = [
    token if position >= len(prompt_tokens) else IGNORE_INDEX
    for position, token in enumerate(kept_tokens)
]
has_answer_label = any(label != IGNORE_INDEX for label in labels)

# A label at position zero has no preceding logit in this isolated sequence.
first_only_labels = [7, IGNORE_INDEX]
has_shifted_target = any(label != IGNORE_INDEX for label in first_only_labels[1:])
assert any(label != IGNORE_INDEX for label in first_only_labels)
assert not has_shifted_target

print("kept_tokens=", kept_tokens)
print("has_answer_label=", has_answer_label)
print("decision=", "keep" if has_answer_label else "reject_or_increase_max_length")
print("first_position_label_has_shifted_target=", has_shifted_target)
Truncation guard output
kept_tokens= ['<system>', 'policy', '<user>', 'stale_key', '<assistant>']
has_answer_label= False
decision= reject_or_increase_max_length
first_position_label_has_shifted_target= False

Pack short examples while preserving their boundaries

Short demonstrations can leave many unused token slots when padded to a large fixed length. The fraction of padded slots isn't a measured compute-waste percentage: dynamic batch padding, length grouping, and attention kernels change the work performed. Packing is another efficiency option. For this run's independent-example objective, preserve each example's attention and loss boundaries.

For independent demonstrations, a reply for case R550 shouldn't condition on T102 merely because the loader put them in one block. Position IDs block attention only if something turns them into a mask. As of Transformers 5.22, models that use its shared causal-mask helper, such as Llama, do that under eager and SDPA attention when you restart the IDs at each example, pass no attention mask, and run with use_cache=False. An all-ones mask or a live cache silently removes the isolation, so set use_cache=False in a hand-written loop. Trainer and gradient checkpointing already turn the cache off during training.[14][15] Otherwise, enforce a causal block-diagonal pattern using an explicit mask or a kernel's segment metadata. The predicate below describes the intended edges; it doesn't execute an attention kernel:

packed_boundary_check.py
segments = [
    ["T102 prompt", "T102 response", "<eos>"],
    ["R550 prompt", "R550 response", "<eos>"],
]

flat = [(sequence_id, token) for sequence_id, row in enumerate(segments) for token in row]
position_ids = [position for row in segments for position, _ in enumerate(row)]

def may_attend(query_index, key_index):
    query_sequence, _ = flat[query_index]
    key_sequence, _ = flat[key_index]
    return key_index <= query_index and query_sequence == key_sequence

r550_response = 4
assert position_ids == [0, 1, 2, 0, 1, 2]
assert not may_attend(r550_response, 1)
print("position_ids=", position_ids)
print("R550_response_sees_T102_response=", may_attend(r550_response, 1))
Packed boundary output
position_ids= [0, 1, 2, 0, 1, 2]
R550_response_sees_T102_response= False

Reset position IDs isolate examples only when the attention path turns them into a segment mask, as described above. The kernel must enforce those boundaries during computation. Resetting positions with no segment mask in the kernel causes cross-example contamination:

Two six-by-six attention masks for packed examples T102 and R550, whose position IDs each reset to 0, 1, and 2. In the block-causal mask, query 4 reads only keys 3 and 4, with values 10 and 20, so its output is 15.0 and stays 15.0 after adding 100 to T102. In the global causal mask, query 4 also reads T102, and its output moves from 7.2 to 67.2. Gold lines mark the boundary between the two examples.
Gold lines mark the boundary between T102 and R550. The R550 query at index 4 reads keys 3 and 4, whose values average to 15.0.
IndexTokenposition_idIntended attend-toBroken path (no segment mask)
0T102 prompt0T102 onlyT102 only
1T102 response1T102 onlyT102 only
2T102 eos2T102 onlyT102 only
3R550 prompt0R550 onlyT102 + R550 (cross-example leak)
4R550 response1R550 onlyT102 + R550 (R550 answer conditions on T102)
5R550 eos2R550 onlyT102 + R550

Test the effect inside an actual attention operation, too. In this CPU fixture, zero queries and keys give uniform attention over permitted positions. The R550 response is at index 4. Changing only T102's three values should leave its output unchanged under the block mask, but change it under global causal attention. PyTorch SDPA Boolean masks use True for permitted edges.[16]

test_packed_attention_isolation.py
import torch
import torch.nn.functional as F

segment_ids = torch.tensor([0, 0, 0, 1, 1, 1])
causal = torch.ones(6, 6, dtype=torch.bool).tril()
isolated = causal & (segment_ids[:, None] == segment_ids[None, :])
queries = keys = torch.zeros(1, 1, 6, 1, dtype=torch.float64)
values = torch.tensor([1., 2., 3., 10., 20., 30.], dtype=torch.float64).view(1, 1, 6, 1)
changed = values.clone()
changed[:, :, :3] += 100

def r550_output(mask, value):
    return F.scaled_dot_product_attention(
        queries, keys, value, attn_mask=mask, dropout_p=0.0,
    )[0, 0, 4, 0].item()

before = r550_output(isolated, values)
after = r550_output(isolated, changed)
leaking = r550_output(causal, values)
leaking_after = r550_output(causal, changed)
assert before == after == 15.0
assert abs(leaking - 7.2) < 1e-12
assert abs(leaking_after - 67.2) < 1e-12
print(f"Isolated R550: {before:.1f} -> {after:.1f}")
print(f"Global causal R550: {leaking:.1f} -> {leaking_after:.1f}")
Packed attention perturbation
Isolated R550: 15.0 -> 15.0
Global causal R550: 7.2 -> 67.2

R550 averages its own values 10 and 20 to obtain 15. Global attention averages all five earlier-or-current values to obtain 7.2; adding 100 to three of them increases the result by 60. This verifies the supplied CPU mask, not TRL's GPU packing kernel. Apply the same perturbation idea to the collated model path with dropout disabled, alongside checking segment-start labels.

Packing can also fail when an example gets truncated before being packed. If T102 survives truncation with every response token stripped away, packing still incorporates the prefix to fill the block, but that segment contributes zero learning signal. Inspect segment boundaries and verify non-zero answer labels before starting a training run.

TRL's bfd packs examples after truncating overlong ones; bfd_split splits overlong sequences into chunks; wrapped concatenates a stream and cuts token sequences.[13] Preserving stored tokens through splitting doesn't preserve every completion's original prompt context or every trainable target. Choose the strategy from the learning objective, then inspect the resulting segments.

As of TRL 1.17, the trainer enables padding-free mode for both bfd and bfd_split. wrapped does not enable padding-free mode by default. Its continuous stream does not retain the original independent-example boundaries; with the default padded collator, examples can read each other and joins are scored.[12] Releases before 1.16 also warned when the attention implementation wasn't a FlashAttention variant. That warning is gone, so don't wait for one.[17] An ordinary global causal kernel can read across examples. Disable packing while debugging if necessary; verify boundary reachability before enabling it. A warning or successful forward pass isn't a boundary test.

Attention isolation alone also isn't enough. A globally shifted loss must not ask the last token of one independent segment to predict the first token of the next. As of TRL 1.17, the padding-free collator masks labels wherever segment position IDs reset to zero.[12] Audit that loss boundary as well as the attention boundary.

Packing changes loader-item counts and can remove or split targets. A fixed max_steps can therefore imply a different number of data passes, without proving overfitting. Log processed input tokens, active shifted targets, and retained original examples. Throughput and learning exposure use different denominators.

Evaluate hyperparameters through controlled experiments

A learning-rate sweep asks how much change this objective and parameterization can tolerate. Check task improvement and regressions on general capabilities together. Neither a frozen base nor a small learning rate guarantees preserved behavior.

The fragment below uses 1e-4 as an adapter candidate. That isn't a universal safe range or a fixed multiplier over full fine-tuning. Compare nearby rates under the same data exposure and evaluation rules. Dataset quality, optimizer, adapter scaling, model size, and loss normalization across microbatches all affect the result.

KnobBaseline to testWhat decides whether it survives
a short sweep for the actual update surfaceheld-out task gains and regressions at matched exposure
Scheduleexplicit warmup duration, decay shape, and final LRobserved early updates and behavior at the chosen checkpoint
Passes over dataa bounded pilot with measured retained targetsvalidation trends and task behavior, not a fixed epoch ceiling
Regularization & clippingrecord decay groups and clipping thresholdcompare changes in gradients, parameter updates, and held-out behavior
PrecisionBF16 where supported; otherwise choose an explicit pathfinite numerics and task parity under the actual hardware

Warmup reduces early step sizes; it doesn't make AdamW's bias correction unnecessary or guarantee stability. A useful historical counterexample is InstructGPT's standalone SFT baseline: 16 epochs, no warmup, and cosine decay to 10% of its initial LR, with learning rates and epochs selected by searches.[3] That recipe isn't a recommendation to copy 16 epochs. It disproves a mandatory 3%-10% warmup or 1-3 epoch rule.

A cosine schedule's floor is configurable; standard Transformers cosine warmup decays to zero, while a minimum-LR variant can use a nonzero floor.[18] Resolve the actual scheduler instead of assuming 10%. AdamW applies decoupled decay separately from moment estimation. Decay doesn't bound distance from the pretrained weights, and clipping raw gradients doesn't directly bound AdamW's parameter update.[19] Inspect both gradients and update norms.

Effective batch size is also a token budget

The batch size configured on a single device isn't the batch size the optimizer sees during an update step. For synchronous training across complete accumulation windows, the number of items per update follows:

text
loader_items_per_update = per_device_batch * gradient_accumulation_steps * world_size

Without packing, a loader item is one demonstration. With packing, each item is a composite block containing multiple short demonstrations, so the formula counts packed blocks rather than raw examples. Track both raw examples and actual supervised tokens per update. The CPU check below models an unpacked dataset across a four-GPU setup:

sft_effective_batch.py
per_device_batch = 4
grad_accum = 8
world_size = 4
examples_per_update = per_device_batch * grad_accum * world_size

supervised_tokens_per_microbatch_on_one_rank = [180, 212, 164, 220, 198, 204, 175, 207]
supervised_tokens_per_update = sum(supervised_tokens_per_microbatch_on_one_rank) * world_size

print(f"examples_per_update={examples_per_update}")
print(f"supervised_tokens_per_update={supervised_tokens_per_update}")
Effective batch output
examples_per_update=128
supervised_tokens_per_update=6240

Always calculate the combined impact of batch changes rather than reasoning about single knobs. Notice how doubling world size or accumulation doubles the update volume:

compare_update_sizes.py
def examples_per_update(per_device_batch, accumulation, world_size):
    return per_device_batch * accumulation * world_size

run_a = examples_per_update(4, 8, 4)
run_b = examples_per_update(2, 16, 8)

print("run_a_examples_per_update=", run_a)
print("run_b_examples_per_update=", run_b)
print("ratio_b_over_a=", run_b / run_a)
assert run_b == 2 * run_a
Run comparison output
run_a_examples_per_update= 128
run_b_examples_per_update= 256
ratio_b_over_a= 2.0

When sharing batch configurations, specify accumulation steps, world size, sequence packing mode, and supervised token counts.

Loss weighting across accumulation steps matters as well. Suppose one microbatch has one target loss of 4.0 and another has nine target losses of 1.0 each. Averaging batch means gives 2.5; the token mean is (4.0+9.0)/10=1.3(4.0+9.0)/10=1.3. Equal example weights and equal token weights are different objectives. For the token mean, normalize loss sums by active shifted targets across the complete update window.

token_weighted_accumulation.py
microbatches = [[4.0], [1.0] * 9]
mean_of_means = sum(sum(batch) / len(batch) for batch in microbatches) / len(microbatches)
token_mean = sum(map(sum, microbatches)) / sum(map(len, microbatches))
assert mean_of_means == 2.5 and token_mean == 1.3
print(f"mean_of_means={mean_of_means:.1f}; token_mean={token_mean:.1f}")
Token-weighted accumulation
mean_of_means=2.5; token_mean=1.3

In ordinary distributed data parallel training, rank gradients are averaged. If each rank divides its local loss sum by a globally summed token count, that averaging introduces another factor of world_size. Compensate for it, or use a trainer that performs the normalization correctly. Don't add the factor twice when the trainer already handles it. Unequal target counts, partial accumulation windows, and actual loss gradients need checks; the scalar-loss fixture above doesn't execute distributed training.

What a resumable run must save

A model.safetensors file doesn't encode the optimizer history or next data batch. It may be part of an inference artifact, but serving also needs compatible architecture and tokenization. An adapter export additionally needs its base model and adapter configuration.

To reconstruct the continuation, retain the state your recipe actually uses:

  • model weights or LoRA adapter weights
  • optimizer state (AdamW first moment mtm_t, second moment vtv_t, and step counters)
  • scheduler state (step counter, warmup phase, decay multiplier)
  • global step and completed epoch indices
  • sampler progress and random generator states, not only original seeds
  • tokenizer revision and chat-template hash
  • resolved training configuration and dataset manifests
  • best validation metric achieved so far
  • loss-scaler state if using FP16 dynamic scaling, plus any other stateful training machinery

Optimizer history determines direction and scaling. Scheduler progress determines the next LR, which needn't be the peak if reset incorrectly. Sampler and generator states determine upcoming data and randomness. Preserve the exact tokenizer/template artifacts and the adapter's base revision. Metadata labels or filenames alone don't guarantee identical contents or complete worker state.

For ordinary AdamW without AMSGrad, the update uses newly computed, bias-corrected moments:

mt=β1mt−1+(1−β1)gt,vt=β2vt−1+(1−β2)gt2m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\qquad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 m^t=mt1−β1t,v^t=vt1−β2t\widehat m_t=\frac{m_t}{1-\beta_1^t},\qquad \widehat v_t=\frac{v_t}{1-\beta_2^t} θt=(1−ηtλ)θt−1−ηtm^tv^t+ϵ\theta_t=(1-\eta_t\lambda)\theta_{t-1}-\eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon}

Here gtg_t is the current gradient and squaring is elementwise. A fresh optimizer with nonzero g1g_1 first computes m^1=g1\widehat m_1=g_1 and v^1=g12\widehat v_1=g_1^2. Its update doesn't divide the current gradient by epsilon alone. Dropping state changes the trajectory, but doesn't imply a mandatory spike.[19]

Predict the direction after gradients 2.0, then -1.0. A fresh AdamW step follows the negative gradient upward, while retained positive momentum can still move the weight downward:

resume_actual_adamw.py
import copy
import math
import torch

weight = torch.nn.Parameter(torch.tensor(1.0, dtype=torch.float64))
optimizer = torch.optim.AdamW([weight], lr=0.1, weight_decay=0.0)

def update(parameter, opt, gradient):
    opt.zero_grad(set_to_none=True)
    parameter.grad = torch.tensor(gradient, dtype=torch.float64)
    opt.step()

update(weight, optimizer, 2.0)
saved_weight = weight.detach().clone()
saved_optimizer = copy.deepcopy(optimizer.state_dict())
update(weight, optimizer, -1.0)

resumed = torch.nn.Parameter(saved_weight.clone())
resumed_optimizer = torch.optim.AdamW([resumed], lr=0.1, weight_decay=0.0)
resumed_optimizer.load_state_dict(saved_optimizer)
update(resumed, resumed_optimizer, -1.0)

fresh = torch.nn.Parameter(saved_weight.clone())
fresh_optimizer = torch.optim.AdamW([fresh], lr=0.1, weight_decay=0.0)
update(fresh, fresh_optimizer, -1.0)
assert torch.equal(resumed, weight)
assert resumed.item() < saved_weight.item() < fresh.item()
assert math.isclose(fresh.item() - saved_weight.item(), 0.1, abs_tol=1e-8)
print(f"uninterrupted={weight.item():.6f}; resumed={resumed.item():.6f}")
print(f"fresh_optimizer={fresh.item():.6f}; all_finite={bool(torch.isfinite(fresh))}")
Actual AdamW continuation
uninterrupted=0.873366; resumed=0.873366
fresh_optimizer=1.000000; all_finite=True

This controlled CPU experiment restores optimizer history while supplying the same next gradient; it doesn't replay a stochastic LLM job. Prefer saving at completed optimizer-step boundaries. Mid-accumulation continuation also needs partial gradients and microbatch progress. Compare the next batch, LR, loss, and update against an uninterrupted reference. Bitwise parity depends on deterministic operations and a compatible environment.[20][21]

A weight number line from 0.85 to 1.05. Gradient plus 2 moved the weight from 1.000000 to 0.900000 and stored moments 0.200000 and 0.004000 at step 1. Both lanes start at 0.900000 and receive the same next gradient, minus 1, at learning rate 0.1. With restored moments the weight moves down to 0.873366. With a fresh optimizer it moves up to 1.000000. The CPU example isolates optimizer history; real continuation also needs scheduler, sampler, RNG, and artifact identity.
Restoring AdamW moments continues the saved parameter along the uninterrupted CPU path. The comparison holds the next gradient and recipe fixed; full training also needs scheduler, sampler, RNG and artifact identity.

The simpler SGD-with-momentum fixture below isolates the same dependency: restoring velocity gives a next weight of 0.52; resetting it gives 0.70. The difference is 0.18, about 34.6% of 0.52. These are scalar update values, not an AdamW variance calculation or a measured LLM loss jump.

A manifest can list required continuation artifacts. This schema fixture checks key presence only; it doesn't open files, validate hashes, or demonstrate a successful resume:

resume_bundle_manifest_check.py
required = {
    "model_state",
    "optimizer_state",
    "scheduler_state",
    "global_step",
    "rng_state",
    "sampler_state",
    "tokenizer_version",
    "chat_template_version",
    "data_manifest",
    "training_config",
    "eval_manifest",
    "best_metric",
}

resume_bundle = {
    "model_state": "weights/step_600.safetensors",
    "optimizer_state": "optimizer/step_600.pt",
    "scheduler_state": "scheduler/step_600.pt",
    "global_step": 600,
    "rng_state": "rng/step_600.pt",
    "sampler_state": {"epoch": 1, "batches_consumed": 120},
    "tokenizer_version": "policy-sft-tokenizer-v3",
    "chat_template_version": "llama3-access-policy-v2",
    "data_manifest": "data/sft_manifest_2026-05-20.json",
    "training_config": "config/resolved_training.json",
    "eval_manifest": "eval/access_policy_behavior_v4.json",
    "best_metric": {"name": "policy_pass_rate", "value": 0.97},
}

missing = sorted(required - resume_bundle.keys())
print("schema_complete=", not missing)
print("best_metric=", resume_bundle["best_metric"])
Checkpoint manifest output
schema_complete= True
best_metric= {'name': 'policy_pass_rate', 'value': 0.97}

The scalar SGD-with-momentum fixture below makes the dependency on velocity explicit. Its fixed next gradient isolates optimizer state; no training data or model loss is evaluated:

resume_next_update.py
import json

def step(state, gradient):
    velocity = 0.9 * state["velocity"] + gradient
    return {"weight": state["weight"] - 0.1 * velocity,
            "velocity": velocity, "step": state["step"] + 1}

saved = step({"weight": 1.0, "velocity": 0.0, "step": 0}, 2.0)
reference = step(saved, 1.0)
restored = json.loads(json.dumps(saved))
resumed = step(restored, 1.0)
weights_only = step({**restored, "velocity": 0.0}, 1.0)
assert resumed == reference and weights_only != reference
print(f"uninterrupted={reference['weight']:.2f}; resumed={resumed['weight']:.2f}")
print(f"reset_momentum={weights_only['weight']:.2f}")
Resume next-update check
uninterrupted=0.52; resumed=0.52
reset_momentum=0.70

Choosing the best checkpoint

Choose checkpoints against the deployment objective on validation data. Teacher-forced loss measures likelihood of reference tokens given reference prefixes; it doesn't directly measure whether a freely generated reply opens the required ticket. Generate outputs for task evaluation too, and keep a final test separate from repeated selection.

Unreliable selection criteria:

  • lowest training loss alone (doesn't measure unseen-case performance)
  • latest training step (risks overfitting or late degradation)
  • file size or raw token count

Rigorous selection criteria:

  • strict syntax and format pass rate (JSON schema compliance, valid function call formatting)
  • domain policy pass rate on held-out cases via deterministic rule verifiers or calibrated judges
  • human evaluation on blinded, randomized completion pairs
  • validation loss with a declared role in selection, rather than an assumed universal priority

For the authored selection fixture below, require format validity of at least 98%, then maximize policy compliance and break ties with validation loss, followed by the earlier step. These rates and this threshold are supplied data and policy choices, not trained-model results or a universal release standard. In a real evaluation, report denominators, uncertainty, important failure slices, and regressions on general behavior.

select_sft_checkpoint.py
checkpoints = [
    {"step": 200, "val_loss": 1.91, "policy_pass_rate": 0.93, "format_pass_rate": 0.99},
    {"step": 400, "val_loss": 1.77, "policy_pass_rate": 0.91, "format_pass_rate": 1.00},
    {"step": 600, "val_loss": 1.79, "policy_pass_rate": 0.97, "format_pass_rate": 0.98},
]

def select_checkpoint(rows):
    eligible = [row for row in rows if row["format_pass_rate"] >= 0.98]
    return max(eligible, key=lambda row: (row["policy_pass_rate"], -row["val_loss"], -row["step"]), default=None)

best = select_checkpoint(checkpoints)
assert best is not None
assert select_checkpoint([]) is None
assert select_checkpoint([{**checkpoints[0], "format_pass_rate": 0.97}]) is None
assert select_checkpoint([{**checkpoints[0], "step": 600}, checkpoints[0]])["step"] == 200
assert select_checkpoint([{**checkpoints[0], "val_loss": 2.0}, {**checkpoints[0], "step": 400}])["step"] == 400

print("best_step=", best["step"])
print("best_policy_pass_rate=", best["policy_pass_rate"])
print("lowest_loss_step=", min(checkpoints, key=lambda row: row["val_loss"])["step"])
Checkpoint selection output
best_step= 600
best_policy_pass_rate= 0.97
lowest_loss_step= 400

Step 400 shows the lowest validation loss (1.771.77), but step 600 achieves the highest policy compliance rate (0.970.97) while clearing the format threshold (0.980.98). The selection gate picks step 600. If no checkpoint meets the format threshold, the gate yields None, signaling that the recipe needs fixing before anything ships.

Three checkpoints, steps 200, 400, and 600, are rows plotted against three aligned scales. All clear the inclusive 0.98 format gate with 0.99, 1.00, and 0.98. Policy pass rates are 0.93, 0.91, and 0.97, so step 600 wins. Validation losses are 1.91, 1.77, and 1.79. Step 400 is lowest, but validation loss only breaks a policy tie.
All three checkpoints clear the inclusive 0.98 format gate. Policy pass rate selects step 600; validation loss breaks only a policy tie.

Single GPU first, then scale out

Don't start debugging thread T102 across an eight-GPU cluster. If prompt templating or label masking contains subtle bugs, scaling out distributes those defects across collective communication calls rather than resolving them. Prove your pipeline on a single device first:

  1. single GPU execution
  2. small, representative evaluation split
  3. frequent checkpointing
  4. verified chat templating and response-only label masking
  5. clear task compliance metric

Validate the smallest feasible run before adding scale. Some required models can't fit one device at all; debug data contracts with small fixtures, then use the minimum necessary sharding. Test distributed loss normalization and checkpoint restore independently of the single-device run.

Signs that one GPU is still fine

A single device is feasible when weights, gradients, optimizer state, activations, and temporary buffers fit at the chosen precision and the run meets its time budget. A model that fits for inference may still fail this training-memory test.

Signs that you need FSDP or ZeRO

Consider FSDP or ZeRO when persistent model weights, gradients, and AdamW optimizer states exceed available GPU memory. Measure activation memory independently: sharding model parameters doesn't automatically eliminate activation overhead. Longer contexts often require activation checkpointing, reduced micro-batch sizes, or context parallelism. Adding more GPUs introduces network communication latency and won't speed up an improperly configured small job.

FSDP and ZeRO offer different state-sharding strategies; the chosen stage determines what is sharded.[22][23] Specify precision explicitly. As of TRL 1.17, TRL defaults to BF16 when FP16 isn't set; verify hardware support and numerical behavior on the target stack.[1]

Minimal operational skeleton

The row still follows the same path, whichever library applies the update:

sft_pipeline_shape.txt
dataset -> template -> tokenizer -> collator/mask -> trainer
       -> periodic eval -> checkpoint save -> best-checkpoint export

TRL, torchtune, and the Alignment Handbook implement variations of this workflow.[1][24][25]

The configuration fragment below is an unexecuted integration blueprint for conversational prompt-completion rows. Its pilot disables packing until boundaries are verified. API fields were checked against TRL 1.17.1 on November 4, 2026; pin a tested package version and model revision for a real run.

trl_sft_config_fragment.py
from trl import SFTConfig, SFTTrainer

args = SFTConfig(
    output_dir="runs/access-policy-sft-lora",
    learning_rate=1e-4,              # adapter baseline; tune for the target task
    max_length=1024,
    bf16=True,                     # only on supported hardware; verify numerics
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    max_steps=200,                  # authored pilot budget, not a recommended duration
    lr_scheduler_type="linear",    # explicit schedule; reaches zero at the budget endpoint
    warmup_steps=0,                 # authored candidate to compare, not a stability promise
    seed=42,
    data_seed=42,
    packing=False,                 # enable only after attention and loss-boundary checks
    eval_packing=False,              # keep validation examples separate
    completion_only_loss=True,       # train on the completion field, not the prompt
    loss_type="nll",                 # explicit standard loss for this adapter fragment
    eval_strategy="steps",
    eval_steps=100,
    save_steps=100,
)

trainer = SFTTrainer(
    model=model_with_lora_adapters,
    args=args,
    train_dataset=train_rows,        # prompt/completion are role/content message lists
    eval_dataset=held_out_rows,
    processing_class=tokenizer,
)

This fragment omits model loading, adapter initialization, and dataset construction. Convert the earlier authoring rows to message lists so the trainer can apply the checkpoint's template; raw string fields don't automatically become a chat conversation. We explicitly choose loss_type="nll", which is also the current default. TRL now documents "chunked_nll" as a deprecated alias of "nll", rather than a separate objective.[1] The implementation can compute NLL through a chunked projection path; pin and inspect that implementation separately from the target objective.[26] The evaluation loop computes teacher-forced loss, not the supplied policy/format rates. Generate and grade validation replies separately before applying the selection rule.

Common pitfalls

A regression does not name the boundary that failed. Check the split, the mask, and the checkpoint before you spend a sweep on the learning rate.

Confusing the objective with the update surface

Saying "we tried LoRA instead of SFT" mixes two distinct decisions. LoRA and full fine-tuning describe which weights are updated; SFT defines the supervised next-token cross-entropy objective. Define the behavioral target and objective first. Compare adapter tuning against full-weight tuning only when parameter capacity or memory budget is the variable under test.

Evaluating examples that share a case with training

High validation scores followed by failures on new documents can reflect leakage, distribution shift, or a mismatched metric. Check whether related cases crossed splits and whether the grouping actually tests the deployment claim. A thread-disjoint split doesn't test unseen documents shared across those threads.

Reporting micro-batch as if it were the real batch

When two experiments claim "batch size 4" but demonstrate drastically different stability, check accumulation steps and GPU count before modifying the optimizer. Always publish per-device batch size, gradient accumulation steps, world size, packing mode, and total supervised tokens per update step.

Saving weights without resume state

Changed loss or batches after resume prompts a state comparison; it doesn't uniquely diagnose an omitted checkpoint field. Check optimizer counters, LR, upcoming data, RNG states, precision/scaler state, and artifact identities against the uninterrupted reference.

Using a packing path that crosses example boundaries

Incoherent answers don't uniquely diagnose packing leakage. Test attention reachability and shifted-label boundaries directly. For independent demonstrations, use a verified segment-aware path; for intentional continuous-text training, cross-document context may be part of the chosen objective. A reset position ID or EOS token isn't itself an enforced barrier.

Selecting the best checkpoint by train loss

Training loss measures reference-token likelihood on seen examples, not memorization by itself. Use validation behavior and declared constraints for selection, then assess the frozen choice on the final test. Checks need a task-appropriate grader; fluent output and low loss don't certify policy compliance.

Complete the lesson

Mastery Check

Answer every question, then check your score. Score 75% or higher to mark this lesson complete.

1.An assistant understands a retention policy but repeatedly demonstrates the wrong escalation procedure. The team wants supervised response targets and a frozen base with trainable low-rank modules. Which objective/update-surface pair describes that experiment?
2.The deployment claim is performance on unseen policy documents. Threads A and B both cite document D1; thread C cites D2. A proposed split trains on A/C and validates on B. What split property and claim follow?
3.An isolated sequence has six ignored context labels followed by two answer-token labels and one stop label at positions 6,7,8. The trainer shifts once internally. Which predictor rows and active-term count enter response loss?
4.Token ID2 is both the actual stop at position5 and right padding at positions6/7. Answer text is at position4. Under response-only training, which active target positions and padding labels are correct?
5.Two packed examples have values A=[3,6,9], B=[8,24,0]. Query4 belongs to B and uses uniform weights over allowed earlier/current positions. Add50 to each A value. Which isolated and global-causal before→after outputs follow?
6.Microbatch A has target losses [3,1]; B has four losses of0.5. In order mean of microbatch means, complete-window token mean, what follows?
7.Three DDP workers have local loss-sum gradients3,6,9 and a global active-target count6. Each divides by6, then DDP averages worker gradients. What gradient results, what was intended, and which compensation is needed if the trainer has applied none?
8.A supplied momentum-SGD recipe uses velocity_new=0.5×velocity_old+gradient and weight_new=weight_old−0.2×velocity_new. From weight1, velocity0, first gradient3 is applied and saved. The next gradient is−1. What next weights follow with saved velocity versus reset velocity?
9.Require format>=0.98, then maximize policy rate, then minimize validation loss, then prefer the earlier step. Supplied (format,policy,loss) values are step100=(.99,.92,1.2),300=(.97,.98,.9),500=(.98,.94,1.4),700=(.99,.94,1.3). Which step wins?
10.Checkpoint A assigns high likelihood to approved reference replies but freely generates defer rotation on held-out stale-key requests. Which result and next measurement fit this observation?

10 questions remaining.

References

TRL Documentation: SFT Trainer.

Hugging Face · 2026

https://huggingface.co/docs/trl/sft_trainer

Fine-Tune Your First LLM.

PyTorch Contributors · 2026

https://meta-pytorch.org/torchtune/stable/tutorials/first_finetune_tutorial.html

Training Language Models to Follow Instructions with Human Feedback (InstructGPT).

Ouyang, L., et al. · 2022 · NeurIPS 2022

https://arxiv.org/abs/2203.02155

Don't Stop Pretraining: Adapt Language Models to Domains and Tasks.

Gururangan, S., Marasovic, A., Swayamdipta, S., et al. · 2020 · ACL 2020

https://aclanthology.org/2020.acl-main.740/

Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

Rafailov, R., et al. · 2023

https://arxiv.org/abs/2305.18290

LoRA: Low-Rank Adaptation of Large Language Models.

Hu, E. J., et al. · 2021 · ICLR

https://arxiv.org/abs/2106.09685

QLoRA: Efficient Finetuning of Quantized LLMs

Dettmers, T., et al. · 2023 · NeurIPS

https://arxiv.org/abs/2305.14314

Transformers Documentation: Writing a chat template.

Hugging Face · 2026

https://huggingface.co/docs/transformers/main/en/chat_templating_writing

Transformers Documentation: Chat templates.

Hugging Face · 2026

https://huggingface.co/docs/transformers/main/en/chat_templating

CrossEntropyLoss

PyTorch Contributors · 2026

https://docs.pytorch.org/docs/2.14/generated/torch.nn.CrossEntropyLoss.html

Instruction Fine-Tuning: Does Prompt Loss Matter?

Huerta-Enochian, M., & Ko, S. Y. · 2024 · EMNLP 2024

https://arxiv.org/abs/2401.13586v4

TRL 1.17.1: SFT Trainer Implementation

Hugging Face TRL Contributors · 2026

https://github.com/huggingface/trl/blob/v1.17.1/trl/trainer/sft_trainer.py

Reducing Memory Usage: Packing and Padding-free

Hugging Face · 2026

https://huggingface.co/docs/trl/reducing_memory_usage

Transformers 5.22.1: Attention Mask Utilities (masking_utils.py)

Hugging Face Contributors · 2026

https://github.com/huggingface/transformers/blob/v5.22.1/src/transformers/masking_utils.py

Transformers 5.22.1: Trainer

Hugging Face Contributors · 2026

https://github.com/huggingface/transformers/blob/v5.22.1/src/transformers/trainer.py

torch.nn.functional.scaled_dot_product_attention

PyTorch Contributors · 2026

https://docs.pytorch.org/docs/2.14/generated/torch.nn.functional.scaled_dot_product_attention.html

Drop the FlashAttention warnings for padding-free and packing in SFT (Pull Request 7620)

Hugging Face TRL Contributors · 2026

https://github.com/huggingface/trl/pull/7620

Transformers: Optimization and Learning-Rate Schedules

Hugging Face · 2026

https://huggingface.co/docs/transformers/main_classes/optimizer_schedules

PyTorch 2.14: AdamW

PyTorch Contributors · 2026

https://docs.pytorch.org/docs/2.14/generated/torch.optim.AdamW.html

Saving and Loading Models

PyTorch Contributors · 2026

https://docs.pytorch.org/tutorials/beginner/saving_loading_models.html

Universal Checkpointing with DeepSpeed: A Practical Guide.

DeepSpeed Team · 2026

https://www.deepspeed.ai/tutorials/universal-checkpointing/

FullyShardedDataParallel

PyTorch Contributors · 2026

https://docs.pytorch.org/docs/stable/fsdp.html

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models.

Rajbhandari, S., et al. · 2020 · SC 2020

https://arxiv.org/abs/1910.02054

alignment-handbook: Robust recipes to align language models with human and AI preferences.

Hugging Face · 2025

https://github.com/huggingface/alignment-handbook

Welcome to the torchtune Documentation.

PyTorch Contributors · 2026

https://meta-pytorch.org/torchtune/stable/index.html

TRL: SFT Trainer Implementation (September 22, 2026 source checkpoint)

Hugging Face Contributors · 2026

https://github.com/huggingface/trl/blob/5bb945c38f319259eddaad7bb3838e4aaf590c49/trl/trainer/sft_trainer.py

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.