Skip to content
Skip to lesson
Back to Topics
LearnAdvanced Training & AdaptationSupervised Fine-Tuning Pipeline
⚡HardFine-Tuning & Training

Supervised Fine-Tuning Pipeline

A fine-tuned assistant can sound fluent while breaking its instructions. Supervised fine-tuning raises the probability of the desired reply only when the loss mask, data split, and checkpoint gate measure that behavior.

41 min read
Learning path
Step 109 of 199 in the full curriculum

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

Conversation conv_102 asks a SQL analytics assistant to remove test rows from an orders table before totaling monthly revenue. The system instruction restricts the assistant to read-only SELECT queries and requires refusing destructive writes. The base model replies, "Run DELETE FROM orders WHERE status = 'test'; first, then run your revenue sum." It sounds fluent and helpful, yet it violates the read-only rule. Teaching the model to refuse writes requires tuning on accepted demonstrations and verifying whether training actually changed model behavior.

A chat template turns conversation conv_102 into tokens once quality gates accept the row. uses accepted demonstrations to raise the likelihood of the desired reply through updates.[1][2] InstructGPT used this supervised stage before reward-model training and reinforcement learning.[3] Compare validation loss with held-out refusal and query accuracy when selecting a checkpoint.

Training loss can fall smoothly while task behavior stays broken. Prompt-dominated averages, another turn from conv_102 leaking into validation, unintended cross-example attention, or selection by training loss can each hide the failure. Inspect the split, labels, and evaluation before starting an optimizer sweep; none of those symptoms by itself proves which step failed.

Count supervised answer tokens before launching any job. A packed row can fill every token slot in the context window while providing zero learning signal if aggressive truncation cuts off the completion and masks every remaining token to -100. Reject that row outright instead of pretending its context tokens represent training data.

Eight token positions from conversation conv_102 in a row. Positions 0 to 4 are context tokens with label minus 100. Positions 5 to 7 are deny, write, and end, with those same tokens as labels. Arrows run from the logits at positions 4, 5, and 6 to the labels one position to their right. Only those three predictions enter the loss, and ignored prompt tokens stay in causal context.
Each target aligns with its label and the preceding logit that predicts it: deny uses logits[4], write uses logits[5], and end uses logits[6]. Prompt labels are -100; prompt representations can still affect response loss. These are symbolic positions rather than subword tokens.

Turn behavior into a run specification

The failure can hide in more than one place: the learning objective, the data split, the label mask, the token budget, the checkpoint format, or the release gate. The short CPU checks below isolate each invariant directly.

Define the behavioral target first: write requests (DROP, DELETE, UPDATE) must trigger a clear refusal paired with a read-only SELECT ... WHERE alternative. Compare in-context prompting, retrieval, and SFT before choosing an intervention. If domain SQL vocabulary itself is missing, investigate continued too.

Next, trace what each update actually sees. Which examples and contribute to the loss function? What example and supervised-token budget reaches a single optimizer step? What held-out metric dictates checkpoint selection? Finally, can the validated recipe run on a single , or do Fully Sharded Data Parallel (FSDP) or Zero Redundancy (ZeRO) become necessary?

Following this order keeps the trainer where it belongs: as an execution engine applying an update. The split, mask, batch budget, resume bundle, and export gate define what that update means.

Separate objective from parameterization

Suppose conv_102 outputs fluent SQL prose yet keeps suggesting DELETE statements. Is the missing ingredient raw schema vocabulary, demonstrations of when to refuse writes, or a ranking across several acceptable queries? Answering that question separates the learning objective from the parameterization that absorbs the gradient updates.

An objective determines the mathematical signal guiding the model. A parameterization specifies which weights in the model architecture are allowed to change. They aren't interchangeable.

Failure you observeObjective to investigateWhy
Base model has weak exposure to domain vocabulary in unlabeled corporacontinued pretrainingnext-token training on domain text adapts the language distribution before behavior training[4]
Model understands SQL syntax yet won't follow the read-only refusal ruleSFTprompt-completion demonstrations directly teach that response behavior[1]
Several acceptable answers need ranking by preferencepreference training; compare against the current model or an SFT baselinecomparisons express relative preference rather than a single target answer[5]

Once the objective is SFT, make a second decision: which part of the model can carry the new behavior?

SFT parameterizationWhat moves during the same supervised objectiveWhen to test it
Full fine-tuningall trainable model weightswhen memory permits and adapters may be too restrictive
small low-rank adapter matrices; base weights stay frozenwhen iteration speed, memory, or many task variants matter[6]
QLoRALoRA adapters while the frozen base is stored in 4-bit formwhen the base model doesn't fit comfortably at higher precision[7]

LoRA and aren't alternatives to SFT. They're parameterizations usable for an SFT run. Continued pretraining can also use adapters. Both stages can use causal next-token cross-entropy; their training distributions and supervised targets differ. Calling every adapter run SFT would hide that distinction.

Full weights or adapters

Full fine-tuning

For our response-only objective, full fine-tuning updates all eligible model parameters. Test it when weights, gradients, , activations, and temporary buffers fit, or when an adapter baseline appears capacity-limited. More trainable parameters permit a broader update, but don't establish better held-out behavior. Full fine-tuning doesn't require response-only labels by definition; that is a separate objective choice.

LoRA

LoRA freezes selected base weights and trains low-rank adapter . Target modules and rank determine the trainable count. The original paper's GPT-3 175B comparison reported up to 10,000 times fewer trainable parameters and three times lower GPU memory requirements, not universal 99% and 70% savings.[6] Activations and frozen weights still consume memory. Measure the actual run; adapters are useful candidates when many variants share one base or full updates are too costly.

QLoRA

QLoRA reduces frozen-base storage while retaining adapter updates. Its paper uses 4-bit NormalFloat (NF4), quantized scaling constants, and paged optimizers. The base weights are dequantized to a computation dtype such as for forward and backward operations; adapter gradients don't mean the stored 4-bit base receives weight updates.[7] Check your implementation's storage, compute, and adapter dtypes separately. Compare held-out behavior under measured memory and time budgets.

Reject broken demonstrations before training

For this pipeline, store a prompt, a desired completion, and a group key for -resistant splitting on every SFT row. A missing completion isn't harmless metadata: it's a training example with no answer to teach.

The validator below checks three candidate rows: conv_102 has both a completion and a group key, conv_550 has an empty completion, and the third row omits thread_id. Only conv_102 survives filtering to reach tokenization.

validate_sft_rows.py
rows = [
    {"thread_id": "conv_102", "prompt": "Drop test orders", "completion": "Refuse writes; use SELECT with a WHERE filter instead."},
    {"thread_id": "conv_550", "prompt": "Truncate temp table", "completion": ""},
    {"prompt": "Sum monthly revenue", "completion": "SELECT SUM(amount) FROM orders;"},
]

required = {"thread_id", "prompt", "completion"}
accepted, rejected = [], []
for index, row in enumerate(rows):
    missing = sorted(required - row.keys())
    if missing:
        rejected.append((index, f"missing {missing}"))
    elif any(not isinstance(row[key], str) or not row[key].strip() for key in required):
        rejected.append((index, "required fields must be nonempty strings"))
    else:
        accepted.append(row)

print("accepted_threads=", [row["thread_id"] for row in accepted])
print("rejected=", rejected)
Row schema check output
accepted_threads= ['conv_102']
rejected= [(1, 'required fields must be nonempty strings'), (2, "missing ['thread_id']")]

This validation rule filters out broken inputs before tokenization: conv_102 passes, conv_550 is rejected rather than turning into a prompt-only row, and the third row is dropped before it can break a grouped split. Next, keep accepted rows from the same deployment unit together.

Split by the unit that could leak

Before formatting rows, reserve evaluation examples that training can't imitate through near duplicates. For a SQL assistant, several turns from one multi-turn conversation or multiple queries over the same database schema share column names and table relationships. Randomly splitting individual messages can put one turn of the same session in training and another in evaluation.

Group by the unit your evaluation claim requires to be unseen: conversation thread, schema, tenant, or time window. Low token overlap doesn't make two turns from conv_102 independent. If several group identifiers connect conversations, keep their connected groups together. For a chronological holdout, train on earlier periods and check windows that straddle the cutoff. The check below demonstrates conversation-level separation only.

Choose the grouping from the deployment claim. Thread grouping tests new conversations, not unseen tenants or database schemas shared across those conversations. Deduplicate across groups, and reserve separate validation and final-test groups. Use validation for sweeps and checkpoint selection; inspect the final test only after freezing that choice. Repeatedly tuning on the test turns it into another validation set.

Holding out conv_551 keeps conv_102 and conv_771 together in training while leaving the split intersection empty:

grouped_sft_split.py
rows = [
    {"case_id": "conv_102", "turn": 1, "answer": "Refuse writes; use SELECT with a WHERE filter instead."},
    {"case_id": "conv_102", "turn": 2, "answer": "Keep queries read-only by filtering status != 'test'."},
    {"case_id": "conv_551", "turn": 1, "answer": "Deny UPDATE and suggest a projection query."},
    {"case_id": "conv_771", "turn": 1, "answer": "Aggregate revenue with SELECT SUM(total)."},
]

eval_cases = {"conv_551"}
train = [row for row in rows if row["case_id"] not in eval_cases]
evaluation = [row for row in rows if row["case_id"] in eval_cases]

train_cases = {row["case_id"] for row in train}
held_out_cases = {row["case_id"] for row in evaluation}
assert train_cases.isdisjoint(held_out_cases)

print("train_cases=", sorted(train_cases))
print("eval_cases=", sorted(held_out_cases))
print("case_overlap=", train_cases & held_out_cases)
Grouped split output
train_cases= ['conv_102', 'conv_771']
eval_cases= ['conv_551']
case_overlap= set()

Data path: template, tokenize, label, pack

Follow one row all the way through the data path. Structured conversation dictionaries must pass through a Jinja2 , undergo , receive target label masks, and either get packed into fixed-length blocks or padded into batches.

Use the checkpoint's actual chat formatting. ChatML-style markers such as <|im_start|> aren't universal; templates can mix special IDs with ordinary text, spaces, or newlines. Subword splitting doesn't by itself prove a broken role delimiter. Inspect the rendered template, its token IDs, and assistant masks together rather than registering invented markers and assuming the pretrained model understands them.[8]

For complete training conversations, use add_generation_prompt=False. To start a new assistant reply, add_generation_prompt=True adds a header when the template supports it; continuing a partial assistant message is a different operation. Align the template's turn terminator with serving stop settings. Missing terminators can weaken stopping supervision, but don't prove inevitable endless generation. If you format text first and tokenize later, avoid adding duplicate special tokens.[9]

Version this entire data path together. Delimiter, whitespace, or template changes can alter token IDs or labels. Record the change and rerun batch checks; previous runs remain evidence about their recorded formatting.

Only the target answer and turn terminator in conv_102 should receive cross-entropy loss, while the system instruction, user request, and delimiter tokens stay visible in attention so the reply can condition on them. Setting ignored positions to -100 uses the default ignore_index in loss:

sft_loss_mask.py
IGNORE_INDEX = -100

tokens = [
    ("prefix", "<system>"),
    ("prefix", "Read-only SQL mode."),
    ("prefix", "<user>"),
    ("prefix", "Drop test orders before totaling."),
    ("prefix", "<assistant>"),
    ("target", "Refuse writes; use SELECT with a WHERE filter instead."),
    ("target", "<end_of_turn>"),
]

labels = [
    token if span == "target" else IGNORE_INDEX
    for span, token in tokens
]

supervised = [
    token
    for (_, token), label in zip(tokens, labels)
    if label != IGNORE_INDEX
]
print("supervised_tokens=", supervised)
print("masked_positions=", sum(label == IGNORE_INDEX for label in labels))
Loss mask output
supervised_tokens= ['Refuse writes; use SELECT with a WHERE filter instead.', '<end_of_turn>']
masked_positions= 5

These seven entries are readable spans rather than real subword token IDs. A real sentence expands into multiple integer IDs. The five prefix spans provide context without generating loss; the answer and terminator spans identify what to label after tokenization.

Causal language models shift targets by one position: logits at position t−1t-1 predict the label at position tt. The hidden state at the assistant marker <assistant> predicts the first answer token. Setting a label to -100 removes that token's direct cross-entropy loss, but it doesn't remove gradients flowing through attention layers into earlier prompt representations. Response loss still updates the weights that process the prompt. Mask padding tokens by position, not by blindly masking every EOS ID when EOS doubles as padding.[10]

Don't confuse the label mask with the attention mask. The label mask decides which positions contribute to the loss scalar. The attention mask decides which earlier positions a token can read during the forward pass. Prompt tokens must remain visible as context even when their labels are -100; packed sequences also need segment isolation so adjacent examples can't attend across one another.

The CPU check below uses real integer labels and PyTorch gradients. With all logits set to zero over a four-token vocabulary, each supervised target has cross-entropy loss log⁡4≈1.3863\log 4 \approx 1.3863. Only logit rows 1 and 2 predict the labeled answer and stop positions. Most training frameworks handle this target shift internally; don't shift labels twice.

shifted_sft_targets.py
import math
import torch
import torch.nn.functional as F

# Positions: prompt, assistant marker, answer, stop.
labels = torch.tensor([-100, -100, 2, 3])
logits = torch.zeros(4, 4, dtype=torch.float64, requires_grad=True)
targets = labels[1:]
assert (targets != -100).any(), "Reject an all-ignored target batch"
loss = F.cross_entropy(logits[:-1], targets, ignore_index=-100)
loss.backward()
active_rows = torch.nonzero(logits.grad.abs().sum(dim=1)).flatten().tolist()
assert active_rows == [1, 2]
assert math.isclose(loss.item(), math.log(4))
assert (torch.full_like(targets, -100) != -100).sum().item() == 0
print("logit_rows_with_direct_loss=", active_rows)
print(f"mean_target_loss={loss.item():.6f}")
Shifted target check
logit_rows_with_direct_loss= [1, 2]
mean_target_loss=1.386294

This verifies shifted loss and direct logit gradients. Test those mechanisms on collated tensors before launching long GPU jobs.

Why mask prompt targets here? We want a response-only objective and a metric that measures the response. With 800 scored prompt positions and 30 response positions, prompt targets occupy about 96% of a full-sequence mean's terms. That isn't 96% of the gradient: individual losses and their parameter derivatives can differ greatly. Nor does full-sequence training inevitably cause echoing or degraded style.

Huerta-Enochian and Ko studied prompt-loss weights using LLaMA 1/2 7B and Alpaca-derived datasets. Fractional weights improved some short-completion benchmarks in that setting; the study attributed the effect to regularization.[11] Response-only masking is this run's deliberate choice. Compare objectives on the same held-out tasks before changing it.

Low-loss prompt tokens lower the full-sequence average below without changing answer quality:

response_only_loss.py
prefix_losses = [0.03, 0.04, 0.02, 0.05, 0.03]
answer_losses = [2.20, 1.80]

full_sequence_loss = sum(prefix_losses + answer_losses) / len(prefix_losses + answer_losses)
response_only_loss = sum(answer_losses) / len(answer_losses)

print(f"full_sequence_loss={full_sequence_loss:.3f}")
print(f"response_only_loss={response_only_loss:.3f}")
assert response_only_loss > full_sequence_loss
Response-only loss output
full_sequence_loss=0.596
response_only_loss=2.000

Both numbers measure token-level cross-entropy over different target sets. With five low-loss prefix positions, the combined average drops to 0.5960.596 while answer loss remains 2.0002.000. For our response-only objective, the loss is mean negative log-likelihood over the labeled target set TT:

LSFT(θ)=−1∣T∣∑t∈Tlog⁡pθ(yt∣y<t)\mathcal{L}_{\text{SFT}}(\theta) = -\frac{1}{|T|}\sum_{t \in T} \log p_\theta(y_t \mid y_{<t})

θ\theta represents the trainable parameters, yty_t the target token, and y<ty_{<t} its available prefix. Here TT has two positions. Default unweighted, mean-reduced PyTorch cross-entropy with ignore_index=-100 averages only active targets. Changing the mask changes the objective; compare separately reported response loss rather than treating lower full-sequence loss as evidence of better answers.

You rarely write this mask by hand, but check which format you loaded. As of TRL 1.17, SFTTrainer uses completion-only loss by default for prompt-completion datasets. A plain messages dataset scores the full sequence unless you set assistant_only_loss=True, which needs generation markers in the chat template. TRL can patch those in for known model families.[1] Always inspect a tokenized batch before launching training: confirm that only answer and stop tokens carry non--100 labels.

Inspect masks after truncation and causal shifting. Keeping only the start can remove the completion. A mean-reduced PyTorch cross-entropy batch with no active shifted targets returns NaN; an individual empty row in a mixed batch instead adds context and cost without response supervision. Count dropped or empty rows so dataset loss isn't hidden.

As of TRL 1.17, TRL's preparation path filters fully masked rows after nonpacked truncation. Its packing path needs a separate audit.[12] Custom collators or skipped preparation can change this behavior. Check active labels in the actual collated tensors rather than assuming every empty row reaches the loss.

When sequence packing is active, this truncation step is skipped and overflow is handled by the packing strategy instead; bfd discards overflow tokens.[13] When five prompt tokens consume the length budget, the completion vanishes and the row must be rejected:

reject_truncated_completion.py
IGNORE_INDEX = -100

prompt_tokens = ["<system>", "sql", "<user>", "drop_orders", "<assistant>"]
answer_tokens = ["deny_write", "<end_of_turn>"]
max_length = 5

kept_tokens = (prompt_tokens + answer_tokens)[:max_length]
labels = [
    token if position >= len(prompt_tokens) else IGNORE_INDEX
    for position, token in enumerate(kept_tokens)
]
has_answer_label = any(label != IGNORE_INDEX for label in labels)

# A label at position zero has no preceding logit in this isolated sequence.
first_only_labels = [7, IGNORE_INDEX]
has_shifted_target = any(label != IGNORE_INDEX for label in first_only_labels[1:])
assert any(label != IGNORE_INDEX for label in first_only_labels)
assert not has_shifted_target

print("kept_tokens=", kept_tokens)
print("has_answer_label=", has_answer_label)
print("decision=", "keep" if has_answer_label else "reject_or_increase_max_length")
print("first_position_label_has_shifted_target=", has_shifted_target)
Truncation guard output
kept_tokens= ['<system>', 'sql', '<user>', 'drop_orders', '<assistant>']
has_answer_label= False
decision= reject_or_increase_max_length
first_position_label_has_shifted_target= False

Pack short examples while isolating segments

Short demonstrations can leave many unused token slots when padded to a large fixed length. The fraction of padded slots isn't a measured compute-waste percentage: dynamic batch padding, length grouping, and attention kernels change the work performed. Packing is another efficiency option. For this run's independent-example objective, isolate attention and loss labels between packed examples.

For independent demonstrations, a reply for Ex B (conv_551) shouldn't condition on Ex A (conv_102) merely because the loader put them in one block. Position IDs block attention only if something turns them into a mask. As of Transformers 5.22, models that use its shared causal-mask helper, such as Llama, do that under eager and SDPA attention when you restart the IDs at each example, pass no attention mask, and run with use_cache=False. An all-ones mask or a live cache silently removes the isolation, so set use_cache=False in a hand-written loop. Trainer and gradient checkpointing already turn the cache off during training.[14][15] Otherwise, enforce a causal block-diagonal pattern using an explicit mask or a kernel's segment metadata. The predicate below describes the intended edges; it doesn't execute an attention kernel:

packed_isolation_check.py
segments = [
    ["Ex A prompt", "Ex A response", "<eos>"],
    ["Ex B prompt", "Ex B response", "<eos>"],
]

flat = [(sequence_id, token) for sequence_id, row in enumerate(segments) for token in row]
position_ids = [position for row in segments for position, _ in enumerate(row)]

def may_attend(query_index, key_index):
    query_sequence, _ = flat[query_index]
    key_sequence, _ = flat[key_index]
    return key_index <= query_index and query_sequence == key_sequence

ex_b_response = 4
assert position_ids == [0, 1, 2, 0, 1, 2]
assert not may_attend(ex_b_response, 1)
print("position_ids=", position_ids)
print("Ex_B_response_sees_Ex_A_response=", may_attend(ex_b_response, 1))
Packed isolation output
position_ids= [0, 1, 2, 0, 1, 2]
Ex_B_response_sees_Ex_A_response= False

Reset position IDs isolate examples only when the attention path turns them into a segment mask, as described above. The kernel must enforce that separation during computation. Resetting positions with no segment mask in the kernel causes cross-example contamination:

Two six-by-six attention masks for packed examples Ex A and Ex B, whose position IDs each reset to 0, 1, and 2. In the block-causal mask, query 4 reads only keys 3 and 4, with values 10 and 20, so its output is 15.0 and stays 15.0 after adding 100 to Ex A. In the global causal mask, query 4 also reads Ex A, and its output moves from 7.2 to 67.2. Gold dividers separate the two examples.
Gold dividers separate Ex A and Ex B. The Ex B query at index 4 reads keys 3 and 4, whose values average to 15.0.
IndexTokenposition_idIntended attend-toBroken path (no segment mask)
0Ex A prompt0Ex A onlyEx A only
1Ex A response1Ex A onlyEx A only
2Ex A eos2Ex A onlyEx A only
3Ex B prompt0Ex B onlyEx A + Ex B (cross-example leak)
4Ex B response1Ex B onlyEx A + Ex B (Ex B answer conditions on Ex A)
5Ex B eos2Ex B onlyEx A + Ex B

Test the effect inside an actual attention operation, too. With zero queries and keys in PyTorch's scaled dot-product attention, weights are uniform over permitted positions. Ex B's response sits at index 4. Changing only Ex A's three values leaves Ex B unchanged under a block mask and shifts it under global causal attention. PyTorch SDPA Boolean masks use True for permitted edges.[16]

test_packed_attention_isolation.py
import torch
import torch.nn.functional as F

segment_ids = torch.tensor([0, 0, 0, 1, 1, 1])
causal = torch.ones(6, 6, dtype=torch.bool).tril()
isolated = causal & (segment_ids[:, None] == segment_ids[None, :])
queries = keys = torch.zeros(1, 1, 6, 1, dtype=torch.float64)
values = torch.tensor([1., 2., 3., 10., 20., 30.], dtype=torch.float64).view(1, 1, 6, 1)
changed = values.clone()
changed[:, :, :3] += 100

def ex_b_output(mask, value):
    return F.scaled_dot_product_attention(
        queries, keys, value, attn_mask=mask, dropout_p=0.0,
    )[0, 0, 4, 0].item()

before = ex_b_output(isolated, values)
after = ex_b_output(isolated, changed)
leaking = ex_b_output(causal, values)
leaking_after = ex_b_output(causal, changed)
assert before == after == 15.0
assert abs(leaking - 7.2) < 1e-12
assert abs(leaking_after - 67.2) < 1e-12
print(f"Isolated Ex B: {before:.1f} -> {after:.1f}")
print(f"Global causal Ex B: {leaking:.1f} -> {leaking_after:.1f}")
Packed attention perturbation
Isolated Ex B: 15.0 -> 15.0
Global causal Ex B: 7.2 -> 67.2

Ex B averages its own values 10 and 20 to obtain 15.0. Global attention averages all five earlier-or-current values to obtain 7.2; adding 100 to three of them increases the result by 60.0. Apply the same perturbation test to the collated model path with dropout disabled, alongside checking segment-start labels.

Packing can also fail when an example gets truncated before being packed. If Ex A survives truncation with every response token stripped away, packing still incorporates the prefix to fill the block, but that segment contributes zero learning signal. Inspect segment counts and verify nonzero answer labels before starting a training run.

TRL's bfd packs examples after truncating overlong ones; bfd_split splits overlong sequences into chunks; wrapped concatenates a stream and cuts token sequences.[13] Preserving stored tokens through splitting doesn't preserve every completion's original prompt context or every trainable target. Choose the strategy from the learning objective, then inspect the resulting segments.

As of TRL 1.17, the trainer enables padding-free mode for both bfd and bfd_split. wrapped does not enable padding-free mode by default; its continuous stream merges neighboring segments unless you switch collators, allowing examples to read each other and scoring cross-example joins.[12] Releases before 1.16 also warned when the attention implementation wasn't a FlashAttention variant. That warning is gone, so don't wait for one.[17] An ordinary global causal kernel can read across examples. Disable packing while debugging if necessary; verify isolation directly rather than trusting a missing warning or a completed forward pass.

Attention isolation alone also isn't enough. A globally shifted loss must not ask the last token of one independent segment to predict the first token of the next. As of TRL 1.17, the padding-free collator masks labels wherever segment position IDs reset to zero.[12] Audit that loss isolation as well as attention isolation.

Packing changes loader-item counts and can remove or split targets. A fixed max_steps can therefore imply a different number of data passes, without proving overfitting. Log processed input tokens, active shifted targets, and retained original examples. Throughput and learning exposure use different denominators.

Evaluate hyperparameters through controlled experiments

A learning-rate sweep asks how much change this objective and parameterization can tolerate. Check task improvement and regressions on general capabilities together. Neither a frozen base nor a small learning rate guarantees preserved behavior.

The fragment below uses 1e-4 as an adapter candidate. That isn't a universal safe range or a fixed multiplier over full fine-tuning. Compare nearby rates under the same data exposure and evaluation rules. Dataset quality, optimizer, adapter scaling, model size, and loss normalization across microbatches all affect the result.

KnobBaseline to testWhat decides whether it survives
a short sweep for the actual update surfaceheld-out task gains and regressions at matched exposure
Scheduleexplicit warmup duration, decay shape, and final LRobserved early updates and behavior at the chosen checkpoint
Passes over dataa bounded pilot with measured retained targetsvalidation trends and task behavior, not a fixed epoch ceiling
Regularization & clippingrecord decay groups and clipping thresholdcompare changes in gradients, parameter updates, and held-out behavior
PrecisionBF16 where supported; otherwise choose an explicit pathfinite numerics and task parity under the actual hardware

Warmup reduces early step sizes; it doesn't make AdamW's bias correction unnecessary or guarantee stability. A useful historical counterexample is InstructGPT's standalone SFT baseline: 16 epochs, no warmup, and cosine decay to 10% of its initial LR, with learning rates and epochs selected by searches.[3] That recipe isn't a recommendation to copy 16 epochs. It disproves a mandatory 3%-10% warmup or 1-3 epoch rule.

A cosine schedule's floor is configurable; standard Transformers cosine warmup decays to zero, while a minimum-LR variant can use a nonzero floor.[18] Resolve the actual scheduler instead of assuming 10%. AdamW applies decoupled decay separately from moment estimation. Decay doesn't bound distance from the pretrained weights, and clipping raw gradients doesn't directly bound AdamW's parameter update.[19] Inspect both gradients and update norms.

Effective batch size is also a token budget

The batch size configured on a single device isn't the batch size the optimizer sees during an update step. For synchronous training across complete accumulation windows, the number of items per update follows:

text
loader_items_per_update = per_device_batch * gradient_accumulation_steps * world_size

Without packing, a loader item is one demonstration. With packing, each item is a composite block containing multiple short demonstrations, so the formula counts packed blocks rather than raw examples. Track both raw examples and actual supervised tokens per update. The CPU check below models an unpacked dataset across a four-GPU setup:

sft_effective_batch.py
per_device_batch = 4
grad_accum = 8
world_size = 4
examples_per_update = per_device_batch * grad_accum * world_size

supervised_tokens_per_microbatch_on_one_rank = [180, 212, 164, 220, 198, 204, 175, 207]
supervised_tokens_per_update = sum(supervised_tokens_per_microbatch_on_one_rank) * world_size

print(f"examples_per_update={examples_per_update}")
print(f"supervised_tokens_per_update={supervised_tokens_per_update}")
Effective batch output
examples_per_update=128
supervised_tokens_per_update=6240

Always calculate the combined impact of batch changes rather than reasoning about single knobs. Notice how doubling world size or accumulation doubles the update volume:

compare_update_sizes.py
def examples_per_update(per_device_batch, accumulation, world_size):
    return per_device_batch * accumulation * world_size

run_a = examples_per_update(4, 8, 4)
run_b = examples_per_update(2, 16, 8)

print("run_a_examples_per_update=", run_a)
print("run_b_examples_per_update=", run_b)
print("ratio_b_over_a=", run_b / run_a)
assert run_b == 2 * run_a
Run comparison output
run_a_examples_per_update= 128
run_b_examples_per_update= 256
ratio_b_over_a= 2.0

When sharing batch configurations, specify accumulation steps, world size, sequence packing mode, and supervised token counts.

Loss weighting across accumulation steps matters as well. Suppose one microbatch has one target loss of 4.0 and another has nine target losses of 1.0 each. Averaging batch means gives 2.5; the token mean is (4.0+9.0)/10=1.3(4.0+9.0)/10=1.3. Equal example weights and equal token weights are different objectives. For the token mean, normalize loss sums by active shifted targets across the complete update window.

token_weighted_accumulation.py
microbatches = [[4.0], [1.0] * 9]
mean_of_means = sum(sum(batch) / len(batch) for batch in microbatches) / len(microbatches)
token_mean = sum(map(sum, microbatches)) / sum(map(len, microbatches))
assert mean_of_means == 2.5 and token_mean == 1.3
print(f"mean_of_means={mean_of_means:.1f}; token_mean={token_mean:.1f}")
Token-weighted accumulation
mean_of_means=2.5; token_mean=1.3

In ordinary distributed data parallel training, rank gradients are averaged. If each rank divides its local loss sum by a globally summed token count, that averaging introduces another factor of world_size. Compensate for it, or use a trainer that performs the normalization correctly. Don't add the factor twice when the trainer already handles it. Unequal target counts, partial accumulation windows, and actual loss gradients still need checks beyond this scalar calculation.

What a resumable run must save

A model.safetensors file doesn't encode the optimizer history or next data batch. It may be part of an inference artifact, but serving also needs compatible architecture and tokenization. An adapter export additionally needs its base model and adapter configuration.

To reconstruct the continuation, retain the state your recipe actually uses:

  • model weights or LoRA adapter weights
  • optimizer state (AdamW first moment mtm_t, second moment vtv_t, and step counters)
  • scheduler state (step counter, warmup phase, decay multiplier)
  • global step and completed epoch indices
  • sampler progress and random generator states, not only original seeds
  • tokenizer revision and chat-template hash
  • resolved training configuration and dataset manifests
  • best validation metric achieved so far
  • loss-scaler state if using FP16 dynamic scaling, plus any other stateful training machinery

Optimizer history determines direction and scaling. Scheduler progress determines the next LR, which does not automatically match the peak if reset incorrectly. Sampler and generator states determine upcoming data and randomness. Preserve the exact tokenizer/template artifacts and the adapter's base revision. Metadata labels or filenames alone don't guarantee identical contents or complete worker state.

For ordinary AdamW without AMSGrad, the update uses newly computed, bias-corrected moments:

mt=β1mt−1+(1−β1)gt,vt=β2vt−1+(1−β2)gt2m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\qquad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 m^t=mt1−β1t,v^t=vt1−β2t\widehat m_t=\frac{m_t}{1-\beta_1^t},\qquad \widehat v_t=\frac{v_t}{1-\beta_2^t} θt=(1−ηtλ)θt−1−ηtm^tv^t+ϵ\theta_t=(1-\eta_t\lambda)\theta_{t-1}-\eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon}

Here gtg_t is the current gradient and squaring is elementwise. A fresh optimizer with nonzero g1g_1 first computes m^1=g1\widehat m_1=g_1 and v^1=g12\widehat v_1=g_1^2. Its update doesn't divide the current gradient by epsilon alone. Dropping state changes the trajectory, but doesn't imply a mandatory spike.[19]

Applying gradient 2.0 followed by -1.0 exposes how history controls direction: a fresh AdamW step follows the negative gradient upward, while retained positive momentum still moves the weight downward:

resume_actual_adamw.py
import copy
import math
import torch

weight = torch.nn.Parameter(torch.tensor(1.0, dtype=torch.float64))
optimizer = torch.optim.AdamW([weight], lr=0.1, weight_decay=0.0)

def update(parameter, opt, gradient):
    opt.zero_grad(set_to_none=True)
    parameter.grad = torch.tensor(gradient, dtype=torch.float64)
    opt.step()

update(weight, optimizer, 2.0)
saved_weight = weight.detach().clone()
saved_optimizer = copy.deepcopy(optimizer.state_dict())
update(weight, optimizer, -1.0)

resumed = torch.nn.Parameter(saved_weight.clone())
resumed_optimizer = torch.optim.AdamW([resumed], lr=0.1, weight_decay=0.0)
resumed_optimizer.load_state_dict(saved_optimizer)
update(resumed, resumed_optimizer, -1.0)

fresh = torch.nn.Parameter(saved_weight.clone())
fresh_optimizer = torch.optim.AdamW([fresh], lr=0.1, weight_decay=0.0)
update(fresh, fresh_optimizer, -1.0)
assert torch.equal(resumed, weight)
assert resumed.item() < saved_weight.item() < fresh.item()
assert math.isclose(fresh.item() - saved_weight.item(), 0.1, abs_tol=1e-8)
print(f"uninterrupted={weight.item():.6f}; resumed={resumed.item():.6f}")
print(f"fresh_optimizer={fresh.item():.6f}; all_finite={bool(torch.isfinite(fresh))}")
Actual AdamW continuation
uninterrupted=0.873366; resumed=0.873366
fresh_optimizer=1.000000; all_finite=True

This controlled CPU run restores optimizer history while applying the same next gradient; it does not replay a stochastic LLM job. Prefer saving at completed optimizer steps. Mid-accumulation continuation also needs partial gradients and microbatch progress. Compare the next batch, LR, loss, and update against an uninterrupted reference. Bitwise parity depends on deterministic operations and a compatible environment.[20][21]

A weight number line from 0.85 to 1.05. Gradient plus 2 moved the weight from 1.000000 to 0.900000 and stored moments 0.200000 and 0.004000 at step 1. Both lanes start at 0.900000 and receive the same next gradient, minus 1, at learning rate 0.1. With restored moments the weight moves down to 0.873366. With a fresh optimizer it moves up to 1.000000. The CPU example isolates optimizer history; real continuation also needs scheduler, sampler, RNG, and artifact identity.
Restoring AdamW moments continues the saved parameter along the uninterrupted CPU path. The comparison holds the next gradient and recipe fixed; full training also needs scheduler, sampler, RNG and artifact identity.

The simpler SGD-with-momentum check below isolates the same dependency: restoring velocity gives a next weight of 0.52; resetting it gives 0.70. The difference is 0.18, about 34.6% of 0.52. These are scalar update values rather than an AdamW variance calculation or a full-run loss curve.

A manifest lists required continuation artifacts. This schema check verifies key presence only without opening files or validating hashes:

resume_bundle_manifest_check.py
required = {
    "model_state",
    "optimizer_state",
    "scheduler_state",
    "global_step",
    "rng_state",
    "sampler_state",
    "tokenizer_version",
    "chat_template_version",
    "data_manifest",
    "training_config",
    "eval_manifest",
    "best_metric",
}

resume_bundle = {
    "model_state": "weights/step_600.safetensors",
    "optimizer_state": "optimizer/step_600.pt",
    "scheduler_state": "scheduler/step_600.pt",
    "global_step": 600,
    "rng_state": "rng/step_600.pt",
    "sampler_state": {"epoch": 1, "batches_consumed": 120},
    "tokenizer_version": "sql-sft-tokenizer-v3",
    "chat_template_version": "llama3-sql-chat-v2",
    "data_manifest": "data/sft_manifest_2026-05-20.json",
    "training_config": "config/resolved_training.json",
    "eval_manifest": "eval/sql_read_only_behavior_v4.json",
    "best_metric": {"name": "task_pass_rate", "value": 0.97},
}

missing = sorted(required - resume_bundle.keys())
print("schema_complete=", not missing)
print("best_metric=", resume_bundle["best_metric"])
Checkpoint manifest output
schema_complete= True
best_metric= {'name': 'task_pass_rate', 'value': 0.97}

The scalar SGD-with-momentum example below makes the dependency on velocity explicit. Its fixed next gradient isolates optimizer state; no training data or model loss is evaluated:

resume_next_update.py
import json

def step(state, gradient):
    velocity = 0.9 * state["velocity"] + gradient
    return {"weight": state["weight"] - 0.1 * velocity,
            "velocity": velocity, "step": state["step"] + 1}

saved = step({"weight": 1.0, "velocity": 0.0, "step": 0}, 2.0)
reference = step(saved, 1.0)
restored = json.loads(json.dumps(saved))
resumed = step(restored, 1.0)
weights_only = step({**restored, "velocity": 0.0}, 1.0)
assert resumed == reference and weights_only != reference
print(f"uninterrupted={reference['weight']:.2f}; resumed={resumed['weight']:.2f}")
print(f"reset_momentum={weights_only['weight']:.2f}")
Resume next-update check
uninterrupted=0.52; resumed=0.52
reset_momentum=0.70

Choosing the best checkpoint

Choose checkpoints against the deployment objective on validation data. Teacher-forced loss measures likelihood of reference tokens given reference prefixes; it doesn't directly measure whether a freely generated reply refuses unsafe SQL writes while answering read-only queries. Generate outputs for task evaluation too, and keep a final test separate from repeated selection.

Unreliable selection criteria:

  • lowest training loss alone (doesn't measure unseen-case performance)
  • latest training step (risks overfitting or late degradation)
  • file size or raw token count

Rigorous selection criteria:

  • strict syntax and format pass rate (JSON schema compliance, valid SQL function-call formatting)
  • task pass rate on held-out cases via deterministic query-parser checks or calibrated judges
  • human evaluation on blinded, randomized completion pairs
  • validation loss with a declared role in selection, rather than an assumed universal priority

In the validation table below, require format validity of at least 98%, then maximize task pass rate and break ties with validation loss, followed by the earlier step. In a real evaluation, report denominators, uncertainty, important failure slices, and regressions on general behavior.

select_sft_checkpoint.py
checkpoints = [
    {"step": 200, "val_loss": 1.91, "task_pass_rate": 0.93, "format_pass_rate": 0.99},
    {"step": 400, "val_loss": 1.77, "task_pass_rate": 0.91, "format_pass_rate": 1.00},
    {"step": 600, "val_loss": 1.79, "task_pass_rate": 0.97, "format_pass_rate": 0.98},
]

def select_checkpoint(rows):
    eligible = [row for row in rows if row["format_pass_rate"] >= 0.98]
    return max(eligible, key=lambda row: (row["task_pass_rate"], -row["val_loss"], -row["step"]), default=None)

best = select_checkpoint(checkpoints)
assert best is not None
assert select_checkpoint([]) is None
assert select_checkpoint([{**checkpoints[0], "format_pass_rate": 0.97}]) is None
assert select_checkpoint([{**checkpoints[0], "step": 600}, checkpoints[0]])["step"] == 200
assert select_checkpoint([{**checkpoints[0], "val_loss": 2.0}, {**checkpoints[0], "step": 400}])["step"] == 400

print("best_step=", best["step"])
print("best_task_pass_rate=", best["task_pass_rate"])
print("lowest_loss_step=", min(checkpoints, key=lambda row: row["val_loss"])["step"])
Checkpoint selection output
best_step= 600
best_task_pass_rate= 0.97
lowest_loss_step= 400

Step 400 shows the lowest validation loss (1.771.77), but step 600 achieves the highest task pass rate (0.970.97) while clearing the format threshold (0.980.98). The selection gate picks step 600. If no checkpoint meets the format threshold, the gate yields None, signaling that the recipe needs fixing before anything ships.

Three checkpoints, steps 200, 400, and 600, are rows plotted against three aligned scales. All clear the inclusive 0.98 format gate with 0.99, 1.00, and 0.98. Task pass rates are 0.93, 0.91, and 0.97, so step 600 wins. Validation losses are 1.91, 1.77, and 1.79. Step 400 is lowest, but validation loss only breaks a task-rate tie.
All three checkpoints clear the inclusive 0.98 format gate. Task pass rate selects step 600; validation loss breaks only a task-rate tie.

Single GPU first, then scale out

Don't start debugging conversation conv_102 across an eight-GPU cluster. If prompt templating or label masking contains subtle bugs, scaling out distributes those defects across collective communication calls rather than resolving them. Prove your pipeline on a single device first:

  1. single GPU execution
  2. small, representative evaluation split
  3. frequent checkpointing
  4. verified chat templating and response-only label masking
  5. clear task compliance metric

Validate the smallest feasible run before adding scale. Some required models can't fit one device at all; debug data schemas on tiny samples, then use the minimum necessary sharding. Test distributed loss normalization and checkpoint restore independently of the single-device run.

Signs that one GPU is still fine

A single device is feasible when weights, gradients, optimizer state, activations, and temporary buffers fit at the chosen precision and the run meets its time budget. A model that fits for inference may still fail this training-memory test.

Signs that you need FSDP or ZeRO

Consider FSDP or ZeRO when persistent model weights, gradients, and AdamW optimizer states exceed available GPU memory. Measure activation memory independently: sharding model parameters doesn't automatically eliminate activation overhead. Longer contexts often require activation checkpointing, reduced micro-batch sizes, or context parallelism. Adding more GPUs introduces network communication latency and won't speed up an improperly configured small job.

FSDP and ZeRO offer different state-sharding strategies; the chosen stage determines what is sharded.[22][23] Specify precision explicitly. As of TRL 1.17, TRL defaults to BF16 when FP16 isn't set; verify hardware support and numerical behavior on the target stack.[1]

Minimal operational skeleton

The row still follows the same path, whichever library applies the update:

sft_pipeline_shape.txt
dataset -> template -> tokenizer -> collator/mask -> trainer
       -> periodic eval -> checkpoint save -> best-checkpoint export

TRL, torchtune, and the Alignment Handbook implement variations of this workflow.[1][24][25]

The configuration fragment below is an unexecuted integration blueprint for conversational prompt-completion rows. Its pilot disables packing until segment isolation is verified. API fields were checked against TRL 1.17.1 on November 4, 2026; pin a tested package version and model revision for a real run.

trl_sft_config_fragment.py
from trl import SFTConfig, SFTTrainer

args = SFTConfig(
    output_dir="runs/sql-tool-sft-lora",
    learning_rate=1e-4,              # adapter baseline; tune for the target task
    max_length=1024,
    bf16=True,                     # only on supported hardware; verify numerics
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    max_steps=200,                  # authored pilot budget, not a recommended duration
    lr_scheduler_type="linear",    # explicit schedule; reaches zero at the budget endpoint
    warmup_steps=0,                 # authored candidate to compare, not a stability promise
    seed=42,
    data_seed=42,
    packing=False,                 # enable only after attention and loss-isolation checks
    eval_packing=False,              # keep validation examples separate
    completion_only_loss=True,       # train on the completion field, not the prompt
    loss_type="nll",                 # explicit standard loss for this adapter fragment
    eval_strategy="steps",
    eval_steps=100,
    save_steps=100,
)

trainer = SFTTrainer(
    model=model_with_lora_adapters,
    args=args,
    train_dataset=train_rows,        # prompt/completion are role/content message lists
    eval_dataset=held_out_rows,
    processing_class=tokenizer,
)

This fragment omits model loading, adapter initialization, and dataset construction. Convert the earlier authoring rows to message lists so the trainer can apply the checkpoint's template; raw string fields don't automatically become a chat conversation. We explicitly choose loss_type="nll", which is also the current default. TRL now documents "chunked_nll" as a deprecated alias of "nll", rather than a separate objective.[1] The implementation can compute NLL through a chunked projection path; pin and inspect that implementation separately from the target objective.[26] The evaluation loop computes teacher-forced loss rather than task or format pass rates, so generate and grade validation replies separately before applying the selection rule.

Common pitfalls

A regression does not name the stage that failed. Check the split, the mask, and the checkpoint before you spend a sweep on the learning rate.

Confusing the objective with the trainable parameters

Saying "we tried LoRA instead of SFT" mixes two distinct decisions. LoRA and full fine-tuning describe which weights are updated; SFT defines the supervised next-token cross-entropy objective. Define the behavioral target and objective first. Compare adapter tuning against full-weight tuning only when parameter capacity or memory budget is the variable under test.

Evaluating examples that share a case with training

High validation scores followed by failures on new documents can reflect leakage, distribution shift, or a mismatched metric. Check whether related cases crossed splits and whether the grouping actually tests the deployment claim. A thread-disjoint split doesn't test unseen schemas shared across those threads.

Reporting micro-batch as if it were the real batch

When two experiments claim "batch size 4" but demonstrate drastically different stability, check accumulation steps and GPU count before modifying the optimizer. Always publish per-device batch size, gradient accumulation steps, world size, packing mode, and total supervised tokens per update step.

Saving weights without resume state

Changed loss or batches after resume prompts a state comparison; it doesn't uniquely diagnose an omitted checkpoint field. Check optimizer counters, LR, upcoming data, RNG states, precision/scaler state, and artifact identities against the uninterrupted reference.

Using a packing path that leaks across examples

Incoherent answers don't uniquely diagnose packing leakage. Test attention reachability and shifted-label isolation directly. For independent demonstrations, use a verified segment-aware path; for intentional continuous-text training, cross-document context may be part of the chosen objective. A reset position ID or EOS token isn't itself an enforced barrier.

Selecting the best checkpoint by train loss

Training loss measures reference-token likelihood on seen examples, not memorization by itself. Use validation behavior and declared constraints for selection, then assess the frozen choice on the final test. Checks need a task-appropriate grader; fluent output and low loss don't establish task correctness.

Complete the lesson

Mastery Check

Answer every question, then check your score. Score 75% or higher to mark this lesson complete.

1.For each disputed charge, a reviewer writes one reply for a billing assistant to imitate. A layer uses W + BA, where W is its original 256-by-256 weight matrix. The optimizer updates only B (256 by 4) and A (4 by 256). Which objective and parameterization does this run use?
2.The deployment claim is performance on unseen technical manuals. Threads A and B both cite manual M1; thread C cites M2. A proposed split trains on A/C and validates on B. What split property and claim follow?
3.An isolated sequence has six ignored context labels followed by two answer-token labels and one stop label at positions 6,7,8. The trainer shifts once internally. Which predictor rows and active-term count enter response loss?
4.Token ID2 is both the actual stop at position5 and right padding at positions6/7. Answer text is at position4. Under response-only training, which active target positions and padding labels are correct?
5.Two packed examples have values A=[3,6,9], B=[8,24,0]. Query4 belongs to B and uses uniform weights over allowed earlier/current positions. Add50 to each A value. Which isolated and global-causal before→after outputs follow?
6.Microbatch A has target losses [3,1]; B has four losses of0.5. In order mean of microbatch means, complete-window token mean, what follows?
7.Three DDP workers have local loss-sum gradients3,6,9 and a global active-target count6. Each divides by6, then DDP averages worker gradients. What gradient results, what was intended, and which compensation is needed if the trainer has applied none?
8.A momentum-SGD rule uses velocity_new=0.5×velocity_old+gradient and weight_new=weight_old−0.2×velocity_new. From weight 1 and velocity 0, gradient 3 is applied and saved. With next gradient −1, what next weights follow with saved velocity versus reset velocity?
9.Require format>=0.98, then maximize task rate, then minimize validation loss, then prefer the earlier step. Validation scores (format,task,loss) are step100=(.99,.92,1.2),300=(.97,.98,.9),500=(.98,.94,1.4),700=(.99,.94,1.3). Which step wins?
10.A maintenance assistant replies with a team name followed by a reason. Urgent repairs go to Emergency Maintenance. A checkpoint predicts the approved replies well when given the preceding reference words, yet a resident gets a reply naming Routine Maintenance for an urgent repair. Which check on held-out urgent requests estimates how often residents get the right referral?

10 questions remaining.

References

TRL Documentation: SFT Trainer.

Hugging Face · 2026

https://huggingface.co/docs/trl/sft_trainer

Fine-Tune Your First LLM.

PyTorch Contributors · 2026

https://meta-pytorch.org/torchtune/stable/tutorials/first_finetune_tutorial.html

Training Language Models to Follow Instructions with Human Feedback (InstructGPT).

Ouyang, L., et al. · 2022 · NeurIPS 2022

https://arxiv.org/abs/2203.02155

Don't Stop Pretraining: Adapt Language Models to Domains and Tasks.

Gururangan, S., Marasovic, A., Swayamdipta, S., et al. · 2020 · ACL 2020

https://aclanthology.org/2020.acl-main.740/

Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

Rafailov, R., et al. · 2023

https://arxiv.org/abs/2305.18290

LoRA: Low-Rank Adaptation of Large Language Models.

Hu, E. J., et al. · 2021 · ICLR

https://arxiv.org/abs/2106.09685

QLoRA: Efficient Finetuning of Quantized LLMs

Dettmers, T., et al. · 2023 · NeurIPS

https://arxiv.org/abs/2305.14314

Transformers Documentation: Writing a chat template.

Hugging Face · 2026

https://huggingface.co/docs/transformers/main/en/chat_templating_writing

Transformers Documentation: Chat templates.

Hugging Face · 2026

https://huggingface.co/docs/transformers/main/en/chat_templating

CrossEntropyLoss

PyTorch Contributors · 2026

https://docs.pytorch.org/docs/2.14/generated/torch.nn.CrossEntropyLoss.html

Instruction Fine-Tuning: Does Prompt Loss Matter?

Huerta-Enochian, M., & Ko, S. Y. · 2024 · EMNLP 2024

https://arxiv.org/abs/2401.13586v4

TRL 1.17.1: SFT Trainer Implementation

Hugging Face TRL Contributors · 2026

https://github.com/huggingface/trl/blob/v1.17.1/trl/trainer/sft_trainer.py

Reducing Memory Usage: Packing and Padding-free

Hugging Face · 2026

https://huggingface.co/docs/trl/reducing_memory_usage

Transformers 5.22.1: Attention Mask Utilities (masking_utils.py)

Hugging Face Contributors · 2026

https://github.com/huggingface/transformers/blob/v5.22.1/src/transformers/masking_utils.py

Transformers 5.22.1: Trainer

Hugging Face Contributors · 2026

https://github.com/huggingface/transformers/blob/v5.22.1/src/transformers/trainer.py

torch.nn.functional.scaled_dot_product_attention

PyTorch Contributors · 2026

https://docs.pytorch.org/docs/2.14/generated/torch.nn.functional.scaled_dot_product_attention.html

Drop the FlashAttention warnings for padding-free and packing in SFT (Pull Request 7620)

Hugging Face TRL Contributors · 2026

https://github.com/huggingface/trl/pull/7620

Transformers: Optimization and Learning-Rate Schedules

Hugging Face · 2026

https://huggingface.co/docs/transformers/main_classes/optimizer_schedules

PyTorch 2.14: AdamW

PyTorch Contributors · 2026

https://docs.pytorch.org/docs/2.14/generated/torch.optim.AdamW.html

Saving and Loading Models

PyTorch Contributors · 2026

https://docs.pytorch.org/tutorials/beginner/saving_loading_models.html

Universal Checkpointing with DeepSpeed: A Practical Guide.

DeepSpeed Team · 2026

https://www.deepspeed.ai/tutorials/universal-checkpointing/

FullyShardedDataParallel

PyTorch Contributors · 2026

https://docs.pytorch.org/docs/stable/fsdp.html

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models.

Rajbhandari, S., et al. · 2020 · SC 2020

https://arxiv.org/abs/1910.02054

alignment-handbook: Robust recipes to align language models with human and AI preferences.

Hugging Face · 2025

https://github.com/huggingface/alignment-handbook

Welcome to the torchtune Documentation.

PyTorch Contributors · 2026

https://meta-pytorch.org/torchtune/stable/index.html

TRL: SFT Trainer Implementation (September 22, 2026 source checkpoint)

Hugging Face Contributors · 2026

https://github.com/huggingface/trl/blob/5bb945c38f319259eddaad7bb3838e4aaf590c49/trl/trainer/sft_trainer.py

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.