Personalize this lesson
Adapt explanations and teaching visuals to your background and preferred voice.
Conversation conv_102 asks a SQL analytics assistant to remove test rows from an orders table before totaling monthly revenue. The system instruction restricts the assistant to read-only SELECT queries and requires refusing destructive writes. The base model replies, "Run DELETE FROM orders WHERE status = 'test'; first, then run your revenue sum." It sounds fluent and helpful, yet it violates the read-only rule. Teaching the model to refuse writes requires tuning on accepted demonstrations and verifying whether training actually changed model behavior.
A chat template turns conversation conv_102 into tokens once quality gates accept the row. Supervised fine-tuning (SFT) uses accepted demonstrations to raise the likelihood of the desired reply through gradient updates.[1][2] InstructGPT used this supervised stage before reward-model training and reinforcement learning.[3] Compare validation loss with held-out refusal and query accuracy when selecting a checkpoint.
Training loss can fall smoothly while task behavior stays broken. Prompt-dominated averages, another turn from conv_102 leaking into validation, unintended cross-example attention, or selection by training loss can each hide the failure. Inspect the split, labels, and evaluation before starting an optimizer sweep; none of those symptoms by itself proves which step failed.
Count supervised answer tokens before launching any job. A packed row can fill every token slot in the context window while providing zero learning signal if aggressive truncation cuts off the completion and masks every remaining token to -100. Reject that row outright instead of pretending its context tokens represent training data.

Turn behavior into a run specification
The failure can hide in more than one place: the learning objective, the data split, the label mask, the token budget, the checkpoint format, or the release gate. The short CPU checks below isolate each invariant directly.
Define the behavioral target first: write requests (DROP, DELETE, UPDATE) must trigger a clear refusal paired with a read-only SELECT ... WHERE alternative. Compare in-context prompting, retrieval, and SFT before choosing an intervention. If domain SQL vocabulary itself is missing, investigate continued pretraining too.
Next, trace what each update actually sees. Which examples and tokens contribute to the loss function? What example and supervised-token budget reaches a single optimizer step? What held-out metric dictates checkpoint selection? Finally, can the validated recipe run on a single GPU, or do Fully Sharded Data Parallel (FSDP) or Zero Redundancy Optimizer (ZeRO) become necessary?
Following this order keeps the trainer where it belongs: as an execution engine applying an update. The split, mask, batch budget, resume bundle, and export gate define what that update means.
Separate objective from parameterization
Suppose conv_102 outputs fluent SQL prose yet keeps suggesting DELETE statements. Is the missing ingredient raw schema vocabulary, demonstrations of when to refuse writes, or a ranking across several acceptable queries? Answering that question separates the learning objective from the parameterization that absorbs the gradient updates.
An objective determines the mathematical signal guiding the model. A parameterization specifies which weights in the model architecture are allowed to change. They aren't interchangeable.
| Failure you observe | Objective to investigate | Why |
|---|---|---|
| Base model has weak exposure to domain vocabulary in unlabeled corpora | continued pretraining | next-token training on domain text adapts the language distribution before behavior training[4] |
| Model understands SQL syntax yet won't follow the read-only refusal rule | SFT | prompt-completion demonstrations directly teach that response behavior[1] |
| Several acceptable answers need ranking by preference | preference training; compare against the current model or an SFT baseline | comparisons express relative preference rather than a single target answer[5] |
Once the objective is SFT, make a second decision: which part of the model can carry the new behavior?
| SFT parameterization | What moves during the same supervised objective | When to test it |
|---|---|---|
| Full fine-tuning | all trainable model weights | when memory permits and adapters may be too restrictive |
| LoRA | small low-rank adapter matrices; base weights stay frozen | when iteration speed, memory, or many task variants matter[6] |
| QLoRA | LoRA adapters while the frozen base is stored in 4-bit form | when the base model doesn't fit comfortably at higher precision[7] |
LoRA and QLoRA aren't alternatives to SFT. They're parameterizations usable for an SFT run. Continued pretraining can also use adapters. Both stages can use causal next-token cross-entropy; their training distributions and supervised targets differ. Calling every adapter run SFT would hide that distinction.
Full weights or adapters
Full fine-tuning
For our response-only objective, full fine-tuning updates all eligible model parameters. Test it when weights, gradients, optimizer state, activations, and temporary buffers fit, or when an adapter baseline appears capacity-limited. More trainable parameters permit a broader update, but don't establish better held-out behavior. Full fine-tuning doesn't require response-only labels by definition; that is a separate objective choice.
LoRA
LoRA freezes selected base weights and trains low-rank adapter matrices. Target modules and rank determine the trainable count. The original paper's GPT-3 175B comparison reported up to 10,000 times fewer trainable parameters and three times lower GPU memory requirements, not universal 99% and 70% savings.[6] Activations and frozen weights still consume memory. Measure the actual run; adapters are useful candidates when many variants share one base or full updates are too costly.
QLoRA
QLoRA reduces frozen-base storage while retaining adapter updates. Its paper uses 4-bit NormalFloat (NF4), quantized scaling constants, and paged optimizers. The base weights are dequantized to a computation dtype such as BF16 for forward and backward operations; adapter gradients don't mean the stored 4-bit base receives weight updates.[7] Check your implementation's storage, compute, and adapter dtypes separately. Compare held-out behavior under measured memory and time budgets.
Reject broken demonstrations before training
For this pipeline, store a prompt, a desired completion, and a group key for leakage-resistant splitting on every SFT row. A missing completion isn't harmless metadata: it's a training example with no answer to teach.
The validator below checks three candidate rows: conv_102 has both a completion and a group key, conv_550 has an empty completion, and the third row omits thread_id. Only conv_102 survives filtering to reach tokenization.
rows = [
{"thread_id": "conv_102", "prompt": "Drop test orders", "completion": "Refuse writes; use SELECT with a WHERE filter instead."},
{"thread_id": "conv_550", "prompt": "Truncate temp table", "completion": ""},
{"prompt": "Sum monthly revenue", "completion": "SELECT SUM(amount) FROM orders;"},
]
required = {"thread_id", "prompt", "completion"}
accepted, rejected = [], []
for index, row in enumerate(rows):
missing = sorted(required - row.keys())
if missing:
rejected.append((index, f"missing {missing}"))
elif any(not isinstance(row[key], str) or not row[key].strip() for key in required):
rejected.append((index, "required fields must be nonempty strings"))
else:
accepted.append(row)
print("accepted_threads=", [row["thread_id"] for row in accepted])
print("rejected=", rejected)accepted_threads= ['conv_102']
rejected= [(1, 'required fields must be nonempty strings'), (2, "missing ['thread_id']")]This validation rule filters out broken inputs before tokenization: conv_102 passes, conv_550 is rejected rather than turning into a prompt-only row, and the third row is dropped before it can break a grouped split. Next, keep accepted rows from the same deployment unit together.
Split by the unit that could leak
Before formatting rows, reserve evaluation examples that training can't imitate through near duplicates. For a SQL assistant, several turns from one multi-turn conversation or multiple queries over the same database schema share column names and table relationships. Randomly splitting individual messages can put one turn of the same session in training and another in evaluation.
Group by the unit your evaluation claim requires to be unseen: conversation thread, schema, tenant, or time window. Low token overlap doesn't make two turns from conv_102 independent. If several group identifiers connect conversations, keep their connected groups together. For a chronological holdout, train on earlier periods and check windows that straddle the cutoff. The check below demonstrates conversation-level separation only.
Choose the grouping from the deployment claim. Thread grouping tests new conversations, not unseen tenants or database schemas shared across those conversations. Deduplicate across groups, and reserve separate validation and final-test groups. Use validation for sweeps and checkpoint selection; inspect the final test only after freezing that choice. Repeatedly tuning on the test turns it into another validation set.
Holding out conv_551 keeps conv_102 and conv_771 together in training while leaving the split intersection empty:
rows = [
{"case_id": "conv_102", "turn": 1, "answer": "Refuse writes; use SELECT with a WHERE filter instead."},
{"case_id": "conv_102", "turn": 2, "answer": "Keep queries read-only by filtering status != 'test'."},
{"case_id": "conv_551", "turn": 1, "answer": "Deny UPDATE and suggest a projection query."},
{"case_id": "conv_771", "turn": 1, "answer": "Aggregate revenue with SELECT SUM(total)."},
]
eval_cases = {"conv_551"}
train = [row for row in rows if row["case_id"] not in eval_cases]
evaluation = [row for row in rows if row["case_id"] in eval_cases]
train_cases = {row["case_id"] for row in train}
held_out_cases = {row["case_id"] for row in evaluation}
assert train_cases.isdisjoint(held_out_cases)
print("train_cases=", sorted(train_cases))
print("eval_cases=", sorted(held_out_cases))
print("case_overlap=", train_cases & held_out_cases)train_cases= ['conv_102', 'conv_771']
eval_cases= ['conv_551']
case_overlap= set()Two turns from conversation conv_102 have different wording. Can one train and the other evaluate if their token overlap is low?
Answer
No. They share the conversation being held out, so both turns stay in the same split. Group by schema or tenant when the evaluation needs to estimate performance on unseen schemas or tenants.
Data path: template, tokenize, label, pack
Follow one row all the way through the data path. Structured conversation dictionaries must pass through a Jinja2 chat template, undergo tokenization, receive target label masks, and either get packed into fixed-length blocks or padded into batches.
Use the checkpoint's actual chat formatting. ChatML-style markers such as <|im_start|> aren't universal; templates can mix special IDs with ordinary text, spaces, or newlines. Subword splitting doesn't by itself prove a broken role delimiter. Inspect the rendered template, its token IDs, and assistant masks together rather than registering invented markers and assuming the pretrained model understands them.[8]
For complete training conversations, use add_generation_prompt=False. To start a new assistant reply, add_generation_prompt=True adds a header when the template supports it; continuing a partial assistant message is a different operation. Align the template's turn terminator with serving stop settings. Missing terminators can weaken stopping supervision, but don't prove inevitable endless generation. If you format text first and tokenize later, avoid adding duplicate special tokens.[9]
Version this entire data path together. Delimiter, whitespace, or template changes can alter token IDs or labels. Record the change and rerun batch checks; previous runs remain evidence about their recorded formatting.
Only the target answer and turn terminator in conv_102 should receive cross-entropy loss, while the system instruction, user request, and delimiter tokens stay visible in attention so the reply can condition on them. Setting ignored positions to -100 uses the default ignore_index in PyTorch cross-entropy loss:
IGNORE_INDEX = -100
tokens = [
("prefix", "<system>"),
("prefix", "Read-only SQL mode."),
("prefix", "<user>"),
("prefix", "Drop test orders before totaling."),
("prefix", "<assistant>"),
("target", "Refuse writes; use SELECT with a WHERE filter instead."),
("target", "<end_of_turn>"),
]
labels = [
token if span == "target" else IGNORE_INDEX
for span, token in tokens
]
supervised = [
token
for (_, token), label in zip(tokens, labels)
if label != IGNORE_INDEX
]
print("supervised_tokens=", supervised)
print("masked_positions=", sum(label == IGNORE_INDEX for label in labels))supervised_tokens= ['Refuse writes; use SELECT with a WHERE filter instead.', '<end_of_turn>']
masked_positions= 5These seven entries are readable spans rather than real subword token IDs. A real sentence expands into multiple integer IDs. The five prefix spans provide context without generating loss; the answer and terminator spans identify what to label after tokenization.
Causal language models shift targets by one position: logits at position predict the label at position . The hidden state at the assistant marker <assistant> predicts the first answer token. Setting a label to -100 removes that token's direct cross-entropy loss, but it doesn't remove gradients flowing through attention layers into earlier prompt representations. Response loss still updates the weights that process the prompt. Mask padding tokens by position, not by blindly masking every EOS ID when EOS doubles as padding.[10]
Don't confuse the label mask with the attention mask. The label mask decides which positions contribute to the loss scalar. The attention mask decides which earlier positions a token can read during the forward pass. Prompt tokens must remain visible as context even when their labels are -100; packed sequences also need segment isolation so adjacent examples can't attend across one another.
The CPU check below uses real integer labels and PyTorch gradients. With all logits set to zero over a four-token vocabulary, each supervised target has cross-entropy loss . Only logit rows 1 and 2 predict the labeled answer and stop positions. Most training frameworks handle this target shift internally; don't shift labels twice.
import math
import torch
import torch.nn.functional as F
# Positions: prompt, assistant marker, answer, stop.
labels = torch.tensor([-100, -100, 2, 3])
logits = torch.zeros(4, 4, dtype=torch.float64, requires_grad=True)
targets = labels[1:]
assert (targets != -100).any(), "Reject an all-ignored target batch"
loss = F.cross_entropy(logits[:-1], targets, ignore_index=-100)
loss.backward()
active_rows = torch.nonzero(logits.grad.abs().sum(dim=1)).flatten().tolist()
assert active_rows == [1, 2]
assert math.isclose(loss.item(), math.log(4))
assert (torch.full_like(targets, -100) != -100).sum().item() == 0
print("logit_rows_with_direct_loss=", active_rows)
print(f"mean_target_loss={loss.item():.6f}")logit_rows_with_direct_loss= [1, 2]
mean_target_loss=1.386294This verifies shifted loss and direct logit gradients. Test those mechanisms on collated tensors before launching long GPU jobs.
Why mask prompt targets here? We want a response-only objective and a metric that measures the response. With 800 scored prompt positions and 30 response positions, prompt targets occupy about 96% of a full-sequence mean's terms. That isn't 96% of the gradient: individual losses and their parameter derivatives can differ greatly. Nor does full-sequence training inevitably cause echoing or degraded style.
Huerta-Enochian and Ko studied prompt-loss weights using LLaMA 1/2 7B and Alpaca-derived datasets. Fractional weights improved some short-completion benchmarks in that setting; the study attributed the effect to regularization.[11] Response-only masking is this run's deliberate choice. Compare objectives on the same held-out tasks before changing it.
Low-loss prompt tokens lower the full-sequence average below without changing answer quality:
prefix_losses = [0.03, 0.04, 0.02, 0.05, 0.03]
answer_losses = [2.20, 1.80]
full_sequence_loss = sum(prefix_losses + answer_losses) / len(prefix_losses + answer_losses)
response_only_loss = sum(answer_losses) / len(answer_losses)
print(f"full_sequence_loss={full_sequence_loss:.3f}")
print(f"response_only_loss={response_only_loss:.3f}")
assert response_only_loss > full_sequence_lossfull_sequence_loss=0.596
response_only_loss=2.000Both numbers measure token-level cross-entropy over different target sets. With five low-loss prefix positions, the combined average drops to while answer loss remains . For our response-only objective, the loss is mean negative log-likelihood over the labeled target set :
represents the trainable parameters, the target token, and its available prefix. Here has two positions. Default unweighted, mean-reduced PyTorch cross-entropy with ignore_index=-100 averages only active targets. Changing the mask changes the objective; compare separately reported response loss rather than treating lower full-sequence loss as evidence of better answers.
You rarely write this mask by hand, but check which format you loaded. As of TRL 1.17, SFTTrainer uses completion-only loss by default for prompt-completion datasets. A plain messages dataset scores the full sequence unless you set assistant_only_loss=True, which needs generation markers in the chat template. TRL can patch those in for known model families.[1] Always inspect a tokenized batch before launching training: confirm that only answer and stop tokens carry non--100 labels.
Inspect masks after truncation and causal shifting. Keeping only the start can remove the completion. A mean-reduced PyTorch cross-entropy batch with no active shifted targets returns NaN; an individual empty row in a mixed batch instead adds context and cost without response supervision. Count dropped or empty rows so dataset loss isn't hidden.
As of TRL 1.17, TRL's preparation path filters fully masked rows after nonpacked truncation. Its packing path needs a separate audit.[12] Custom collators or skipped preparation can change this behavior. Check active labels in the actual collated tensors rather than assuming every empty row reaches the loss.
When sequence packing is active, this truncation step is skipped and overflow is handled by the packing strategy instead; bfd discards overflow tokens.[13] When five prompt tokens consume the length budget, the completion vanishes and the row must be rejected:
IGNORE_INDEX = -100
prompt_tokens = ["<system>", "sql", "<user>", "drop_orders", "<assistant>"]
answer_tokens = ["deny_write", "<end_of_turn>"]
max_length = 5
kept_tokens = (prompt_tokens + answer_tokens)[:max_length]
labels = [
token if position >= len(prompt_tokens) else IGNORE_INDEX
for position, token in enumerate(kept_tokens)
]
has_answer_label = any(label != IGNORE_INDEX for label in labels)
# A label at position zero has no preceding logit in this isolated sequence.
first_only_labels = [7, IGNORE_INDEX]
has_shifted_target = any(label != IGNORE_INDEX for label in first_only_labels[1:])
assert any(label != IGNORE_INDEX for label in first_only_labels)
assert not has_shifted_target
print("kept_tokens=", kept_tokens)
print("has_answer_label=", has_answer_label)
print("decision=", "keep" if has_answer_label else "reject_or_increase_max_length")
print("first_position_label_has_shifted_target=", has_shifted_target)kept_tokens= ['<system>', 'sql', '<user>', 'drop_orders', '<assistant>']
has_answer_label= False
decision= reject_or_increase_max_length
first_position_label_has_shifted_target= FalsePack short examples while isolating segments
Short demonstrations can leave many unused token slots when padded to a large fixed length. The fraction of padded slots isn't a measured compute-waste percentage: dynamic batch padding, length grouping, and attention kernels change the work performed. Packing is another efficiency option. For this run's independent-example objective, isolate attention and loss labels between packed examples.
For independent demonstrations, a reply for Ex B (conv_551) shouldn't condition on Ex A (conv_102) merely because the loader put them in one block. Position IDs block attention only if something turns them into a mask. As of Transformers 5.22, models that use its shared causal-mask helper, such as Llama, do that under eager and SDPA attention when you restart the IDs at each example, pass no attention mask, and run with use_cache=False. An all-ones mask or a live cache silently removes the isolation, so set use_cache=False in a hand-written loop. Trainer and gradient checkpointing already turn the cache off during training.[14][15] Otherwise, enforce a causal block-diagonal pattern using an explicit mask or a kernel's segment metadata. The predicate below describes the intended edges; it doesn't execute an attention kernel:
segments = [
["Ex A prompt", "Ex A response", "<eos>"],
["Ex B prompt", "Ex B response", "<eos>"],
]
flat = [(sequence_id, token) for sequence_id, row in enumerate(segments) for token in row]
position_ids = [position for row in segments for position, _ in enumerate(row)]
def may_attend(query_index, key_index):
query_sequence, _ = flat[query_index]
key_sequence, _ = flat[key_index]
return key_index <= query_index and query_sequence == key_sequence
ex_b_response = 4
assert position_ids == [0, 1, 2, 0, 1, 2]
assert not may_attend(ex_b_response, 1)
print("position_ids=", position_ids)
print("Ex_B_response_sees_Ex_A_response=", may_attend(ex_b_response, 1))position_ids= [0, 1, 2, 0, 1, 2]
Ex_B_response_sees_Ex_A_response= FalseReset position IDs isolate examples only when the attention path turns them into a segment mask, as described above. The kernel must enforce that separation during computation. Resetting positions with no segment mask in the kernel causes cross-example contamination:

| Index | Token | position_id | Intended attend-to | Broken path (no segment mask) |
|---|---|---|---|---|
| 0 | Ex A prompt | 0 | Ex A only | Ex A only |
| 1 | Ex A response | 1 | Ex A only | Ex A only |
| 2 | Ex A eos | 2 | Ex A only | Ex A only |
| 3 | Ex B prompt | 0 | Ex B only | Ex A + Ex B (cross-example leak) |
| 4 | Ex B response | 1 | Ex B only | Ex A + Ex B (Ex B answer conditions on Ex A) |
| 5 | Ex B eos | 2 | Ex B only | Ex A + Ex B |
Test the effect inside an actual attention operation, too. With zero queries and keys in PyTorch's scaled dot-product attention, weights are uniform over permitted positions. Ex B's response sits at index 4. Changing only Ex A's three values leaves Ex B unchanged under a block mask and shifts it under global causal attention. PyTorch SDPA Boolean masks use True for permitted edges.[16]
import torch
import torch.nn.functional as F
segment_ids = torch.tensor([0, 0, 0, 1, 1, 1])
causal = torch.ones(6, 6, dtype=torch.bool).tril()
isolated = causal & (segment_ids[:, None] == segment_ids[None, :])
queries = keys = torch.zeros(1, 1, 6, 1, dtype=torch.float64)
values = torch.tensor([1., 2., 3., 10., 20., 30.], dtype=torch.float64).view(1, 1, 6, 1)
changed = values.clone()
changed[:, :, :3] += 100
def ex_b_output(mask, value):
return F.scaled_dot_product_attention(
queries, keys, value, attn_mask=mask, dropout_p=0.0,
)[0, 0, 4, 0].item()
before = ex_b_output(isolated, values)
after = ex_b_output(isolated, changed)
leaking = ex_b_output(causal, values)
leaking_after = ex_b_output(causal, changed)
assert before == after == 15.0
assert abs(leaking - 7.2) < 1e-12
assert abs(leaking_after - 67.2) < 1e-12
print(f"Isolated Ex B: {before:.1f} -> {after:.1f}")
print(f"Global causal Ex B: {leaking:.1f} -> {leaking_after:.1f}")Isolated Ex B: 15.0 -> 15.0
Global causal Ex B: 7.2 -> 67.2Ex B averages its own values 10 and 20 to obtain 15.0. Global attention averages all five earlier-or-current values to obtain 7.2; adding 100 to three of them increases the result by 60.0. Apply the same perturbation test to the collated model path with dropout disabled, alongside checking segment-start labels.
Packing can also fail when an example gets truncated before being packed. If Ex A survives truncation with every response token stripped away, packing still incorporates the prefix to fill the block, but that segment contributes zero learning signal. Inspect segment counts and verify nonzero answer labels before starting a training run.
TRL's bfd packs examples after truncating overlong ones; bfd_split splits overlong sequences into chunks; wrapped concatenates a stream and cuts token sequences.[13] Preserving stored tokens through splitting doesn't preserve every completion's original prompt context or every trainable target. Choose the strategy from the learning objective, then inspect the resulting segments.
As of TRL 1.17, the trainer enables padding-free mode for both bfd and bfd_split. wrapped does not enable padding-free mode by default; its continuous stream merges neighboring segments unless you switch collators, allowing examples to read each other and scoring cross-example joins.[12] Releases before 1.16 also warned when the attention implementation wasn't a FlashAttention variant. That warning is gone, so don't wait for one.[17] An ordinary global causal kernel can read across examples. Disable packing while debugging if necessary; verify isolation directly rather than trusting a missing warning or a completed forward pass.
Attention isolation alone also isn't enough. A globally shifted loss must not ask the last token of one independent segment to predict the first token of the next. As of TRL 1.17, the padding-free collator masks labels wherever segment position IDs reset to zero.[12] Audit that loss isolation as well as attention isolation.
Packing changes loader-item counts and can remove or split targets. A fixed max_steps can therefore imply a different number of data passes, without proving overfitting. Log processed input tokens, active shifted targets, and retained original examples. Throughput and learning exposure use different denominators.
A packed batch resets position IDs for Ex B, but Ex B's response can still attend to Ex A. Is the packing path correct?
Answer
No. Reset positions isolate examples only when the attention path turns them into a segment mask, and here it evidently doesn't. The attention implementation must honor segment isolation, and shifted labels must retain the intended answer targets without creating a target across adjacent examples.
Evaluate hyperparameters through controlled experiments
A learning-rate sweep asks how much change this objective and parameterization can tolerate. Check task improvement and regressions on general capabilities together. Neither a frozen base nor a small learning rate guarantees preserved behavior.
The fragment below uses 1e-4 as an adapter candidate. That isn't a universal safe range or a fixed multiplier over full fine-tuning. Compare nearby rates under the same data exposure and evaluation rules. Dataset quality, optimizer, adapter scaling, model size, and loss normalization across microbatches all affect the result.
| Knob | Baseline to test | What decides whether it survives |
|---|---|---|
| Learning rate | a short sweep for the actual update surface | held-out task gains and regressions at matched exposure |
| Schedule | explicit warmup duration, decay shape, and final LR | observed early updates and behavior at the chosen checkpoint |
| Passes over data | a bounded pilot with measured retained targets | validation trends and task behavior, not a fixed epoch ceiling |
| Regularization & clipping | record decay groups and clipping threshold | compare changes in gradients, parameter updates, and held-out behavior |
| Precision | BF16 where supported; otherwise choose an explicit path | finite numerics and task parity under the actual hardware |
Warmup reduces early step sizes; it doesn't make AdamW's bias correction unnecessary or guarantee stability. A useful historical counterexample is InstructGPT's standalone SFT baseline: 16 epochs, no warmup, and cosine decay to 10% of its initial LR, with learning rates and epochs selected by searches.[3] That recipe isn't a recommendation to copy 16 epochs. It disproves a mandatory 3%-10% warmup or 1-3 epoch rule.
A cosine schedule's floor is configurable; standard Transformers cosine warmup decays to zero, while a minimum-LR variant can use a nonzero floor.[18] Resolve the actual scheduler instead of assuming 10%. AdamW applies decoupled decay separately from moment estimation. Decay doesn't bound distance from the pretrained weights, and clipping raw gradients doesn't directly bound AdamW's parameter update.[19] Inspect both gradients and update norms.
Effective batch size is also a token budget
The batch size configured on a single device isn't the batch size the optimizer sees during an update step. For synchronous training across complete accumulation windows, the number of items per update follows:
loader_items_per_update = per_device_batch * gradient_accumulation_steps * world_sizeWithout packing, a loader item is one demonstration. With packing, each item is a composite block containing multiple short demonstrations, so the formula counts packed blocks rather than raw examples. Track both raw examples and actual supervised tokens per update. The CPU check below models an unpacked dataset across a four-GPU setup:
per_device_batch = 4
grad_accum = 8
world_size = 4
examples_per_update = per_device_batch * grad_accum * world_size
supervised_tokens_per_microbatch_on_one_rank = [180, 212, 164, 220, 198, 204, 175, 207]
supervised_tokens_per_update = sum(supervised_tokens_per_microbatch_on_one_rank) * world_size
print(f"examples_per_update={examples_per_update}")
print(f"supervised_tokens_per_update={supervised_tokens_per_update}")examples_per_update=128
supervised_tokens_per_update=6240Always calculate the combined impact of batch changes rather than reasoning about single knobs. Notice how doubling world size or accumulation doubles the update volume:
def examples_per_update(per_device_batch, accumulation, world_size):
return per_device_batch * accumulation * world_size
run_a = examples_per_update(4, 8, 4)
run_b = examples_per_update(2, 16, 8)
print("run_a_examples_per_update=", run_a)
print("run_b_examples_per_update=", run_b)
print("ratio_b_over_a=", run_b / run_a)
assert run_b == 2 * run_arun_a_examples_per_update= 128
run_b_examples_per_update= 256
ratio_b_over_a= 2.0When sharing batch configurations, specify accumulation steps, world size, sequence packing mode, and supervised token counts.
Loss weighting across accumulation steps matters as well. Suppose one microbatch has one target loss of 4.0 and another has nine target losses of 1.0 each. Averaging batch means gives 2.5; the token mean is . Equal example weights and equal token weights are different objectives. For the token mean, normalize loss sums by active shifted targets across the complete update window.
microbatches = [[4.0], [1.0] * 9]
mean_of_means = sum(sum(batch) / len(batch) for batch in microbatches) / len(microbatches)
token_mean = sum(map(sum, microbatches)) / sum(map(len, microbatches))
assert mean_of_means == 2.5 and token_mean == 1.3
print(f"mean_of_means={mean_of_means:.1f}; token_mean={token_mean:.1f}")mean_of_means=2.5; token_mean=1.3In ordinary distributed data parallel training, rank gradients are averaged. If each rank divides its local loss sum by a globally summed token count, that averaging introduces another factor of world_size. Compensate for it, or use a trainer that performs the normalization correctly. Don't add the factor twice when the trainer already handles it. Unequal target counts, partial accumulation windows, and actual loss gradients still need checks beyond this scalar calculation.
What a resumable run must save
A model.safetensors file doesn't encode the optimizer history or next data batch. It may be part of an inference artifact, but serving also needs compatible architecture and tokenization. An adapter export additionally needs its base model and adapter configuration.
To reconstruct the continuation, retain the state your recipe actually uses:
- model weights or LoRA adapter weights
- optimizer state (AdamW first moment , second moment , and step counters)
- scheduler state (step counter, warmup phase, decay multiplier)
- global step and completed epoch indices
- sampler progress and random generator states, not only original seeds
- tokenizer revision and chat-template hash
- resolved training configuration and dataset manifests
- best validation metric achieved so far
- loss-scaler state if using FP16 dynamic scaling, plus any other stateful training machinery
Optimizer history determines direction and scaling. Scheduler progress determines the next LR, which does not automatically match the peak if reset incorrectly. Sampler and generator states determine upcoming data and randomness. Preserve the exact tokenizer/template artifacts and the adapter's base revision. Metadata labels or filenames alone don't guarantee identical contents or complete worker state.
For ordinary AdamW without AMSGrad, the update uses newly computed, bias-corrected moments:
Here is the current gradient and squaring is elementwise. A fresh optimizer with nonzero first computes and . Its update doesn't divide the current gradient by epsilon alone. Dropping state changes the trajectory, but doesn't imply a mandatory spike.[19]
Applying gradient 2.0 followed by -1.0 exposes how history controls direction: a fresh AdamW step follows the negative gradient upward, while retained positive momentum still moves the weight downward:
import copy
import math
import torch
weight = torch.nn.Parameter(torch.tensor(1.0, dtype=torch.float64))
optimizer = torch.optim.AdamW([weight], lr=0.1, weight_decay=0.0)
def update(parameter, opt, gradient):
opt.zero_grad(set_to_none=True)
parameter.grad = torch.tensor(gradient, dtype=torch.float64)
opt.step()
update(weight, optimizer, 2.0)
saved_weight = weight.detach().clone()
saved_optimizer = copy.deepcopy(optimizer.state_dict())
update(weight, optimizer, -1.0)
resumed = torch.nn.Parameter(saved_weight.clone())
resumed_optimizer = torch.optim.AdamW([resumed], lr=0.1, weight_decay=0.0)
resumed_optimizer.load_state_dict(saved_optimizer)
update(resumed, resumed_optimizer, -1.0)
fresh = torch.nn.Parameter(saved_weight.clone())
fresh_optimizer = torch.optim.AdamW([fresh], lr=0.1, weight_decay=0.0)
update(fresh, fresh_optimizer, -1.0)
assert torch.equal(resumed, weight)
assert resumed.item() < saved_weight.item() < fresh.item()
assert math.isclose(fresh.item() - saved_weight.item(), 0.1, abs_tol=1e-8)
print(f"uninterrupted={weight.item():.6f}; resumed={resumed.item():.6f}")
print(f"fresh_optimizer={fresh.item():.6f}; all_finite={bool(torch.isfinite(fresh))}")uninterrupted=0.873366; resumed=0.873366
fresh_optimizer=1.000000; all_finite=TrueThis controlled CPU run restores optimizer history while applying the same next gradient; it does not replay a stochastic LLM job. Prefer saving at completed optimizer steps. Mid-accumulation continuation also needs partial gradients and microbatch progress. Compare the next batch, LR, loss, and update against an uninterrupted reference. Bitwise parity depends on deterministic operations and a compatible environment.[20][21]

The simpler SGD-with-momentum check below isolates the same dependency: restoring velocity gives a next weight of 0.52; resetting it gives 0.70. The difference is 0.18, about 34.6% of 0.52. These are scalar update values rather than an AdamW variance calculation or a full-run loss curve.
A manifest lists required continuation artifacts. This schema check verifies key presence only without opening files or validating hashes:
required = {
"model_state",
"optimizer_state",
"scheduler_state",
"global_step",
"rng_state",
"sampler_state",
"tokenizer_version",
"chat_template_version",
"data_manifest",
"training_config",
"eval_manifest",
"best_metric",
}
resume_bundle = {
"model_state": "weights/step_600.safetensors",
"optimizer_state": "optimizer/step_600.pt",
"scheduler_state": "scheduler/step_600.pt",
"global_step": 600,
"rng_state": "rng/step_600.pt",
"sampler_state": {"epoch": 1, "batches_consumed": 120},
"tokenizer_version": "sql-sft-tokenizer-v3",
"chat_template_version": "llama3-sql-chat-v2",
"data_manifest": "data/sft_manifest_2026-05-20.json",
"training_config": "config/resolved_training.json",
"eval_manifest": "eval/sql_read_only_behavior_v4.json",
"best_metric": {"name": "task_pass_rate", "value": 0.97},
}
missing = sorted(required - resume_bundle.keys())
print("schema_complete=", not missing)
print("best_metric=", resume_bundle["best_metric"])schema_complete= True
best_metric= {'name': 'task_pass_rate', 'value': 0.97}A resumed run reloads model weights and global step but not optimizer, scheduler, sampler, or tokenizer-template versions. Can its next metrics be compared as uninterrupted continuation?
Answer
No. The next update may use different momentum, learning rate, data position, randomness, or token formatting. Treat it as a new run unless the complete resume manifest is restored and validated.
The scalar SGD-with-momentum example below makes the dependency on velocity explicit. Its fixed next gradient isolates optimizer state; no training data or model loss is evaluated:
import json
def step(state, gradient):
velocity = 0.9 * state["velocity"] + gradient
return {"weight": state["weight"] - 0.1 * velocity,
"velocity": velocity, "step": state["step"] + 1}
saved = step({"weight": 1.0, "velocity": 0.0, "step": 0}, 2.0)
reference = step(saved, 1.0)
restored = json.loads(json.dumps(saved))
resumed = step(restored, 1.0)
weights_only = step({**restored, "velocity": 0.0}, 1.0)
assert resumed == reference and weights_only != reference
print(f"uninterrupted={reference['weight']:.2f}; resumed={resumed['weight']:.2f}")
print(f"reset_momentum={weights_only['weight']:.2f}")uninterrupted=0.52; resumed=0.52
reset_momentum=0.70Choosing the best checkpoint
Choose checkpoints against the deployment objective on validation data. Teacher-forced loss measures likelihood of reference tokens given reference prefixes; it doesn't directly measure whether a freely generated reply refuses unsafe SQL writes while answering read-only queries. Generate outputs for task evaluation too, and keep a final test separate from repeated selection.
Unreliable selection criteria:
- lowest training loss alone (doesn't measure unseen-case performance)
- latest training step (risks overfitting or late degradation)
- file size or raw token count
Rigorous selection criteria:
- strict syntax and format pass rate (JSON schema compliance, valid SQL function-call formatting)
- task pass rate on held-out cases via deterministic query-parser checks or calibrated judges
- human evaluation on blinded, randomized completion pairs
- validation loss with a declared role in selection, rather than an assumed universal priority
In the validation table below, require format validity of at least 98%, then maximize task pass rate and break ties with validation loss, followed by the earlier step. In a real evaluation, report denominators, uncertainty, important failure slices, and regressions on general behavior.
checkpoints = [
{"step": 200, "val_loss": 1.91, "task_pass_rate": 0.93, "format_pass_rate": 0.99},
{"step": 400, "val_loss": 1.77, "task_pass_rate": 0.91, "format_pass_rate": 1.00},
{"step": 600, "val_loss": 1.79, "task_pass_rate": 0.97, "format_pass_rate": 0.98},
]
def select_checkpoint(rows):
eligible = [row for row in rows if row["format_pass_rate"] >= 0.98]
return max(eligible, key=lambda row: (row["task_pass_rate"], -row["val_loss"], -row["step"]), default=None)
best = select_checkpoint(checkpoints)
assert best is not None
assert select_checkpoint([]) is None
assert select_checkpoint([{**checkpoints[0], "format_pass_rate": 0.97}]) is None
assert select_checkpoint([{**checkpoints[0], "step": 600}, checkpoints[0]])["step"] == 200
assert select_checkpoint([{**checkpoints[0], "val_loss": 2.0}, {**checkpoints[0], "step": 400}])["step"] == 400
print("best_step=", best["step"])
print("best_task_pass_rate=", best["task_pass_rate"])
print("lowest_loss_step=", min(checkpoints, key=lambda row: row["val_loss"])["step"])best_step= 600
best_task_pass_rate= 0.97
lowest_loss_step= 400Step 400 shows the lowest validation loss (), but step 600 achieves the highest task pass rate () while clearing the format threshold (). The selection gate picks step 600. If no checkpoint meets the format threshold, the gate yields None, signaling that the recipe needs fixing before anything ships.

Single GPU first, then scale out
Don't start debugging conversation conv_102 across an eight-GPU cluster. If prompt templating or label masking contains subtle bugs, scaling out distributes those defects across collective communication calls rather than resolving them. Prove your pipeline on a single device first:
- single GPU execution
- small, representative evaluation split
- frequent checkpointing
- verified chat templating and response-only label masking
- clear task compliance metric
Validate the smallest feasible run before adding scale. Some required models can't fit one device at all; debug data schemas on tiny samples, then use the minimum necessary sharding. Test distributed loss normalization and checkpoint restore independently of the single-device run.
Signs that one GPU is still fine
A single device is feasible when weights, gradients, optimizer state, activations, and temporary buffers fit at the chosen precision and the run meets its time budget. A model that fits for inference may still fail this training-memory test.
Signs that you need FSDP or ZeRO
Consider FSDP or ZeRO when persistent model weights, gradients, and AdamW optimizer states exceed available GPU memory. Measure activation memory independently: sharding model parameters doesn't automatically eliminate activation overhead. Longer contexts often require activation checkpointing, reduced micro-batch sizes, or context parallelism. Adding more GPUs introduces network communication latency and won't speed up an improperly configured small job.
FSDP and ZeRO offer different state-sharding strategies; the chosen stage determines what is sharded.[22][23] Specify precision explicitly. As of TRL 1.17, TRL defaults to BF16 when FP16 isn't set; verify hardware support and numerical behavior on the target stack.[1]
Minimal operational skeleton
The row still follows the same path, whichever library applies the update:
dataset -> template -> tokenizer -> collator/mask -> trainer
-> periodic eval -> checkpoint save -> best-checkpoint exportTRL, torchtune, and the Alignment Handbook implement variations of this workflow.[1][24][25]
The configuration fragment below is an unexecuted integration blueprint for conversational prompt-completion rows. Its pilot disables packing until segment isolation is verified. API fields were checked against TRL 1.17.1 on November 4, 2026; pin a tested package version and model revision for a real run.
from trl import SFTConfig, SFTTrainer
args = SFTConfig(
output_dir="runs/sql-tool-sft-lora",
learning_rate=1e-4, # adapter baseline; tune for the target task
max_length=1024,
bf16=True, # only on supported hardware; verify numerics
per_device_train_batch_size=1,
gradient_accumulation_steps=8,
max_steps=200, # authored pilot budget, not a recommended duration
lr_scheduler_type="linear", # explicit schedule; reaches zero at the budget endpoint
warmup_steps=0, # authored candidate to compare, not a stability promise
seed=42,
data_seed=42,
packing=False, # enable only after attention and loss-isolation checks
eval_packing=False, # keep validation examples separate
completion_only_loss=True, # train on the completion field, not the prompt
loss_type="nll", # explicit standard loss for this adapter fragment
eval_strategy="steps",
eval_steps=100,
save_steps=100,
)
trainer = SFTTrainer(
model=model_with_lora_adapters,
args=args,
train_dataset=train_rows, # prompt/completion are role/content message lists
eval_dataset=held_out_rows,
processing_class=tokenizer,
)This fragment omits model loading, adapter initialization, and dataset construction. Convert the earlier authoring rows to message lists so the trainer can apply the checkpoint's template; raw string fields don't automatically become a chat conversation. We explicitly choose loss_type="nll", which is also the current default. TRL now documents "chunked_nll" as a deprecated alias of "nll", rather than a separate objective.[1] The implementation can compute NLL through a chunked projection path; pin and inspect that implementation separately from the target objective.[26] The evaluation loop computes teacher-forced loss rather than task or format pass rates, so generate and grade validation replies separately before applying the selection rule.
Common pitfalls
A regression does not name the stage that failed. Check the split, the mask, and the checkpoint before you spend a sweep on the learning rate.
Confusing the objective with the trainable parameters
Saying "we tried LoRA instead of SFT" mixes two distinct decisions. LoRA and full fine-tuning describe which weights are updated; SFT defines the supervised next-token cross-entropy objective. Define the behavioral target and objective first. Compare adapter tuning against full-weight tuning only when parameter capacity or memory budget is the variable under test.
Evaluating examples that share a case with training
High validation scores followed by failures on new documents can reflect leakage, distribution shift, or a mismatched metric. Check whether related cases crossed splits and whether the grouping actually tests the deployment claim. A thread-disjoint split doesn't test unseen schemas shared across those threads.
Reporting micro-batch as if it were the real batch
When two experiments claim "batch size 4" but demonstrate drastically different stability, check accumulation steps and GPU count before modifying the optimizer. Always publish per-device batch size, gradient accumulation steps, world size, packing mode, and total supervised tokens per update step.
Saving weights without resume state
Changed loss or batches after resume prompts a state comparison; it doesn't uniquely diagnose an omitted checkpoint field. Check optimizer counters, LR, upcoming data, RNG states, precision/scaler state, and artifact identities against the uninterrupted reference.
Using a packing path that leaks across examples
Incoherent answers don't uniquely diagnose packing leakage. Test attention reachability and shifted-label isolation directly. For independent demonstrations, use a verified segment-aware path; for intentional continuous-text training, cross-document context may be part of the chosen objective. A reset position ID or EOS token isn't itself an enforced barrier.
Selecting the best checkpoint by train loss
Training loss measures reference-token likelihood on seen examples, not memorization by itself. Use validation behavior and declared constraints for selection, then assess the frozen choice on the final test. Checks need a task-appropriate grader; fluent output and low loss don't establish task correctness.