Training Run Operations

The weights directory for policy-sft-v4 is still on disk after a 4-GPU job dies at step 1840. The next morning someone launches with resume_from_checkpoint=weights/step_1840. Loss jumps, the learning rate restarts, and yesterday's shuffled rows come around again. The file was an export, not a resume.

LoRA already showed that on policy-lora-v1: adapters can replace full-weight updates when the task is a thin policy overlay. This chapter treats the run as the system: what "resume" means on a sharded SFT save, how batch math changes the update, and which method to launch given data volume, domain shift, and GPU budget.

Three operations people call resume

People use "resume" for three different jobs. Mixing them is how policy-sft-v4 looked like it continued when it actually restarted.

Continue is the only operation that preserves Adam moments, the learning-rate schedule, and which rows have already been seen. Initialize is valid, but its first metrics aren't comparable to the interrupted curve. Export shouldn't be fed back into a trainer as if the optimizer still existed. A merged LoRA dump is initialize or export, not continue: torchtune resumes adapters from adapter_model.pt plus the original base, and loading merged weights together with adapters is an error.^{[1]Reference 1Checkpointing in torchtune.https://meta-pytorch.org/torchtune/stable/deep_dives/checkpointer.html}

The SFT resume list still holds: model, optimizer, scheduler, step, RNG and sampler, tokenizer and chat-template versions, data and eval manifests, and the best held-out metric so far. The new question is whether that bundle is one complete distributed artifact, and which of the three operations you are actually performing.

If you load weights_only and keep plotting loss on the old chart, you are lying about continuity. Start a new run id.

A trainer reloads model weights and global_step=1840, but not optimizer, scheduler, or sampler state. Can the next 200 steps be compared as uninterrupted continuation of policy-sft-v4?

Answer

No. Call it initialize, not continue. Momentum, learning rate, and data position all changed, so the curve is a new experiment that happens to start from those weights.

A checkpoint is a distributed artifact

On one GPU, torch.save of a dict can be "the checkpoint." On four FSDP ranks, each rank holds a shard of parameters, gradients, and Adam moments. ZeRO exists largely because those moments dominate model-state memory.^{[2]Reference 2ZeRO: Memory Optimizations Toward Training Trillion Parameter Models.https://arxiv.org/abs/1910.02054} A save that finishes on ranks 0-2 and dies on rank 3 isn't a checkpoint. It's a partial directory.

A checkpoint manifest is the inventory that says which files, sizes, hashes, step, topology, and related state belong together. Directory existence isn't completion. Publish complete=true only after every required shard verifies. Refuse to load if a shard is missing, unexpected, or hash-mismatched.

Two save formats show up in real jobs:

DCP isn't torch.load(path). It still writes multiple files (at least one per rank), but it operates in place: you allocate the current model's state_dict so DCP has sharding info and storage to fill, then it reshards into that layout.^{[3]Reference 3PyTorch Distributed Checkpoint (DCP) Recipehttps://docs.pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html} That's how a 4-GPU policy-sft-v4 dump can continue on 8 GPUs, as long as the new process group can read the checkpoint directory (usually a shared filesystem). Naive rank files don't do that. Changing data-parallel width is the DCP path. Changing tensor-parallel or pipeline layout usually needs an explicit converter, which is what DeepSpeed's universal export is for.

The topology itself is part of the resume contract. Record world_size, FSDP/ZeRO stage, and whether the dump is DCP, universal, or rank-local. A loader that ignores those fields will either crash or silently rebuild optimizer state.

policy-sft-v4 saved four rank-local files at step 1840. You restart on eight GPUs and torch.load each rank file by rank index. Why is that not a DCP-style resume?

Answer

Rank-local files are bound to the layout that wrote them. DCP (or a DeepSpeed universal checkpoint) reshards tensors using the current model's sharding. Indexing rank0.pt on an 8-GPU job either crashes or loads the wrong slice of optimizer state.

Preemption saves the last complete step

Cluster jobs get a terminate signal (SIGTERM). Spot VMs disappear. A rank hits an NCCL timeout (NVIDIA's collective communication library giving up on a stuck all-reduce). The useful unit isn't "the process was running." It's "the last optimizer step whose bundle is durable."

Write checkpoints at optimizer-step boundaries, not mid-backward. Gradient accumulation and GradScaler share that same boundary, from the mixed-precision chapter. If accumulation takes eight microbatches, a kill during microbatch 5 has no new complete step. Keep the previous complete dump. Then:

1. Drain in-flight microbatches only if you can finish the accumulation window before the kill deadline.
2. Save model, optimizer, scheduler, sampler, and RNG together.
3. Verify shards and hashes.
4. Publish the manifest with complete=true and point latest at that step.
5. Only then is it legal for a new process group to continue.

Cadence is a cost tradeoff. Saving every step on a 70B ZeRO-3 job can dominate the step time. Saving once per hour on a 20-minute preemptible slice wastes the slice. A practical rule: the expected lost work after a kill should stay small versus the save cost. For policy-sft-v4 on four GPUs with a 90-second save, a checkpoint every 50-100 steps is usually enough. Measure it.

Final export for eval still isn't a resume bundle. torchtune distinguishes intermediate checkpoints from the artifact you ship.^{[1]Reference 1Checkpointing in torchtune.https://meta-pytorch.org/torchtune/stable/deep_dives/checkpointer.html} Keep both names in the manifest so nobody resumes from an eval export.

What "the batch" is

SFT already used this identity. It's the run's unit of update, so it belongs in the ops card, not only in the trainer config:

$$B_{\mathrm{global}} = B_{\mathrm{micro}} \times K \times D$$

$B_{\mathrm{micro}}$ is the per-device microbatch, $K$ is accumulation steps, and $D$ is the data-parallel world size. For policy-sft-v4:

$$B_{\mathrm{global}} = 2 \times 8 \times 4 = 64 \text{ examples per optimizer step}$$

If packing is on, also log supervised tokens per update. Two runs with 64 examples can still differ by 2× in tokens if replies are longer.

When someone says "we doubled GPUs and loss exploded," expand the identity before touching Adam. Doubling $D$ while leaving $B_{\mathrm{micro}}$ and $K$ fixed doubles $B_{\mathrm{global}}$. That's a different recipe.

halved_accum_to_keep_64 keeps the example budget and still doubles tokens if each rank still sees 5,200 supervised tokens per window. Publish all four numbers: microbatch, $K$, $D$, and supervised tokens.

Learning-rate scaling, warmup, and when the rule breaks

Goyal et al. trained ImageNet with large minibatches by pairing two rules: when the minibatch grows by $k$, multiply the learning rate by $k$, and warm that rate up instead of starting at the scaled peak.^{[5]Reference 5Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.https://arxiv.org/abs/1706.02677} The intuition is that $k$ averaged gradients are a less noisy estimate of the full-data gradient, so a proportionally larger step keeps the expected update similar. Goyal's measurements were large-batch SGD on ImageNet. The same linear rule is the usual first bet for AdamW SFT, then you measure clip-hit rate and held-out accuracy instead of assuming it transferred.

Their gradual warmup starts at the small-batch learning rate $\eta$ and ramps to $k\eta$. For policy-sft-v4, a LoRA peak of $2\times 10^{-5}$ at 64 examples/update becomes $4\times 10^{-5}$ at 128 examples/update. The first warmup steps stay near $2\times 10^{-5}$ and rise. Jumping straight to $4\times 10^{-5}$ on step 1 is the failure that warmup exists to prevent. Many trainers instead warm from near zero to the new peak, which is even more conservative. Either way, don't open at the scaled peak.

The rule isn't unbounded. McCandlish et al. describe a critical batch size: past that scale, extra batch growth buys less progress per compute.^{[6]Reference 6An Empirical Model of Large-Batch Training.https://arxiv.org/abs/1812.06162} Goyal's ResNet-50 runs matched small-batch ImageNet accuracy up to minibatch 8192 on 256 GPUs, then error rose. An SFT job with a few thousand labeled rows can leave the useful region much earlier. If you 8× the batch, 8× the LR, skip warmup, and watch NaNs, you didn't disprove Adam. You left the region where linear scaling is a reasonable first bet.

Warmup and AdamW weight decay are part of the same recipe, not decorations. AdamW decouples weight decay from the adaptive update so "L2" doesn't get rescaled by the second moment.^{[7]Reference 7Decoupled Weight Decay Regularization.https://arxiv.org/abs/1711.05101} Changing decay when you change batch size is a second experiment. Keep it fixed unless you are studying regularization.

The first warmup step stays at $2\times 10^{-5}$, the old peak, and only then climbs to $4\times 10^{-5}$. Freshly initialized adapter rows make that even more true: keep early steps small on purpose.

Instability: data, optimizer, or scaler

Loss went to NaN at step 1847 after the 8-GPU restart. Clipping fired on 40% of steps. More than one layer can produce that screenshot.

Clipping rescales a gradient whose norm exceeds a threshold. It can stop one pathological batch from destroying Adam moments. It can't tell you whether that batch was garbage text, an FP16 overflow, or a learning-rate mistake. If clip-hit rate jumps from 1% to 40% after a topology change, don't raise the clip. Find the layer in the table.

The mixed-precision chapter already showed the accumulation bug: step/update belongs on the optimizer boundary, not each microbatch. An 8× accumulation window that steps eight times has quietly turned $K$ into 1 and multiplied the effective learning rate.

After doubling world_size, clip-hit rate jumps to 40% and the first ten steps look noisy. Warmup is 0 and peak LR was multiplied by 2. Is this evidence that AdamW is unstable on access-policy SFT?

Answer

No. You changed $B_{\mathrm{global}}$ and the peak rate together and skipped warmup. Treat it as a schedule/batch-card failure until a matched warmup run still explodes.

Which adaptation method to launch

Resume and batch math tell you how to operate a job. They don't tell you which job to start. Full SFT, LoRA, QLoRA, continued pretraining, and distillation are answers to different constraints. The SFT chapter separated objective from parameterization. Here the same access-policy assistant has to pick under data volume, domain shift, and GPU budget.

Work the same product through five honest cards:

1. 2,000 labeled escalation replies, 1× 24GB GPU, base model already speaks support English. The missing piece is the new rotation procedure. Launch LoRA SFT. Full FT wastes memory on a small behavior delta.
2. Same 2,000 rows, 70B base, 1× 24GB GPU. LoRA optimizer state still may not fit with activations. QLoRA is the method that exists for that memory wall, not a better algorithm than LoRA. The paper's headline was 65B on one 48GB GPU. 70B on 24GB is the same idea, tighter.
3. Unlabeled internal access-policy corpus, ~2B tokens, 8× 80GB, and the base model mangles product nouns. That's continued pretraining, then a small SFT. SFT on 2,000 rows won't teach a dialect the checkpoint never saw.
4. 200,000 high-quality replies that redefine tone, tools, and refusal policy, 8× 80GB. Full SFT is in play. LoRA can still be the first experiment, but don't pretend a rank-8 adapter is guaranteed to match full-weight capacity on a broad rewrite.
5. A strong 70B teacher already follows policy, and production needs a 7B on-prem student. Distill after the teacher is good. Distillation doesn't replace getting the teacher right.

QLoRA is quantized LoRA, not a third objective. If LoRA already fits, QLoRA's job is over. Distillation is Knowledge Distillation. CPT is Continued Pretraining. Full SFT vs adapters is the parameterization split from Supervised Fine-Tuning. This chapter's job is to pick the card before you spend the cluster.

After SFT quality exists, preference is a different decision

Once policy-sft-v4 can produce acceptable answers, the next training question is often "which answer is preferred?" That's not a resume problem and not a batch-card problem. Reward models, DPO, GRPO, KTO, and ORPO are post-training families that rank or reinforce among candidates. GRPO already shows up inside RLVR and the slime deep dive. Don't smuggle them into the LoRA vs CPT table.

A compact gate:

Operate those runs with the same contracts: a continue-capable checkpoint, an honest batch card, and a method that matches the signal you actually have.

A run card you can fail closed

Before policy-sft-v4 is allowed to consume another slice, fill this card. If a field is unknown, the job doesn't start.

A production check that is small enough to run in CI: restore the last complete dump on a different world size (4 → 2 or 4 → 8) with DCP, confirm optimizer step count and scheduler LR match the manifest, and refuse to continue if the sampler would replay the first 368 batches.

That is training ops. The model file is one tensor dump. The run is a system that can die, restart, keep its update honest, and justify why those weights were the ones that moved.

Mastery check

Key concepts

• Continue, initialize, and export are different operations
• Sharded checkpoints need a complete manifest; DCP reshards, rank-local files don't
• Preemption restores the last complete optimizer step
• Global batch is microbatch × accumulation × world size
• Linear LR scaling needs warmup and stops being a free lunch at large batches
• Instability is data, schedule, or scaler/accumulation until proven otherwise
• Full SFT, LoRA, QLoRA, CPT, and distillation bind different constraints

Evaluation rubric

• Foundational: Classify a weights-only reload as initialize, not continue, and compute $B_{\mathrm{global}}$ from microbatch, $K$, and $D$.
• Intermediate: Reject an incomplete rank-local dump and a world-size mismatch, and scale peak LR with warmup when $D$ doubles.
• Advanced: Route a NaN postmortem to data vs schedule vs scaler, and choose LoRA, QLoRA, CPT, full SFT, or distillation from a constraint card.

Follow-up questions

Why does DCP ask for the current model state_dict before loading, unlike torch.load(path)?

Answer

DCP fills pre-allocated tensors using the live sharding. The new world size's layout has to exist first so load-time resharding has somewhere to put each slice.

You keep B_micro=2 and K=8, raise D from 4 to 8, and keep peak LR at 2e-5 with 200 warmup steps. What did you change, and is that automatically safe?

Answer

You doubled examples per update to 128. Leaving LR fixed is a different recipe from Goyal's linear rule, not automatically safer. Compare against a warmed 4e-5 run and watch clip-hit rate and held-out accuracy.

Common pitfalls

• Plotting an initialized run on the interrupted loss curve
• Pointing latest at a directory before rank 3 finishes writing
• Doubling GPUs and calling the old batch size "unchanged"
• Raising grad clip until NaNs stop, without inspecting the microbatch
• Using QLoRA when LoRA already fits, or LoRA when the base model doesn't speak the domain