Instruction Tuning & Chat Templates

The answer can be right and the assistant can still fail. A model trained with one set of speaker labels may receive another at serving time, then continue with a new user turn instead of answering. The text looks plausible, so a normal spot check can miss the boundary error.

The last chapter scored CodeAssist on private policy cases: freeze-deploy, rollback-runbook, and secret-rotation. Keep those rows as holdout. They tell you whether an answer is grounded, not how to teach the assistant to write that answer. This lab takes one related policy row and follows it through data curation, serialization, loss, generation, and release review.

A base language model predicts plausible next tokens. It isn't reliably trained to read a user turn, answer as a policy assistant, cite the evidence, and stop. Instruction tuning adapts the model toward that behavior; supervised fine-tuning (SFT) does so from approved instruction-response examples. The same chat template has to survive from data prep to serving.

Use a separate demonstration so training never collides with the eval set you just built:

Policy evidence: Service-account keys older than 14 days require a rotation ticket before continued use.

Before moving on, predict the first supervised choice. In the row above, should training raise the probability of the user's question, the grounded assistant answer, or both? The table below turns that choice into a concrete contract.

One example defines a behavior contract

A pretrained causal language model has one basic mechanism: given preceding tokens, assign probabilities to the next token. It can sometimes answer a question because conversations appeared in pretraining data, but answering isn't yet a dependable product contract.

For an access-policy assistant, one SFT training row can look like this:

During SFT, the optimizer increases the probability of target response tokens conditioned on the turns that came before them. Prompt text remains useful context even when it isn't a supervised target.

In InstructGPT, supervised demonstrations were the first post-training stage before preference feedback was used to refine response quality.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} That sequence gives this lab its order: make one grounded answer and token contract correct before adding preference judgments.

SFT isn't a guarantee that the model only changes style or formatting. Fine-tuning can alter factual behavior and task competence too. LIMA provides a useful, narrower result: with a capable base model, a carefully curated set of 1,000 demonstrations produced strong response-format and instruction-following behavior in its experiments.^{[2]Reference 2LIMA: Less Is More for Alignment.https://arxiv.org/abs/2305.11206}

Treat that result as evidence for data quality, not a promise that every task needs little data. A policy assistant still needs coverage, grounding checks, and holdouts that are separate from training rows.

Chat messages eventually become one token stream

Applications store a conversation as structured records. The language model receives tokens. A chat template is the serialization rule between those two representations: it adds role markers, delimiters, whitespace, and, when appropriate, the cue that a new assistant response should begin.^{[3]Reference 3Transformers Documentation: Chat templates.https://huggingface.co/docs/transformers/main/en/chat_templating}

Take the policy row from above with a complete user turn. Before looking at the figure, predict the final token sequence for generation: should it close the user message and stop, or close it and open an assistant turn? That one boundary decides whether the model answers or keeps writing the transcript.

Model families don't share one universal chat serialization:

Those strings are examples of checkpoint-specific formats, not a menu of interchangeable wrappers. Feeding [INST] formatting to a checkpoint trained with another role-token layout may still produce text, but you've changed its input distribution. That is the serving failure from the opening: readable text doesn't mean the model saw the context it was trained to continue.

This lesson focuses on system, user, and assistant turns. Production templates often also serialize tool messages, tool_calls, and multimodal parts; those schemas are out of scope here and are covered when you wire tool-using agents. Don't treat three-role ChatML as the full production contract.

Render training, fresh generation, and prefill separately

The smallest useful template exercise is to render the same support task in three modes. Before running it, predict which ending belongs to each mode:

1. A completed training transcript contains the known assistant answer.
2. A new inference request ends at an assistant-start cue.
3. A prefill already contains the beginning of an assistant answer and asks the model to continue it.

The last two modes aren't the same. Starting a new assistant turn and continuing an existing assistant message should never happen at once. One adds an assistant boundary; the other leaves the existing assistant text open.

Hugging Face tokenizers implement this idea with apply_chat_template. For generation, add_generation_prompt=True appends a start-of-assistant sequence only when that particular template defines one. For a response prefill, continue_final_message=True keeps the final assistant content open, and the documentation treats combining both flags as an error.^{[3]Reference 3Transformers Documentation: Chat templates.https://huggingface.co/docs/transformers/main/en/chat_templating}

The flag names describe intent, not a universal string. A model can need a visible assistant header, or it can begin the response immediately after a user delimiter. Let the checkpoint's tokenizer decide which tokens belong at the boundary.

This snippet intentionally uses the tokenizer attached to the checkpoint rather than recreating the format with string concatenation. For Mistral-7B-Instruct-v0.1, the shipped template already places generation after [/INST]; add_generation_prompt=True doesn't append a separate assistant-start token. Llama 3 Instruct does append <code>&lt;&#124;start_header_id&#124;&gt;assistant&lt;&#124;end_header_id&#124;&gt;</code>.^{[3]Reference 3Transformers Documentation: Chat templates.https://huggingface.co/docs/transformers/main/en/chat_templating}

If you render text with tokenize=False and tokenize it in a later step, pass add_special_tokens=False; otherwise BOS or EOS tokens may be inserted twice. The safer path is to let apply_chat_template tokenize the rendered sequence in one call.^{[3]Reference 3Transformers Documentation: Chat templates.https://huggingface.co/docs/transformers/main/en/chat_templating}

⚠️ Common mistake: Hugging Face's text-generation pipeline defaults to add_generation_prompt=True. If the last message is already an assistant turn, the pipeline switches to continue_final_message=True and treats that text as a prefill. Set the flags yourself when you want a new assistant turn, or when you want a completed assistant message to stay closed.^{[3]Reference 3Transformers Documentation: Chat templates.https://huggingface.co/docs/transformers/main/en/chat_templating}

Whitespace changes the token contract

Jinja renders indentation, line breaks, and trailing spaces unless its whitespace-control markers remove them. Hugging Face warns that extra whitespace absent from training can harm performance and recommends - markers around template blocks and expressions.^{[5]Reference 5Transformers Documentation: Writing a chat template.https://huggingface.co/docs/transformers/main/en/chat_templating_writing} A particularly quiet failure is spaces immediately before a special token. The transcript still looks readable, but the tokenizer receives a different byte sequence than the training run.

Compare a loose template with an exact one. Before running the check, predict whether .strip() can repair the loose result. It can't: the three spaces sit inside the sequence, before <|eot|>.

Store a template revision or digest in the training manifest, but also keep golden rendered strings and token IDs. A matching filename doesn't prove that a serving engine uses the same Jinja whitespace semantics. Render the same messages through training and serving paths, compare bytes and token IDs, and block deployment on any unexplained difference.

Before rendering thousands of rows, reject malformed dialogue structure. Predict which of the three role lists below should fail: a missing assistant answer, two user turns in a row, or a valid system-user-assistant exchange. An assistant reply without a preceding user request is a bad supervised example even if its text is fluent.

Fine-tuning data must teach supported answers

Good formatting can't rescue bad supervision. If a training row claims that stale keys can wait 30 days when the policy says 14 days with a rotation ticket, SFT reinforces an unsupported answer.

Decide what behavior each accepted row is allowed to teach before you worry about how many rows you can generate.

Self-Instruct demonstrated a repeatable way to expand instruction data: start from human-written tasks, generate new instructions and responses, then filter invalid or similar rows before fine-tuning.^{[7]Reference 7Self-Instruct: Aligning Language Models with Self-Generated Instructions.https://aclanthology.org/2023.acl-long.754/} A product team can use the same pattern with policy questions, but it must validate answers against source clauses rather than trusting a generator's confidence.

The next lab treats each assistant answer as a candidate SFT row. A row is accepted only if it contains required policy facts and omits a known unsupported claim.

The filter is deliberately small. A real data pipeline also checks duplicated instructions, personal information, unsafe replies, role ordering, length limits, and human-review requirements for high-risk cases. Version the evidence snapshot, generator prompt, filters, and accepted dataset together. Otherwise you won't know which change caused a behavior regression.

The output gives us a useful failure: two accepted stale-key paraphrases and one admin-access row still don't demonstrate how the assistant should answer a source-citation question. More accepted rows aren't the fix; missing intent coverage is.

Which tokens should receive the loss?

Every completed chat transcript contains tokens from the system prompt, user question, and assistant answer. Before choosing an objective, predict which tokens should move the weights when the policy answer is wrong: the prompt, the answer, or both. That choice determines what the gradient budget emphasizes.

Two choices matter:

Assistant-only loss is common, but it isn't the definition of SFT. TRL exposes it as assistant_only_loss=True for conversational data only when the chat template can identify assistant spans through {% generation %} and {% endgeneration %} markers. For known families such as Qwen3, current TRL releases patch the template when that flag is on. Inspect the resolved template for the checkpoint you train.^{[8]Reference 8TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer} If the span mask is wrong, you may silently train on user text or mask out the answer you meant to learn.

The prompt still stays visible to the model. Setting a label to -100 removes that position from cross-entropy; it doesn't erase the context that lets the assistant answer.

In a causal language model, each target is the token predicted from the tokens before it. Trainers usually shift labels internally so labels[i] is what the model should emit after position i-1. The lab below shows those supervised targets after the shift: assistant words and the assistant turn terminator receive loss, while role markers, system text, and user text get the ignore label -100.

Use this kind of inspected tiny batch before launching training. A training-loss curve can't reveal that your boundary finder shifted one token too far and trained the wrong span.

This calculation makes the objective choice visible. Full-sequence loss scores nearly the whole serialized row; assistant-only loss scores only the desired answer span and its terminator. If the assistant target count unexpectedly drops to zero, stop before tuning the learning rate: the mask, not optimization, is broken.

Packing saves padding, but isolation is a separate setting

Once you know which tokens are targets, the next waste is padding around short chats. If every short SFT row is padded to a long context window, the accelerator spends much of its work processing padding. Packing fills a window with several short sequences instead. TRL supports packing as a training configuration for this reason.^{[8]Reference 8TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer}

Before turning it on, predict the safety question: may a token in conversation B attend to earlier tokens in conversation A? Packing answers how rows share capacity; isolation answers whether their histories can mix.

Current TRL makes that distinction in the packing strategy. packing=True defaults to packing_strategy="bfd" (best-fit decreasing), and that path enables padding-free batching. With FlashAttention 2 or 3, the trainer passes position_ids that restart at each packed sample, so attention stays inside the sample.^{[8]Reference 8TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer}

The wrapped strategy concatenates then splits at max_length; ordinary causal attention can leak across that cut. Without a FlashAttention kernel, padding-free packing can contaminate even when you meant to isolate.^{[8]Reference 8TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer}

Don't treat packing=True as a synonym for block-diagonal attention. If each policy conversation must be an independent supervised example, read the trainer docs for the version you run and verify the boundary with a tiny packed batch.

First measure why packing is tempting. Four short training rows padded separately to a 16-token window waste most of their capacity; filling shared windows reduces that waste.

The utilization check makes the tradeoff tangible: packing raises this example from 42.2% to 84.4% of each allocated window. That gain says nothing about cross-row visibility, so now inspect the boundary mask separately.

Suppose two three-token rows are packed into one sequence:

For strict isolation, the causal self-attention matrix contains two blocks:

$$M = \begin{bmatrix} 1 & 0 & 0 & 0 & 0 & 0 \\ 1 & 1 & 0 & 0 & 0 & 0 \\ 1 & 1 & 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 & 1 & 0 \\ 0 & 0 & 0 & 1 & 1 & 1 \end{bmatrix}$$

Each row is a query position; each column is a possible earlier key position. The zeros at query position 4 under key positions 0 through 2 stop the second conversation from reading the first one. The lab uses 0-based indexes, so position 4 is the middle token of conversation B.

Don't assume an option named packing=True constructs this matrix in every engine. TRL's default BFD path isolates sequences when FlashAttention is in use; wrapped packing doesn't. Inspect the implementation you'll train and run a small boundary test.

Production failures are usually contract failures

Once a checkpoint has been fine-tuned, evaluate it with the private cases you built in the previous chapter. A drifted endpoint might still return fluent text while forgetting the policy evidence or continuing as another role. Log the rendered prompt, tokenizer revision, template revision, truncation policy, and generation settings alongside every result. Otherwise an answer regression could be a template deployment bug rather than a model change.

Before reading the table, predict the first diagnostic for that endpoint. Check the final rendered tokens and special-token count before blaming weights or generation temperature.

This minimal deployment manifest check can't measure model quality, but it prevents shipping an endpoint whose serialization contract is known to differ from the fine-tuning run. Predict which candidate should be blocked before running it: the one with a different template revision and duplicate-special-token setting.

A matching manifest earns the right to run quality evaluations; it doesn't prove the model is ready. Re-run grounded policy cases, critical failure slices, p95 latency checks, and human review checks after every fine-tune or serving change.

The final executable check joins template work back to evaluation. Before running it, predict whether a faster candidate can release with one critical policy error. Template parity is mandatory, but a correctly formatted candidate still ships only if grounded cases and operational limits pass.

At release time, the checks form a dependency chain: validate rows, render exact bytes, supervise intended spans, isolate packed histories, then run grounded and operational evaluations. When a chat checkpoint changes, that order tells you where to look before retraining.

Mastery check

Key concepts

• Base-model continuation versus instruction-tuned response behavior
• SFT examples grounded in approved evidence
• Chat-template serialization and checkpoint-specific markers
• Fresh generation versus assistant prefill
• Full-sequence versus assistant-only loss
• Synthetic-data acceptance checks
• Packed-sequence isolation as a verified design choice
• Training and serving template parity

Evaluation rubric

• Foundational: Explains why a base model can complete dialogue text without reliably acting as a support assistant.
• Intermediate: Renders completed, generation, and prefill versions of one conversation without mixing their boundary rules.
• Intermediate: Builds an assistant-only target mask and explains why that's a choice rather than a universal SFT requirement.
• Advanced: Designs a data and serving audit that rejects unsupported policy rows and template drift before evaluation.

Common pitfalls

• Training on plausible synthetic answers without checking them against policy evidence.
• Hand-formatting messages with delimiters that don't match the checkpoint's tokenizer template.
• Treating assistant_only_loss or packed isolation as automatic behavior without inspecting the trainer.
• Using a new-assistant generation cue when the request already contains an assistant prefill.
• Declaring a fine-tuned checkpoint better before rerunning grounded private evaluations.

Follow-up questions

Why can a base model produce conversation-looking text without being a reliable assistant?

Answer

It predicts likely continuations from its token context. Dialogue appeared in pretraining data, so it may continue a conversation, but no dependable product contract says it must answer the user's request, follow policy evidence, or stop at an assistant boundary. SFT supplies examples that reward those continuations.

When do you use add_generation_prompt=True, and when do you use continue_final_message=True?

Answer

Use a generation prompt when the final input turn is complete and you need to start a fresh assistant answer, if the checkpoint's template uses such a cue. Use final-message continuation when an assistant response has already begun, such as a JSON prefill. They represent different contexts and shouldn't be enabled together.

Why isn't assistant-only loss the definition of supervised fine-tuning?

Answer

SFT means optimizing a model on supervised example sequences. A trainer can compute loss across the full transcript or only selected assistant spans. Assistant-only loss is useful when you want gradient focused on responses, but it requires correct template-aware span masks and remains an objective choice.

What must you verify before using packed support conversations as independent examples?

Answer

Verify whether tokens in one packed conversation can attend to tokens from an earlier conversation. Isolation comes from the packing strategy and attention kernel, not from packing itself. Current TRL BFD packing plus FlashAttention isolates packed rows; wrapped packing can leak. Test a tiny packed batch instead of assuming the flag name implies a block-diagonal mask.

Practice extension

Add a third policy row for a source-citation requirement and a second rejected answer that invents automatic approval. Extend filter-grounded-sft-rows.py to print which required fact is missing or which forbidden claim was found. Then add its accepted answer to the rendering and label-building labs. Show exactly which assistant tokens become supervised targets and exactly which evidence allowed the row into training.