Instruction Tuning & Chat Templates

The policy-answering assistant now has evaluations. One private case asks about a stale service-account key, with this evidence:

Policy evidence: Service-account keys older than 14 days require a rotation ticket before continued use.

A candidate model may retrieve the right clause and still respond badly. A base language model predicts plausible next tokens; it isn't reliably trained to interpret a user turn, answer as a policy assistant, cite the policy, and stop. Supervised fine-tuning (SFT) teaches that response behavior, and the same chat template must survive from data preparation to serving.

One example defines a behavior contract

A pretrained causal language model has one basic mechanism: given preceding tokens, assign probabilities to the next token. It can sometimes answer a question because conversations appeared in pretraining data, but answering isn't yet a dependable product contract.

For an access-policy assistant, one SFT training row can look like this:

During SFT, the optimizer increases the probability of target response tokens conditioned on the turns that came before them. In InstructGPT, supervised demonstrations were the first post-training stage before preference feedback was used to refine response quality.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

SFT isn't a guarantee that the model only changes style or formatting. Fine-tuning can alter factual behavior and task competence too. LIMA provides a useful, narrower result: with a capable base model, a carefully curated set of 1,000 demonstrations produced strong response-format and instruction-following behavior in its experiments.^{[2]Reference 2LIMA: Less Is More for Alignment.https://arxiv.org/abs/2305.11206} Treat that as evidence for data quality, not a promise that every task needs little data.

Chat messages eventually become one token stream

Applications store a conversation as structured records. The language model receives tokens. A chat template is the serialization rule between those two representations: it adds role markers, delimiters, whitespace, and, when appropriate, the cue that a new assistant response should begin.^{[3]Reference 3Transformers Documentation: Chat templates.https://huggingface.co/docs/transformers/main/en/chat_templating}

Model families don't share one universal chat serialization:

Those strings are examples of checkpoint-specific formats, not a menu of interchangeable wrappers. Feeding [INST] formatting to a checkpoint trained with another role-token layout may still produce text, but you have changed its input distribution.

Render training, fresh generation, and prefill separately

The smallest useful template exercise is to render the same support task in three modes:

1. A completed training transcript contains the known assistant answer.
2. A new inference request ends at an assistant-start cue.
3. A prefill already contains the beginning of an assistant answer and asks the model to continue it.

The last two modes aren't the same. Starting a new assistant turn and continuing an existing assistant message should never happen at once.

Hugging Face tokenizers implement this idea with apply_chat_template. For generation, add_generation_prompt=True appends a start-of-assistant sequence only when that particular template defines one. For a response prefill, continue_final_message=True keeps the final assistant content open, and the documentation treats combining both flags as an error.^{[3]Reference 3Transformers Documentation: Chat templates.https://huggingface.co/docs/transformers/main/en/chat_templating}

This snippet intentionally uses the tokenizer attached to the checkpoint rather than recreating the format with string concatenation. For Mistral-7B-Instruct-v0.1, the shipped template already places generation after [/INST]; add_generation_prompt=True doesn't append a separate assistant-start token. Other checkpoint templates do append a cue. If you render text with tokenize=False and tokenize it in a later step, pass add_special_tokens=False; otherwise BOS or EOS tokens may be inserted twice.^{[3]Reference 3Transformers Documentation: Chat templates.https://huggingface.co/docs/transformers/main/en/chat_templating}

Before rendering thousands of rows, reject malformed dialogue structure. An assistant reply without a preceding user request is a bad supervised example even if its text is fluent.

Fine-tuning data must teach supported answers

Good formatting can't rescue bad supervision. If a training row claims that stale keys can wait 30 days when the policy says 14 days with a rotation ticket, SFT reinforces an unsupported answer.

Your first data design decision isn't "How many rows can I generate?" It's "What behavior is each accepted row allowed to teach?"

Self-Instruct demonstrated a repeatable way to expand instruction data: start from human-written tasks, generate new instructions and responses, then filter invalid or similar rows before fine-tuning.^{[7]Reference 7Self-Instruct: Aligning Language Models with Self-Generated Instructions.https://aclanthology.org/2023.acl-long.754/} A product team can use the same pattern with policy questions, but it must validate answers against source clauses rather than trusting a generator's confidence.

The next lab treats each assistant answer as a candidate SFT row. A row is accepted only if it contains required policy facts and omits a known unsupported claim.

This filter is deliberately small. A real data pipeline also checks duplicated instructions, personal information, unsafe replies, role ordering, length limits, and human-review requirements for high-risk cases. Version the evidence snapshot, generator prompt, filters, and accepted dataset together. Otherwise you won't know which change caused a behavior regression.

An accepted dataset also needs coverage. Rows for stale-key and admin-access cases don't demonstrate how the assistant should answer a source-citation question.

Which tokens should produce gradient?

Every completed chat transcript contains tokens from the system prompt, user question, and assistant answer. You must decide which of those tokens are supervised targets.

Two choices matter:

Assistant-only loss is common, but it isn't the definition of SFT. TRL exposes it as assistant_only_loss=True for conversational data only when the chat template can identify assistant spans through {% generation %} and {% endgeneration %} markers. Current TRL releases automatically patch templates for some bundled model families; inspect the resolved template for the checkpoint you train.^{[8]Reference 8TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer} If the span mask is wrong, you may silently train on user text or mask out the answer you meant to learn.

In a causal language model, the target for a token is evaluated after the preceding tokens. The tiny preprocessing lab below marks assistant words and the assistant turn terminator as targets; role markers, system text, and user text receive the usual ignore label -100.

Use this kind of inspected tiny batch before launching training. A training-loss curve can't reveal that your boundary finder shifted one token too far and trained the wrong span.

This calculation makes the objective choice visible. Full-sequence loss scores nearly the whole serialized row; assistant-only loss scores only the desired answer span and its terminator.

Packing saves padding, but test isolation explicitly

SFT rows have different lengths. If every short chat is padded to a long context window, the accelerator spends much of its work processing padding. Packing fills a window with several short sequences instead. TRL supports packing as a training configuration for this reason.^{[8]Reference 8TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer}

Packing introduces a decision that teams often miss: may a token in conversation B attend to earlier tokens in conversation A? Some concatenated language-model recipes separate samples with an end token while retaining ordinary causal attention. If you need each support conversation to be an independent supervised example, use a trainer or attention kernel that supports isolation and verify its boundary behavior.

First measure why packing is tempting. Four short training rows padded separately to a 16-token window waste most of their capacity; filling shared windows reduces that waste.

Suppose two three-token rows are packed into one sequence:

For strict isolation, the causal self-attention matrix contains two blocks:

$$M = \begin{bmatrix} 1 & 0 & 0 & 0 & 0 & 0 \\ 1 & 1 & 0 & 0 & 0 & 0 \\ 1 & 1 & 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 & 1 & 0 \\ 0 & 0 & 0 & 1 & 1 & 1 \end{bmatrix}$$

Each row is a query position; each column is a possible earlier key position. The zeros in row 4 under columns 0 through 2 prevent the second conversation from reading the first one.

Don't assume an option named packing=True automatically constructs this matrix. Inspect your trainer's documented semantics and run a small boundary test with the implementation you'll train.

Production failures are usually contract failures

Once a checkpoint has been fine-tuned, evaluate it with the private cases you built in the previous chapter. Log the rendered prompt, tokenizer revision, template revision, truncation policy, and generation settings alongside every result. Otherwise an answer regression could be a template deployment bug rather than a model change.

This minimal deployment manifest check can't measure model quality, but it prevents shipping an endpoint whose serialization contract is known to differ from the fine-tuning run.

A matching manifest earns the right to run quality evaluations; it doesn't prove the model is ready. Re-run grounded policy cases, critical failure slices, p95 latency checks, and human review checks after every fine-tune or serving change.

The final executable check joins template work back to evaluation. Template parity is mandatory, but a correctly formatted candidate still ships only if grounded cases and operational limits pass.

From chat foundations to applied engineering

Instruction tuning connects data to behavior:

1. A conversation row demonstrates the response you want.
2. A chat template turns that row into the exact token context used for training and serving.
3. A chosen label mask decides where supervised gradient is spent.
4. Data filtering, packing checks, and private evaluations keep the learned behavior honest.

You don't need to fine-tune a large model to practice. Render transcripts, inspect labels, validate synthetic rows, and treat the serving manifest as part of your experiment record. Those habits scale from a tiny exercise to a serious training run.

Mastery check

Key concepts

• Base-model continuation versus instruction-tuned response behavior
• SFT examples grounded in approved evidence
• Chat-template serialization and checkpoint-specific markers
• Fresh generation versus assistant prefill
• Full-sequence versus assistant-only loss
• Synthetic-data acceptance checks
• Packed-sequence isolation as a verified design choice
• Training and serving template parity

Evaluation rubric

• Foundational: Explains why a base model can complete dialogue text without reliably acting as a support assistant.
• Intermediate: Renders completed, generation, and prefill versions of one conversation without mixing their boundary rules.
• Intermediate: Builds an assistant-only target mask and explains why that's a choice rather than a universal SFT requirement.
• Advanced: Designs a data and serving audit that rejects unsupported policy rows and template drift before evaluation.

Common pitfalls

• Training on plausible synthetic answers without checking them against policy evidence.
• Hand-formatting messages with delimiters that don't match the checkpoint's tokenizer template.
• Treating assistant_only_loss or packed isolation as automatic behavior without inspecting the trainer.
• Using a new-assistant generation cue when the request already contains an assistant prefill.
• Declaring a fine-tuned checkpoint better before rerunning grounded private evaluations.

Follow-up questions

Practice extension

Add a third policy row for a source-citation requirement and a second rejected answer that invents automatic approval. Extend filter-grounded-sft-rows.py to print which required fact is missing or which forbidden claim was found. Then add its accepted answer to the rendering and label-building labs. Show exactly which assistant tokens become supervised targets and exactly which evidence allowed the row into training.