Tokens, Chat Templates, and the Causal-LM Objective

A decoder-only LM maps token ids to embeddings, sends them through \(L\) identical blocks, applies a final norm, and projects to vocabulary logits with the LM head. Each block is pre-norm attention followed by a pre-norm MLP, each with a residual connection. The Transformers course derives every piece: block anatomy covers the residual stream and RMSNorm, and the end-to-end forward pass traces shapes from token to logit. What a fine-tuner needs on top of that is the names. PEFT configs, FSDP wrap policies, quantization skip-lists and merge tools all refer to modules by their attribute names in the Hugging Face modeling code.

The table uses the published Qwen3-4B config: \(d = 2560\), \(d_{ff} = 9728\), \(L = 36\), 32 query heads and 8 KV heads of size \(d_h = 128\), \(V = 151{,}936\), and tie_word_embeddings: true. Shapes are written as input → output. nn.Linear stores the weight transposed, as (out, in). Here \(d_q = n_h d_h = 4096\) and \(d_{kv} = n_{kv} d_h = 1024\). Qwen3-4B is a reminder that \(d_q\) need not equal \(d\).

Table 1. Module names and Qwen3-4B shapes

Qwen3 (and Gemma 3) also have per-head q_norm and k_norm of size \(d_h\) (QK-norm); Llama 3 and Mistral do not.

Per block, the four attention projections hold \(2 \cdot 2560 \cdot 4096 + 2 \cdot 2560 \cdot 1024 = 26.2\)M parameters. The three MLP projections hold \(3 \cdot 2560 \cdot 9728 = 74.7\)M, which is 74% of the block. Over 36 blocks that comes to 3.63B, and the tied embedding adds another 0.39B, for about 4.02B in total. Two things follow. MLP projections hold most of the parameters, which is why module 06 targets all-linear rather than only q_proj/v_proj. And because the embedding is tied, any change to embed_tokens also changes the output layer. Module 06 owns weight tying and ensure_weight_tying.

MoE layers

In a mixture-of-experts model, each block's single MLP is replaced by \(E\) expert MLPs plus a router: a linear layer \(d \to E\) whose softmax picks the top-\(k\) experts for each token. The Transformers course MoE module derives the routing. Two parts matter to a fine-tuner.

The first is the auxiliary load-balancing loss. Without it the router collapses onto a few experts. In the Switch Transformer form (Fedus et al., 2021):

Here \(f_i\) is the fraction of tokens in the batch dispatched to expert \(i\) (a hard count, so it has no gradient), and \(P_i\) is the mean router probability assigned to expert \(i\) (soft, so it does have a gradient). Under perfect balance \(f_i = P_i = 1/E\), the sum is \(E \cdot 1/E^2 = 1/E\), and \(\mathcal{L}_{\text{aux}} = \alpha_{\text{aux}}\). Any imbalance makes it larger. In HF code, \(\alpha_{\text{aux}}\) is router_aux_loss_coef, and the term is only computed when output_router_logits=True (modeling_qwen3_moe.py at v5.19.0). Qwen3-30B-A3B's config ships with \(E = 128\), \(k = 8\), router_aux_loss_coef: 0.001 and output_router_logits: false. If you fine-tune it with a plain loop and leave that flag at its default, the router trains with no balancing pressure at all. TRL's SFTConfig exposes router_aux_loss_coef and logs the unscaled aux_loss (TRL SFT docs, v1.15.0).

The second is targeting and memory. Experts are often stored as fused 3D parameters, not as separate nn.Linear modules, so target_modules name-matching can silently skip them. Memory scales with total parameters (all 128 experts are resident), not with the 3B active ones. Module 10 owns MoE fine-tuning: router stability, expert parallelism and LoRA on experts.

Attention implementations

Every backend computes the same softmax attention. They differ in memory traffic and in which masks they accept. In Transformers v5 the choice goes through the AttentionInterface registry: you pick one with attn_implementation= in from_pretrained, or switch at runtime with model.set_attn_implementation(...) (attention backends docs, v5.19.0).

The packed-sequence column needs one more detail. In v5, when you pass position_ids and no 2D attention mask, the masking utilities detect where positions reset to 0 and build a per-document mask for any backend (masking_utils.py at v5.19.0). Only FlashAttention does this without materializing a dense mask. That is why TRL documents padding_free=True as supported only with FlashAttention 2 or 3. FlashAttention-2 needs an Ampere-or-newer GPU, so a T4 cannot use it. Module 03 covers the padding-free memory math and module 05 covers packing strategies.

BPE and SentencePiece in one page

A tokenizer maps text to ids from a fixed vocabulary of size \(V\). Byte-pair encoding (Sennrich et al., 2016) starts from characters, or from bytes in byte-level BPE, and repeatedly merges the most frequent adjacent pair. The learned merge list is the tokenizer. SentencePiece (Kudo and Richardson, 2018) treats input as a raw stream with whitespace encoded as ▁, so it needs no language-specific pre-tokenizer, and it can train BPE or unigram models. Qwen3 and Llama 3 use byte-level BPE with vocabularies of about 150k and 128k. Gemma 3 uses a SentencePiece-derived vocabulary of 262k. The Transformers course tokenization module trains a BPE tokenizer from scratch, and the text preprocessing note covers Unicode and normalization traps.

Transformers v5 made two changes that affect old tutorials. It is PyTorch-only: the release "sunset[s] Flax/TensorFlow support in favor of focusing on PyTorch as the sole backend" (Transformers v5 blog, Dec 2025). The stable v5.0.0 shipped on 26 January 2026 (release page). It also unified tokenizers on the Rust tokenizers backend, "removing the concept of 'Fast' and 'Slow' tokenizers". So use_fast= and *TokenizerFast class choices no longer mean anything, and code that relied on a slow tokenizer's quirks may tokenize differently after upgrading. Re-check any cached tokenized dataset after a major-version bump.

BOS, EOS, PAD, UNK and added tokens

Table 2 was read directly from each repository's tokenizer_config.json and generation_config.json at a pinned revision. The Llama and Gemma repositories are gated, so you need to log in to view their files.

Table 2. Special tokens by family (pinned revisions)

Three facts from this table drive the rest of the module:

• Qwen3 has no BOS and Llama and Gemma add BOS twice if you let them. Section 2.4 shows the double BOS.
• The token that ends a chat turn often differs from the tokenizer's eos_token. The Qwen3 base tokenizer says EOS is <|endoftext|>, but its chat template closes turns with <|im_end|>.
• Stopping is a list. Qwen's hybrid models stop on either <|im_end|> or <|endoftext|>, and Gemma 3 stops on <eos> or <end_of_turn>. Any server or eval harness that reads only tokenizer.eos_token_id will miss one.

Adding tokens and resizing embeddings

Suppose you want dedicated tokens for the extraction task, such as <ticket> and </ticket>. Add them to the tokenizer, then grow the embedding (and the untied LM head) to match:

Teaching fragment (not executed; Transformers v5 API):

tok.add_special_tokens({"additional_special_tokens": ["<ticket>", "</ticket>"]})
model.resize_token_embeddings(len(tok), pad_to_multiple_of=64)   # (V', d), V' rounded up to a multiple of 64

pad_to_multiple_of=64 rounds the new vocabulary up so the LM-head matmul stays tensor-core friendly. The new rows are not random. With the default mean_resizing=True, Transformers samples them from a Gaussian with the old embeddings' mean and covariance (Hewitt: the mean plus small noise) (resize_token_embeddings at v5.19.0). This follows Hewitt's analysis: mean initialization keeps the next-token distribution almost unchanged, so the new token does not hog probability mass before training. Mean initialization only makes the new token harmless. It does not make the token mean anything. A frozen-base LoRA run never updates those rows, and the model will never learn to emit </ticket> at the right time. Train them explicitly, with trainable_token_indices or modules_to_save (module 06), or as part of full or continued pretraining (module 05). Often the better choice is to add no tokens at all and express the structure in plain text the tokenizer already handles well.

Perplexity depends on the tokenizer

Perplexity is the exponential of the mean per-token negative log-likelihood (in nats):

It is a per-token quantity, so it depends on what a token is. A tokenizer that splits text into more, easier pieces gets lower perplexity on the same text without predicting it any better. The language models note and the information theory note derive PPL from cross-entropy. To compare models with different tokenizers, normalize by a unit they share, such as the UTF-8 byte. Bits-per-byte is the total NLL converted to bits and divided by byte count:

Here is a worked example on the same 100-byte support email. Tokenizer A splits it into 25 tokens with total NLL 50 nats: PPL \(= e^{2.0} = 7.39\) and BPB \(= 50 / (100 \ln 2) = 0.721\). Tokenizer B splits it into 40 tokens with total NLL 52 nats: PPL \(= e^{1.3} = 3.67\) and BPB \(= 0.750\). B halves the perplexity while compressing the text worse. System A (model plus tokenizer A) assigns this text the higher probability.

The check below confirms the arithmetic and the reversal.

import math

n_bytes = 100
tok_a, nll_a = 25, 50.0   # tokenizer A: fewer, harder tokens
tok_b, nll_b = 40, 52.0   # tokenizer B: more, easier tokens

ppl_a, ppl_b = math.exp(nll_a / tok_a), math.exp(nll_b / tok_b)
bpb_a, bpb_b = nll_a / (math.log(2) * n_bytes), nll_b / (math.log(2) * n_bytes)

assert abs(ppl_a - 7.389) < 1e-3 and abs(ppl_b - 3.669) < 1e-3
assert abs(bpb_a - 0.721) < 1e-3 and abs(bpb_b - 0.750) < 1e-3
assert ppl_b < ppl_a          # B "wins" on perplexity ...
assert bpb_a < bpb_b          # ... but A assigns the text higher probability
# BPB is per-token NLL rescaled by tokens per byte, so the tokenizer cancels out
assert abs(bpb_b - (tok_b / n_bytes) * (nll_b / tok_b) / math.log(2)) < 1e-12
print("perplexity vs bits-per-byte verified")

It does not prove that BPB is enough on its own to compare models. Eval sets, context lengths and byte normalization (NFC vs raw) still have to match.

Next-token cross-entropy and label shifting

The model reads tokens \(x_{b,1..T}\) for each sequence \(b\) and, at every position \(t\), outputs logits for token \(t+1\). Training minimizes the mean negative log-likelihood over the set \(\mathcal{M}\) of positions whose target is not masked:

The Transformers course introduces this as the pretraining objective. Fine-tuning changes only \(\mathcal{M}\). In Hugging Face code you pass labels aligned with input_ids, unshifted, and the model shifts them internally. ForCausalLMLoss right-pads labels with one -100, takes labels[..., 1:], upcasts logits to fp32, and calls cross_entropy(..., ignore_index=-100) (loss_utils.py at v5.19.0). That is equivalent to scoring logits[:, :-1] against labels[:, 1:]. If you shift the labels yourself before passing them in, the model shifts again and every position is trained to predict two tokens ahead. The loss still goes down, but more slowly, and nothing errors.

Take a 6-token tail of an assistant turn: <|im_start|> assistant {"refund_requested": true} <|im_end|> <pad>. Only the content and the closing <|im_end|> are trained, so labels = [-100, -100, id, id, id, -100].

Read the table by rows. The label stored at position \(t\) is scored at position \(t-1\). Position 1 has a masked label of its own, yet it is the position that learns to start the answer, and position 3 is the one that learns to emit <|im_end|>. "Mask the header" therefore means "do not train the model to predict the header tokens". It does not mean "the header positions get no gradient".

The ignore index -100

F.cross_entropy(..., ignore_index=-100) with mean reduction drops masked positions from both the numerator and the denominator. The result is the mean over trained tokens only, not a mean diluted by zeros. Their logits get exactly zero gradient. The check below verifies this, along with the equivalence between the HF pad-and-shift form and the slice form.

import torch
import torch.nn.functional as F

torch.manual_seed(0)
B, T, V = 2, 6, 11
logits = torch.randn(B, T, V, dtype=torch.float64, requires_grad=True)  # (B, T, V); f64 so op order can't flake
labels = torch.randint(0, V, (B, T))                       # (B, T), unshifted
labels[0, :2] = -100                                        # prompt tokens of row 0
labels[1, :3] = -100                                        # prompt tokens of row 1
labels[1, 5] = -100                                         # right padding of row 1

# manual: log_softmax + gather on shifted targets, divide by the trained count
tgt = labels[:, 1:]                                         # (B, T-1)
keep = tgt.ne(-100)
logp = F.log_softmax(logits[:, :-1], dim=-1)                # (B, T-1, V)
nll = -logp.gather(-1, tgt.clamp(min=0).unsqueeze(-1)).squeeze(-1)
manual = nll[keep].sum() / keep.sum()
assert int(keep.sum()) == 6                                 # 4 trained targets in row 0, 2 in row 1

ref = F.cross_entropy(logits[:, :-1].reshape(-1, V), tgt.reshape(-1), ignore_index=-100)
torch.testing.assert_close(manual, ref, atol=1e-12, rtol=0)

# HF style: pad labels with one -100 on the right, shift, score all T positions
hf_tgt = F.pad(labels, (0, 1), value=-100)[:, 1:]
hf = F.cross_entropy(logits.reshape(-1, V), hf_tgt.reshape(-1), ignore_index=-100)
torch.testing.assert_close(hf, ref, atol=1e-12, rtol=0)

ref.backward()
row_grad = logits.grad.abs().sum(-1)                        # (B, T)
for b in range(B):
    for t in range(T):
        trained = t < T - 1 and labels[b, t + 1].item() != -100
        assert (row_grad[b, t] > 0) == trained, (b, t)

# token-level vs sequence-level averaging, the worked example from the prose
l1 = torch.full((2,), 1.0)                                  # sequence 1: 2 trained tokens, loss 1.0 each
l2 = torch.full((6,), 0.5)                                  # sequence 2: 6 trained tokens, loss 0.5 each
tok_level = (l1.sum() + l2.sum()) / (l1.numel() + l2.numel())
seq_level = (l1.mean() + l2.mean()) / 2
torch.testing.assert_close(tok_level, torch.tensor(0.625))
torch.testing.assert_close(seq_level, torch.tensor(0.75))
print("shift, ignore index, zero gradients and averaging verified")

It does not prove that a particular trainer averages this way across gradient-accumulation steps or devices. That depends on num_items_in_batch, which is covered in module 03.

Token-level vs sequence-level averaging

When sequences in a batch have different numbers of trained tokens \(n_b\), there are two reasonable "means" with per-token losses \(\ell_{b,t}\):

Take two sequences with 2 and 6 trained tokens, per-token losses 1.0 in the first and 0.5 in the second. Token-level: \((2 \cdot 1.0 + 6 \cdot 0.5)/8 = 5/8 = 0.625\). Sequence-level: \((1.0 + 0.5)/2 = 0.75\). Token-level weights every token equally, so long responses dominate the gradient. Sequence-level weights every example equally, so each token of a short response counts for more. Neither one is a bug. They are different objectives. A dataset that mixes one-line JSON answers with long explanations will learn different things under each.

Hugging Face Trainer and TRL's SFTTrainer use token-level averaging. The model's loss is sum / num_items_in_batch, where the trainer counts trained tokens across the whole accumulated batch and, by default, across devices (average_tokens_across_devices=True in SFTConfig). TRL logs loss as "the average cross-entropy loss computed over non-masked tokens" (TRL SFT docs). The 2024 gradient-accumulation bug came from mixing the two levels: each micro-batch took a token mean, and those means were then averaged. Module 03 works through it. RL losses reopen the same choice (GRPO's sequence-level vs DAPO's token-level normalization, module 13).

The whole data path, from messages to loss:

Jinja templates and roles

A chat model is still a next-token predictor over a single string. The chat template is a Jinja program stored in the tokenizer config that turns a list of {"role": ..., "content": ...} messages into that string, with the control tokens the model saw in post-training (chat templating docs, v5.19.0). The roles are system, user, assistant and tool. Templates also take tools= (JSON schemas) and per-message fields such as tool_calls and reasoning_content. Module 15 covers tool formats.

apply_chat_template has four switches that matter:

You cannot combine the last two: one opens a new turn and the other leaves the previous one open, and passing both raises an error. In v5, continue_final_message also accepts a field name: continue_final_message="reasoning_content" prefills Qwen's reasoning block and leaves it open, whereas prefilling content closes the reasoning block first. Gemma's equivalent field is thinking (same docs). Thinking formats and Qwen3's thinking-mode fusion (Qwen3 technical report) belong to module 14.

This is the real Qwen3 rendering, at the pinned revision above, of our two-turn conversation (system, user, assistant, user, assistant) with tokenize=False:

<|im_start|>system
Extract the ticket as JSON.<|im_end|>
<|im_start|>user
Hi, order ORD-01142 arrived broken. Ana<|im_end|>
<|im_start|>assistant
{"order_id": "ORD-01142", "issue_type": "defect"}<|im_end|>
<|im_start|>user
Add whether I want a refund: yes.<|im_end|>
<|im_start|>assistant
<think>

</think>

{"refund_requested": true}<|im_end|>

Walk through the special tokens. <|im_start|> (151644) opens each turn and is followed by the role name as ordinary text and a newline. <|im_end|> (151645) closes each turn and is the token the model must learn to emit. A \n follows every <|im_end|>. There is no BOS. The last assistant turn received an empty <think>\n\n</think>\n\n block that the earlier assistant turn did not get. That asymmetry matters below. The same conversation through Llama 3.2's template starts with <|begin_of_text|> and uses <|start_header_id|>assistant<|end_header_id|>\n\n headers with <|eot_id|> endings. It also injects Cutting Knowledge Date: December 2023 and Today Date: <today> into the system turn, so the same dataset renders differently on different days unless you pass date_string. Gemma 3 has no system role: it folds the system prompt into the first user turn and calls the assistant model.

Training templates and assistant masks

For SFT on conversations you usually train only assistant tokens (assistant_only_loss, module 05; completion_only_loss is the default for prompt-completion data). To know which tokens are assistant tokens, the template must mark them. {% generation %} ... {% endgeneration %} blocks around the assistant output make apply_chat_template(..., return_assistant_tokens_mask=True) return assistant_masks. None of the three shipped templates in Table 2 contain these markers. On the pinned revisions, all three return an all-zero mask with a warning. TRL handles this by swapping in patched training templates for known families (Qwen3, Llama 3.x, Gemma 3 and others) when assistant_only_loss=True (TRL chat templates, v1.15.0). For any other model, check the mask yourself. The lab prints it.

The check below builds the mask the way a {% generation %} template does. A hand-written ChatML renderer emits (text, trainable) pieces, and a toy tokenizer splits on whitespace while keeping <|im_start|> and <|im_end|> whole. The fixture has 29 tokens, and exactly 8 are trained: 4 content tokens plus <|im_end|> in the first assistant turn, and 2 plus <|im_end|> in the second.

import re

SPECIAL = ("<|im_start|>", "<|im_end|>")
SPLIT = re.compile(r"(<\|im_start\|>|<\|im_end\|>)")

def render(messages, add_generation_prompt=False):
    """ChatML renderer returning (text, trainable) pieces, like a {% generation %} template."""
    pieces = []
    for m in messages:
        if m["role"] == "assistant":
            pieces.append(("<|im_start|>assistant\n", False))         # header: never trained
            pieces.append((m["content"] + "<|im_end|>", True))        # content + turn end: trained
            pieces.append(("\n", False))
        else:
            pieces.append(("<|im_start|>" + m["role"] + "\n" + m["content"] + "<|im_end|>\n", False))
    if add_generation_prompt:
        pieces.append(("<|im_start|>assistant\n", False))
    return pieces

def tokenize(text):
    out = []
    for part in SPLIT.split(text):
        out.extend([part] if part in SPECIAL else part.split())
    return out

convo = [
    {"role": "system", "content": "Extract JSON."},
    {"role": "user", "content": "Order ORD-01142 arrived broken."},
    {"role": "assistant", "content": '{"order_id": "ORD-01142", "issue_type": "defect"}'},
    {"role": "user", "content": "Refund please."},
    {"role": "assistant", "content": '{"refund_requested": true}'},
]
pieces = render(convo)
text = "".join(p for p, _ in pieces)
tokens = tokenize(text)
mask = [flag for p, flag in pieces for _ in tokenize(p)]
assert [t for p, _ in pieces for t in tokenize(p)] == tokens   # piecewise == whole-string tokenization
vocab = {t: i for i, t in enumerate(sorted(set(tokens)))}
ids = [vocab[t] for t in tokens]
labels = [i if m else -100 for i, m in zip(ids, mask)]

assert len(tokens) == 29 and sum(mask) == 8
for k, t in enumerate(tokens):                                 # headers are never trained
    if t == "assistant":
        assert tokens[k - 1] == "<|im_start|>" and labels[k - 1] == labels[k] == -100
ends = [k for k, t in enumerate(tokens) if t == "<|im_end|>"]
trained_ends = [k for k in ends if labels[k] != -100]
assert len(ends) == 5 and len(trained_ends) == 2               # only assistant turn-ends are targets
assert all(labels[k] == ids[k] for k in trained_ends)
assert all(k == len(tokens) - 1 or tokens[k + 1] == "<|im_start|>" for k in trained_ends)
assert [tokens[k] for k in range(len(tokens)) if mask[k]] == [
    '{"order_id":', '"ORD-01142",', '"issue_type":', '"defect"}', "<|im_end|>",
    '{"refund_requested":', "true}", "<|im_end|>"]

prompt = "".join(p for p, _ in render(convo[:4], add_generation_prompt=True))
assert prompt.endswith("<|im_start|>assistant\n")
assert "<|im_end|>" not in prompt.rsplit("<|im_start|>", 1)[1]  # the open turn has no end token
assert tokenize(prompt)[-2:] == ["<|im_start|>", "assistant"]
print("assistant-only label mask verified")

It does not prove that a real BPE tokenizer splits at the same boundaries. Byte-level BPE can merge a header's trailing newline with the first content character, which is why real masks come from character offsets or template markers rather than from per-piece tokenization.

Pitfall 1: double BOS

The Llama and Gemma templates start with {{ bos_token }}. Their tokenizers also prepend BOS by default: Llama through a TemplateProcessing post-processor, Gemma through add_bos_token: true. Render with tokenize=False, tokenize the string with a plain tok(text) call, and you get two BOS tokens. On the pinned revisions this gives 97 ids instead of 96 for Llama 3.2 and 81 instead of 80 for Gemma 3 (rendered on 10 October 2026; Llama's date line can shift the count by a token). Qwen3 has no BOS and is unaffected. The model never saw <bos><bos> in pretraining, so every training example begins out of distribution, and inference with the correct single BOS then differs from training. The Transformers docs state the fix directly: "make sure you set add_special_tokens=False if you tokenize later ... This isn't an issue if you use apply_chat_template(tokenize=True)" (chat templating docs). TRL's trainers tokenize conversational data through the template, so the bug shows up mainly in hand-written preprocessing.

Pitfall 2: EOS alignment and pad = eos

Many tokenizers ship without a pad token. The usual fix is tok.pad_token = tok.eos_token, and TRL itself falls back to the EOS token when no pad is set (SFTTrainer processing_class, TRL v1.15.0). On its own that is harmless. The damage comes from the collator. The classic causal-LM collator builds labels by copying input_ids and setting every position equal to pad_token_id to -100. Transformers' own DataCollatorForLanguageModeling(mlm=False) still does exactly this (data_collator.py at v5.19.0). If pad and EOS share an id, the real EOS at the end of each example is masked too. The model never receives a gradient toward stopping, so at inference it keeps going: it produces another turn, another JSON object, or filler until max_new_tokens.

TRL's collator avoids this by construction. It builds labels before padding and pads them with -100, so the mask depends on position, not on id (TRL DataCollatorForLanguageModeling, v1.15.0). The check below reproduces both behaviors, and the double-BOS bug, on toy ids.

import torch

BOS, EOS, PAD = 1, 2, 2                                       # the bug: pad id == eos id
seqs = [[BOS, 5, 6, 7, EOS], [BOS, 8, EOS]]
T = max(len(s) for s in seqs)
input_ids = torch.tensor([s + [PAD] * (T - len(s)) for s in seqs])        # right padding, (2, 5)
attention = torch.tensor([[1] * len(s) + [0] * (T - len(s)) for s in seqs])

def eos_targets(labels):
    """Number of positions whose shifted target is EOS and enters the loss."""
    return int(labels[:, 1:].eq(EOS).sum())

by_id = input_ids.clone()
by_id[input_ids == PAD] = -100                                # mask by token id
by_position = input_ids.clone()
by_position[attention == 0] = -100                            # mask by position (TRL's behavior)

assert eos_targets(by_id) == 0                                # the model never learns to stop
assert eos_targets(by_position) == 2                          # exactly one EOS target per sequence
assert by_position[0, 4] == EOS and by_position[1, 2] == EOS and by_position[1, 3] == -100

def toy_tokenize(text_ids, add_special_tokens=True):
    """Stands in for tok(text): prepends BOS when add_special_tokens=True."""
    return ([BOS] if add_special_tokens else []) + list(text_ids)

templated = [BOS, 30, 31, 32, EOS]                            # chat template already emitted BOS
assert toy_tokenize(templated)[:2] == [BOS, BOS]              # double BOS
assert toy_tokenize(templated, add_special_tokens=False).count(BOS) == 1
print("pad=eos masking and double BOS verified")

It does not prove that a real model trained on by_id labels fails to stop. The --never-stop extension of the lab shows that on real weights, and the smoke mode shows it on a tiny model.

There are two fixes. You can mask by attention mask or position, never by id, or you can use a pad token distinct from every stop token: Llama 3.x has <|finetune_right_pad_id|>, and Qwen3 instruct already pads with <|endoftext|>, which differs from its EOS <|im_end|>. There is also a second form of this misalignment. The tokenizer's eos_token must be the token your template ends turns with. Qwen/Qwen3-0.6B-Base has eos_token = pad_token = <|endoftext|>, but its chat template closes turns with <|im_end|>. Train on that template while appending <|endoftext|> as "EOS", and the model learns to end with <|im_end|><|endoftext|>. A server that stops only on <|endoftext|> will then print the <|im_end|>. TRL's docs say to set eos_token="<|im_end|>" in SFTConfig for Qwen bases. The trainer then also prepends that id to generation_config.eos_token_id (sft_trainer.py L719-L737). Module 05 gives the full base-model recipe.

Pitfall 3: base models without a template

A base model may ship with no template, or with one it was never trained on. TRL's SFTConfig(chat_template_path=...) and the clone_chat_template helper copy a template, plus any special tokens it needs, from another tokenizer. If a Jinja file references tokens that are new to the vocabulary, you must add them and resize the embeddings (section 2.2). Module 05 owns this workflow.

Prefix-preserving templates

A template is prefix-preserving if rendering turns \(1..k\) always produces a string prefix of rendering turns \(1..k{+}1\). Multi-turn RL and agent loops depend on this property. They generate turn \(k\), append a tool result, and keep the tokens they already have, including their log-probs, as the prefix of the next step. If appending a message changes how earlier messages render, the trainer is computing log-probs for a sequence the model never actually saw.

The Qwen3 rendering above breaks the property. Its template prints the <think>...</think> block only for assistant turns after the last user query: the original reads {%- if loop.index0 > ns.last_query_index %} and then {%- if loop.last or ... %}. When the conversation ends at assistant turn 1, that turn carries a <think> block. Append a user message or a tool response, render again, and turn 1 loses the block. Many reasoning templates follow the same pattern of stripping old reasoning, and they all break prefix preservation. TRL's patched qwen3_training.jinja drops that conditional "so the thinking block is always emitted", and TRL exposes is_chat_template_prefix_preserving to test a template (TRL chat templates; chat_template_utils.py L854). Module 15 builds on this for tool-use RL.

Table 3. Template pitfalls

Batches need rectangular tensors, so short sequences get padding, and which side you pad on depends on what happens next. The Hugging Face note covers the basic tokenizer call.

Right padding is normal for SFT training. The loss looks at every position independently and pad positions carry label -100. With right padding, real tokens start at position 0, so position_ids = 0, 1, 2, ... match pretraining without any extra work. TRL's SFT collator pads on the right.

Left padding is required for batched generation. generate appends each new token after the last column of the batch. With right padding, a short prompt's last column is a pad token, so the model would continue from padding. With left padding, every row's last column is its real final token, usually the end of <|im_start|>assistant\n. This is why trainers that generate, such as GRPO/RLOO rollouts, online preference methods (module 12) and evaluation loops, pad prompts on the left (tok.padding_side = "left"), while the scoring forward passes can still use any side.

Left padding needs correct position ids. If a prompt has 3 pad tokens, its first real token must get position 0, not 3, or RoPE rotates every token by the wrong angle. Transformers' generate derives positions from the attention mask: position_ids = attention_mask.cumsum(-1) - 1, with pad positions filled with a dummy value (they are masked anyway). Custom rollout code that passes input_ids without attention_mask silently gets positions offset by the pad count, and padded keys are attended to as well.

Packed sequences go one step further. Several short examples are concatenated into one row with no padding, and position ids reset to 0 at each example boundary ([0,1,2,3,0,1,2,0,1,...]). The resets tell the attention backend where documents end (section 2.1), and the first label of each packed example is masked so no example learns to predict across the boundary. Module 05 covers packing strategies (bfd, bfd_split, wrapped), and module 03 covers what padding-free saves.

The Transformers course section on mixed precision explains why bf16 replaced fp16 for pretraining: the same 8-bit exponent as fp32, so no overflow and no loss scaling. Table 4 adds the formats you meet in 2026 fine-tuning. The inference course numerics chapter covers them from the serving side.

Table 4. Numerical formats

Sources: FP8 formats from Micikevicius et al. (2022), MXFP4 from the OCP Microscaling spec v1.0, and NVFP4 from NVIDIA Transformer Engine docs (release 2.18).

The T4 case. Free Colab and Kaggle T4s are compute capability 7.5 (NVIDIA CUDA GPU list), and native bf16 tensor-core math starts at 8.0 (Ampere). Check torch.cuda.get_device_capability() >= (8, 0) rather than torch.cuda.is_bf16_supported(), which can report emulated support. On a T4, train in fp16 with dynamic loss scaling. fp16's 5-bit exponent underflows small gradients, so it needs dynamic loss scaling (torch.amp.GradScaler; module 03 shows the mechanics). In Trainer terms this means fp16=True. TRL's SFTConfig defaults bf16 to True only when fp16 is not set, so passing fp16=True turns the bf16 default off. Check the logged config anyway. Module 03 owns mixed-precision mechanics and master weights, and module 11 owns FP8 training and the 4-bit formats.

Loading. In Transformers v5, torch_dtype= became dtype=, and from_pretrained infers the dtype from the checkpoint config when you do not pass one. A bf16 checkpoint loads as bf16. TRL trainers that receive a model name deliberately differ: "If dtype is not specified in args.model_init_kwargs, it defaults to float32" (TRL SFTTrainer docs, v1.15.0). A 4B model then takes 16 GB of weights before any optimizer state.

Teaching fragment (not executed; Transformers v5 / TRL v1.15 APIs, as of October 2026):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B")              # no use_fast= in v5
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-4B",
    dtype=torch.bfloat16,        # omit -> inferred from config (bf16 here) in v5
    device_map="auto",           # place layers on available devices
    attn_implementation="sdpa",  # or "flash_attention_2" on Ampere+
)

# Passing a model *name* to TRL: set the dtype explicitly or you get float32
args = SFTConfig(output_dir="out", model_init_kwargs={"dtype": torch.bfloat16})
trainer = SFTTrainer(model="Qwen/Qwen3-4B", args=args, train_dataset=ds, processing_class=tok)

TRL, Unsloth and Axolotl wrap these same loader arguments (module 08). device_map="auto" is for inference and single-process training. Under FSDP or DeepSpeed, the launcher places the shards. trust_remote_code=True runs arbitrary Python from the model repository on your machine. Leave it off unless you have read the code and pinned the revision (module 20).

Evaluation and RL both sample from your model, so the decoding settings belong to the experiment just as much as the learning rate does. Write them down. The decoding note derives each method. In one line each, for logits \(z\) and probabilities \(p\):

Two rules. Evaluation uses greedy decoding, or a fixed temperature with \(n\) samples per prompt and a reported mean (pass@k and its variance are in module 09). RL rollouts sample at the temperature the policy-gradient math assumes. Sampling with top-\(p\) or a penalty that the trainer's log-prob computation does not apply makes the importance ratio wrong (module 13). Model repositories ship defaults in generation_config.json. Qwen3 ships temperature=0.6, top_p=0.95, top_k=20 (pinned file above). generate() uses those unless you override them, so "I evaluated with defaults" means different settings for different models.

Stop tokens. generation_config.eos_token_id can be a list, and for all three families in Table 2 it is. A Python model.generate call reads the list. A serving stack, eval harness or hand-rolled loop that takes tokenizer.eos_token_id gets one id. For Qwen3 hybrid that id happens to be the right one (<|im_end|>). For a Qwen3 base model trained on ChatML it is the wrong id (section 2.4, Pitfall 2), which is the classic "works in Python, rambles in the server" bug. Check that the fine-tuned checkpoint's generation_config.json lists your template's turn-end token, and that the server reads it (module 20; the inference course detokenization chapter covers stop handling on the server side).

KV cache. Generation is one token at a time. Without a cache, step \(t\) would recompute keys and values for all \(t\) previous tokens. With a cache, each step computes K and V for one new token and attends over the stored ones, so each step does the weight matmuls for one token only; attention still reads all \(t\) cached K/V, so its cost grows linearly with position. The cache grows linearly in memory: \(2 \cdot L \cdot n_{kv} \cdot d_h\) values per token. For Qwen3-4B in bf16 that is \(2 \cdot 36 \cdot 8 \cdot 128 \cdot 2 = 147{,}456\) bytes, about 0.15 MB, per token. One GRPO prompt with \(G = 8\) completions of 4,096 tokens holds \(8 \cdot 4096 \cdot 0.147\text{ MB} \approx 4.8\) GB of cache. That is why RL rollouts are memory-bound and why TRL's GRPO can hand generation to vLLM's paged cache (module 13). The decode loop and KV cache chapter derives this in detail.

Goal. Render one support-email conversation through three real chat templates, tokenize it, print a per-token table (index, token, label id or -100, in loss?), reproduce the double-BOS and pad=eos bugs, fix them, and confirm a hand-computed loss against the model's own loss.

Tier. CPU for both modes. --full downloads tokenizers only, plus Qwen/Qwen3-0.6B weights for the loss comparison (about 1.5 GB; works on CPU, faster on a T0 T4). The --never-stop extension is T0.

Script. template_mask_inspector.py. Smoke mode has no dependencies beyond torch. Full mode imports transformers (v5.x) inside the functions that need it.

python content/courses/finetuning/code/template_mask_inspector.py --smoke
hf auth login        # Llama 3.2 and Gemma 3 are gated: accept their licenses on the Hub first
python content/courses/finetuning/code/template_mask_inspector.py --full
python content/courses/finetuning/code/template_mask_inspector.py --full --never-stop   # T0 GPU

Step 1: render with each template

Full mode loads the tokenizers of Qwen/Qwen3-0.6B, meta-llama/Llama-3.2-1B-Instruct and google/gemma-3-1b-it at the revisions in Table 2 and prints apply_chat_template(messages, tokenize=False) for each. Record the differences: where BOS appears, the header strings, the empty <think> block on Qwen's last turn, Llama's date lines, and Gemma's folded system prompt.

Step 2: count BOS twice

Tokenize each rendered string with add_special_tokens=True and then False, and compare both against apply_chat_template(tokenize=True). Expect 2/1/1 BOS for Llama and Gemma and 0/0/0 for Qwen. Record the id counts.

Step 3: build the assistant mask

The script first asks for return_assistant_tokens_mask=True. On these shipped templates it returns all zeros. That is your evidence that the markers are missing. It then falls back to offsets: it finds each assistant header in the rendered string, marks characters from the header's end through the turn-end token, and maps tokens to characters with return_offsets_mapping. It prints the per-token table and the trained-token count per template. On the pinned revisions the counts are 33, 26 and 31 for Qwen3, Llama 3.2 and Gemma 3, and Qwen's count includes the empty <think> block on the last turn. The script also flags any template where rendering turns \(1..k\) is not a prefix of rendering turns \(1..k{+}1\). Qwen3 gets flagged.

Step 4: simulate pad = eos

The script sets pad_token = eos_token, pads a 3-example batch on the right, and builds labels twice: by id, the way transformers.DataCollatorForLanguageModeling does, and by position, the way TRL does. Record how many turn-end targets survive in each. By id it should be zero.

Step 5: hand loss vs model(**batch).loss

With Qwen/Qwen3-0.6B weights, the script runs the 3-example position-masked batch, recomputes the loss from the returned logits with log_softmax + gather over shifted, unmasked targets, and asserts agreement with model(**batch).loss to \(10^{-4}\).

Smoke mode reproduces steps 2-5 with a toy ChatML renderer and vocabulary, and adds a tiny GRU never-stop demo (only the position-masked model emits EOS).

Optional extension (--never-stop, T0). This trains Qwen/Qwen3-0.6B-Base for 100 steps on ChatML-formatted support-email examples, using pad_token = <|im_end|> and id-based masking, then again with a distinct pad and position masking. It generates 20 replies from each and reports how many stop on <|im_end|> before max_new_tokens. Expect the fixed run to stop reliably. Expect fewer stops in the id-masked run, not necessarily zero, and record the gap. The counts depend on hardware and seed.

What to record. BOS counts per template, trained-token counts, prefix-preservation flags, the number of EOS targets under each masking scheme, the hand-vs-model loss difference, and (with the extension) stop rates.