LLM Fine-Tuning Deep Dive — A 2026 Course

• You can program in Python and have trained a small neural network in PyTorch.
• You know what a transformer block is at the level of the Transformers course; module 02 recaps the parts a fine-tuner touches, and its module 13 covers the training mechanics this course extends.
• You do not need prior RL. Module 13 builds policy gradients for LLMs from scratch and links the deep RL note for the classical derivation.
• The transfer learning, prompting and alignment and Hugging Face notes are the recaps this course goes beneath.
• Image and video diffusion fine-tuning (DreamBooth, FLUX/SDXL LoRA) is out of scope; the LoRA math in module 06 transfers, the data and loss do not.

Three reader paths:

• The practitioner shipping a task model. Read 01–09 in order, then 11 (quantize for export) and 20 (serve it and cost it).
• The post-training engineer. Read everything; 12–15 (preference optimisation, RLVR, distillation, agents) are your core.
• The interview candidate. Read module 21 first, then the module each question points back to.

CORE RECIPE          01  When to Fine-Tune: What It Is, and When Not to Do It
                     02  Tokens, Chat Templates, and the Causal-LM Objective
                     03  Training Mechanics: Optimizers, Schedules, and Memory Math
                     04  Data Curation and Synthesis: Formats, Dedup, Decontamination, and Synthesis
                     05  Supervised Fine-Tuning and Continued Pretraining
                     06  LoRA and QLoRA in Depth
                     07  Beyond LoRA: The PEFT Landscape and Memory-Efficient Full Tuning
                     08  Frameworks: TRL, Unsloth, Axolotl, LLaMA-Factory, and Training Services
                     09  Evaluating Fine-Tuned Models

SCALE AND PRECISION  10  Scaling Out: Distributed Fine-Tuning, Parallelism, and MoE
                     11  Quantization Across the Lifecycle: QLoRA, QAT, FP8, and Export

POST-TRAINING        12  Preference Optimization: Reward Models, RLHF, DPO and Its Family
                     13  RL with Verifiable Rewards: GRPO and Its Successors
                     14  Distillation and Reasoning Fine-Tuning
                     15  Agentic and Tool-Use Fine-Tuning
                     16  Multimodal Fine-Tuning: Vision-Language, Audio, and Embedding Models
                     17  Long-Context Extension and Long-Sequence Training

SHIP AND SYNTHESISE  18  Model Merging and Adapter Composition
                     19  Safety, Forgetting, and Responsible Fine-Tuning
                     20  Production: Serving Adapters, MLOps, and Cost Planning
                     21  Glossary, Cheat Sheet & Interview Bank

Every silent bug lives in 02; 03 tells you whether a run fits; 06 is how most people fine-tune; 13 is the 2025–26 centre of gravity.

Modules

Dependency graph

Dotted edges are needed only for the scale-out or alignment sections of the target module. Module 21 has no prerequisites: it is a standalone reference to 01–20.

1. Do not skip 02 and 03. Template, masking and memory mistakes cause most failed runs; every later module assumes you can read a masked, templated example and predict whether a run fits.
2. Run the checks and the smoke labs. Every content module (01–20) has finetuning-check fences (CPU, seconds) and at least one lab script with a --smoke CPU mode; the full labs state their GPU tier.
3. Evaluate before you train. Labs report against module 01's eval set (see below). Read 09 early if you are impatient.
4. Answer the self-check questions before moving on; module 21 holds model answers.

Versions and what the prose targets

TRL reached v1.0 on 31 March 2026 with semantic versioning for its stable core (TRL v1 blog); the course teaches the stable trainers (SFT, DPO, Reward, RLOO, GRPO, KTO, Distillation) and flags the ones in trl.experimental. Transformers v5 is PyTorch-only, and PEFT 0.18 is the minimum PEFT release that works with it. Two tools common in older tutorials are not used: torchtune (no longer actively maintained) and TRL's PPOTrainer (removed in TRL v1.13.0; module 12 teaches PPO conceptually).

The checks import only torch, numpy and the standard library in the pinned CPU environment; the prose teaches the October 2026 APIs and names the release behind every quoted default.

Running the checks and labs

python -m venv .venv-finetuning
.venv-finetuning/bin/python -m pip install torch==2.8.0 numpy
.venv-finetuning/bin/python site/test_finetuning_corrections.py
.venv-finetuning/bin/python content/courses/finetuning/code/lora_from_scratch.py --smoke

Run these from the repository root. Fences marked finetuning-check are independent CPU programs with asserts, run with networking refused, that assert the numbers in the surrounding prose; other code blocks are teaching fragments. Lab scripts in code/, such as lora_from_scratch.py, have a --smoke mode (CPU, no downloads, asserts its own result) and a --full mode (real models, GPU tier in the module's Lab section).

Hardware tiers

Why T0 is fp16-only and what that costs: a T4 lacks bf16, and fp16 overflows at 65,504, while bf16 keeps fp32's range with fewer mantissa bits (epsilon \(2^{-7}\) vs \(2^{-10}\)). An 8B model's weights take \(8\times10^9 \times 2 / 10^9 = 16\) GB in bf16, a full T4, but about 4.13 GB in NF4 with double quantization at 4.127 bits/param for the quantized linear weights; embeddings and norms stay 16-bit, so the real footprint is somewhat higher (module 06). The check below proves both, but not that a given GPU kernel supports a dtype.

import torch

f16, bf16 = torch.finfo(torch.float16), torch.finfo(torch.bfloat16)
# fp16 tops out at 65504; bf16 shares fp32's 8-bit exponent but has 7 mantissa bits to fp16's 10.
assert f16.max == 65504.0
assert bf16.max > 3e38
assert bf16.eps == 2 ** -7 and f16.eps == 2 ** -10
x = torch.tensor(70000.0)
assert torch.isinf(x.to(torch.float16)).item()
assert torch.isfinite(x.to(torch.bfloat16)).item()
# Weights-only memory for an 8B model, decimal GB.
bf16_gb = 8e9 * 2 / 1e9
nf4_dq_gb = 8e9 * 4.127 / 8 / 1e9
assert bf16_gb == 16.0
assert abs(nf4_dq_gb - 4.127) < 0.01
assert bf16_gb / nf4_dq_gb > 3.8
print("fp16/bf16 limits and 8B weight arithmetic verified")

The running example

Where a module needs a task, it uses support-email JSON extraction: turn a customer email into one JSON object with exactly five keys: customer_name, order_id (canonical ORD-NNNNN), issue_type (refund, shipping, defect, account or other), refund_requested (boolean) and sentiment (positive, neutral or negative). Module 01 owns the dataset contract. On CPU, python content/courses/finetuning/code/fit_or_not_baseline.py --data-only writes ft_data/train.jsonl (200 rows) and ft_data/eval.jsonl (the 50-row frozen golden eval set, built from held-out templates), one {"id", "email", "gold"} object per line. Modules 04, 05, 09, 11, 13, 19 and 20 reuse these files, module 01's system prompt and its scorer (parse rate, schema validity, per-field and all-fields exact match) unchanged. Module 01 also sets the prompting baseline every fine-tune must beat. The default model family is Qwen3 (0.6B to 8B, base and instruct), because it ships both checkpoints, a documented thinking toggle, and first-class support in TRL, Unsloth, Axolotl and vLLM.

Symbols

Milestones

1. After module 05: a templated, masked, packed SFT batch whose trained tokens you can list by hand, and an SFT run that beats the module-01 prompting baseline.
2. After module 09: an eval report with bootstrap confidence intervals, a judge with measured position bias, and a regression suite.
3. After module 13: a GRPO run whose reward, length, entropy and clip-ratio curves you can explain, with a reward-hacking audit.

Use modules 11 and 20 alongside the inference course for serving.