Preference Optimization: Reward Models, RLHF, DPO and Its Family

Every method in this module optimizes, exactly or approximately, one objective. Given a prompt distribution, a reward function \(r(x,y)\) and a frozen reference policy \(\pi_{\text{ref}}\) (usually the SFT model):

The KL term does three jobs. It keeps the policy near the SFT model, which already writes fluent text. It limits reward hacking: the reward is a learned proxy whose blind spots (hedging, flattery, length) an unconstrained optimizer finds. And it preserves diversity, since without it the policy collapses onto the single highest-scoring response. \(\beta\) is the exchange rate between reward and drift (KL in the notes).

The InstructGPT pipeline

Ouyang et al. (2022), InstructGPT, made this concrete in three stages: (1) SFT on labeler demonstrations; (2) a reward model trained on labeler rankings of several model outputs per prompt; (3) PPO against that reward model, with a per-token KL penalty folded into the reward, \(r_t = r_\phi(x,y)\,\mathbb{1}[t=T] - \beta\log\frac{\pi(y_t\mid x,y_{<t})}{\pi_{\text{ref}}(y_t\mid x,y_{<t})}\). Labelers preferred the 1.3B InstructGPT model's outputs to the 175B GPT-3's.

PPO is expensive for structural reasons, not tuning reasons. You hold four networks: the policy (trained), the reference (frozen, for the KL), the reward model (frozen) and a value model (trained, for advantages), and every step pays for autoregressive sampling. PPO's clipped surrogate and GAE are derived in the deep-RL notes; their LLM form, and where the KL lives and how it is estimated, are module 13's topic.

PPOTrainer, PPOConfig and the value-head model classes were removed from TRL in v1.13 (September 2026), citing near-zero usage. Treat RLHF-with-PPO as a concept; online RL in TRL now means GRPO or RLOO (module 13).

Best-of-N: the baseline every trained method must beat

Sample \(N\) responses from \(\pi_{\text{ref}}\), return the one the reward model scores highest. No training. The commonly quoted KL of the best-of-\(N\) policy from the base policy is

Beirami et al. (2024), Theoretical guarantees on the best-of-n alignment policy, prove this is an upper bound, not an equality, and propose a tighter estimator. The formula is still useful for budgeting:

The KL grows only logarithmically in \(N\), and in Gao et al. (2022), Scaling Laws for Reward Model Overoptimization, best-of-\(N\) reached a given gold reward at a much smaller KL than RL did (they caution that KL is an imperfect common currency). If your DPO model does not beat best-of-16 from the SFT model under the same reward model, the training bought you only lower inference cost. Best-of-\(N\) distillation (SFT or DPO on the selected samples) keeps that quality without the \(N\times\) cost.

Bradley-Terry

Humans judge "which is better" more reliably than "rate this 1 to 10". Bradley-Terry turns pairwise judgments into a scalar score:

where \(m \ge 0\) is an optional per-pair margin (for example, larger when annotators were more confident). This is logistic regression on reward differences: the label is always 1, the logit is \(r_w - r_l\) (the logistic link is reviewed in probability). Only differences matter, so adding a constant to every reward changes nothing; because the offset is free, a reward model can drift to large means that hurt downstream RL. Eisenstein et al. (2023) add a centering term \(\eta\,\mathbb{E}[(r_w + r_l)^2]\) to pin the mean near zero.

The architecture is a causal LM with its LM head replaced by a scalar head that reads the hidden state at the last non-padding token: AutoModelForSequenceClassification with num_labels=1. TRL's stable RewardTrainer (v1.15.0 docs) does exactly this. It takes the same preference format as DPO (prompt/chosen/rejected, standard or conversational, implicit or explicit prompt), accepts an optional margin column, defaults to learning_rate=1e-4, and exposes centering as center_rewards_coefficient (recommended 0.01; loss lines).

The check below fits a known linear reward from 64 pairs by gradient descent and confirms the BT loss is literally binary cross-entropy on the reward difference.

import torch
import torch.nn.functional as F

# Fit a 3-feature linear reward model on 64 synthetic pairs from a known weight vector
g = torch.Generator().manual_seed(0)
w_true = torch.tensor([1.0, -2.0, 0.5])
a, b = torch.randn(64, 3, generator=g), torch.randn(64, 3, generator=g)
a_wins = (a @ w_true) > (b @ w_true)
x_w = torch.where(a_wins[:, None], a, b)
x_l = torch.where(a_wins[:, None], b, a)
w = torch.zeros(3, requires_grad=True)
for _ in range(300):
    loss = -F.logsigmoid(x_w @ w - x_l @ w).mean()
    loss.backward()
    with torch.no_grad():
        w -= 0.5 * w.grad
        w.grad.zero_()
cos = F.cosine_similarity(w.detach(), w_true, dim=0).item()
assert cos > 0.95, cos
assert ((x_w @ w) > (x_l @ w)).float().mean().item() > 0.95
diff = (x_w @ w - x_l @ w).detach()                     # BT == BCE with label 1; shift-invariant
torch.testing.assert_close(-F.logsigmoid(diff), F.binary_cross_entropy_with_logits(diff, torch.ones(64), reduction="none"))
torch.testing.assert_close(-F.logsigmoid((diff + 5.0) - 5.0), -F.logsigmoid(diff))
print("Bradley-Terry == BCE on differences; linear RM recovered, cosine", round(cos, 4))

It does not show what makes real reward models fail: a linear reward on Gaussian features has no spurious correlates (length, style) for the model to latch onto.

Over-optimization and ensembles

The reward model is a proxy, and optimizing a proxy hard enough makes the true objective worse. Gao et al. (2022) trained proxy RMs on labels from a fixed "gold" RM. As the policy moves away from the initial model (\(d=\sqrt{\mathrm{KL}}\)), the proxy score keeps rising while the gold score rises, peaks and falls. Larger RMs and more RM data push the peak out; they do not remove it.

The best-evidenced mitigation is an ensemble: Coste et al. (2023), Reward Model Ensembles, optimize against worst-case or uncertainty-weighted aggregates. Eisenstein et al. (2023), Helping or Herding?, find pretraining-seed diversity beats fine-tuning-seed diversity, but ensembles mitigate and do not eliminate reward hacking: a shared error gets exploited. Pair ensembles with a KL budget and periodic human or strong-judge spot checks (module 09).

Outcome, process and generative rewards

An outcome reward model (ORM) scores the final response; a process reward model (PRM) scores each reasoning step, which gives denser credit and catches right-answer-wrong-reasoning. Lightman et al. (2023), Let's Verify Step by Step, found a PRM beat an ORM at selecting best-of-\(N\) solutions on MATH, but step labels are expensive, and TRL's experimental PRMTrainer was removed in v1.14 (September 2026). A generative reward prompts a strong LLM to judge: no reward training, but it inherits judge biases (module 09). A programmatic verifier, where one exists, beats all three; that is module 13.

Rafailov et al. (2023), Direct Preference Optimization, observed that you can skip both the reward model and the RL loop. Four steps.

Step 1: the optimum of the KL-regularized objective has a closed form. Fix \(x\) and write \(\pi(y)\) for \(\pi(y\mid x)\). Divide the objective by \(-\beta\) and rearrange:

\(Z(x)\) does not depend on \(\pi\), and \(\pi^*\) is a valid distribution. By Gibbs' inequality the KL is minimized, at zero, exactly when \(\pi = \pi^*\). So

Step 2: invert it. Take logs and solve for the reward:

Any reward can be written as a log-ratio of its own optimal policy, plus a prompt-only term.

Step 3: substitute into Bradley-Terry. BT only uses reward differences for the same prompt, so \(\beta\log Z(x)\) cancels. Parameterize \(\pi^*\) by the policy you are training, \(\pi_\theta\), and maximize the likelihood of the observed preferences:

The policy is the reward model. Its implicit reward is

Throughout this module \(h\) is the margin in implicit-reward units.

Step 4: the gradient. With \(\frac{d}{dh}[-\log\sigma(h)] = -\sigma(-h)\) and \(\nabla_\theta h = \beta\,(\nabla_\theta\log\pi_\theta(y_w) - \nabla_\theta\log\pi_\theta(y_l))\) (the reference is constant):

The bracket raises the chosen log-likelihood and lowers the rejected one. The weight \(\sigma(\hat r_l - \hat r_w)\) is near 1 for pairs the implicit reward ranks wrong and near 0 for pairs it already ranks right by a wide margin. DPO is a self-paced curriculum: pairs it has learned stop contributing. At initialization \(\pi_\theta = \pi_{\text{ref}}\), so every log-ratio is 0: the loss is \(\log 2 \approx 0.6931\) and the weight is \(\sigma(0)=0.5\), giving gradients of \(-\beta/2 = -0.05\) on the chosen sequence log-prob and \(+0.05\) on the rejected one at \(\beta=0.1\). At a margin \(h = 2\) the weight drops to \(\sigma(-2) = 0.1192\).

The check builds a tiny token-level "policy" for one chosen and one rejected sequence, makes the reference a detached copy, and verifies those numbers with autograd.

import math
import torch
import torch.nn.functional as F

torch.manual_seed(0)
beta, V = 0.1, 5
chosen, rejected = torch.tensor([1, 3, 0]), torch.tensor([2, 2, 4])  # token ids, T=3
logits_w = torch.randn(3, V, requires_grad=True)                      # (T, V) policy
logits_l = torch.randn(3, V, requires_grad=True)
ref_w, ref_l = logits_w.detach().clone(), logits_l.detach().clone()   # reference = frozen copy

def seq_logp(logits, toks):                                           # summed token log-probs
    return F.log_softmax(logits, -1).gather(-1, toks[:, None]).sum()

lp_w, lp_l = seq_logp(logits_w, chosen), seq_logp(logits_l, rejected)
lp_w.retain_grad(); lp_l.retain_grad()
h = beta * ((lp_w - seq_logp(ref_w, chosen)) - (lp_l - seq_logp(ref_l, rejected)))
loss = -F.logsigmoid(h)
loss.backward()
assert abs(h.item()) < 1e-7
assert abs(loss.item() - math.log(2)) < 1e-6                          # 0.6931
assert abs(lp_w.grad.item() - (-0.05)) < 1e-7                         # -beta * sigma(0)
assert abs(lp_l.grad.item() - 0.05) < 1e-7
# Chain rule to the logits: -0.05 * (onehot - softmax)
expected = -0.05 * (F.one_hot(chosen, V).float() - F.softmax(ref_w, -1))
torch.testing.assert_close(logits_w.grad, expected)

# Perturb so the margin is h = 2.0: the weight is sigma(-2) and the gradient beta * sigma(-2)
lw = torch.tensor(2.0 / beta, requires_grad=True)                     # chosen log-ratio = 20
ll = torch.tensor(0.0, requires_grad=True)
loss2 = -F.logsigmoid(beta * (lw - ll))
loss2.backward()
weight = torch.sigmoid(torch.tensor(-2.0)).item()
assert abs(weight - 0.1192) < 1e-4
assert abs(lw.grad.item() - (-beta * weight)) < 1e-7                  # -0.0119
assert abs(ll.grad.item() - beta * weight) < 1e-7
print("DPO: loss log 2 at init, weight 0.5 -> sigma(-2) at margin 2")

It does not show what happens to the probability of other sequences when shared parameters move; that is the likelihood-displacement story in 12.9.

The role of \(\beta\)

Higher \(\beta\) keeps the policy closer to the reference, not "learns preferences harder". In the objective, \(\beta\) multiplies the KL penalty. In the loss, \(\beta\) scales the log-ratios inside the sigmoid, so a given margin \(h\) is reached with a log-ratio gap of only \(h/\beta\): at \(\beta = 0.5\) the loss saturates after the policy moves 5 times less than at \(\beta = 0.1\), and then \(\sigma(-h)\) switches the gradient off. The DPO paper used 0.1 and 0.5; TRL's default is 0.1.

Assumptions and limits

The closed form holds for the unrestricted policy class, on the support of \(\pi_{\text{ref}}\), with the true reward. In practice: the policy is a finite network, so \(\pi_\theta\) cannot equal \(\pi^*\) and a gradient on one pair moves others; the data is finite, so the loss is minimized by driving \(h\to\infty\) on training pairs (overfitting, which IPO targets); and the pairs are off-policy, sampled from some other model, so DPO never sees what the current policy would say. These gaps explain most of the variants below.

DPOTrainer and DPOConfig are stable in TRL v1. As of October 2026 (v1.15.0, dpo_config.py):

max_length is v1.15's only truncation knob. use_liger_kernel=True is deprecated in DPOTrainer; pass model_init_kwargs={"use_kernels": True} instead. A PEFT adapter targeting lm_head raises, because log-probs are computed from lm_head.weight directly by the fused LM head; use modules_to_save=["lm_head"].

Where the reference log-probs come from

Standard DPO needs reference log-probs. It does not necessarily need a second model in memory. TRL picks the cheapest source available:

The "continue an existing adapter" row is the SFT-then-DPO path of the lab: load the SFT adapter with is_trainable=True, and the reference is the SFT policy, not the base model (dpo_from_scratch.py's full() shows both PEFT paths). precompute_ref_log_probs is incompatible with sync_ref_model (a moving reference makes cached values stale) and with IterableDataset.

Two ways to move the reference. TR-DPO (Gorbatovski et al. 2024) argues a reference that tracks the policy lets you train longer before over-optimization. It moves in jumps: frozen for \(\tau\) steps, then mixed with \(\alpha=0.6\), so it lags the policy and keeps 40% of the old reference after each sync (no PEFT support without a standalone reference). BEMA (Block and Zhang 2025) uses a bias-corrected running average of the policy's iterates: an EMA \(\bar\theta_t = (1-\omega_t)\bar\theta_{t-1} + \omega_t\theta_t\), \(\omega_t = (\rho+\gamma t)^{-\kappa}\), plus a correction, \(\theta'_t = \bar\theta_t + a_t(\theta_t - \theta_0)\) with \(a_t = (\rho+\gamma t)^{-\eta}\). A plain EMA lags the current weights because it averages old iterates; the \(a_t(\theta_t-\theta_0)\) term cancels that lag and keeps the smoothing. The experimental trl.experimental.bema_for_ref_model.BEMACallback(update_ref_model=True) copies \(\theta'_t\) into the reference every 400 steps by default (docs).

Padding

In v1.15 the text DPO collator right-pads prompt plus completion and builds a completion_mask so only completion tokens enter the sequence log-prob. Wherever you generate (iterative DPO, best-of-\(N\)), use left padding, or new tokens follow pad tokens (module 02).

loss_type accepts 15 values, each a different function of the log-ratios. Notation: \(\Delta_w = \log\frac{\pi_\theta(y_w)}{\pi_{\text{ref}}(y_w)}\) and \(\Delta_l\) likewise (summed over completion tokens), \(\hat r_w = \beta\Delta_w\), \(\hat r_l = \beta\Delta_l\), \(h = \hat r_w - \hat r_l\), and \(\bar\Delta = \Delta_w/\lvert y_w\rvert - \Delta_l/\lvert y_l\rvert\), the difference of per-token average log-ratios. Every formula below was read off the v1.15.0 loss block.

Three things to notice. ipo is the only one that stops at a finite margin: its minimum sits at \(\bar\Delta = 1/(2\tau)\), so with the default \(\beta=0.1\) it targets 5 nats per token, which is far too aggressive; IPO runs use a larger \(\tau\) (0.5 gives a target of 1). apo_zero, bco_pair and nca_pair act on each response's own log-ratio, so they can force the chosen likelihood up; sigmoid, hinge and robust only see the difference. sigmoid_norm is not SimPO: it still uses the reference model (it length-normalizes \(\log\pi_\theta/\pi_{\text{ref}}\)), and has no margin \(\gamma\). Exact, reference-free SimPO lives in the experimental CPO trainer (12.6).

Multi-loss and variant knobs

Several losses are averaged per pair and summed with loss_weights. MPO (Wang et al. 2024), the standard VLM recipe in module 16, combines three:

DPOConfig(loss_type=["sigmoid", "bco_pair", "sft"], loss_weights=[0.8, 0.2, 1.0])  # MPO

Three knobs modify the log-ratios before any loss sees them:

The check evaluates five TRL losses, plus SimPO from 12.6, exactly as written. Fixture: \(\beta=0.1\), \(\Delta_w = 3\), \(\Delta_l = -2\) (so \(h=0.5\)), \(\lvert y_w\rvert = 20\), \(\lvert y_l\rvert = 10\) (so \(\bar\Delta = 0.35\)). Values: sigmoid \(0.4741\), hinge \(0.5000\), robust (\(\epsilon=0.1\)) \(0.4116\), IPO (\(\tau=0.5\)) \(0.4225\), sigmoid_norm \(0.6758\). For SimPO (12.6) with \(\beta_s = 2.0\), \(\gamma = 0.5\), policy log-probs \(-12\) and \(-9\): \(0.6444\).

import math
import torch
import torch.nn.functional as F

ls = F.logsigmoid
beta, eps = 0.1, 0.1
d_w, d_l, len_w, len_l = torch.tensor(3.0), torch.tensor(-2.0), 20.0, 10.0

def sigmoid_loss(h):            return -ls(h)
def hinge_loss(h):              return torch.relu(1 - h)
def robust_loss(h, e):          return (-(1 - e) * ls(h) + e * ls(-h)) / (1 - 2 * e)
def ipo_loss(avg_delta, tau):   return (avg_delta - 1 / (2 * tau)) ** 2
def sigmoid_norm_loss(avg_delta, b): return -ls(b * avg_delta)
def simpo_loss(lp_w, lp_l, n_w, n_l, b, gamma):
    return -ls(b * lp_w / n_w - b * lp_l / n_l - gamma)

h = beta * (d_w - d_l)
avg = d_w / len_w - d_l / len_l
assert abs(h.item() - 0.5) < 1e-7 and abs(avg.item() - 0.35) < 1e-7
vals = {
    "sigmoid": sigmoid_loss(h), "hinge": hinge_loss(h), "robust": robust_loss(h, eps),
    "ipo": ipo_loss(avg, 0.5), "sigmoid_norm": sigmoid_norm_loss(avg, beta),
    "simpo": simpo_loss(torch.tensor(-12.0), torch.tensor(-9.0), len_w, len_l, 2.0, 0.5),
}
stated = {"sigmoid": 0.4741, "hinge": 0.5000, "robust": 0.4116,
          "ipo": 0.4225, "sigmoid_norm": 0.6758, "simpo": 0.6444}
for k, v in stated.items():
    assert round(vals[k].item(), 4) == v, (k, vals[k].item())

# At h = 0: sigmoid = log 2, hinge = 1, robust reduces to sigmoid when eps = 0
z = torch.tensor(0.0)
assert abs(sigmoid_loss(z).item() - math.log(2)) < 1e-7
assert hinge_loss(z).item() == 1.0
for hv in (-1.0, 0.0, 0.5, 3.0):
    hv = torch.tensor(hv)
    torch.testing.assert_close(robust_loss(hv, 0.0), sigmoid_loss(hv))
# IPO is minimized (zero loss, zero gradient) at its target average margin 1/(2 tau)
tau = 0.5
a = torch.tensor(1 / (2 * tau), requires_grad=True)
loss = ipo_loss(a, tau)
loss.backward()
assert loss.item() == 0.0 and a.grad.item() == 0.0
assert ipo_loss(torch.tensor(1.5), tau) > 0 and ipo_loss(torch.tensor(0.5), tau) > 0
print("loss zoo fixture values verified")

It does not tell you which loss trains the better model; that is an empirical question, and L12.2 is how you answer it for your data.

ORPO (Hong et al. 2024) folds SFT and preference learning into one stage with no reference model. With \(p(y) = \exp\big(\frac{1}{\lvert y\rvert}\log\pi_\theta(y\mid x)\big)\) (the per-token geometric-mean probability) and \(\mathrm{odds}(y) = \frac{p}{1-p}\):

The NLL term teaches the format; the odds-ratio term pushes the rejected style down. In TRL it is trl.experimental.orpo.ORPOTrainer, and ORPOConfig.beta is \(\lambda\) (default 0.1; config, loss). Use it when you have preference data and a base model, and want one run instead of two.

SimPO (Meng et al. 2024) drops the reference and uses the length-normalized policy log-likelihood as the reward, with a target margin \(\gamma\):

Length normalization removes the incentive to pad; \(\gamma\) demands a gap, not just an ordering. SimPO's per-token \(\beta\) is much larger than DPO's (the paper sweeps roughly 2 to 10). In TRL, exact SimPO is trl.experimental.cpo.CPOTrainer with loss_type="simpo", cpo_alpha=0.0, and simpo_gamma (default 0.5) (config, loss). DPO's sigmoid_norm is the nearest stable-trainer option, but it keeps the reference.

CPO (Xu et al. 2024) came from machine translation: \(-\log\sigma\big(\beta\log\pi_\theta(y_w) - \beta\log\pi_\theta(y_l)\big) + \alpha\,\mathcal{L}_{\text{NLL}}(y_w)\), reference-free DPO plus a behavior-cloning term (cpo_alpha=1.0 by default); loss_type="simpo" with non-zero cpo_alpha gives CPO-SimPO. All three trainers are experimental (trl.experimental, no API stability promise).

Pairs are expensive; thumbs are free. Production logs say "good" or "bad", rarely two answers to one prompt. KTO (Ethayarajh et al. 2024) learns from that unpaired signal with a prospect-theory value function. With implicit reward \(\hat r = \beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)}\) and a reference point \(z_0\):

where \(\Delta\) is the log-ratio and \(z_0\) is a batch estimate of \(\mathrm{KL}(\pi_\theta\,\|\,\pi_{\text{ref}})\), computed on deliberately mismatched (prompt, completion) pairs from the same batch, clamped at 0 and detached. A desirable output earns loss reduction only for beating the policy's typical log-ratio. The sigmoid saturates in both directions, which is the "loss aversion" story and why KTO is reported robust to noisy labels.

TRL's KTOTrainer is stable (v1.15.0 docs). Data is {"prompt", "completion", "label": True/False}; a paired dataset is split automatically. desirable_weight and undesirable_weight (both 1.0) are \(\lambda_D\), \(\lambda_U\); the paper's guidance, which TRL warns on, is \(\frac{\lambda_D n_D}{\lambda_U n_U} \in [1, \tfrac{4}{3}]\), so with 1,000 thumbs-up and 9,000 thumbs-down set desirable_weight between 9.0 and 12.0. The KL estimate pairs each prompt with a neighbor's completion, so KTO needs per_device_train_batch_size > 1 and the default sequential sampling. TRL's KTO docs advise a learning rate of about 1e-6 at the default \(\beta = 0.1\), and never outside 5e-7 to 5e-6. Use KTO for thumbs-up/down logs (module 04); use DPO when you genuinely have pairs.

Offline DPO trains on responses the current policy would never write. Fix: label fresh ones.

• Online DPO (Guo et al. 2024): each step, sample two completions per prompt from the current policy, have an LLM annotator pick the winner, take a DPO step on that pair. They report it beats offline DPO and RLHF in human evaluation. TRL: trl.experimental.online_dpo.OnlineDPOTrainer, which also accepts a reward model (docs).
• Iterative DPO: sample \(k\) completions per prompt from the current model, rank them with an RM, keep (best, worst), run offline DPO, repeat. This is the rounds recipe of Llama 3 and Tulu 3 (module 01). L12.3 runs one round.
• Self-rewarding loops (Yuan et al. 2024): the model generates candidates and judges them, then trains on its own preferences with DPO. The judge's biases (verbosity, self-preference) become the target and nothing external catches drift, so anchor every round with a fixed external eval set and a human or strong-judge check.
• RLOO samples \(k\) completions and uses the mean reward of the other \(k-1\) as a baseline for a policy-gradient step, with no value model; it and GRPO are module 13.

RLAIF and beyond chat

RLAIF replaces human preference labels with AI feedback (Bai et al. 2022, Constitutional AI; its safety use is module 19). It now powers most synthetic preference data: an LLM judge scores several candidates and the dataset keeps (best, worst), as in UltraFeedback (module 04). For VLMs, RLAIF-V supplies AI-labeled image pairs and MPO is the usual loss mix (module 16); DPOTrainer takes an image/images column, with max_length=None so truncation cannot cut image tokens. For tool calling, pairs carry a tools column of JSON schemas and the rejected response is often a malformed call (module 15).

DPOTrainer logs these, per the metrics code:

A healthy start: rewards and margins at 0, accuracy at 0 (ties count as wrong), loss \(\log 2\). A healthy run: margins rise smoothly, held-out accuracy plateaus well above 0.5, loss falls without collapsing to zero, and samples improve under a judge.

Both log-probs go down

In almost any DPO run rewards/chosen goes negative: the chosen response becomes less likely than under the reference, while the rejected one falls faster. TRL's own docs say DPO typically widens the margin "by suppressing the likelihood of dispreferred completions rather than by increasing the likelihood of preferred ones".

This is not a bug; it is what the loss allows. \(\mathcal{L}_{\text{DPO}}\) depends only on \(h\), so moving both log-ratios down, the rejected one faster, lowers it. The worked fixture: chosen log-prob falls from \(-20\) to \(-22\) and rejected from \(-20\) to \(-30\) relative to the reference. Then rewards/chosen \(= 0.1 \times (-2) = -0.2\), rewards/rejected \(= -1.0\), margin \(0.8\), accuracy 1, and the loss drops from \(0.6931\) to \(-\log\sigma(0.8) = 0.3711\).

Where does the mass go? Not to \(y_w\) and \(y_l\), so to other sequences. Pal et al. (2024), Smaug / DPO-Positive, prove standard DPO can reduce the chosen likelihood whenever the relative probability increases, and see it most on pairs with small edit distance. Razin et al. (2024), Likelihood Displacement, show it can be catastrophic: training to prefer "No" over "Never" can sharply raise "Yes", and in a refusal-training experiment it cut Llama-3-8B-Instruct's refusal rate from 74.4% to 33.4%. The driver is preferences whose responses have similar hidden embeddings: the gradient pushing \(y_l\) down also pushes \(y_w\) down, and frees mass for whatever response loads most on the direction separating them.

When it is harmless: the chosen log-prob falls a few nats, judge win rate rises, samples look right. When it is harmful: the chosen log-prob keeps sliding by tens of nats, samples degrade, safety evals regress, or outputs drift to an unseen mode. Then intervene:

The check computes the TRL metrics on a 4-pair fixture (\(\beta=0.1\); rewards/chosen \(0.0125\), rewards/rejected \(-0.2000\), margins \(0.2125\), accuracy \(0.75\), loss \(0.5984\)), then the both-fall pair, then takes one real SGD step on a linear-feature policy where both log-probs fall and the loss still drops.

import math
import torch
import torch.nn.functional as F

def metrics(pol_w, pol_l, ref_w, ref_l, beta):
    cr, rr = beta * (pol_w - ref_w), beta * (pol_l - ref_l)
    m = cr - rr
    return {"chosen": cr.mean().item(), "rejected": rr.mean().item(), "margins": m.mean().item(),
            "accuracies": (cr > rr).float().mean().item(), "loss": (-F.logsigmoid(m)).mean().item()}

beta = 0.1
ref_w = torch.tensor([-30., -25., -40., -18.]); pol_w = torch.tensor([-28., -25., -41., -18.5])
ref_l = torch.tensor([-32., -27., -35., -20.]); pol_l = torch.tensor([-35., -26., -39., -22.])
m = metrics(pol_w, pol_l, ref_w, ref_l, beta)
for k, v in {"chosen": 0.0125, "rejected": -0.2, "margins": 0.2125,
             "accuracies": 0.75, "loss": 0.5984}.items():
    assert round(m[k], 4) == v, (k, m[k])

# The "both fall" pair: chosen -20 -> -22, rejected -20 -> -30 relative to the reference
b = metrics(torch.tensor([-22.]), torch.tensor([-30.]), torch.tensor([-20.]), torch.tensor([-20.]), beta)
assert round(b["chosen"], 4) == -0.2 and round(b["rejected"], 4) == -1.0
assert round(b["margins"], 4) == 0.8 and b["accuracies"] == 1.0
assert round(b["loss"], 4) == 0.3711 and b["loss"] < math.log(2)

# One real gradient step that lowers BOTH log-probs and still lowers the loss.
phi = torch.tensor([[1., 1.],    # chosen
                    [1., 0.],    # rejected (shares feature 0 with chosen)
                    [0., 4.],    # unseen response, loads on the chosen-minus-rejected direction
                    [0., 0.]])
theta = torch.zeros(2, requires_grad=True)
ref = F.log_softmax(phi @ torch.zeros(2), -1)
def step_stats(t):
    lp = F.log_softmax(phi @ t, -1)
    loss = -F.logsigmoid(beta * ((lp[0] - ref[0]) - (lp[1] - ref[1])))
    return lp, loss
lp0, loss0 = step_stats(theta)
loss0.backward()
with torch.no_grad():
    theta1 = theta - 1.0 * theta.grad
    lp1, loss1 = step_stats(theta1)
assert lp1[0] < lp0[0] and lp1[1] < lp0[1]           # chosen AND rejected log-probs fall
assert loss1 < loss0                                  # yet the DPO loss improves
assert lp1[2].exp() > lp0[2].exp()                    # the mass went to the unseen response
print("metrics fixture, both-fall pair and displacement step verified")

It does not show whether the unseen response that gained mass is better; only a judge or a human can tell you, which is why diagnostics never replace evaluation.

Length exploitation

Summed log-probs give long responses larger log-ratios, and chosen responses are usually longer, so DPO can learn "longer is better". Symptom: mean output length climbs while a length-controlled judge's win rate stays flat (module 09). Fix with sigmoid_norm, ld_alpha (sweep values below 1), SimPO, or length-balanced pairs.

Most failed preference runs are data failures. Five recur (construction is module 04):

• Off-policy pairs. Text from other models is text your policy would not write; ranking it transfers imperfectly. Use WPO weighting, or regenerate responses with your SFT model.
• Noisy labels. InstructGPT's trained labelers agreed on only about 73% of comparisons (Ouyang et al. 2022). Use robust with label_smoothing at your estimated flip rate (0.1 is typical), or ipo, which does not chase infinite margins.
• Near-identical pairs. A one-word difference carries little signal and, per Razin et al., drives likelihood displacement. Filter by edit distance or embedding similarity.
• Chosen from a much stronger model. It teaches "be the stronger model", which the policy cannot do, so the shortcut is to push its own outputs down. Keep both responses near on-policy.
• Length bias. If chosen is systematically longer, the cheapest margin is verbosity.

Preference tuning can erode upstream safety behavior; re-run module 19's safety evals after every stage.

Goal: run SFT → DPO, read its diagnostics, and measure what the loss choice does. Tier: CPU smoke for both scripts; T1 (24 GB) for L12.1 to L12.3; T0 (free T4) for L12.4. Models: Qwen/Qwen3-1.7B (or Qwen/Qwen3-4B on T1 with LoRA). Dataset: trl-lib/ultrafeedback_binarized.

Scripts: dpo_from_scratch.py, reward_model_bt.py. Smoke first:

python content/courses/finetuning/code/dpo_from_scratch.py --smoke
python content/courses/finetuning/code/reward_model_bt.py --smoke

The DPO smoke trains a tabular policy (16 prompts, 6 responses each) on 48 synthetic pairs, asserts accuracy reaches 1.0 with margins rising every interval, then reproduces the both-fall fixture. rewards/chosen goes negative even here, because a response chosen in one pair is rejected in another. The RM smoke asserts held-out Bradley-Terry pairwise accuracy above 0.9.

L12.1 SFT → DPO (T1)

Train DPO on module 05's SFT LoRA adapter (or from Qwen/Qwen3-1.7B). With --sft-adapter, TRL loads a frozen "ref" copy of the adapter, so the reference is your SFT policy:

python content/courses/finetuning/code/dpo_from_scratch.py --full --sft-adapter ./sft-out --loss-type sigmoid

Record rewards/margins, rewards/accuracies and both logps/* every 10 steps, plus mean generated length on 200 held-out prompts. Evaluate SFT and DPO with module 09's tooling: a length-controlled pairwise judge and IFEval (google/IFEval). Expect: eval accuracy well above 0.5, chosen log-probs drifting down slowly, a judge win-rate gain, some length growth.

L12.2 Loss-type ablation (T1)

Run four arms from the same starting point and step budget:

python content/courses/finetuning/code/dpo_from_scratch.py --full --loss-type sigmoid
python content/courses/finetuning/code/dpo_from_scratch.py --full --loss-type ipo --beta 0.5
python content/courses/finetuning/code/dpo_from_scratch.py --full --loss-type sigmoid_norm --beta 2.0
python content/courses/finetuning/code/dpo_from_scratch.py --full --loss-type apo_zero

ipo and sigmoid_norm use per-token averages, so they need a larger \(\beta\). Plot mean output length (x) against LC win rate (y), one point per arm. Expect: sigmoid longest; sigmoid_norm shorter at a similar win rate; apo_zero with the chosen log-prob rising instead of falling.

L12.3 Reward model and one round of iterative DPO (T1)

python content/courses/finetuning/code/reward_model_bt.py --full --step train --rm-model Qwen/Qwen3-0.6B
python content/courses/finetuning/code/reward_model_bt.py --full --step rank --rm-path rm-out --policy Qwen/Qwen3-1.7B

The rank step samples 4 completions per prompt (left-padded), scores them with the RM, and writes (best, worst) pairs to onpolicy_pairs.jsonl, skipping near-ties. Train DPO on them:

python content/courses/finetuning/code/dpo_from_scratch.py --full --dataset onpolicy_pairs.jsonl --output-dir dpo-iter1

Compare against the offline L12.1 model and against best-of-4 from the starting model under the same judge. If iterative DPO does not beat best-of-4, say so.

L12.4 KTO on simulated thumbs (T0)

Take the module 01 support-email JSON extraction outputs, label each True when the JSON parses and every field matches the gold record, False otherwise, and train KTO with Qwen/Qwen3-0.6B in fp16 (T4 has no bf16). Teaching fragment (not executed); pass args to KTOTrainer with a LoRA peft_config exactly as for DPO:

from trl import KTOConfig

# rows: {"prompt": [...messages...], "completion": [{"role": "assistant", "content": json_str}], "label": bool}
n_good, n_bad = 1200, 4800
args = KTOConfig(output_dir="kto-thumbs", beta=0.1, learning_rate=5e-6,   # LoRA: top of TRL's range
                 per_device_train_batch_size=8,            # > 1: the KL estimate needs neighbors
                 desirable_weight=n_bad / n_good,          # 4.0: ratio lambda_D n_D / lambda_U n_U = 1
                 undesirable_weight=1.0, bf16=False, fp16=True)

Record JSON validity and field accuracy on the 50-example golden set before and after. Expect: validity up; if field accuracy drops, your "good" label is too lenient.