Post-Training and Alignment: SFT, RLHF, DPO and RL for Reasoning

A pretrained base model is a superb text continuer and a poor assistant. Ask it a question and it may continue with three more questions, because that is what a forum page looks like. Post-training turns the base model into something that follows instructions, holds a conversation, refuses genuinely harmful requests, uses tools, and reasons step by step. It uses a tiny fraction of pretraining compute but determines most of what users experience. Whether you fine-tune open models or simply choose between hosted ones, you need to know what supervised fine-tuning, RLHF, DPO, Constitutional AI and reinforcement learning with verifiable rewards actually optimise, and what failure modes each introduces.

Intuition: from a well-read apprentice to a professional

Think of the base model as an apprentice who has read every book in the library but has never worked with a client. Supervised fine-tuning is showing the apprentice hundreds of thousands of example consultations: "here is a request, here is a good answer." Preference training is a senior colleague comparing two of the apprentice's drafts and saying which is better, over and over, so the apprentice internalises taste that is hard to write down. Reinforcement learning with verifiable rewards is handing the apprentice a stack of problems with answer keys and letting them practise until they find methods that work. Safety training is the professional code of conduct. Each stage builds on the last.

Stage 1: supervised fine-tuning (SFT)

SFT, also called instruction tuning, continues training with the same next-token loss, but on curated (prompt, response) pairs formatted as conversations. The crucial difference from pretraining is loss masking: only the response tokens contribute to the loss, so the model learns to produce answers, not to imitate users.

$\mathcal{L}_{\text{SFT}}(\theta) = -\sum_{t \in \text{response}} \log \pi_\theta(y_t \mid x, y_{<t})$

Here $x$ is the prompt (including system message and earlier turns), $y$ the target response, and $\pi_\theta$ the model viewed as a policy over tokens. Data comes from human-written demonstrations, filtered outputs of stronger models, and task collections reformatted as instructions. Quality and diversity dominate quantity: tens of thousands of excellent examples often beat millions of mediocre ones. SFT teaches format and behaviour, and it mostly surfaces knowledge already present from pretraining. Fine-tuning on facts the model never learned tends to increase hallucination, because it teaches the model to answer confidently beyond its knowledge.

Chat templates and special tokens

Conversations must be serialised into a single token sequence. A chat template defines how: special tokens mark the start and end of each turn and the role of the speaker (system, user, assistant, tool). For example, the ChatML convention wraps turns in <|im_start|> and <|im_end|>, while Llama 3 models use <|start_header_id|>role<|end_header_id|> headers and end turns with <|eot_id|>. These special tokens are single vocabulary items that never appear in normal text, which is what lets the model reliably know who is speaking and when to stop.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
messages = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "What is 2 + 2?"},
]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(text)  # shows the exact special-token format the model was trained on

The template is stored with the tokenizer (in its configuration) and apply_chat_template renders it. add_generation_prompt=True appends the header for the assistant's turn so the model starts answering. Using the wrong template is one of the most common causes of mysteriously bad fine-tuned or served models.

Stage 2: RLHF

Reinforcement learning from human feedback was developed by Christiano et al. (2017) and brought to language models at scale by OpenAI's InstructGPT work (Ouyang et al., 2022), which reported that labelers preferred outputs of a 1.3B-parameter InstructGPT model over those of the 175B GPT-3. The pipeline has three steps.

Step A: collect comparisons

For a prompt $x$, sample two or more responses from the SFT model and ask human labelers which they prefer. This yields triples $(x, y_w, y_l)$ with $y_w$ preferred ("winner") over $y_l$ ("loser"). Comparisons are much easier and more consistent for people than writing ideal answers or assigning absolute scores.

Step B: train a reward model with the Bradley-Terry loss

The Bradley-Terry model says the probability that $y_w$ beats $y_l$ is a logistic function of the difference of their scalar "strengths":

$P(y_w \succ y_l \mid x) = \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big), \qquad \sigma(z) = \frac{1}{1+e^{-z}}$

The reward model $r_\phi$ is usually the SFT model with its output layer replaced by a single scalar head. It is trained by maximum likelihood on the comparisons:

$\mathcal{L}_{\text{RM}}(\phi) = -\mathbb{E}_{(x, y_w, y_l)}\Big[\log \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big)\Big]$

Only reward differences are identified, so the absolute level is arbitrary and is typically normalised.

Step C: optimise the policy with PPO and a KL penalty

Now treat the language model as a policy $\pi_\theta$ and maximise reward, while staying close to the SFT model $\pi_{\text{ref}}$:

$\max_{\theta}\ \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot\mid x)}\big[r_\phi(x, y)\big] - \beta\, \mathbb{D}_{\text{KL}}\big[\pi_\theta(\cdot \mid x)\,\|\,\pi_{\text{ref}}(\cdot \mid x)\big]$

The coefficient $\beta$ controls how far the policy may drift. Without the KL term the policy finds reward hacking exploits: responses that score highly on the imperfect reward model but are verbose, sycophantic or nonsensical. In practice the KL term is applied per token as a reward penalty, $r_t = -\beta \log\frac{\pi_\theta(y_t\mid\cdot)}{\pi_{\text{ref}}(y_t\mid\cdot)}$, plus the reward model score at the final token. Proximal Policy Optimization (Schulman et al., 2017) then updates the policy with its clipped surrogate objective,

$\mathcal{L}^{\text{CLIP}}(\theta) = \mathbb{E}_t\Big[\min\big(\rho_t \hat{A}_t,\ \text{clip}(\rho_t, 1-\epsilon, 1+\epsilon)\,\hat{A}_t\big)\Big], \quad \rho_t = \frac{\pi_\theta(y_t \mid s_t)}{\pi_{\theta_{\text{old}}}(y_t \mid s_t)}$

where $\hat{A}_t$ is an advantage estimate from a learned value (critic) network and $\epsilon$ is around 0.2. RLHF with PPO is effective but heavy: four models in memory (policy, reference, reward, critic), on-policy generation in the loop, and many sensitive hyperparameters.

Stage 2, alternative: Direct Preference Optimization

Rafailov et al. (2023) showed that the KL-regularised objective above has a closed-form optimal policy, $\pi^*(y\mid x) \propto \pi_{\text{ref}}(y\mid x)\exp\big(r(x,y)/\beta\big)$. Inverting it expresses the reward in terms of the policy, $r(x,y) = \beta \log\frac{\pi(y\mid x)}{\pi_{\text{ref}}(y\mid x)} + \beta\log Z(x)$, and the intractable partition function $Z(x)$ cancels when you take a difference of two responses in the Bradley-Terry model. The result is a simple classification loss on preference pairs:

$\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x, y_w, y_l)}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\right)\right]$

No reward model, no sampling during training, two models in memory. The authors reported DPO matching or beating PPO-based RLHF on sentiment control, summarisation and dialogue. The trade-off is that DPO learns only from a fixed (offline) dataset, so it cannot explore beyond the responses it was given, and it can over-optimise, for example by pushing down the likelihood of both responses. Many variants (IPO, KTO, ORPO, SimPO and others) adjust the loss; online and iterative DPO, which regenerates pairs from the current policy, recovers some of RL's benefits.

Worked example: one DPO loss term

Let $\beta = 0.1$. For a preference pair, log-probabilities summed over response tokens are:

1. Implicit reward margin: $\beta(3.0 - (-2.0)) = 0.1 \times 5.0 = 0.5$.
2. $\sigma(0.5) = 1/(1+e^{-0.5}) = 1/(1+0.6065) = 0.6225$.
3. Loss: $-\ln 0.6225 = 0.474$.

At initialisation $\pi_\theta = \pi_{\text{ref}}$, so the margin is 0 and the loss is $\ln 2 \approx 0.693$. Training raises the margin, pushing the loss toward zero.

import torch
import torch.nn.functional as F

def dpo_loss(pol_w, pol_l, ref_w, ref_l, beta=0.1):
    # each argument: tensor of summed log-probs per example, shape [batch]
    margin = beta * ((pol_w - ref_w) - (pol_l - ref_l))
    return -F.logsigmoid(margin).mean()

print(dpo_loss(torch.tensor([-42.0]), torch.tensor([-50.0]),
               torch.tensor([-45.0]), torch.tensor([-48.0])))  # ≈ 0.474

Libraries such as Hugging Face TRL provide production trainers for SFT, DPO, PPO-style RLHF and GRPO.

RLAIF and Constitutional AI

Human preference labels are slow, expensive and can expose labelers to disturbing content. Anthropic's Constitutional AI paper (Bai et al., 2022) replaced human harmfulness labels with AI feedback guided by a short written list of principles, the "constitution". It has two phases:

1. Supervised phase: the model generates responses to harmful prompts, critiques its own response against a randomly chosen principle, revises it, and is fine-tuned on the revisions.
2. RL phase: the fine-tuned model produces pairs of responses; an AI model judges which better follows the constitution; those AI preferences train a preference model used as the reward for RL. This is reinforcement learning from AI feedback (RLAIF).

The aims were a model that is harmless but not evasive, one that explains objections rather than simply refusing, with behaviour governed by explicit, inspectable principles. AI feedback is now used throughout the industry, typically mixed with human data, and "LLM as judge" reward models are standard.

RL with verifiable rewards for reasoning

For math, code and other tasks with checkable answers, you can skip the learned reward model and use a verifier: does the final answer match the reference, do the unit tests pass, is the output in the required format? This removes a major source of reward hacking and lets RL run at very large scale.

GRPO (Group Relative Policy Optimization), introduced in DeepSeekMath (Shao et al., 2024), is a PPO variant that drops the critic. For each prompt $q$ it samples a group of $G$ outputs from the current policy, scores them, and uses the group's statistics as the baseline:

$\hat{A}_i = \frac{r_i - \text{mean}(r_1,\dots,r_G)}{\text{std}(r_1,\dots,r_G)}$

Every token of output $i$ receives advantage $\hat{A}_i$, plugged into the PPO clipped objective averaged over tokens and outputs. Instead of putting the KL penalty into the reward, GRPO adds $\beta\,\mathbb{D}_{\text{KL}}[\pi_\theta\|\pi_{\text{ref}}]$ directly to the loss, using the estimator $\frac{\pi_{\text{ref}}}{\pi_\theta} - \log\frac{\pi_{\text{ref}}}{\pi_\theta} - 1$, which is always non-negative. Dropping the critic saves a model's worth of memory and avoids training a value function on long, sparse-reward sequences.

Worked example: group advantages

A prompt gets $G = 4$ sampled solutions with verifier rewards $r = (1, 0, 0, 1)$. Mean $= 0.5$; population standard deviation $= 0.5$. Advantages: $(+1, -1, -1, +1)$. The two correct solutions are reinforced and the two wrong ones discouraged. If all four were correct, every advantage would be zero and the prompt teaches nothing, which is why practitioners filter out prompts that are always or never solved.

DeepSeek-R1 (DeepSeek-AI, 2025) showed what this can do at scale. R1-Zero applied GRPO with rule-based accuracy and format rewards directly to a base model with no SFT, and long chains of thought, self-verification and backtracking emerged on their own, though the outputs were hard to read and mixed languages. The released R1 added a small "cold-start" SFT set, reasoning RL, a further round of SFT on filtered samples and general data, and a final RL stage for helpfulness and harmlessness. Its reasoning traces were also distilled into smaller dense models. Successor methods (for example DAPO and Dr. GRPO) adjust GRPO's normalisation and clipping. See the reasoning-models lesson for inference-side behaviour.

Safety training

Safety work runs through every stage: SFT on examples of good refusals and safe completions, preference data that penalises harmful help and also over-refusal, red-teaming (human and automated) to find failures, adversarial training against jailbreaks, and system-level layers such as input and output classifiers. A key tension is helpfulness versus harmlessness: pushing too hard on refusals yields a model that declines benign requests ("how do I kill a Python process?"). Modern practice measures both harmful-compliance and over-refusal rates and trains on borderline cases. Alignment training shapes behaviour but does not guarantee it; fine-tuning an aligned open model on a small amount of data can remove safety behaviour, which matters when you release weights.

Common pitfalls

• Training on prompt tokens in SFT, which teaches the model to imitate users; mask them.
• Wrong or inconsistent chat template between training and serving.
• Reward hacking: length bias (longer answers score higher), sycophancy, formatting tricks. Monitor length and KL, and audit high-reward samples by hand.
• Too small a KL coefficient or too many DPO epochs, causing drift, repetition or collapsed diversity.
• Noisy preference data: inter-annotator agreement on comparisons is often only around 60 to 75 percent; measure it.
• Verifier loopholes in RLVR: answer extractors that accept malformed output, or unit tests the model can game.
• Evaluating only helpfulness and forgetting over-refusal and safety regressions, or the reverse.

In practice

A typical open-model recipe today: SFT on a curated mixture (often partly synthetic and filtered), then preference optimisation (DPO or a variant, sometimes online), then RL with verifiable rewards for math and code, then safety tuning, with evaluations after every stage. Small teams adapting an open model usually need only SFT, frequently with LoRA adapters, plus possibly DPO on a few thousand preference pairs. Use a maintained library (TRL, or similar frameworks) rather than writing PPO from scratch, keep a frozen reference model for KL, log reward, KL and response length together, and always read samples. Alignment methods change quickly; the stable ideas are the objectives above.