Post

LLMs from Scratch Part 4: Making LLMs Useful with RLHF, InstructGPT, and DPO

Posted Updated
By Jayden
12 min read

Part 3 ended with a question: GPT-3 is capable, so why isn’t it ChatGPT? The answer is alignment, the gap between a model that predicts likely text and one that follows what people actually want. This post covers the three-stage pipeline that narrowed that gap, the math behind it, and DPO, the 2023 method that removes two of its stages.

Table of Contents

0. The Alignment Problem

Say a quant fund wants to use an LLM for financial research. It needs to answer questions about balance sheets, summarize earnings calls, and flag risk disclosures. GPT-3 can do all of these, sometimes. Ask it to summarize a 10-K and it might summarize, critique the filing, or keep writing the 10-K. Ask it a direct question and it might answer, rephrase the question, or list related questions it finds more interesting.

Nothing is wrong with the model in a technical sense. It is doing what it was trained to do: produce text that plausibly continues the input. “Plausible continuation” and “useful response” are just different objectives. To GPT-3, a prompt is more text; it has no built-in idea of a user, a question, or an expected format.

GPT-3 objective: predict likely next tokens Prompt: "What drove Apple's Q3 margin expansion?" "What drove Apple's Q3 margin expansion? What drove Apple's Q3 margin expansion? Some analysts believe the key drivers were..." echoes the prompt instead of answering InstructGPT objective: follow human intent Prompt: "What drove Apple's Q3 margin expansion?" "Three main factors drove the expansion: services mix shift (+2.1pp), component cost relief, and favorable FX hedging gains." answers the question directly

Figure 1. The alignment gap in a financial context. GPT-3 predicts a plausible continuation, which for a repeated question often means echoing it. InstructGPT has learned that a question calls for an answer.

Making the model useful means training it on a different target: not “what text is likely to come next?” but “what would a knowledgeable, helpful person say?” Those are different distributions, and learning the second one requires human feedback that says which responses are better, not just which are probable.

That is the alignment problem in practice. InstructGPT (Ouyang et al., 2022) tackled it with a three-stage pipeline that became the basis for ChatGPT and most later instruction-tuned models.

1. Supervised Fine-Tuning

The first stage is the simplest: collect demonstrations of good behavior and fine-tune the pre-trained model on them.

OpenAI hired a team of about 40 contractors, screened for their ability to judge sensitive content and follow detailed guidelines. They wrote responses to prompts, most of them sampled from real API usage (plus some the labelers wrote themselves). The resulting (prompt, ideal response) pairs were used to fine-tune GPT-3 with a standard supervised cross-entropy objective

\[L_{SFT} = -\sum_{t} \log P_\theta(y_t \mid y_{<t}, x)\]

where $x$ is the prompt and $y$ is the demonstrated response.

Prompt Pool "Summarize this earnings call." "What is the FCF yield of AAPL?" "Explain duration risk simply." real API queries 👩‍💼 Human Labeler writes ideal response SFT Dataset (x) "Explain duration risk" (y) "Duration measures how sensitive a bond's price..." (x) "What is FCF yield?" (y) "Free cash flow yield divides FCF by market cap..." SFT Model GPT-3 fine-tuned ~13K examples

Figure 2. SFT data collection. Human labelers write ideal responses to real API prompts. The resulting (prompt, response) pairs are used to fine-tune GPT-3 with standard supervised learning.

SFT improves instruction-following a lot. The model picks up the format of a helpful response: answer the question directly, use sensible structure, stay on topic. But it has a basic limitation.

Labelers can only write so many demonstrations. The model learns to imitate them, and on prompts outside that distribution it can extrapolate in harmful, misleading, or unhelpful ways. SFT also has no way to say which of several plausible responses is better, because it provides examples, not rankings.

If the demonstrations are high quality, why isn’t SFT enough? Because the model generalizes from them in ways that can go wrong. A model trained on thousands of good financial explanations can still state an incorrect earnings figure with confidence: SFT teaches it the style of a good answer, not when it doesn’t know. Preference training gives a richer signal (better vs. worse) but doesn’t fully fix this either. The GPT-4 Technical Report found that its post-training actually made the model’s confidence less well calibrated.

2. The Reward Model

The second stage targets what SFT can’t do: teaching the model which responses are better.

Instead of writing responses, labelers now rank them. For each prompt, several completions (between 4 and 9 in InstructGPT) are sampled from the SFT model and ranked from best to worst. This data trains a Reward Model, a separate network that assigns a scalar score to any (prompt, response) pair, predicting which responses people would prefer.

The Reward Model is trained on pairwise comparisons. For a prompt $x$, a preferred response $y_w$ (the winner), and a rejected response $y_l$ (the loser), the loss function pushes the model to assign a higher score to $y_w$ than to $y_l$

\[\mathcal{L}_{RM}(\theta) = -\mathbb{E}_{(x,\, y_w,\, y_l) \sim \mathcal{D}} \left[\log \sigma\!\left(r_\theta(x, y_w) - r_\theta(x, y_l)\right)\right]\]
"Explain why the Fed raised rates in 2022." Response A (y_w) "The Fed raised rates to combat 40-year high CPI of 8.5%, tightening financial Response B (y_l) "The Federal Reserve is the central bank of the United States. It was... 👩‍💼 A > B Reward Model r_θ(x, y) → scalar r = 8.3 r = 2.1 σ(8.3 − 2.1) = σ(6.2) ≈ 1.0 loss ≈ 0 ✓

Figure 3. Reward model training on a financial example. Response A gives a specific, accurate answer about Fed policy; Response B gives a generic definition. The labeler ranks A above B, and the loss pushes the RM to score A higher. When the score gap is large and in the right direction, σ approaches 1 and the loss approaches zero.

The sigmoid $\sigma$ turns the score difference into a probability: $\sigma(r_\theta(x, y_w) - r_\theta(x, y_l))$ is the probability the model gives to the correct ranking. The loss is $-\log$ of that, which is smallest when preferred responses get much higher scores.

Once trained, the Reward Model can score any response without a human in the loop. That is the central design choice: humans label a limited set of comparisons to train a stand-in for their preferences, and that stand-in is used at scale during reinforcement learning. The quality of the whole pipeline depends on how well the Reward Model has captured what people actually value.

Why train a separate Reward Model instead of having people rate responses during training? Scale. PPO generates a huge number of responses and needs to score every one, and human annotation can’t keep up in real time. The Reward Model reuses the human feedback: label tens of thousands of prompts’ worth of comparisons once, then score as many samples as needed cheaply.

3. PPO: Optimizing Against the Reward

With a Reward Model in place, the third stage uses reinforcement learning to push the SFT model toward responses that score highly. The algorithm is Proximal Policy Optimization (PPO), with the language model treated as the policy.

The objective has three components

\[\text{objective}(\phi) = \mathbb{E}_{(x,y) \sim \pi_\phi^{RL}}\!\left[r_\theta(x, y) - \beta \log \frac{\pi_\phi^{RL}(y \mid x)}{\pi^{SFT}(y \mid x)}\right] + \gamma\, \mathbb{E}_{x \sim \mathcal{D}_{pretrain}}\!\left[\log \pi_\phi^{RL}(x)\right]\]

The first term, $r_\theta(x, y)$, rewards a high Reward Model score. This is the main training signal.

The second term, $-\beta \log(\pi_\phi^{RL} / \pi^{SFT})$, is a KL penalty. It measures how far the RL policy has moved away from the SFT model and subtracts a cost proportional to that drift. The third term, weighted by $\gamma$, keeps the model performing well on the original pre-training distribution (InstructGPT calls this variant PPO-ptx).

Prompt x "Summarize 10-K" π^RL_φ RL policy trainable Response y "Strong margins, guidance..." Reward Model r_θ(x, y) = 7.8 π^SFT frozen reference Final reward r − β·KL = 7.8 − 1.2 PPO gradient update → π^RL improves

Figure 4. The PPO training loop. The RL policy generates a response, gets a reward from the frozen RM, pays a KL penalty for moving away from the SFT reference, and updates its weights. The loop repeats for many steps.

The KL penalty does two jobs. First, it limits reward hacking, where an RL policy finds outputs that exploit gaps in the reward model. Without it, the model could learn responses that score well but are useless or incoherent. A financial RM trained on a limited comparison set has blind spots, and an unconstrained policy will find them.

Second, it guards against catastrophic forgetting of general language ability. If the RL objective is pushed too hard, the model can drift so far from its SFT starting point that it stops being coherent outside the fine-tuning distribution. The penalty keeps the policy near the SFT model, where general language quality is preserved.

Does the KL penalty stop the model from ever improving much over SFT? No. Its strength is set by $\beta$. A small $\beta$ gives the model more freedom; a large one keeps it close to SFT. In practice $\beta$ is set small enough to allow real gains in instruction-following while still preventing the worst reward hacking.

4. DPO: Skipping the Reward Model

By 2023, the RLHF pipeline’s costs were clear. It needs two models, a Reward Model and a policy, plus a reinforcement learning loop between them. RL training is unstable, sensitive to hyperparameters, and slow, and the Reward Model brings its own failure modes. The whole thing is expensive to run and hard to debug.

Rafailov et al. (2023) asked whether the Reward Model needs to exist as a separate model at all. Their answer was no: under the same KL-constrained objective as RLHF, the language model itself implicitly defines a reward function.

The derivation starts from the RLHF objective. Maximizing expected reward under a KL constraint has a closed-form optimal solution

\[\pi^*(y \mid x) \propto \pi^{SFT}(y \mid x) \cdot \exp\!\left(\frac{1}{\beta} r(x, y)\right)\]

Rearranging, the reward can be expressed in terms of the policy ratio

\[r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi^{SFT}(y \mid x)} + \beta \log Z(x)\]

The partition function $Z(x)$ cancels in pairwise comparisons. Substituting into the pairwise ranking loss yields the DPO objective directly

\[\mathcal{L}_{DPO}(\pi_\theta;\, \pi_{ref}) = -\mathbb{E}_{(x,\, y_w,\, y_l) \sim \mathcal{D}}\!\left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{ref}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{ref}(y_l \mid x)}\right)\right]\]
RLHF Pref. data Train Reward Model Sample + Score PPO Update Aligned LM step 1 step 2 step 3 step 4 result DPO Pref. data Direct Optimization L_DPO on language model Aligned LM step 1 step 2 (no RM, no RL loop) result Same inputs and output: 2 steps instead of 4, no RL loop

Figure 5. RLHF vs DPO pipelines. Both start from preference data and produce an aligned language model. DPO drops the Reward Model training, the sampling loop, and the PPO step, and treats alignment as a supervised learning problem on the language model directly.

The structure inside $\sigma$ is what makes the DPO loss easy to read. The term $\beta \log(\pi_\theta(y_w \mid x) / \pi_{ref}(y_w \mid x))$ measures how much more likely the model is to produce the preferred response than the reference model is. That is the implicit reward for $y_w$. The second term does the same for $y_l$, and the loss widens the gap between the two in the direction of the human preference.

In other words, the language model plays the role of the reward model. DPO writes the preferences directly into the model’s probabilities, with no separately trained scorer in between.

Do you lose anything by dropping the explicit Reward Model? Less than you might expect. The implicit reward $\beta \log(\pi_\theta(y \mid x) / \pi_{ref}(y \mid x))$ can be computed for any (prompt, response) pair, and DPO-trained models are in fact evaluated this way as reward models (for example on RewardBench). What you don’t get is a separate, independently trained judge, and you need the reference model on hand to compute the score.

DPO doesn’t strictly beat RLHF. Xu et al. (2024) found that a carefully tuned PPO pipeline can outperform DPO on harder tasks such as code generation, and DPO is sensitive to the gap between its preference data and the model’s own outputs. For many practical applications at moderate scale, though, DPO gets competitive results with far less engineering.

The move from SFT → RM → PPO to SFT → DPO fits a familiar pattern in deep learning: a method that works is followed by a simpler one with fewer moving parts that gets similar results. The simpler method often wins in practice because it is easier to tune, debug, and scale. Neither one “solves” alignment, but DPO made preference training far more accessible.

References

  • Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022. arxiv.org/abs/2203.02155
  • Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. arxiv.org/abs/2305.18290
  • Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arxiv.org/abs/1707.06347
  • Stiennon, N., Ouyang, L., Wu, J., et al. (2020). Learning to summarize with human feedback. NeurIPS 2020. arxiv.org/abs/2009.01325
  • Bai, Y., Jones, A., Ndousse, K., et al. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. Anthropic. arxiv.org/abs/2204.05862
  • Ziegler, D. M., Stiennon, N., Wu, J., et al. (2019). Fine-tuning language models from human preferences. arxiv.org/abs/1909.08593
  • OpenAI (2023). GPT-4 Technical Report. arxiv.org/abs/2303.08774
  • Xu, S., Fu, W., Gao, J., et al. (2024). Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024. arxiv.org/abs/2404.10719
  • Lambert, N., Pyatkin, V., Morrison, J., et al. (2024). RewardBench: Evaluating Reward Models for Language Modeling. arxiv.org/abs/2403.13787
AI, LLM
rlhf instructgpt chatgpt reward-model ppo dpo alignment sft human-feedback
This post is licensed under CC BY 4.0 by the author.