RLHF: The Three-Stage Training Pipeline That Made Language Models Follow Instructions

Language models trained on text corpora learn to predict the next token well, but predicting what comes next in a training corpus is not the same as being helpful to a user. A model pre-trained on internet text learns to produce fluent text that matches statistical patterns in that corpus. Nothing in that objective tells it to be accurate rather than plausible, concise rather than verbose, or safe rather than harmful. OpenAI's InstructGPT research, published in March 2022 as 'Training language models to follow instructions with human feedback' (arXiv:2203.02155, NeurIPS 2022) by Long Ouyang, Jeff Wu, and colleagues, addressed this gap directly. The paper introduced a three-stage training process called reinforcement learning from human feedback (RLHF) and demonstrated a result that challenged common assumptions about scale: a 1.3 billion parameter InstructGPT model was preferred by human evaluators to the 175 billion parameter GPT-3, despite having 100 times fewer parameters.

Where the idea came from

The conceptual root of RLHF as applied to language models is a 2017 paper by Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei: 'Deep reinforcement learning from human preferences' (arXiv:1706.03741, submitted June 2017). That paper was not about language models. It was about controlling RL agents in Atari games and simulated robot locomotion. Its core finding was that human comparisons of pairs of short behavior clips could train a reward model, and that reward model could then guide standard RL training without the agent ever needing access to a programmed reward function. The paper showed this worked using less than one percent of all agent-environment interactions, making human annotation cost practical. Paul Christiano and Jan Leike were both on the team that later applied these ideas to language models. The leap from control tasks to large language models took five years and required solving additional problems the 2017 paper did not face, including the much larger output space of text generation and the difficulty of ranking open-ended responses.

Stage one: supervised fine-tuning on human demonstrations

InstructGPT's training starts from GPT-3, which has already been pre-trained on web-scale text. In the first stage, human contractors recruited through Upwork and Scale AI write example responses to a set of prompts representing tasks users actually submit to the API: brainstorming, summarization, coding questions, creative writing, and open-ended Q and A. These demonstration responses are used to fine-tune GPT-3 through ordinary supervised learning with cross-entropy loss on the labeler-written outputs. The model that comes out of stage one has started to mirror the behavior pattern in the demonstrations and is more instruction-following than raw GPT-3, but its quality ceiling is set entirely by what the labelers wrote. It has not yet been optimized against comparative human judgments across many candidate responses.

Stage two: training a reward model on human comparisons

The second stage trains a separate reward model. The SFT model from stage one generates multiple candidate responses to a set of prompts. Human labelers rank those responses from best to worst. The paper converts these rankings into pairwise training examples (A is better than B, B is better than C, and so on), and a separate model initialized from the SFT weights is trained to predict which response a human would prefer. This reward model takes a prompt and a response as input and outputs a scalar score. It learns to assign higher scores to responses that ranked higher in human evaluations. The reward model is what encodes human judgment as a function that the RL stage can query at training time, once direct human feedback becomes too expensive to collect for every update step.

Stage three: PPO with a KL penalty

The third stage runs reinforcement learning. The SFT model serves as the starting policy. At each step, it generates a response to a prompt from a held-out dataset. The reward model scores that response and the score becomes the RL reward signal. The update algorithm is Proximal Policy Optimization (PPO), introduced by John Schulman and colleagues at OpenAI in 2017. PPO constrains how much the policy is allowed to change in each update, preventing catastrophically large steps that destabilize training. InstructGPT adds a second constraint on top of PPO: a KL divergence penalty that measures how far the current policy has drifted from the frozen SFT reference model and penalizes large divergences. Without this penalty, the policy would quickly learn to produce text that scores well on the reward model while becoming increasingly incoherent, a failure mode called reward hacking. The KL penalty is the primary mechanism the InstructGPT pipeline uses to suppress it.

What the results showed

• Outputs from the 175B InstructGPT model were preferred to outputs from the 175B GPT-3 roughly 85 percent of the time (85 plus or minus 3 percent) by human labelers. The 1.3B InstructGPT was also preferred to the full 175B GPT-3, despite having 100 times fewer parameters.
• InstructGPT generated approximately 25 percent fewer toxic outputs than GPT-3 when both models were prompted to be respectful, as measured by the Perspective API toxicity classifier.
• InstructGPT showed improvements on TruthfulQA, a benchmark measuring whether models produce factually accurate answers rather than plausible-sounding falsehoods.
• Performance regressions on public NLP benchmarks were minimal, indicating that RLHF fine-tuning did not significantly degrade the base capabilities established during pretraining.

The parameter count result is the most frequently cited finding from the paper. A model 100 times smaller, when aligned to human preferences through RLHF, outperformed a much larger model that had only been optimized for next-token prediction. This reframed what it meant to improve a language model, shifting attention from raw scale to training objective and alignment quality.

DPO: replacing the RL stage with supervised learning

Running RLHF requires managing multiple models simultaneously: a frozen SFT reference, an updating policy, and a reward model. The RL optimization itself is sensitive to hyperparameters and can be unstable. A May 2023 paper from Stanford, 'Direct Preference Optimization: Your Language Model is Secretly a Reward Model' (arXiv:2305.18290) by Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn, showed that the full RLHF objective could be reformulated as a supervised learning problem with no explicit reward model and no RL loop. DPO works directly on preference pairs. Given a preferred response and a dispreferred response to the same prompt, DPO increases the relative log-probability the model assigns to the preferred response while implicitly keeping divergence from the reference model in check. The authors proved this is mathematically equivalent to optimizing the RLHF objective under standard assumptions. DPO does not require running PPO, does not require a separate reward model, and trains with the same infrastructure used for supervised fine-tuning. By late 2023 it had become a standard component in open-weight model alignment pipelines, partly because smaller teams without the infrastructure to run stable PPO could reproduce aligned model quality using only a preference dataset and a standard training loop. PromptingIndex covers RLHF, DPO, and the broader alignment training landscape across its technical deep-dives.