How RLHF aligns LLMs with human preferences
Imagine training a world-class language model on hundreds of billions of words scraped from the internet books, forums, code repositories, scientific papers and then deploying it as a conversational assistant. The model would be extraordinarily knowledgeable, fluent, and technically capable. It would also, in all likelihood, be deeply unreliable as a helpful assistant. It might refuse nothing, agree with everything, generate plausible-sounding falsehoods with total confidence, and occasionally produce outputs that range from unhelpful to genuinely harmful.
This is the core alignment problem: a model that is good at predicting text is not automatically good at being useful, honest, or safe. The two objectives predicting the next token accurately, and behaving helpfully in a human context are related but not identical. The gap between them is real, significant, and the subject of one of the most important practical advances in the field: Reinforcement Learning from Human Feedback, or RLHF.
RLHF is not a single trick or a patch applied after training. It is a multi-stage learning framework that fundamentally reshapes the distribution of a model’s outputs by encoding human judgment into its behavior. Understanding how it works and why it works illuminates much of what distinguishes the modern generation of AI assistants from their predecessors.
Why Pretraining Alone Isn’t Enough
To understand RLHF’s role, it helps to be precise about what pretraining actually teaches a model to do.
During pretraining, a language model learns a single objective: given a sequence of tokens, predict the most probable next token. Over billions of examples, the model learns a compressed statistical model of the training corpus. It acquires grammar, world knowledge, reasoning patterns, and stylistic registers but it acquires all of them in service of one goal: be a good probability estimator over text.
Here’s the problem. The internet’s text does not predominantly consist of helpful, accurate, thoughtful responses to user questions. It contains arguments, misinformation, sarcasm, harmful content, political rhetoric, and an enormous amount of writing that is merely fluent rather than true or useful. A model trained to mimic this distribution faithfully will mimic all of it, including the parts you don’t want.
Moreover, pretraining provides no mechanism for the model to distinguish between what a user is literally asking and what they actually need. A user asking “how do I fix this error?” might benefit from a correction, an explanation, or a counter-question but the pretrained model has no concept of a user, a conversation, or a goal. It has tokens and probability distributions.
Key insight: Pretraining teaches a model to model text. RLHF teaches a model to model human preferences about text a different and more specific objective.
The Three Stages of RLHF
RLHF as typically implemented involves three sequential phases. Each builds on the previous and serves a distinct purpose.
Stage 1: Supervised Fine-Tuning
The process begins with supervised fine-tuning (SFT) on a curated dataset of prompt-response pairs written or selected by human annotators. These are examples of what good responses look like: clear, helpful, honest, appropriately calibrated in confidence. The model is fine-tuned on this data using standard supervised learning the same cross-entropy objective used in pretraining, but applied to a much smaller, higher-quality dataset.
This stage accomplishes something important. The model’s behavior is steered toward the target distribution conversational, assistant-like, responsive. But supervised fine-tuning has a ceiling: it can only teach the model to replicate the specific examples it’s shown. It cannot generalize the underlying principle of what makes a response good in situations the annotators didn’t anticipate.
Stage 2: Reward Model Training
Here is where RLHF becomes genuinely distinctive. Rather than labeling responses as good or bad in absolute terms, human annotators are asked to make comparative judgments: given two (or more) responses to the same prompt, which one is better?
Comparison is cognitively easier and more reliable than absolute scoring. It also implicitly captures preference gradients the fact that one response can be better than another without either being perfect.
These pairwise comparisons are used to train a separate model called the reward model (RM). The reward model takes a prompt and a response as input and outputs a scalar score representing how much a human rater would prefer that response. It is trained to assign higher scores to the preferred response in each comparison pair, using a contrastive loss function.
Consider a schematic example. A prompt asks: “Explain why the sky is blue.” Two responses are generated:
Response A: "The sky is blue because of light scattering."
Response B: "The sky appears blue due to Rayleigh scattering — shorter wavelengths of visible
light scatter more strongly as sunlight passes through the atmosphere. This preferentially
scatters blue light in all directions, making it visible from any viewing angle."
An annotator marks B as preferred. The reward model learns that responses with accurate, elaborated explanations receive higher scores than vague, technically incomplete ones. After training on thousands of such comparisons, the reward model has internalized a human-aligned quality signal.
Key insight: The reward model converts qualitative human judgments into a differentiable scalar signal a necessary bridge between human preference and gradient-based optimization.
Stage 3: Policy Optimization via Reinforcement Learning
With a trained reward model in hand, the language model can now be optimized directly against human preference. The language model is treated as a policy: it takes a prompt as input (the state) and generates a response (the action). The reward model scores that response (the reward). The policy is then updated to increase the probability of generating high-reward responses.
The reinforcement learning algorithm used here is typically Proximal Policy Optimization (PPO), a technique designed to make policy updates that improve reward while avoiding large, destabilizing changes to model behavior.
There is one critical subtlety. Left unconstrained, the model would quickly learn to produce outputs that achieve high reward model scores but diverge substantially from coherent, natural language. It would exploit weaknesses in the reward model rather than genuinely satisfying human preference a failure mode known as reward hacking or reward overoptimization.
To prevent this, a penalty term is added to the reward signal based on the KL divergence between the current policy and the original SFT model. In practical terms, this acts as a leash: the model is allowed to move toward higher-reward outputs, but only within a bounded distance from its starting behavior. The KL penalty prevents the model from collapsing into degenerate, reward-gaming outputs.
Effective Reward = Reward Model Score - beta * KL(current policy || SFT policy)
Where:
beta controls the strength of the constraint
KL measures how far the current policy has drifted from the reference
Key insight: The KL penalty is what keeps RLHF grounded it balances the incentive to maximize human preference scores against the imperative to remain a coherent, general-purpose language model.
What RLHF Actually Changes in Model Behavior
RLHF is often described as making models “safer” or “more helpful,” but these are downstream effects of a more specific change: the model’s output distribution shifts to be more aligned with the implicit quality signal encoded by human annotators.
In practice, RLHF-trained models tend to exhibit several observable behavioral changes compared to their SFT or pretrained counterparts. They are more likely to refuse requests they’ve been trained to decline, more likely to express appropriate uncertainty rather than confabulating confidently, and more likely to produce structured, well-reasoned answers to complex questions.
But RLHF also imports the biases and limitations of its annotators. If annotators systematically prefer longer responses, the model will learn to write longer responses even when brevity is optimal. If annotators have a cultural or linguistic background that isn’t universally represented, the model’s behavior will reflect those preferences. The reward model is not an objective measure of quality; it is a learned approximation of the preferences of the specific humans who participated in the labeling process.
This means RLHF is not a solution to alignment so much as a powerful tool for specifying it. What gets aligned depends entirely on what gets measured and how, and by whom.
RLHF represents one of the most consequential ideas in applied AI research: the insight that a model’s behavior can be shaped not just by the data it was trained on, but by a structured learning process that directly incorporates human judgment. By decomposing alignment into supervised fine-tuning, reward modeling from comparative feedback, and constrained policy optimization, RLHF gives practitioners a practical mechanism for steering large language models toward behavior that is genuinely useful, rather than merely fluent.
Understanding RLHF conceptually matters for more than historical or technical curiosity. The choices embedded in its three stages who provides feedback, what comparisons are made, how the KL penalty is weighted are not neutral engineering decisions. They are value choices that determine what the model will and won’t do, who it will and won’t serve, and what quality means in the context of AI-generated text. For anyone seeking to understand why modern AI assistants behave as they do, RLHF is the essential lens.
Reinforcement Learning from Human Feedback was originally published in AI Evergreen on Medium, where people are continuing the conversation by highlighting and responding to this story.