What is RLHF (reinforcement learning from human feedback)?

RLHF, reinforcement learning from human feedback, is a training method that makes a language model behave the way people prefer. Humans compare pairs of model answers, a reward model learns to predict their preferences, and the language model is then trained to produce answers that score highly. It is a big part of why chat assistants are helpful rather than just fluent.

3 min read·Checked September 25, 2026·Also called reinforcement learning from human feedback, preference tuning, RLAIF

Read the free chapter →How AI Actually Thinks guide

Pretraining teaches a model to write like the internet; preference training teaches it to write like a good assistant. RLHF is the best-known way of doing the second part.

Why do language models need RLHF?

A model that has only been pretrained is a text-completion engine. Ask it "How do I bake bread?" and it might continue with more questions, because that is what often follows a question on a web page. Instruction fine-tuning teaches it to answer, but it is hard to write example answers for every quality you care about: being accurate, admitting uncertainty, declining harmful requests, not rambling. It is much easier for people to look at two answers and say which is better. RLHF turns those comparisons into a training signal.

How does RLHF work?

RLHF has three main stages, described clearly in OpenAI's 2022 InstructGPT paper, which built on earlier work from OpenAI and DeepMind in 2017.

1
Supervised fine-tuning
Start from a pretrained model and fine-tune it on example prompts with high-quality answers written by people.
2
Train a reward model
Show people several model answers to the same prompt and ask them to rank them. Train a separate model to predict which answer a person would prefer, producing a score.
3
Optimise with reinforcement learning
Have the language model generate answers, score them with the reward model, and update the model to make high-scoring answers more likely. A penalty keeps it from drifting too far from the original model.

The InstructGPT results showed how much this matters: human raters preferred answers from a 1.3 billion parameter RLHF model over the 175 billion parameter GPT-3 base model. The same approach, with more data and refinement, was used to build ChatGPT.

What are the alternatives to RLHF?

Several newer methods reach similar goals with less cost or complexity:

Method	What changes	Why teams use it
RLHF (PPO)	Reward model plus reinforcement learning	The original approach, flexible but complex and unstable to train
DPO (Direct Preference Optimization)	Trains directly on preference pairs, no separate reward model or RL loop	Simpler and cheaper; widely used for open models since 2023
RLAIF	An AI model, guided by written principles, produces the preference labels	Scales feedback beyond what human raters can provide
Constitutional AI	Anthropic's method using a written set of principles for self-critique and AI feedback	Makes the values being trained for explicit and inspectable
RL with verifiable rewards	Rewards come from checking answers, such as passing tests or matching a maths result	Used to train reasoning models

In practice, labs combine several of these, and the details of their pipelines are often not published.

What are the downsides of RLHF?

RLHF optimises for what raters prefer, and what raters prefer is not always what is true or useful. Known side effects include:

Sycophancy: Models learn that agreeing with the user is rewarded, so they may endorse a wrong claim
Verbosity: Longer, more thorough-looking answers often win comparisons, so models pad
Reward hacking: The model finds outputs that score well with the reward model without actually being better
Overconfidence: Confident answers can be preferred over honest uncertainty, feeding hallucinations

Labs counter these with better rater guidelines, targeted training data and evaluations that specifically test for them.

Why does RLHF matter to AI engineers?

You will rarely run RLHF yourself, but it shapes the models you build on. Their default tone, how readily they refuse, their tendency to agree with you, and their fondness for bullet points and long answers all come partly from preference training. Knowing that helps you write prompts that push against those defaults: asking for concise answers, telling the model to disagree when the user is wrong, or explicitly allowing "I don't know".

Frequently asked questions

Is ChatGPT trained with RLHF?

Yes. OpenAI described using RLHF to train ChatGPT, following the approach from its InstructGPT research. Other assistants, including Claude and Gemini, also use preference-based training, with their own variations.

What is the difference between RLHF and fine-tuning?

Supervised fine-tuning trains on examples of correct outputs. RLHF trains on comparisons between outputs, using a reward model and reinforcement learning. RLHF is usually applied after supervised fine-tuning.

What is DPO in machine learning?

DPO, Direct Preference Optimization, is a 2023 method that trains a model directly on pairs of preferred and rejected answers, without training a separate reward model or running reinforcement learning. It is simpler than classic RLHF and widely used for open models.

Why do AI assistants agree with me even when I am wrong?

This tendency, called sycophancy, is partly a side effect of preference training, since raters often favour answers that agree with them. Asking the model to critique your idea or point out errors directly helps counter it.

Last checked for accuracy on September 25, 2026. Written by the solidcoder team.