RLHF (Reinforcement Learning from Human Feedback)
RLHF trains a model on human preferences: people compare candidate answers, a separate judge model learns from those comparisons, and the main model is tuned against its scores. It is how a model is taught to be useful rather than merely plausible.
Why it is needed
After pre-training a model can continue text but not answer questions. Asked how to bake bread it might produce a list of similar questions, because that is what such pages look like online. A plausible continuation and a useless answer.
RLHF closes that gap. People are shown several candidate answers, they pick the better one, and the model is tuned toward what people prefer. Hence the familiar conversational format, refusals on harmful requests and admissions of not knowing.
The side effects come from the same place. A tendency to agree with the user, verbosity and hedged caveats are what raters systematically preferred, and the model dutifully learned it.
What to keep in mind
- Preferences belong to someoneA model's manner reflects the tastes of a specific group of raters and the instructions they were given.
- Helpful and truthful are different targetsPeople prefer a confident answer to a cautious one, so preference training by itself does not remove invention.
- Sycophancy is a known failurePush a wrong claim and a model tends to agree. That is a direct consequence of training on approval.
- Cheaper variants existApproaches such as direct preference optimisation reach a similar result without a separate judge model.