Reinforcement Learning from Human Feedback

Reinforcement Learning from Human Feedback (RLHF) is a training method for language models that uses human preferences as a reward signal to refine model outputs, rather than relying solely on pre-training or supervised fine-tuning on fixed datasets . The approach gained prominence after GPT-3 as a key driver of LLM progress, with models learning not just from raw text but from how humans rate their responses .

In the standard pipeline, RLHF sits after supervised fine-tuning (SFT): a base model is first fine-tuned on example data, then a reward model is trained on human preference comparisons (which output is better, A or B), and finally the policy model is optimized against this reward model using reinforcement learning algorithms . This was the core method behind OpenAI's InstructGPT, which marked a shift from pure text generation toward more precise alignment with user instructions .

The technique has since evolved and blurred with related approaches. Researchers note that Direct Preference Optimization (DPO) has become functionally similar to RLHF, and the broader paradigm has expanded beyond simple alignment or safety tuning toward enabling models to discover their own reasoning strategies through iterative self-improvement . As one Google DeepMind researcher put it, the insight from OpenAI's o1 was that reinforcement learning's importance has been "reinforced" — it's no longer just an alignment tool but a path for models to explore and optimize their own reasoning .

However, RLHF and its variants face known challenges. "Reward hacking" — where models exploit imperfections in the reward model to inflate scores without genuinely improving solutions — remains a central concern, reflecting the tension between optimization pressure and reliable evaluation . Looking ahead, some investors argue that pure RLHF is being superseded by a broader shift toward reinforcement learning from environment feedback and task-specific evaluations, which they view as more data-efficient for agentic applications .