RLHF
Reinforcement learning from human feedback (RLHF) fine-tunes a model to maximise a reward model trained on human preference comparisons, usually with PPO and a penalty for drifting from the original model.
Christiano et al. introduced learning rewards from human comparisons; Ouyang et al. applied it to InstructGPT in three steps: supervised fine-tuning, a reward model trained on ranked outputs, and PPO against that reward. Labelers preferred the 1.3B InstructGPT model's outputs to the 175B GPT-3's.
RLHF is expensive and sensitive to reward hacking, which is why simpler methods such as DPO and verifiable rewards are often used instead.
Learn more on Kojen
Related terms
- DPODirect Preference Optimization (DPO) trains a model on pairs of preferred and rejected responses with a simple classification-style loss, matching RLHF's objective without a separate reward model or reinforcement learning..
- RLVRReinforcement learning with verifiable rewards (RLVR) trains a model with rewards from an automatic checker, such as a correct maths answer or passing unit tests, instead of a learned reward model..
- LLM-as-a-judgeLLM-as-a-judge is the practice of using a language model to grade another model's outputs against a rubric or a reference answer, instead of a human grader..
Sources
- Christiano et al., 2017. Deep reinforcement learning from human preferences (arXiv:1706.03741)
- Ouyang et al., 2022. Training language models to follow instructions with human feedback (arXiv:2203.02155)
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free