Skip to content

Glossary · Training

RLHF

Reinforcement learning from human feedback (RLHF) fine-tunes a model to maximise a reward model trained on human preference comparisons, usually with PPO and a penalty for drifting from the original model.

Also called Reinforcement learning from human feedback · Reviewed 15 November 2026

Christiano et al. introduced learning rewards from human comparisons; Ouyang et al. applied it to InstructGPT in three steps: supervised fine-tuning, a reward model trained on ranked outputs, and PPO against that reward. Labelers preferred the 1.3B InstructGPT model's outputs to the 175B GPT-3's.

RLHF is expensive and sensitive to reward hacking, which is why simpler methods such as DPO and verifiable rewards are often used instead.

Learn more on Kojen

Sources

  1. [1]Christiano et al., 2017. Deep reinforcement learning from human preferences (arXiv:1706.03741)
  2. [2]Ouyang et al., 2022. Training language models to follow instructions with human feedback (arXiv:2203.02155)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free