Prompt Tuning, Fine-Tuning & Reasoning
Lesson 4 of 5Intermediate 10 min

Learning from preferences: RLHF and DPO

Some qualities are easier to compare than to write down. Preference training teaches a model which answer people prefer.

In this lesson you'll

  • Describe the RLHF pipeline
  • Explain what DPO removes from it
  • Recognise reward hacking

It's hard to write the perfect answer to an open-ended question, and easy to say which of two answers is better. Christiano et al. showed that an agent can learn complex behaviour from human comparisons of short clips, needing feedback on less than 1% of its interactions [1].

The RLHF pipeline

  1. Supervised fine-tuning on demonstrations of good answers.
  2. Reward model: train a model to predict which of two outputs labelers prefer.
  3. Reinforcement learning: optimise the policy against the reward model (InstructGPT used PPO) [2].

Feedback doesn't have to come from people. Constitutional AI has a model critique and revise its own outputs against a written list of principles, then trains on AI-generated preferences [3].

DPO: skipping the reward model

Direct Preference Optimization shows the same objective can be optimised directly from preference pairs with a simple classification loss, with no separate reward model and no reinforcement-learning loop. It matched or beat PPO-based RLHF in the paper's experiments and is much simpler to run [4].

Reward hacking

A reward model is only a proxy for what people want. Optimise too hard against it and the true quality goes down even as the proxy score keeps rising, which is Goodhart's law in action [5]. Keep an independent evaluation running alongside training.

In Kojen

RLHF is a preview with simulated metrics. The real feedback loop today is deployment feedback: rate predictions positive or negative, and auto-retraining starts new prompt-tuning runs when negative feedback passes a threshold.

Key takeaways

  • Comparisons are easier to collect than ideal answers.
  • DPO gets RLHF's effect without a reward model or RL loop.
  • Proxy rewards can be gamed; always evaluate independently.

Practise in the dojo

Deploy a model version, send a few requests, and rate the predictions to see how feedback feeds auto-retraining.

Set up a feedback loop

Check your understanding

Score 70 or more to complete the lesson. Progress is saved in this browser.

1. What does DPO remove from the classic RLHF pipeline?
2. What happens when a policy over-optimises a proxy reward model?

Further reading

Sources

Checked against the original sources in November 2026.

  1. [1]Christiano, P. et al. (2017). Deep reinforcement learning from human preferences. NeurIPS 2017 · arXiv:1706.03741.
  2. [2]Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022 · arXiv:2203.02155.
  3. [3]Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073.
  4. [4]Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023 · arXiv:2305.18290.
  5. [5]Gao, L., Schulman, J. & Hilton, J. (2022). Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760.