DPO
Direct Preference Optimization (DPO) trains a model on pairs of preferred and rejected responses with a simple classification-style loss, matching RLHF's objective without a separate reward model or reinforcement learning.
Rafailov et al. showed that the KL-constrained reward-maximisation objective used in RLHF has a closed-form optimal policy, so the preference data can train the policy directly. A frozen reference model keeps the policy from drifting.
DPO is cheaper and more stable than PPO-based RLHF, and became a standard post-training step for open models.
Formula
L = −log σ(β·[log π(y_w|x)/π_ref(y_w|x) − log π(y_l|x)/π_ref(y_l|x)])
Learn more on Kojen
Related terms
- RLHFReinforcement learning from human feedback (RLHF) fine-tunes a model to maximise a reward model trained on human preference comparisons, usually with PPO and a penalty for drifting from the original model..
- RLVRReinforcement learning with verifiable rewards (RLVR) trains a model with rewards from an automatic checker, such as a correct maths answer or passing unit tests, instead of a learned reward model..
- LoRALoRA (low-rank adaptation) fine-tunes a model by freezing its weights and training small low-rank matrices added to chosen layers, so only a tiny fraction of parameters is updated..
Sources
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free