Skip to content

Glossary · Training

DPO

Direct Preference Optimization (DPO) trains a model on pairs of preferred and rejected responses with a simple classification-style loss, matching RLHF's objective without a separate reward model or reinforcement learning.

Also called Direct Preference Optimization · Reviewed 15 November 2026

Rafailov et al. showed that the KL-constrained reward-maximisation objective used in RLHF has a closed-form optimal policy, so the preference data can train the policy directly. A frozen reference model keeps the policy from drifting.

DPO is cheaper and more stable than PPO-based RLHF, and became a standard post-training step for open models.

Formula

L = −log σ(β·[log π(y_w|x)/π_ref(y_w|x) − log π(y_l|x)/π_ref(y_l|x)])

Learn more on Kojen

Sources

  1. [1]Rafailov et al., 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv:2305.18290)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free