DPO (Direct Preference Optimization)

The Problem

RLHF works: InstructGPT showed you can use human preference labels to make models dramatically more useful. But the pipeline is painful.

You need to:

1. Train a reward model (another large model, trained on pairwise preferences)
2. Run PPO against that reward model (unstable, sensitive to hyperparameters, requires 4 models in memory simultaneously: actor, critic, reference, reward)
3. Tune KL penalty, learning rate, clip ratio, value function weight…

PPO instability isn’t just inconvenient — it means teams at large labs spend weeks debugging RL runs that diverge or reward-hack. Smaller teams often can’t run RLHF at all.

DPO asks: what if we didn’t need the RL step?

The Key Insight

The RLHF objective has a known optimal solution. If you’re running KL-constrained reward maximization:

${\max }_{\pi }\mathbb{E}[r(x,y)]-\beta \cdot \text{KL}[\pi \Vert {\pi }_{\text{ref}}]$

The optimal policy is:

${\pi }^{\ast }(y|x)\propto {\pi }_{\text{ref}}(y|x)\cdot \exp \left(\frac{r(x,y)}{\beta }\right)$

This is a closed-form expression — you can solve for it analytically. DPO’s authors looked at this and asked: if we know the optimal policy has this form, can we invert it to express the reward as a function of the policy?

Yes. Rearranging:

$r(x,y)=\beta \cdot \log \frac{{\pi }^{\ast }(y|x)}{{\pi }_{\text{ref}}(y|x)}+\beta \cdot \log Z(x)$

The reward is just the log ratio of the optimal policy to the reference, plus a term $Z(x)$ that doesn’t depend on $y$. Now substitute this into the Bradley-Terry preference model used to define human preferences. The $Z(x)$ terms cancel. What remains is a pure supervised loss.

No reward model. No RL. Just cross-entropy on preference pairs.

Mechanism in Plain English

1. Start with a reference policy π_ref (the SFT model). Freeze it.
2. For each preference pair (prompt x, preferred response y_w, rejected response y_l):
   • Compute the log probability of y_w under your current policy π_θ
   • Compute the log probability of y_w under the reference π_ref
   • Do the same for y_l
3. The training signal: make y_w’s implicit reward (log ratio) higher than y_l’s implicit reward
4. The β parameter controls how far you’re allowed to deviate from π_ref

This is just binary classification. One forward pass per training example.

ASCII Diagram

  RLHF pipeline (what DPO replaces):
  
  Preference data
       ↓
  Train reward model (RM)  ← 6B separate model
       ↓
  PPO loop:
    [Policy] → generate → [RM] → score
    [Reference] → KL penalty
    [Critic] → value function
       ↓ (repeat 10K+ steps, 4 models in memory)
  Aligned policy
  
  ─────────────────────────────────────────────────
  
  DPO pipeline:
  
  Preference data: (x, y_w, y_l)
       ↓
  For each pair, compute:
  
  log π_θ(y_w|x)    log π_θ(y_l|x)
  log π_ref(y_w|x)  log π_ref(y_l|x)
  
  Implicit reward margin:
  Δ = β[log(π_θ/π_ref)(y_w) − log(π_θ/π_ref)(y_l)]
  
  Loss = −log σ(Δ)    ← just binary cross-entropy
       ↓
  Aligned policy  (2 models in memory: policy + frozen reference)

Math with Translation

${\mathcal{L}}_{\text{DPO}}=-{\mathbb{E}}_{(x,y_w,y_l)}\left[\log \sigma \left(\beta \log \frac{{\pi }_{\theta }(y_w|x)}{{\pi }_{\text{ref}}(y_w|x)}-\beta \log \frac{{\pi }_{\theta }(y_l|x)}{{\pi }_{\text{ref}}(y_l|x)}\right)\right]$

• ${\pi }_{\theta }(y_w|x)$ — probability of the preferred completion under the policy being trained
• ${\pi }_{\text{ref}}(y_w|x)$ — probability of the preferred completion under the reference (SFT) model
• $\log ({\pi }_{\theta }/{\pi }_{\text{ref}})$ — the implicit reward: how much more (or less) the policy favors this completion vs the reference
• $\beta$ — KL penalty strength; low $\beta$ = aggressive optimization, high $\beta$ = stay close to reference
• $\sigma$ — sigmoid; makes this look like logistic regression
• The outer term $\log \sigma ({\Delta }_w-{\Delta }_l)$ says: make the implicit reward of the winner higher than the loser’s

Gradient intuition: the gradient up-weights $y_w$ and down-weights $y_l$, with larger updates where the policy currently gets the preference wrong (where ${\Delta }_w-{\Delta }_l$ is small or negative).

Concrete Walkthrough

Prompt x: “Explain what a neural network is.”

y_w (preferred): “A neural network is a computational model inspired by the brain, organized in layers that transform inputs into outputs through learned weights.”

y_l (rejected): “Neural networks are AI systems that are very powerful and used in many applications today.”

Training step:

• $\log {\pi }_{\theta }(y_w|x)=-24.3$ (sum of log probs over all tokens in $y_w$)

• $\log {\pi }_{\text{ref}}(y_w|x)=-25.1$

• log ratio for $y_w$: $-24.3-(-25.1)=+0.8$ (policy slightly prefers $y_w$ more than reference does)

• $\log {\pi }_{\theta }(y_l|x)=-18.2$ (shorter = higher probability, naive)

• $\log {\pi }_{\text{ref}}(y_l|x)=-17.9$

• log ratio for $y_l$: $-18.2-(-17.9)=-0.3$ (policy slightly prefers $y_l$ less than reference does)

With $\beta =0.1$:

• $\Delta =0.1\cdot (0.8-(-0.3))=0.1\cdot 1.1=0.11$
• $\text{Loss}=-\log \sigma (0.11)=-\log (0.527)\approx 0.64$

Gradient will push to widen this margin: increase the implicit reward gap between y_w and y_l.

What’s Clever

The non-obvious move is realizing the reward model and the policy are the same object under a reparameterization. In standard RLHF you train them separately. DPO collapses this into one model that is simultaneously the policy and the implicit reward model.

This is why the paper’s title is “Your Language Model is Secretly a Reward Model.”

The second clever thing: the partition function cancels. The hardest part of the reward-as-log-ratio expression is β log Z(x) — a normalizing constant that’s intractable to compute. But in the Bradley-Terry model, Z(x) appears identically in the numerator and denominator and drops out. This isn’t an approximation; it’s an exact cancellation. PPO’s instability partly comes from trying to estimate quantities analogous to Z(x) with a value function. DPO sidesteps this entirely.

Key Sources

• direct-preference-optimization-your-language-model-is-secretly-a-reward-model — Introduces DPO

Related Concepts

• rlhf — DPO is a simplification of RLHF
• sft — SFT model serves as the reference policy π_ref in DPO
• alignment — DPO is one of the main approaches to aligning LLMs to human preferences
• reward-model — DPO eliminates the need for an explicit reward model by making it implicit in the log-ratio

Open Questions

• Does DPO scale reliably to very large models (>70B)?
• Can iterative/online DPO (collecting preference data from the current policy) improve over static offline DPO?
• How does DPO compare to RLHF on complex reasoning tasks?