Preference tuning: DPO and related methods
Supervised fine-tuning shows the model good answers to copy. Preference tuning shows it pairs of answers, one preferred and one rejected, and trains it to produce more answers like the preferred one. Direct Preference Optimization (DPO) does this directly on the pairs, without training a separate reward model as classic RLHF does. Use it after supervised fine-tuning, when "better" is easier to judge by comparing two answers than to write as one perfect reference.
Why preference tuning exists
Some qualities are hard to capture in a single reference answer: tone, helpfulness, how much detail to give, when to refuse. If you ask a domain expert to write the perfect answer, they struggle. If you show them two answers and ask which is better, they can usually decide in seconds.
Preference tuning uses those comparisons as training data. It is one of the steps that turned base models into the assistants people use today.
Stages of training an assistant model
- Pre-training on a huge amount of text, which produces a base model.
- Supervised fine-tuning (SFT) on instruction and response examples, which teaches the model to follow instructions.
- Preference tuning on pairs of answers, which teaches it which responses people prefer.
InstructGPT, the research behind ChatGPT, used this sequence, with RLHF for step 3.
RLHF
Reinforcement learning from human feedback (RLHF) works in two parts:
- Train a reward model on human preference pairs, so it learns to score answers the way people would.
- Use reinforcement learning (usually an algorithm called PPO) to update the language model so it produces answers the reward model scores highly, while staying close to the original model.
RLHF works, and it's complex: you train and run several models at once, and the training can be unstable.
DPO
Direct Preference Optimization, published in 2023, showed that you can skip the separate reward model. DPO uses a simple classification-style loss directly on the preference pairs: it increases the probability of the chosen answer and decreases the probability of the rejected one, relative to a reference model (usually the model after SFT).
The result is close to supervised fine-tuning in how you run it: one model being trained, one frozen reference model, a dataset of pairs and a standard training loop. That simplicity is why DPO and its variants became popular for open models.
Related methods you'll see:
- IPO, KTO and ORPO, variations that change the loss or the data needed (KTO, for example, works with single answers marked good or bad instead of pairs).
- RLAIF, where an AI model produces the preference labels instead of people, guided by a set of principles.
Data format
Each example has a prompt, a chosen answer and a rejected answer.
{"prompt": "A customer asks to cancel an order that has already shipped.", "chosen": "The order has already shipped, so I can't cancel it, but you can return it within 30 days for a full refund. Here's how to start the return...", "rejected": "Sure, I've cancelled your order."}
Where preference data comes from
Your product may already collect it:
- Thumbs up and thumbs down on answers.
- Edits users make to AI drafts (the original is rejected, the edit is chosen).
- A reviewer choosing between two generated options.
- Regenerations, where the user asks for another answer and keeps the second one.
These signals are noisy, so filter them, and have a domain expert review a sample. See feedback collection.
When to use it
- After SFT, when the model does the task but its answers aren't the ones your reviewers prefer.
- When you have, or can collect, hundreds to a few thousand real preference pairs.
- When you can measure the improvement with blind comparisons between the old and new model.
Don't start here. Prompting, retrieval and SFT solve most problems first, and preference tuning adds a training step and a new kind of data to maintain.
What goes wrong
- Length bias: if longer answers were usually preferred, the model learns to write longer answers, not better ones.
- Sycophancy: if people preferred answers that agreed with them, the model learns to agree.
- Noisy labels: a thumbs down can mean the answer was wrong, slow, or simply not what the user hoped to hear.
- Forgetting: like any fine-tuning, it can weaken behaviours you still need, so rerun your general and safety evals.
Measure the tuned model with its win rate against the previous version, judged blind by your expert or by an LLM judge you've checked against the expert.
From the floor
I haven't covered preference tuning in a cohort session yet, so this page is written from the research and the library documentation. The stages above match what we covered in the fine-tuning module: pre-training gives you a base model, and fine-tuning on question and answer data is what teaches it to respond to instructions.
Related
- Building a fine-tuning dataset
- Full fine-tuning, LoRA and QLoRA
- Feedback collection
- How to check if your LLM judge is right
Sources
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model, Rafailov and others (2023)
- Training language models to follow instructions with human feedback, Ouyang and others (2022)
- DPO Trainer, Hugging Face TRL documentation
- Constitutional AI: Harmlessness from AI Feedback, Bai and others (2022)