When to fine-tune an LLM, and when not to
Fine-tune when you need the model to behave differently in a consistent way (follow a format, use domain language, classify into your labels, match a tone), when a smaller fine-tuned model can replace an expensive large one, or when you have proprietary data that encodes expertise no general model has. Don't fine-tune to add facts that change often, because retrieval handles that better, cites its sources and updates without retraining. And don't fine-tune before you have measured the best prompt plus retrieval baseline, because that baseline is the bar fine-tuning has to beat.
What fine-tuning changes
Fine-tuning changes the model's weights, so it changes the model's default behaviour. It teaches how to respond much more reliably than what is true.
That's why the most common mistake goes like this: a team sees the model get facts wrong, fine-tunes it on their documents, and finds the model still makes up facts, now in their company's tone. The model learned the style of the documents, not a reliable store of every fact in them.
Retrieval supplies what to respond with. Fine-tuning shapes how the model responds. Many good systems use both.
Good reasons to fine-tune
Consistent format or behaviour at volume
If structured outputs, examples and clear instructions still break at volume, for example a specific report format or a classification into 40 internal labels, fine-tuning on a few hundred to a few thousand good examples can make the behaviour consistent.
Domain language and style
A model can learn the language of a domain: legal drafting, clinical notes, your brand's support tone. If your prompt is growing longer and longer to describe a style, and the output is still inconsistent, fine-tuning moves that description into the weights.
Lower cost and latency
A small fine-tuned model can often match a large general model on one narrow task, at a fraction of the cost per call. This is the main reason many production teams fine-tune, and it's closely related to distillation.
Proprietary expertise
Take a firm with 20 years of its own case files, research notes or customer conversations. That knowledge isn't on the public internet, so no public model was trained on it, and you can't put 20 years of work into a prompt. Fine-tuning an open-weight model on that data can produce an assistant a generic API can't match.
Accuracy where mistakes are expensive
In domains like law and medicine, a confident wrong answer costs a lot. Harvey, the legal AI company, worked with OpenAI on a custom model trained on case law for this reason. Fine-tuning alone doesn't remove hallucination, so in these domains it's combined with retrieval, citations and evaluation.
Bad reasons to fine-tune
- Adding facts that change: prices, policies, product catalogues, news. Use RAG.
- Fixing a problem you haven't measured. Without an eval set, you won't know whether fine-tuning helped.
- Having a small amount of data. A few dozen examples belong in a prompt, not a training run.
- Wanting citations. A fine-tuned model can't tell you where an answer came from, a retrieval system can.
- Following a trend. Fine-tuning brings training, evaluation, hosting and retraining work that a prompt doesn't.
Decision table
| Goal | First try | Fine-tune when |
|---|---|---|
| Know current company facts | RAG | Rarely |
| Follow a strict output format | Structured outputs, examples | Format still breaks at volume |
| Domain language and style | Prompt with examples | Prompt gets too long or inconsistent |
| Classification into your labels | Few-shot prompt | You have hundreds of labelled examples per label |
| Lower cost or latency | Smaller model plus prompt | Small model needs help to match the large one |
| Competitive edge from data | RAG over it | The data encodes judgment, not just facts |
Cost over time
Prompting and APIs have a low upfront cost and a cost per call that grows with usage, because you keep sending tokens. Fine-tuning has a high upfront cost (data preparation, labelling, GPUs, evaluation), and if it lets you run a smaller model, the cost per call can drop.
The costs people forget are the ongoing ones:
- Retraining when the data, the task or the base model changes.
- Hosting your own model, if it's open-weight.
- Evaluating every new version before it ships.
- Keeping the training data versioned and clean.
Before you start, estimate the training run with your cloud provider's pricing calculator, and estimate the monthly cost at your expected volume for both options. See LLM pricing math.
Before you fine-tune
- Write down the metric you expect fine-tuning to improve.
- Build an eval set from real inputs.
- Measure the best prompt plus retrieval baseline on it.
- Set the bar fine-tuning must beat, including cost and latency.
- Check you have enough good examples, or a realistic plan to get them.
If the baseline already passes, you're done, and you saved yourself a training pipeline.
From the floor
The example I use with leaders is a law firm with 20 years of its own case files: fine-tuning an open-weight model on that data can produce a drafting assistant no generic API can match, because nobody else has that data, and in a hybrid setup, simple questions go to a cheap external API and complex domain questions go to the fine-tuned model.
For the legal LLM project lab, the reason we fine-tuned was accuracy: in law, mistakes are expensive, and there have been cases where lawyers submitted reports generated with ChatGPT that contained mistakes and were fined by the judge.
When a builder asked whether to train on synthetic data when real data isn't available, my answer was that a model trained on data you made up can give good results on that data and then fail in the real world, so if you don't have real data, fine-tuning is usually not the option yet.
Related
- RAG vs fine-tuning
- Building a fine-tuning dataset
- Cloud APIs vs open-source LLMs
- Evaluating a fine-tuned model