RAG vs fine-tuning: which one does your problem need?
Use RAG when the model needs knowledge it doesn't have, especially knowledge that changes or needs to be cited. Use fine-tuning when the model needs to behave differently: a format, a style, domain language, a classification, or a smaller model doing a large model's job. Many strong systems use both, RAG for what to say and fine-tuning for how to say it.
Comparison
| RAG | Fine-tuning | |
|---|---|---|
| Changes | What the model reads | The model's weights |
| Good for | Facts, documents, current data | Behaviour, format, style, narrow tasks |
| Freshness | Update the index any time | Retrain to update |
| Citations | Natural | Not built in |
| Upfront effort | Ingestion and retrieval | Data collection, labelling, training |
| Running cost | Longer prompts per call | Cheaper calls if a smaller model replaces a larger one |
| Main risk | Retrieval misses | Learning wrong behaviour, forgetting |
| Data privacy | Data stays in your index | Data baked into weights |
Three ways to change what a model does
Prompt engineering guides the model with instructions and examples. RAG connects it to external data at query time. Fine-tuning changes how it behaves by training it further on your examples. Try them in that order, because each is cheaper to change than the next.
Common heuristic, and its limit
"Static data means fine-tune, changing data means RAG" is a common rule of thumb. For facts, retrieval usually still wins even when they rarely change, because it gives citations and can be corrected without retraining, while fine-tuning earns its place for behaviour.
When to choose which
Start with RAG and good prompting, measure on your eval set, and fine-tune only for the gap that remains in behaviour or cost.
From the floor
In the fine-tuning fundamentals session, I put the three side by side: prompt engineering is guiding the model to do something, RAG is connecting it to external data sources, like the Tesla earnings question a model trained last year can't answer, and fine-tuning is changing the model's behaviour or the way it was trained, like writing every article in a particular style.