Model adaptation: prompting, RAG, fine-tuning, distillation and quantization
You can adapt a model to your task in four broad ways: prompting changes the instructions, retrieval changes what the model reads, fine-tuning changes the model's weights, and distillation and quantization change the model's size and cost. Try them roughly in that order, because each one costs more to build and maintain than the one before. Most production systems combine them, for example a fine-tuned small model that answers from retrieved documents with a well-designed prompt.
Three ways to change what a model does
Take one request: "Summarise this article in the style of a pirate."
With prompt engineering, you guide the model with instructions and examples in the prompt. Nothing about the model changes, and it answers from what it already learned in training.
With retrieval (RAG), you connect the model to external data sources. If you ask what Tesla's quarter 3 earnings were, the model most likely wasn't trained on data from this year, so your system retrieves the earnings report and puts the relevant parts in the prompt. The model still doesn't change, it reads new information at the moment of the request.
With fine-tuning, you train the model again on your own examples, so its weights change and its default behaviour changes with them. You use it when you want the model to work in a particular style, learn a particular task or understand a particular domain consistently, without repeating long instructions in every prompt.
| What changes | Cost to start | Cost over time | Updates | |
|---|---|---|---|---|
| Prompting | The instructions | Very low | Tokens on every call | Edit the prompt |
| RAG | What the model reads | Medium | Retrieval plus tokens | Update the documents |
| Fine-tuning | The model's weights | High (data, GPUs, evaluation) | Lower per call with a smaller model, plus retraining | Retrain |
| Distillation | A small model learns a large model's task | High | Much lower per call | Retrain |
| Quantization | The precision of the weights | Low to medium | Lower memory and hardware cost | Re-quantize per model version |
Pre-training and fine-tuning
A model like Llama is first pre-trained on a huge dataset from the internet. FineWeb, one of the public pre-training datasets on Hugging Face, has around 15 trillion tokens of English web text. From pre-training, the model learns the relationships between words, how phrases are written, which words are close to each other, like king and queen. That model is called a base model or foundation model.
A base model knows the language, but it doesn't know what to do with an instruction. So it is fine-tuned on a much smaller, very specific dataset, for example thousands of questions and answers, to learn how an instruction looks and how a response should look. That is how a base model becomes a chat model like the ones behind ChatGPT.
This is transfer learning: the model transfers what it learned in pre-training to a new task, so fine-tuning needs thousands or hundreds of thousands of examples, not trillions of words. When you fine-tune, you are doing a smaller version of that same second step for your own task.
Choosing between them
Start with the cheapest option that can work, measure it, and move to the next only when the measurement says you need to.
- Write the best prompt you can, with examples, and build an eval set.
- If the model lacks knowledge, add retrieval.
- If the model has the knowledge but behaves inconsistently (format, style, domain language, classification into your labels), consider fine-tuning.
- If a large model does the task well but costs too much at your volume, consider distillation into a smaller model.
- If the model is too big for your hardware, consider quantization.
The RAG vs fine-tuning comparison goes deeper into the most common decision.
Combining them
These options are not exclusive. BloombergGPT, Bloomberg's model for finance, was trained on a mix of general and financial data, and a product built on a model like that still uses prompts and retrieval for current information. A common production design is a hybrid fleet: a router sends simple questions to a cheap API or small open model, and domain questions to a fine-tuned model, with retrieval feeding both. See the hybrid fleet pattern.
What this section covers
| Page | The question it answers |
|---|---|
| When to fine-tune | Do I need fine-tuning, or will prompting and RAG do? |
| Full fine-tuning, LoRA and QLoRA | How do I fine-tune without a cluster of GPUs? |
| Preference tuning and DPO | How do I teach a model which answer is better? |
| Distillation | How do I get a small model to do a big model's job? |
| Quantization | How do I make a model fit on smaller hardware? |
| Fine-tuning dataset | How do I build the data? |
| Evaluating fine-tuned models | Did fine-tuning help? |
| Serving adapters | How do I serve one or many fine-tuned models? |
Synthetic data for fine-tuning is covered in Data for AI.
From the floor
In AI Engineer HQ, module 4 covered fine-tuning, and we worked through it in steps: the difference between prompt engineering, RAG and fine-tuning, preparing data, fine-tuning Llama with LoRA on the IMDb review dataset for sentiment, QLoRA, evaluation with perplexity, and then a project lab fine-tuning a small Llama model on legal data.
When builders ask whether I would fine-tune a model on my own teaching, my answer is no, not with the data I have. Even if I extract all the text from the cohort conversations, it's a small dataset, so I would use prompt engineering with that text instead, and keep fine-tuning for organisations with years of specialised data that no prompt can hold.