AI system design interview: fine-tuning or RAG for a domain assistant?
Split the need into knowledge and behaviour. Knowledge (cases, current law, templates) goes to retrieval with citations, because it changes and needs sources. Behaviour (drafting style, structure, judgment on clause choices) can benefit from fine-tuning on examples approved by senior people, so start with RAG plus strong prompting and examples, measure on an eval set they wrote, and fine-tune only if a gap in style or judgment remains.
Question
A law firm wants an assistant that answers like their senior associates. Fine-tune or RAG?
Clarify
- What "answers like" means: the right facts, the right format, or the right judgment?
- Volume and budget.
- Whether data can leave the firm.
- Whether approved examples of good answers exist.
Knowledge vs behaviour
| Need | Best fit | Why |
|---|---|---|
| Current facts, case files, templates | RAG | Changes often, needs citations |
| Format and structure | Prompting with examples, then fine-tuning | Learnable from examples |
| Firm-specific judgment | Fine-tuning on approved examples | Hard to describe in a prompt |
| Lower cost at high volume for a narrow task | Fine-tuning a small model | Replaces a large model on that task |
| Data can't leave the firm | Open-weight model hosted internally | Works with both RAG and fine-tuning |
Decide with a baseline
1 - Associates write 100 to 200 questions with model answers. 2 - Build RAG with good prompting and few-shot examples, and score it. 3 - Look at the failures: missing facts point to retrieval, wrong style or judgment points to fine-tuning. 4 - Fine-tune only for the remaining gap, and compare on the same set.
Common mistakes
- Fine-tuning to add facts, which then go stale and can't be cited.
- Fine-tuning before a baseline exists, so nobody knows if it helped.
- Treating it as either-or: many systems use a fine-tuned model with retrieval.
What interviewers listen for
The knowledge and behaviour split, a baseline before fine-tuning, and an eval set written by the people whose judgment you're copying.
From the floor
I use this law firm example with leaders: 20 years of a firm's own case files is data no general model has, and that's where fine-tuning an open model hosted on-premise can give an edge, while routine questions can still go to a cheap API, with a model router deciding which request goes where.