How to build a fine-tuning dataset

What a fine-tuning example is

A fine-tuning example is a demonstration of the job: here is an input, here is exactly what a good response looks like. It isn't a page of text.

Teams often scrape thousands of documents, call it a dataset, train on it, and get a model that sounds like the documents but doesn't do the task. If an example shows the wrong behaviour, the model learns the wrong behaviour, and it learns it with confidence.

Data formats

Most fine-tuning data is stored as JSONL, JSON Lines: a text file where every line is one complete JSON object. It's easy to stream, append to and split.

The common shapes:

{"prompt": "Review: The plot dragged but the acting was superb.\nSentiment:", "completion": " positive"}
{"messages": [{"role": "system", "content": "You summarise clauses in commercial lease agreements."}, {"role": "user", "content": "<clause text>"}, {"role": "assistant", "content": "<approved summary>"}]}
{"prompt": "<question>", "chosen": "<preferred answer>", "rejected": "<worse answer>"}

• Prompt and completion, for classic supervised fine-tuning and classification.
• Chat messages, for chat models and most hosted fine-tuning APIs.
• Preference pairs, for DPO.

Format the text the way the model was trained to see it. For chat models, apply the model's chat template (the tokenizer has a method for this), so the special tokens around system, user and assistant messages match what the model expects.

Step 1: define the task

Write down, in one or two sentences, the input and the output the model should produce, and how you'll judge it. If you can't define what good looks like, you can't build the dataset or evaluate the model.

For a regulated domain, write the boundaries too: what the model must never do, which jurisdictions it covers, which risks you accept.

Step 2: collect real inputs

Use the inputs your system will see: production logs, support tickets, real documents, real questions. Public datasets are good for learning and for a first version, and they are no replacement for your own distribution of cases.

Be careful with synthetic inputs. A model trained on data you made up can score well on that data and fail on real users. Use synthetic data to fill specific gaps, not as the whole dataset. See synthetic data.

Step 3: write or approve the outputs

The outputs are the most important part. Options:

• A domain expert writes them. Best quality, slowest.
• A strong model drafts them and the expert approves or edits them. Faster, and the expert's edits teach you what the model gets wrong.
• Existing human outputs from your process, like past agent replies or analyst summaries, filtered for quality.

One domain expert should define what good means, with a short written reason for difficult cases, so labels stay consistent. See why one domain expert beats five annotators.

Step 4: cover the cases on purpose

List the categories of input you expect (types of request, document types, languages, difficulty levels) and check that each has enough examples. Add the hard and rare cases deliberately: ambiguous requests, requests the model should refuse, inputs with missing information. A dataset that is 90% easy cases teaches the model the easy cases.

Step 5: clean

• Remove exact and near duplicates.
• Remove contradictions, where similar inputs have conflicting outputs.
• Remove or mask personal data and secrets.
• Normalise whitespace and encoding (UTF-8).
• Drop empty and broken records.

Step 6: split before you train

Split into train, validation and test sets, commonly around 80%, 10% and 10%, with a fixed random seed so the split is reproducible. Don't look at the test set until the end, and make sure near duplicates don't end up on both sides of the split.

Step 7: version everything

Store the dataset with a version, the source of every record, the labelling guidelines and the date. When the model changes behaviour, you'll want to know exactly what it was trained on. See data versioning.

Code: prepare a dataset

import json
import random


def clean(text: str) -> str:
    return " ".join(text.split())


def to_example(review: str, label: int) -> dict:
    return {
        "prompt": f"Review: {clean(review)}\nSentiment:",
        "completion": " positive" if label == 1 else " negative",
    }


def split(rows: list[dict], seed: int = 42) -> tuple[list, list, list]:
    rows = rows[:]
    random.Random(seed).shuffle(rows)
    n = len(rows)
    a, b = int(n * 0.8), int(n * 0.9)
    return rows[:a], rows[a:b], rows[b:]


def save_jsonl(rows: list[dict], path: str) -> None:
    with open(path, "w", encoding="utf-8") as f:
        for row in rows:
            f.write(json.dumps(row, ensure_ascii=False) + "\n")


raw = [("Loved   every minute of it.", 1), ("Two hours I won't get back.", 0)] * 50
seen, examples = set(), []
for review, label in raw:
    ex = to_example(review, label)
    key = ex["prompt"]
    if key not in seen and clean(review):
        seen.add(key)
        examples.append(ex)

train, val, test = split(examples)
print(len(raw), "raw rows ->", len(examples), "unique examples")
print(examples[0])

The demo data has 100 rows but only 2 unique examples, which is exactly what deduplication should catch before you train.

How much data

There's no fixed number. As rough guidance from provider documentation and practice: dozens of examples can show an effect for a simple format, a few hundred good examples often work for a narrow task, and complex domain behaviour needs thousands. Start small, train, evaluate, look at the failures, and add examples for the failures.

From the floor

For the first fine-tuning session, the data preparation pipeline loaded the IMDb review dataset, cleaned the text by removing extra white space, dropped empty and missing rows, formatted each review as a prompt with "positive" or "negative" as the completion, split it 80% for training and 10% each for validation and test with a fixed seed of 42, and saved each split as JSONL.

In the legal LLM project lab, phase 0 was writing down the problem and the boundaries before any code: docs for the problem brief, the legal boundaries the tool must not cross, a risk register (hallucination, biased answers) and the metrics, like how often citations are correct. Phase 1 was the data strategy: the CUAD dataset of commercial legal contracts, India's Digital Personal Data Protection Act 2023 and the California Consumer Privacy Act as public sources, a raw, processed and gold folder structure, normalising the text, splitting by article, section and clause, and labelling in Label Studio, because the better the data, the better the model.

Related

• Labeling and annotation
• Synthetic data
• Data formats
• One domain expert

Sources

• Fine-tuning: preparing your dataset, OpenAI documentation
• Chat templates, Hugging Face Transformers documentation
• CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review, Hendrycks and others (2021)
• Label Studio documentation