Labelling and annotation workflows for AI
Labels define what good means for your system, so the labeller should be a domain expert, guidelines should come from real examples, every label should have a short written reason, and quality should be checked by re-labelling a sample. Use an annotation tool that shows the full context: Label Studio is a common open-source option for structured tasks, and a small custom tool works best for reviewing full LLM traces. Label 50 examples yourself before asking anyone else to label, and you'll write much better guidelines.
Why labels matter
Labels feed three things: eval sets, LLM judges and fine-tuning data. Labels done quickly by people who don't know the domain produce an eval that measures the wrong thing, a judge that agrees with the wrong standard, or a model that's confidently wrong in the same ways.
Who should label
For most teams, one domain expert whose judgment the business trusts: a lawyer for legal outputs, a clinician for medical ones, a senior support lead for support answers. More annotators bring more disagreement, and averaging disagreement produces labels that match nobody. See why one domain expert beats five annotators.
Guidelines
Write guidelines after reading real examples, not before:
- Label 50 examples yourself, writing a reason for each.
- Group the reasons into rules.
- For each rule, add examples of pass, fail and borderline cases.
- Run a pilot round with the labeller, review disagreements, and update the guidelines.
- Then scale.
Version the guidelines, because they will change as you see more cases.
Label formats
- Pass or fail with a reason, for evals and judges. See pass or fail vs a 1 to 5 score.
- Categories or tags, for classification and failure modes.
- Spans, for extraction (highlight the clause, the amount, the date).
- Preferred answer between two, for preference tuning.
- Corrected output, for fine-tuning data.
What to store with every label
{"item_id": "tr_8f3a91", "label": "fail", "reason": "Quoted the US refund window to an EU customer.",
"failure_mode": "wrong_region_policy", "labeller": "support_lead_02", "guideline_version": "v3",
"labelled_at": "2026-09-14T10:22:00Z"}
Tools
- Label Studio: open source, runs locally, supports text, images, audio, spans and custom interfaces. Good for structured labelling.
- A small custom app (Streamlit or FastHTML): best for reviewing LLM traces with the conversation, the retrieved documents and the business context on one screen. See build your own annotation tool.
- Spreadsheets: fine for the first 100 labels.
Checking quality
- Re-label a random sample of 10% later, and measure how often the second label matches the first.
- With more than one labeller, measure agreement with Cohen's kappa.
- Read the disagreements, because they show where the guidelines are unclear.
Measuring it
- Agreement rate on re-labelled samples.
- Guideline changes per round (high early, falling over time).
- Labels per hour, which tells you whether the tool is slowing the expert down.
From the floor
In the legal LLM project lab, labelling was part of the data pipeline from the first phase: we installed Label Studio, which runs locally as an open-source tool, for lightweight labelling tasks like accept or reject, whether a clause is present, and its risk level, and labelled data went into a gold folder, because every company labels its data first, and otherwise the model isn't going to work properly.