Why you should build your own annotation tool for LLM evals
Your eval loop moves at the speed your domain expert can read and label a trace, and the tool they use sets that speed. A generic trace viewer shows JSON and token counts, while your expert needs the conversation, the retrieved documents and the business context on one screen, shown the way your product shows them, with one key for pass and one for fail. With AI coding assistants, a tool like this takes hours to build, and every label it saves becomes training data for your LLM judge.
Why the tool matters
Husain and Shankar call a custom annotation tool the most valuable investment in an eval workflow. Hamel Husain says teams with custom tools iterate about 10 times faster, which is his observation from consulting work, not a measured study.
The reason is simple arithmetic. If labelling one trace takes your expert 3 minutes in a generic viewer and 30 seconds in a good tool, the same hour gives you 20 labels or 120.
What a generic viewer gets wrong
A generic trace viewer is built for engineers debugging systems. It shows nested JSON, span timings and token counts.
Your expert is a lawyer, a doctor or a support lead. To judge one answer, they need:
- The conversation, readable.
- The documents the system retrieved.
- The business context: the customer's plan, the order status, the policy version.
If they have to jump between a CRM, a database and a log viewer for each trace, they'll label fewer traces, and less carefully.
What a good annotation tool has
- Everything on one screen: conversation on one side, documents and context on the other.
- Data shown the way your product shows it: emails look like emails, listings look like listings, SQL is highlighted.
- One-keystroke labelling: one key for pass, one for fail, a text box for the reason, an arrow key to move on.
- Filters by failure mode, tags, and a progress bar for the batch.
- A way to see the judge's label next to the expert's, once you have a judge.
What to save with every label
Save every label with:
- The trace ID.
- The annotator.
- The time.
- The rubric version.
- The reason.
These labels become your judge's train, dev and test sets (play 8), so treat them as data, not as clicks.
How to build one
A small Python web framework is enough for a handful of users. Hamel uses FastHTML in his eval tooling videos, and Streamlit works too. With an AI coding assistant, the first version takes hours, not weeks.
Start with:
- A loader that pulls a batch of traces from your tracing store.
- A page that renders one trace in your product's format.
- Pass and fail buttons with keyboard shortcuts, and a reason box.
- A table of saved labels you can export.
Then add filters and views as your expert asks for them.
When off-the-shelf makes sense
Off-the-shelf platforms make sense mainly when you have many annotators who need enterprise access controls, audit trails and task assignment. General labelling tools like Label Studio are good for structured labelling tasks, like tagging clauses or drawing boxes on documents. For reading full LLM traces in your product's context, a small custom tool usually wins.
A real example
The evals FAQ points to a custom app built to review AI-generated medical flashcards for AnkiHub. The reviewer works through 400 or more results per query, with medical criteria and full keyboard control, seeing each card the way a student would.
From the floor
In the legal LLM project lab, we used Label Studio, an open-source labelling tool that runs locally, for lightweight labelling tasks on legal text: accept or reject, whether a clause is present, and its risk level. That works well for structured tags on documents. For reviewing a system's full answers with the retrieved clauses next to them, a small custom screen built for the reviewer is the next step.