Why one domain expert beats five annotators for LLM evals

Why one person

Husain and Shankar call this person the "benevolent dictator". Five annotators with five ideas of good give you labels that disagree with each other. An LLM judge trained or tuned on those labels learns an average that matches nobody, and your pass rate means nothing.

One expert means one standard. "Benevolent" means the expert listens: engineers, product managers and support staff can push back and add context, but one person makes the final call.

Who the expert should be

Pick the person whose judgment the business already trusts for this kind of output, and who would be accountable if it were wrong:

• A lawyer for contract review.
• A psychologist for a mental health support bot.
• A senior support lead for customer support answers.
• A senior engineer for code review comments.

The engineer who wrote the prompt is usually the wrong choice, because they know what the system was meant to do and read that intention into its outputs.

What the expert produces

For each trace:

• A label: pass or fail (see why pass or fail beats a 1 to 5 score).
• A short written reason: "Fail: quoted the 30-day window, but this customer is in the EU where it's 14 days plus the statutory period."

The reasons matter as much as the labels. They become the examples in your judge prompt, the rules in your rubric, and the explanation when someone asks why a trace failed.

Write the rubric after reading real traces, never before. Rules written in a meeting room are guesses.

When you need more than one

Large companies working across many domains, languages or cultures may need more than one annotator. Then measure how often they agree, using Cohen's kappa, which corrects for agreement that would happen by chance.

def cohens_kappa(a: list[str], b: list[str]) -> float:
    n = len(a)
    observed = sum(x == y for x, y in zip(a, b)) / n
    labels = set(a) | set(b)
    expected = sum((a.count(l) / n) * (b.count(l) / n) for l in labels)
    return (observed - expected) / (1 - expected)


expert_1 = ["pass", "pass", "fail", "pass", "fail", "pass", "pass", "fail", "pass", "pass"]
expert_2 = ["pass", "fail", "fail", "pass", "pass", "pass", "pass", "fail", "pass", "pass"]
agree = sum(x == y for x, y in zip(expert_1, expert_2)) / len(expert_1)
print(f"raw agreement: {agree:.0%}")
print(f"Cohen's kappa: {cohens_kappa(expert_1, expert_2):.2f}")

Raw agreement of 80% looks good, but because both experts say pass most of the time, much of that agreement would happen by chance, and kappa comes out at 0.52. A common rough reading: above 0.8 is strong, 0.6 to 0.8 is substantial, 0.4 to 0.6 is moderate, and below 0.4 means your labels aren't measuring the same thing. At 0.52, these two experts need to sit down and write down the rules they disagree on.

When experts disagree

If two experts keep disagreeing, your rules are unclear. Find the rule they disagree about, decide it, and write it down with an example, instead of averaging their votes. Each of those decisions makes the rubric sharper.

Respect the expert's time

Expert time is expensive and limited. Give them a fast annotation tool, and only send them traces worth reading (play 5).

A real example

AnkiHub hired fourth-year medical students to evaluate the RAG system behind its medical flashcards. The people labelling were the people who knew whether a pharmacology card was correct.

From the floor

In the legal LLM project lab, labelling was part of the data pipeline from the first phase, with lightweight labelling tasks like accept or reject, whether a clause is present, and its risk level, done in Label Studio, because the quality of those labels sets the ceiling for the model.

Related

• Find failure modes in traces
• Build your own annotation tool
• Labeling and annotation
• Check your LLM judge

Sources

• Husain and Shankar, LLM evals FAQ: How many people should annotate my LLM outputs?
• Hamel Husain, A Field Guide to Rapidly Improving AI Products
• Cohen's kappa