Why BERTScore, ROUGE and generic "helpfulness" scores mislead you
ROUGE counts how many words an output shares with a reference answer, and BERTScore compares their meaning using embeddings. Both reward answers that look like the reference, and neither checks whether the answer is correct: "you can" and "you cannot" score almost the same. Ready-made judge metrics like "helpfulness" or "coherence" have the same problem in a different form, because they measure someone else's idea of quality, not your failure modes.
What these metrics measure
ROUGE counts overlapping words or word sequences between the generated answer and a reference answer. It was designed for summarisation research.
BERTScore embeds each token of both texts with a BERT-style model and matches them by similarity, so paraphrases score well even when the words differ.
Both answer the question "how similar is this output to the reference?". That's a different question from "is this output right?".
Why similarity isn't correctness
Take a reference answer: "You cannot terminate the lease before 12 months without paying the penalty." Now two outputs:
- "You can't end the lease before 12 months unless you pay the penalty." (right)
- "You can terminate the lease before 12 months without paying the penalty." (wrong, and dangerous)
The wrong one shares almost every word with the reference. One word separates a safe product from a lawsuit, and similarity metrics barely notice it.
def rouge1_f(candidate: str, reference: str) -> float:
c, r = candidate.lower().split(), reference.lower().split()
overlap = sum(min(c.count(w), r.count(w)) for w in set(c))
if overlap == 0:
return 0.0
p, rec = overlap / len(c), overlap / len(r)
return 2 * p * rec / (p + rec)
reference = "you cannot terminate the lease before 12 months without paying the penalty"
right = "you can't end the lease before 12 months unless you pay the penalty"
wrong = "you can terminate the lease before 12 months without paying the penalty"
print(f"right answer ROUGE-1: {rouge1_f(right, reference):.2f}")
print(f"wrong answer ROUGE-1: {rouge1_f(wrong, reference):.2f}")
The wrong answer scores higher than the right one.
Husain and Shankar say similarity metrics like BERTScore and ROUGE aren't useful for most AI applications, and call the overuse of generic metrics "endemic".
Most traces have no reference answer
Similarity metrics need a reference answer to compare against. Most production traces don't have one, so these metrics can't even run on the data that matters most.
Generic judge metrics
Many tools ship ready-made LLM judges for "helpfulness", "coherence", "relevance" or "toxicity". Someone who has never seen your product wrote those prompts. They measure a general idea of quality, and they can rate a fluent, confident, wrong answer as very helpful.
A generic metric is dangerous because it gives you a number. Teams start chasing the number and stop reading traces. If a tool ships a "quality score", read the prompt behind it before you trust it.
Where similarity is fair to use
Inside retrieval, embedding similarity is a reasonable way to rank and debug which chunks come back. That's a search metric, not a measure of the final answer. See how to evaluate RAG in two halves.
What to use instead
The replacement is always the same:
- Pick one failure mode from your error analysis.
- Write a pass or fail check for it, in code where possible.
- If it needs judgment, write a narrow LLM judge for it and validate it against your expert.
A real incident
In March 2024, The Markup found New York City's MyCity chatbot, built on Microsoft's Azure AI services, telling business owners they could take a cut of workers' tips, refuse tenants with Section 8 vouchers and go cash-free. All three answers break New York law. They were fluent and on-topic, which is exactly what a helpfulness or similarity score rewards.
From the floor
In the fine-tuning evaluation session, we used BERTScore to compare the base model and the fine-tuned model's answers with reference answers, as a meaning metric next to perplexity. It's useful for seeing whether outputs moved toward the references after training, and it can't tell you whether an answer is legally or factually right, which is why task checks come after it.
Related
- Pass or fail vs a 1 to 5 score
- Evaluating a fine-tuned model
- Evaluate RAG in two halves
- Find failure modes in traces