TL;DR
When you ask an LLM judge to flag an answer as a “hallucination,” the result depends on how the criterion is written. Write “the answer must be supported by the reference,” and the judge rejects everything the reference doesn’t mention. Write “the answer is false only if it contradicts the reference or contains a wrong specific fact,” and it grades truthfulness. Change one phrase, and agreement with human annotators jumps several-fold.
Here’s what the pain looks like. The reference says “Paris”; the model answers “Paris, the capital of France, on the Seine, about 2 million people.” All correct, but a judge with a “matches the reference” criterion marks it a “hallucination”: the reference says nothing about the Seine. The errors run almost strictly one way. The judge over-rejects: for the strongest models, up to 99–100% of errors were false alarms. The reason is simple: the model executes the instruction literally. It was told “not supported by the reference means bad,” so that’s what it does.
The fix is to rewrite the criterion in the judge prompt as “factual correctness”: an error exists only when there’s a contradiction or a false specific fact (a date, name, number). Extra correct details don’t count as errors. A long, detailed prompt added nothing: it was the change of criterion that did the work. If you need to check faithfulness to a source (in RAG, say), the “faithfulness to source” criterion is still the right one, but choose it deliberately.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Decide what you’re checking → “faithfulness to the reference” OR “factual correctness”
STEP 2: Write the criterion into the judge prompt explicitly:
what counts as an error / what does NOT count as an error
STEP 3: Run the judge on 30–50 answers you’ve already labeled yourself
STEP 4: Compare its verdicts with yours → look at the direction of the errors
(lots of “false alarms” = the criterion is too strict)
It all happens in one judge prompt; steps 3–4 are a manual check on a sample.
Example in Action
Task: You’re building a trivia bot in the spirit of Jeopardy!. You have 200 questions with short reference answers. The bot answers at length, and an LLM judge has to flag answers containing made-up content. Without the right criterion, the judge will reject half of the perfectly good answers: the bot adds correct details that aren’t in the reference.
Prompt:
You are checking a trivia bot’s answers for factual correctness.
Question: Who founded Philadelphia?
Reference answer: William Penn
Bot’s answer: Philadelphia was founded by William Penn in 1682. He laid out the city’s grid plan that same year, and in 1701 he granted Philadelphia its city charter.
Criterion. Mark the answer as HALLUCINATION (1) only if at least one of the following holds:
— the answer contradicts the reference;
— the answer contains a specific, verifiable claim (a date, name, number, title) that is factually wrong.
Do NOT count correct details that aren’t in the reference as hallucinations.
Evaluate the answer as a whole: a correct core doesn’t cancel out a wrong detail elsewhere in the answer.
Order: first compare the answer with the reference. If the reference isn’t enough to verify a specific claim, check it against your own knowledge.
Output: first a list of the specific claims in the answer with a verdict for each, then the final result: 0 or 1.
Result: The model breaks the answer into claims (the name, the founding year, the charter year) and notes for each whether it’s backed by the reference or by its own knowledge. At the end it outputs a final label of 0 or 1. The judge won’t reject correct details missing from the reference. It will catch a wrong date even when the name is right.
Why It Works
The weakness. An LLM judge doesn’t know what you meant. It executes what’s written. The phrase “the answer must follow from the reference” turns any correct addition into a violation. That’s why the errors are lopsided: nearly all of them are false alarms.
The strength. Modern models are good at following a clearly stated criterion. Spell out what counts as an error and what doesn’t, and the judge checks the truth, not a match with the cheat sheet. Prompt detail is secondary here: a compressed version of the criterion worked as well as the long one, and a prompt with the long structure but the old criterion didn’t help.
Control knobs: - What counts as an error (contradiction, false fact) → the main lever; it changes everything. - What does NOT count as an error (correct details beyond the reference) → removes over-rejection. - Unit of evaluation (the whole answer, not just the main answer) → catches a wrong detail around a correct core. - Order (reference first, then the model’s knowledge) → the reference stays the anchor, but not the ceiling. - Prompt length → barely matters; feel free to trim.
Prompt Template
The authors’ verbatim prompts (p1, p2) are in an appendix that wasn’t in the text we had. The template below is built from the paper’s description of the criterion.
You are checking answers for factual correctness.
Question: {question}
Reference answer: {reference}
Answer to check: {answer}
Criterion. Mark the answer as HALLUCINATION (1) only if:
— the answer contradicts the reference;
— or the answer contains a specific, verifiable claim
({fact_types}) that is factually wrong.
Do NOT count correct details that aren’t in the reference as hallucinations.
Evaluate the answer as a whole: a correct core doesn’t cancel out a wrong detail.
Order: first compare with the reference. If the reference isn’t enough,
check the claim against your own knowledge.
Output: {output_format}
{question},{reference},{answer}— the data for one example.{fact_types}— what counts as verifiable in your field: dates, names, amounts, titles, SKUs.{output_format}— e.g., “only 0 or 1” for bulk checks, or “claims + verdict” for debugging.
Limitations
⚠️ Not for source-grounded checks: if you’re doing RAG, summarization, or document analysis, the right criterion is precisely “faithfulness to the source.” The paper says outright that strict reference-based checking is appropriate for those tasks.
⚠️ Weak small models: local models with 7–9 billion parameters behaved unevenly. After the criterion change, Llama-3-8B swung to the opposite extreme: it started letting real errors slip through. For small judges, test the criterion change separately.
⚠️ Not a silver bullet: even with the best criterion, the automatic judge’s agreement with humans stayed below the agreement between two humans. You still need to spot-check labels on a sample by hand.
⚠️ No further gains: an extended version of the criterion (p3) gave no reliable win over the basic one. Once the wording is right, there’s no need to make the prompt more complex.
⚠️ A narrow test: the test used short questions with short references. The finding wasn’t checked on long texts or complex reasoning.
How They Tested It
Researchers from the University of Helsinki took 300 questions from three popular sets: factoid (TriviaQA), multi-hop (HotpotQA), and “misconception traps” (TruthfulQA). Three small open models answered each question, giving 900 question–answer pairs. Humans then manually labeled whether each answer contained a factual error. A second annotator independently checked a third of them, and human agreement came out high (κ = 0.81).
Then they compared automatic “judges”: text-similarity metrics (ROUGE-L, BERTScore), an NLI model, and seven LLM judges, including GPT-5.4 and Claude Opus 4.7. The main experiment: the same judge gets either the “faithfulness to reference” prompt (p1) or the “factual correctness” prompt (p2). The result was striking: GPT-5.4 with p1 barely agreed with humans (κ around 0.02–0.3); with p2, 0.62–0.78. GPT-5-mini rose from 0.19 to 0.63 on Mistral’s answers. Under p1, nearly all errors were false alarms, up to 100%.
A control experiment ruled out the obvious objection: “p2 is just longer and more detailed.” The researchers rewrote p1 in p2’s structure (no gain) and compressed p2 into p1’s structure (no loss). So it really is the criterion. The surprise: the most capable judge (GPT-5.4) erred the most under p1, while Claude Opus 4.7 was already close to the “right” behavior under p1. Raw model power alone doesn’t protect you from a bad criterion.
Adaptations and Extrapolations
🔧 Technique: add an explicit “don’t count as an error” list → fewer false alarms
The paper shows the effect of rewording the criterion. If the judge still over-rejects, add an exception example to the prompt:
Don’t count as errors: clarifications, synonyms, a more detailed description,
correct facts that aren’t in the reference.
🔧 Technique: sort errors by direction → find a bad criterion fast
After a run on 30–50 answers, split the judge’s errors into two types: “the judge said bad, I think it’s fine” and the reverse. If nearly all errors are of one type, the criterion is to blame, not random noise. This is the paper’s own idea (analyzing false alarms and misses), carried over to manual quality control.
Extrapolation: auditing your own checker prompts for agents. If you tell an agent “check the result against the spec,” it will reject everything the spec doesn’t mention, even useful stuff. Add to the instruction: “Don’t count additions beyond the spec as errors if they don’t contradict the spec and contain no false facts.” This follows from the paper’s logic, but the authors didn’t test it on agents.
Resources
- The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation — Jorma Valjakka, Juha Mylläri, Juhani Kivimäki, Jukka K. Nurminen, Department of Computer Science, University of Helsinki.
- Data: Harvard Dataverse, https://doi.org/10.7910/DVN/PCHISZ
- Code: https://github.com/jova486/LPHB
- Question sets: TriviaQA, HotpotQA, TruthfulQA.