Blind review (label-free protocol): an LLM judge must not see the verdict it is checking

STEP 1 (separate request, WITHOUT the system's label):
  pair data → the LLM decides on its own: MATCH / NON_MATCH + score 0–10 + key differences

STEP 2 (outside the LLM — a spreadsheet, a script or by hand):
  LLM verdict vs system decision → agree / disagree

STEP 3: disagreements → manual review by a human

Step 1 runs in a single request. The system's label must not appear in it at all: not in the text, not in the examples, not in the field names.

Task: You sell on Amazon and Walmart. A script merges listings: "this is the same product on both marketplaces". The script put a MATCH label on a pair whose titles are almost identical. You need to check whether it merged different models.

Prompt (blind):

You are an expert in marketplace product catalogs. You are given two listings.
Decide on your own whether this is the SAME product (the same model, 
capacity/storage, bundle) or DIFFERENT products.

Listing A (Amazon):
Title: Apple iPhone 15 Smartphone 128GB, Black
Brand: Apple
Manufacturer part number: MTLH3
Price: $699

Listing B (Walmart):
Title: Apple iPhone 15 Pro 128 GB, Black Titanium
Brand: Apple
Manufacturer part number: MTUV3
Price: $899

Rules:
- Similar titles on their own are NOT proof. Look for differences in model, 
  capacity, part number, bundle.
- If there is not enough data, say so.

Answer in this format:
1. Key similarities (as a list)
2. Key differences (as a list)
3. Verdict: MATCH or NON_MATCH
4. Confidence that this is the same product, from 0 to 10

Result: The model will give a short breakdown: what matched (brand, color, storage), what differs (the Pro line vs. the regular one, part numbers, the price gap). Then a verdict and a confidence score. You compare this verdict with the script's label. If the script said MATCH and the blind judge said NON_MATCH with a low score, the pair needs a manual look.

The weakness. When the judge sees the system's decision, it turns into a ready-made conclusion that needs justifying. The model generates text consistent with its context, so it hunts for arguments in favor of the label it was shown. The authors call this anchor bias. Instead of an evaluation you get an echo. On top of that, models trust familiar names: the pair "The Fast and the Furious" (the 2001 film and the whole franchise) looks identical, even though these are different entities.

The strength. When the model simply solves the task ("same object or not") without someone else's answer in front of it, it does a decent job of weighing similarities and differences. For the model it is an ordinary task, like any classification.

How the method uses this. We separate "decide" from "compare with the system's decision". The comparison is done by a spreadsheet or a human, and the model takes no part in it. There is nothing for it to anchor on.

Control levers: - The 0–10 score → keep it if you need to rank doubtful pairs. Drop it if you only need the verdict. - The "key differences" block → keep it for analyzing disputed cases and for manual review. Drop it to save tokens at high volume. - The "similar titles are not proof" rule → my addition, it is not in the paper. Helps against the "same name, different entities" trap; test it on your own data. - Disagreement threshold → choose it yourself: for example, send to a human only the pairs where the verdict disagrees with the system or the score is between 3 and 7.

Prompt Template

You are {expert_role}. You are given two objects from different sources.
Decide on your own whether this is the SAME {entity_type} or DIFFERENT ones.

Object A ({source_A}):
{data_A}

Object B ({source_B}):
{data_B}

Rules:
- Similar names on their own are not proof.
- Look for differences in: {what_to_check}.
- If there is not enough data, say so.

Answer in this format:
1. Key similarities
2. Key differences
3. Verdict: MATCH or NON_MATCH
4. Confidence from 0 to 10 that this is the same {entity_type}

What to fill in: {entity_type} — a product, company, customer, term, document. {what_to_check} — the attributes by which objects really differ in your field (EIN and VAT number, SKU, version, date).

The main rule: the prompt contains no system decision, no words like "the system thinks" and no examples with a label filled in. The template is my reconstruction from the protocol description (the paper calls it Prompt III), not a verbatim copy.

🚀 Quick start — paste into the chat:

Here is a blind decision-review template. Adapt it to my task: {describe what 
you are checking: product matching, duplicate customers, classification, etc.}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask which objects are being compared, from which sources, and by which attributes they really differ. This is needed so the prompt contains no system decision yet gives the model the right criteria for an independent judgment.

⚠️ Detailed instructions don't save you: if you keep the label but add a long instruction like "evaluate in 8 steps, relying only on the data", the anchoring to the label stays. The authors stress that step-by-step reasoning (Chain-of-Thought) does not solve this problem. The label itself has to go.

⚠️ Stronger model ≠ more reliable judge: more powerful models react more strongly to the label. "I'll just use a better model" is no defense.

⚠️ Unfamiliar domains: in narrow domains (medical ontologies in the paper) the model leans on how recognizable the names are. When the names are hidden, its scores collapse across the board. Blind mode helps noticeably there too, but it does not reach the ceiling and needs human review.

⚠️ The "same name" trap: even a blind model can mistake a film and a franchise with the same title for the same thing. Explicit difference criteria in the prompt and human control on disputed pairs are a must.

⚠️ This is verification by prediction: you are measuring whether an independent model agrees with the system, not "how right the system is". If the model itself makes mistakes, the disagreements will be noise too. You need a small human-labeled sample to check the blind judge itself.

⚠️ The paper text was truncated: I saw the method description and the conclusions from the abstract and introduction, but not the whole results section or the appendices. The authors' exact prompt wording is not in this text.

The authors took three judge models (GPT-4o-mini, GPT-4o and Claude Opus 4.7) and three sets of entity pairs from different knowledge bases. Two sets are general-purpose (DBpedia with Wikidata and with YAGO); the third is a new biomedical one (MeSH and SNOMED CT, 15 thousand pairs). They tested four alignment systems. In total they made about 153 thousand requests and spent around 577 dollars.

The idea came in three moves. First, perturbations: they hid entity names and watched what the judge relied on. Then three prompt modes: detailed (8 reasoning steps), minimal, and blind, with no label. Quality was measured as the probability that a correct decision gets a higher score than a wrong one. A value below 0.5 is worse than a coin flip. Finally, a counterfactual test: the same data, but with the system's label flipped. If the score follows the label, the label is to blame. Scores shifted by more than 2 points for 53–99% of pairs.

The result: with a visible label, quality fell to 0.12–0.87, meaning in places the judge scored things backwards. Blind mode brought it back to 0.93–1.00 on the general datasets and 0.93–0.95 on the biomedical one. The surprise was that strong models turned out to be more sensitive to the label. As a check, two authors blindly labeled 102 pairs (very high agreement). For "match" pairs, the correlation between models and humans with a visible label was weak (0.23–0.41). The practical takeaway: remove the label, don't complicate the instruction.

This is not from the paper; these are my own transfers of the principle. Test them on your own tasks.

🔧 Technique: remove the label → a blind reviewer. Give any LLM reviewer only the material, not the author's conclusion.

You are an independent reviewer. Below is only the source material: {material}.
Decide for yourself which conclusion is correct, and justify it. You do not see the author's conclusion.

Then compare your verdict with the author's conclusion. A disagreement is a signal to check.

🔧 Technique: separate the roles in an agent chain. If you have an executor agent and a reviewer agent (for example, in Claude Code), write into the reviewer's instructions: "Do not read the author's change description or commit message. Look only at the diff and the tests". Same principle: the reviewer is not shown the "all good" label. The paper did not test agents.

🔧 Technique: selective manual review. Send to a human only the pairs where the blind verdict disagrees with the system or the confidence score is 3–7. This saves labeling, but still do a calibration on a small labeled sample.

• Paper: Reliability of LLM Judges for Evaluating Entity Alignment
• Authors: Vaibhava Lakshmi Ravideshik (University of Michigan, Ann Arbor) and Mayank Kejriwal (Information Sciences Institute, University of Southern California)
• Code and data: https://github.com/vaibhavalakshmiravideshik/llm-as-a-judge-entity-alignment
• Benchmark: MeSH × SNOMED CT (15 thousand pairs); access to part of SNOMED CT requires a UMLS license
• Related work mentioned: GSM-Symbolic (Mirzadeh et al., 2025), O'Leary (2025) on anchoring effects in ChatGPT, Claude, Gemini