TL;DR
When you ask an LLM to pick the best answer out of several, it usually does worse than a simple majority vote, sometimes by as much as 10 points, and does so with confidence and conviction. EL-DGR is a rule that doesn't improve the judge itself; it limits the judge's powers: the judge can override the majority only if it attaches an exact quote from the source as evidence supporting its choice.
The problem is that when you ask a model to rate an answer with a single "good/bad" score or to "pick the best", it mashes different things into one number: the logic of the reasoning and the factual accuracy. That creates a paradox: the chain "The 2022 World Cup was held in Qatar → the capital of Qatar is Dubai → answer: Dubai" sounds logical and coherent, even though the second step is factually wrong (the capital of Qatar is Doha). The judge gives it a high score for "smoothness" and doesn't see the error, because the score doesn't show what exactly made it high.
EL-DGR fixes this by introducing a hierarchy of priorities instead of a single score: by default the consensus wins (the majority of matching answers). The judge can overrule it, but only by producing an exact quote from the source that supports its choice when the consensus has no such quote. If no option is supported, a new, extra-checked attempt is generated (the "repair").
Method Blueprint
STEP 1: Generate N candidate answers to the question → numbered list of candidates
STEP 2: For each candidate, check whether there's an exact (verbatim) quote in the source supporting its answer → label "supported / not supported"
STEP 3: Determine the consensus: which answer appears most often among the candidates
STEP 4: Apply the rule in order (do not add up the criteria!):
→ consensus is supported → take the consensus
→ consensus is not supported, but another candidate is → take that one (with the quote)
→ nobody is supported → generate and check a new answer ("repair"); if it's supported, take it, otherwise honestly say "insufficient evidence"
All the steps can be done within one detailed prompt or through 2-3 sequential requests in a chat.
Example in Practice
Task: A lawyer or business owner uploads an office lease (or a mortgage agreement) into the chat and asks: "What's the penalty for late payment?" Across repeated attempts the model sometimes gets confused and gives different numbers.
Prompt:
Here is the text of the contract:
{contract_text}
Question: what penalty is specified for late rent payment?
Step 1. Give 3 independent attempts to answer the question using only the text of the contract. Call them Candidate_1, Candidate_2, Candidate_3.
Step 2. For each candidate, check whether the contract contains a VERBATIM quote supporting its answer. Give the quote or mark it "not supported".
Step 3. Determine which answer repeats most often (the consensus).
Step 4. Decide which answer to give, strictly in this order:
- if the consensus is supported by a quote, return the consensus
- if the consensus is not supported but another candidate is, return that one with the quote
- if nobody is supported, reread the contract, give a new answer and check it against a quote. If that one isn't supported either, write "the contract has no clear answer"
Don't pick an answer just because it sounds more confident; pick only based on the presence of an exact quote.
Result: The model will show three answer options, each with a quote from the contract or a "not found" mark, determine which answer came up most often, and, following a clear rule, give a final answer that names exactly which line of the contract it relies on. If the contract text is vague, the model will say so honestly instead of making up a confident wrong number.
Why It Works
When an LLM is asked to "rate and pick the best option" on a single scale, it dumps different things into one number: how convincing the text sounds and how factually correct it is. That creates a blind spot: the model gives a high score for eloquence even when the fact is wrong, and the error becomes invisible.
On the other hand, the model is good at a specific, verifiable operation: searching for an exact text match in the source (that's not a judgment call, it's matching). EL-DGR leans on exactly that strength: instead of an abstract "which answer is better" assessment, the model does a concrete check of "is this phrase literally in the text".
Control levers: - Number of candidates (N): more options = higher chance of finding a supported answer, but more expensive in time/tokens. For simple questions 2-3 is enough. - Strictness of the certificate (exact quote vs paraphrase of meaning): a stricter criterion = fewer errors, but more "insufficient evidence" refusals. Loosen it for creative/paraphrasing tasks. - What the default is (consensus or the first answer): you can change the baseline option to fit your task. - The "repair" step can be dropped for simple questions where consensus + the judge's check is enough.
Prompt Template
Source/document: {text}
Question: {question}
Step 1 — Candidates:
Give {N} independent attempts to answer the question using only the source text. Call them Candidate_1, Candidate_2...
Step 2 — Certification:
For each candidate, check whether the source contains a VERBATIM quote supporting the answer. Give the quote or mark it "not supported".
Step 3 — Consensus:
Find the answer that repeats across the majority of candidates.
Step 4 — Selection rule (strictly in order, don't blend the criteria into one score):
- Consensus is supported → return the consensus.
- Consensus is not supported, but another candidate is → return that one, with the quote.
- Nobody is supported → generate and check a new answer. Return it only if it's supported, otherwise write "insufficient evidence in the source".
Never pick an answer just because it sounds convincing; pick only based on the presence of a quote.
Placeholders: {text} is the document/article/report you need an exact answer from. {question} is the question itself. {N} is how many answer options to generate (usually 3).
🚀 Quick start: paste this into the chat:
Here's the EL-DGR answer-selection rule template. Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what your source and question are and how many options to generate, because without a source and an explicit question the "quote = evidence" rule won't work. It will take the tiered logic (consensus → supported choice → repair) and tailor it to your task.
Limitations
⚠️ Only for verifiable facts: the method works where there's a source with specific facts or numbers (a contract, report, article). For creative or subjective tasks (rating writing style, ideas) there's no quote that could serve as evidence, so the method doesn't apply.
⚠️ A strict quote rule loses correct paraphrases: if the model answered correctly but in its own words rather than with an exact quote, the method will reject that answer. That's a deliberate price paid for reliability.
⚠️ It doesn't improve generation itself: the method only chooses among already generated options. If all the options are bad to begin with, the method won't save you.
⚠️ It doesn't work as a training signal: if you try to add up all the evaluation criteria into one weight to "train" the model, the effect is zero. It only works as a selection rule after generation, not as a way to make the model smarter overall.
How They Tested It
The researchers froze a pool of ready-made answers from four different trained models on GSM8K (math) and HotpotQA (questions over documents) and compared four ways of choosing the final answer from that pool: the first one that comes up, majority vote, a judge with full authority, and EL-DGR. The judge was DeepSeek-R1-7B, a model that never saw the correct answer.
The biggest surprise: on a held-out set of 30 questions, the judge with full authority did worse than if nobody had asked it at all, 10 points below majority vote, all while confidently scoring every option. Give the same judge the same data but limit its right to overrule the consensus to cases with evidence, and the result shot up to become the best of all the options.
Separately, the researchers checked whether splitting evaluation into several criteria (logic, facts, confidence, etc.) works as a training signal for the model, and that turned out to be a null result: accuracy didn't change, and an analysis of "which criterion matters most" showed that no single criterion is essential on its own. The conclusion: splitting criteria helps only when used as an admission filter, not when collapsed into one final number for training.
Resources
When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines — Yiyao Zhang, Diksha Goel, Hussain Ahmad, Shixun Huang, Jun Shen. University of Wollongong, CSIRO's Data61, Adelaide University. NeurIPS 2026 Workshop "Can We Trust the Judge?".