TL;DR
When you ask an LLM to evaluate something and give a confidence percentage for its answer, the model almost always spits out round, inflated numbers — 80, 90, 100 — which makes that self-assessment useless as a quality signal. The reason is simple: the model answers on first impulse, with no real internal check, so its "confidence" is more a habit of sounding sure than an honest estimate.
The method fixes this with two additions to an ordinary evaluation prompt. The first is a direct warning to the model: "you have tended to be overconfident in past evaluations." The second is a request to hold a mini-debate before the final answer: first argue "for," then "against," and only then give a verdict with a confidence number. It all happens in one prompt, with no extra requests.
The result: confidence scores become more varied (they stop getting stuck on round numbers) and more accurately reflect the model's real uncertainty. Importantly, on the newest top models (GPT-5.x, Claude Sonnet 4.5+, Gemini 3.1 Pro) this works without losing accuracy on the final answer, while on older models (GPT-4o, Sonnet 4, Gemini 2.5 Pro) accuracy drops noticeably.
Method Blueprint
STEP 1 (in one prompt): Give the evaluation task + add an overconfidence warning
STEP 2: Ask it to argue "Yes/fits" — concrete reasons for
STEP 3: Ask it to argue "No/doesn't fit" — concrete reasons against
STEP 4: Based on both sets of arguments — the final answer + a confidence number (0–100)
All four steps — one model call, one prompt.
Example in Action
Task: A marketer asks an LLM to evaluate a draft social media ad against the criterion "does the copy hook the audience from the first words" — a subjective criterion where you want not just "yes, great" but an honest, calibrated confidence score.
Prompt:
You are an expert at evaluating ad copy. Evaluate the text against this criterion:
"Does this text hook the target audience within the first few seconds?"
Text: [paste the ad post text]
Important: in past evaluations you have tended to be overconfident —
often giving 90–100 out of 100 even when the signals were mixed.
Take this tendency into account and be more honest.
Hold a mini-debate in three steps:
1. Arguments "Yes, it hooks": concrete reasons why the text holds attention.
2. Arguments "No, it doesn't hook": concrete reasons why the text might not work.
3. Final verdict: based on both sets of arguments, give an answer (Yes/No)
and state your confidence as a percentage from 0 to 100.
Result: The model returns two detailed blocks — arguments for and against — followed by a final verdict with a confidence number. Expect not round figures like 90 or 100, but something like 68% or 74% — a more honest reflection of the model's real doubts. This beats a plain "rate it from 1 to 10," because you can see the reasoning behind the verdict and trust the number more.
Why It Works
The LLM's weakness is that when asked directly to "evaluate and say how confident you are," the model gives a superficial number — the first thing that "comes to mind," with no second look. It's like asking someone "are you sure?" right after they answer — they'll reflexively say "yes, 100%."
The LLM's strength is that it's excellent at playing the advocate from two opposing positions: arguing convincingly to "defend" and "attack" the same claim.
The method exploits that strength: by making the model genuinely spell out both camps of arguments before the verdict, we force it to weigh real uncertainty instead of reporting a round number. The overconfidence warning works as an anchor — it nudges the model toward more modest but also more accurate estimates.
Control knobs: - Drop the self-debate and keep only the advisory → faster, weaker effect (less spread in scores, but calibration still improves — the advisory on its own is the main calibration lever). - Swap binary True/False for a 1–5 scale → adapts to your own rating system. - Phrase the criterion as concretely as possible (not "is it good" but "does it hook from the first words") → raises the quality of the arguments in the debate.
Prompt Template
You are a judge evaluating {what is being evaluated} against the criterion: "{evaluation criterion}".
{Material}: {text or content to evaluate}
Important warning: in past evaluations you have tended to be
overconfident — often giving values close to 100 even when there were
reasons for doubt. Take this tendency into account and be more honest in your self-assessment.
Hold a mini-debate in three steps:
1. Arguments "{option_1}": give concrete reasons in favor of this answer.
2. Arguments "{option_2}": give concrete reasons against.
3. Final verdict: weigh both sets of arguments and give a final answer
({option_1}/{option_2}) with confidence from 0 to 100.
Fill in: what you're evaluating (a text, an idea, a decision), the evaluation criterion, the material itself, and two answer options (yes/no, fits/doesn't fit).
🚀 Quick start — paste into the chat:
Here is an evaluation template using self-debate and an overconfidence advisory.
Adapt it to my task: [your task].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask which evaluation criterion to use and which two answer options to compare — because without that the debate can't be set up properly.
Limitations
⚠️ Accuracy drop on older models: on GPT-4o, GPT-4.1, Gemini 2.5 Pro and Claude Sonnet 4, adding the advisory and self-debate noticeably lowers the accuracy of the binary answer itself (not the confidence number, but the verdict). On the new flagships (GPT-5.x, Claude Sonnet 4.5+, Gemini 3.1 Pro) accuracy holds or even rises.
⚠️ Only tested for single evaluations: the method was tested for one call with one answer (True/False + confidence). It wasn't tested for comparing several options against each other (e.g., "which of three texts is best") or for multi-agent scenarios.
⚠️ Not for tasks with a single correct answer: the method wasn't tested on code, math or other verifiable tasks — there, objective checking usually works better than any self-assessment.
How They Studied It
The author ran 18 different models (the GPT, Claude and Gemini families plus several open-weight models) across three datasets: SummEval (experts rate the quality of text summaries), AggreFact (factual-accuracy checks across 9 task types) and HelpSteer2 (around a thousand crowdsourced annotators rate the quality of LLM answers). Four prompt variants were compared: the classic logprob-based G-Eval (reads token probabilities via the API), a simple Rubric (just asks for a confidence number), and two new variants — one with free-form reasoning and one with structured self-debate.
The main discovery came from splitting the models by release date: on models from before 2025, adding the advisory and self-debate improved calibration of the confidence number but dragged down the accuracy of the answer itself. On post-2025 flagships, calibration also improved, but accuracy didn't suffer — it held or even rose. That explains why logprobs used to be considered the "gold standard" — studies were testing on older models, where self-debate came at a steep price. On new models that price disappeared, and verbalized confidence beat the logprob-based approach in robustness to task subjectivity.
Resources
Yu-Chung Hsiao (Cisco Systems). "Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models". Builds on G-Eval (Liu et al. 2023), SummEval (Fabbri et al. 2021), AggreFact (Tang et al. 2023), HelpSteer2 (Wang et al. 2024).