TL;DR
LLMs significantly inflate their confidence in their own answers: up to 26% higher than for identical answers written by the user. Ask ChatGPT "are you sure about that?" and the model will say "yes" even when the answer is wrong. This isn't an interface glitch. It's systematic behavior, and it shows up in every modern LLM.
The headline finding: who "wrote" the answer matters more than what it says. When the model sees its own answer in the "assistant" role, it automatically trusts it more, simply because it generated it. The researchers call this ownership bias: the model implicitly reasons "I said it, so I must have been sure." It's the mirror image of sycophancy: there the model agrees with the user, here it agrees with itself.
The fix is beautifully simple: to get an honest read on how accurate an answer is, hand the answer back to the model as someone else's. Copy what the LLM wrote, paste it back in, but this time frame it as "text from a user / colleague / expert." Ask for an assessment. The model will give you a far more objective and critical review.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
Two requests instead of one:
REQUEST 1: Normal: get the LLM's answer to your question
→ The model answers (with hidden inflated confidence)
REQUEST 2: Verification: copy the answer,
reframe it as "someone else's text"
→ Ask it to rate the accuracy and give a confidence percentage
Both requests can go in the same chat or a new one (no context is better).
Example in Practice
Task: You asked Claude to break down a clause in a lease for an office in Chicago, and now you want to know whether you can trust that breakdown.
Prompt (Request 1, normal):
Here is a clause from a commercial lease:
"The Landlord may unilaterally adjust the rent
no more than once per year, giving the Tenant 30 days' notice."
Explain: is this a normal market clause, or are there risks for the tenant?
The model gives an explanation. Say it concludes "this is standard, risks are minimal."
Prompt (Request 2, verification):
A lawyer sent me this comment on a clause in my lease:
"""
[Paste the model's answer from Request 1 word for word]
"""
How accurate and complete is this analysis? Rate your confidence as a percentage.
What's missing or debatable here?
Result: In the second request the model will very likely flag nuances it skipped the first time: that "30 days' notice" offers no protection against abuse, that there's no cap on the size of the increase, that "no more than once per year" doesn't stop the landlord from doubling the rent. Confidence drops, the critique gets specific. You get an honest assessment instead of self-approval.
Why It Works
LLMs are trained to give confident, coherent answers: that's what human raters rewarded during RLHF (reinforcement learning from human feedback). The upshot: sounding confident is baked into the model's behavior as the "right" thing to do. When the model sees its own answer in the "assistant" role, an internal logic kicks in: "I generated this, so I must have been sure, otherwise I'd have said something else."
This happens regardless of whether the answer is right. The study showed that even for answers known to be wrong, the model stays highly confident as long as it sees them as its own. It isn't lying. It simply can't look at itself from the outside.
When you present the same text as "written by a colleague / lawyer / user," the authorship context changes. The model no longer "defends" the text as its own. It judges the content on its merits and starts spotting holes it "didn't notice" before. A simple reframe removes the blind spot.
Control levers: - Drop the previous conversation → a fresh chat gives a cleaner assessment, with no "memory" that the model said it itself - Ask for a confidence percentage → a number surfaces doubt more sharply than words do - Add a "source" → "a financial analyst sent this," "GPT-4 suggested this": the more specific the "author," the more objective the review - Ask what's missing → steers the model toward hunting for gaps instead of confirming
Prompt Template
[Role name] sent me this answer to the question "{original_question}":
"""
{LLM_answer_text}
"""
Assess: how accurate and complete is this answer?
- Confidence that it's accurate: ?%
- What's missing, debatable, or needs checking?
- What should I double-check with a specialist?
Placeholders:
- {role name}: "A colleague," "An expert," "A consultant," "Another AI model": any source that isn't you and isn't "the assistant"
- {original_question}: the question you asked in the first request
- {LLM_answer_text}: exactly what the model answered, word for word
🚀 Quick start: paste this into the chat:
Help me apply an answer-verification technique. I want to check an answer
I got from an LLM: present it to me as if an outside expert wrote it.
Ask me what the original question was and which answer I want to check.
[paste the template above]
The LLM will ask for the original question and the answer text, because without them it can't write a verification prompt in the voice of an "outside source." It will then adapt the template to your task.
Limitations
⚠️ Takes two requests: The method doesn't kick in on its own; you have to apply it deliberately. If it never occurs to you to check the answer, the ownership bias stays put.
⚠️ Best for verifiable facts: For questions with an objective answer (law, finance, medicine, technical data) the effect is strongest. For subjective judgments ("is this copy any good?") it's weaker.
⚠️ Exposes errors, doesn't fix them: If the model doesn't know something either way, it still doesn't know it. The method helps you find shaky spots, not repair them.
⚠️ Works better in a fresh chat: In the same conversation the model may keep the context "I wrote this earlier"; a new chat is cleaner.
How They Studied It
A team from the universities of Mainz and Colorado took six modern open LLMs (Llama 3.1, Qwen3 and Gemma 3, two sizes each) and ran a series of tightly separated experiments. First they compared base (pretrained-only) models with instruction-tuned ones and confirmed it: accuracy goes up, but confidence stops tracking whether the model is actually right. That backed up what people had suspected.
A neat design choice: the researchers deliberately separated two effects, the instruction-following training itself and the chat format (chat template). The main culprit turned out to be the training, but the chat format makes things worse on top of it. That led to the hypothesis: maybe it's about who is "speaking" inside the chat?
To test ownership bias, they took the same questions from MMLU (57 subjects, about 14,000 questions) and showed the model an answer in two ways: as its own assistant answer or as the user's answer. The result was strikingly consistent: all six models, all three confidence-measurement methods, all sizes. Everywhere, the model gave its own answers markedly higher confidence. The gap averaged 16-27% depending on the measurement method. The wrong-answer zone is especially telling: when the answer was wrong, the model's confidence in its own version stayed high, while for the "user's" version it rightly dropped close to zero.
Original from the Paper
Prompts used to measure confidence in an answer provided by the assistant (left)
and by the user (right) using the P(True) method:
--- ASSISTANT VERSION ---
[User]: What is the answer to {question}?
[Assistant]: The answer is {answer}.
[User]: Is the answer correct? Please respond with "True" or "False".
--- USER VERSION ---
[User]: What is the answer to {question}?
A user answered: The answer is {answer}.
Is the answer correct? Please respond with "True" or "False".
Context: This is the key experimental prompt. The only difference between the two versions: in the first, the answer is presented as the assistant's words; in the second, as the user's. Identical text, different source, and different model confidence.
Adaptations and Extrapolations
💡 Adaptation: Two-Step Check for Important Decisions
Whenever an LLM gives you advice on something serious (strategy, a legal question, medical information), make this automatic:
An expert sent this advice on the question "{your_question}":
"""
{model_answer}
"""
In this advice, identify:
1. What is definitely correct, with confidence above 90%
2. What is debatable or depends on context
3. What needs to be checked with a real specialist
🔧 Technique: Anonymous Source → Sharper Critique
Instead of "a colleague wrote this," use the most neutral source possible:
"Someone copied this answer from the internet: [text]. How accurate is it?"
An anonymous source removes even the slightest deference to a "colleague's authority." The model critiques more harshly.
🔧 Technique: A Competitor as the Source
For business analysis or strategy:
"A rival consultant prepared this analysis: [text]. What's wrong or missing here?"
The "a competitor wrote this" frame switches on critical thinking even harder: the model hunts for weak spots more aggressively.
Resources
Paper: Large Language Models Are Overconfident in Their Own Responses (2025)
Authors: Mario Sanz-Guerrero, Manuel Mager, Katharina von der Wense
Institutions: Johannes Gutenberg University Mainz (Germany), University of Colorado Boulder (USA)
Contact: [email protected]
Related concepts from the paper: ownership bias, calibration, Expected Calibration Error (ECE), Brier score, instruction tuning, RLHF, sycophancy, verbalized confidence