TL;DR
This study shows that LLM confidence estimation methods (how sure a model is of its answer) suffer from a problem that standard metrics don't catch. There are five main ways to get a confidence estimate: take the token probabilities (Prob.), calibrate them (Platt Scaling), ask the model to rate its own confidence (Verbalized Confidence), ask "is this true or false?" (P(True)), or train a separate calibrator model (Calib1). All of them are measured by calibration (does confidence match accuracy) and discrimination (is confidence higher for correct answers).
The headline finding: good calibration doesn't mean reliability in practice. A method can show an excellent match between confidence and accuracy yet give different confidence scores for the same answer when the prompt wording changes, or, conversely, give the same score for answers that mean different things. For example, Verbalized Confidence (asking the model "Rate your confidence from 0 to 1") can give 0.95 when phrased "what is your confidence" and 0.05 when phrased "how sure are you" for the very same answer. P(True) often returns maximum confidence of 1.0 for any answer, even when the model is wrong. Prob. and PS are sensitive to changes but unstable when the prompt is paraphrased.
The researchers introduced three new metrics: Robustness (P-RB), the stability of confidence across different prompt phrasings; Stability (A-STB), the same confidence for semantically equivalent answers ("ovary," "the ovaries," "ovarium"); and Sensitivity (A-SST), different confidence for answers with different meanings. Testing on 9 models (GPT-4o, Mistral, Llama, Qwen, OLMo) showed that no method strikes a perfect balance. Calib1 (a separate trained model) is the most stable but requires training. VC and P(True) are the most unstable to phrasing. Prob. is the sweet spot for most tasks.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Why This Matters
When you use an LLM in high-stakes situations (fact-checking an article, legal consultation, business analysis), you want to know how confident the model is in its answer. The classic approach: "if the model is confident, it's probably right." But that only works if the confidence estimation method is stable and sensitive.
Stability means: if you rephrase the prompt, or the model gives the semantically same answer in different words, the confidence estimate shouldn't jump around. If you ask "Name the capital of France" and "Which city is the capital of France" and confidence leaps from 0.9 to 0.3, the method is broken.
Sensitivity means: if the model gives a different answer (not a paraphrase but a difference in meaning), confidence should change. If the method returns 0.8 for both "Paris" and "Lyon," it's useless.
Calibration and discrimination (the standard metrics) don't catch these problems. A method can be perfectly calibrated (confidence matches accuracy on average across the dataset) yet produce chaos on specific examples. Or the reverse: it always outputs the same value (say, 0.5) and is technically calibrated but completely useless.
Five Ways to Get a Model's Confidence
1. Prob.: token probabilities
The model generates text token by token, and each token has a probability. Prob. takes those probabilities, multiplies them, and normalizes by answer length. The higher the final number, the more confident the model is in its choice of words.
Pros: Sensitive to changes in the answer, works out of the box, no extra requests needed.
Cons: Not very stable to prompt paraphrases (confidence can change even when the answer is the same). Only works if you have access to logits (not available in the ChatGPT/Claude UI; you need the API).
2. Platt Scaling (PS): calibrated probabilities
Takes Prob. and passes it through the formula _c_ = 1/(1 + _e^(a·log(prob) + b)_), where _a_ and _b_ are fitted so that confidence better matches actual accuracy.
Pros: Better calibrated than raw probabilities.
Cons: The same problems as Prob., plus you have to fit the coefficients on data in advance.
3. Verbalized Confidence (VC): ask the model to rate itself
The simplest way: after the answer, you write "Rate your confidence in this answer from 0 to 1" and the model outputs a number.
Pros: Works in a regular chat, no API needed.
Cons: Extremely unstable to phrasing. Ask "what is the probability" instead of "what is your confidence" and the number can shift by 0.5-0.9. Use a 0-100% scale instead of 0-1 and it jumps too. Sensitive to changes, but chaotically.
4. P(True): ask "is this true?"
After the model's answer you write: "Is it true that [model's answer]? Answer True or False." You take the probability of the "True" token as the confidence.
Pros: Stable to phrasing.
Cons: Too conservative or too confident: it often outputs either 1.0 or something close to 0. Poor at distinguishing shades of confidence. Requires an extra request.
5. Calib1: a separate calibrator model
You train a small model (a classifier) that looks at the question and answer and predicts whether the answer is correct. The probability of "correct" = confidence.
Pros: The most stable to prompt phrasings and answer paraphrases. Good calibration and discrimination.
Cons: Requires training on a dataset with correct answers. Not available in a regular chat.
Example in Practice
Task: You're testing a business hypothesis for a client: launching farm-to-door grocery delivery in Chicago. The model produced a demand forecast: "Potential audience of 150-200 thousand people, average order $45." You need to understand how confident the model is in that figure.
Option 1: Verbalized Confidence:
You gave a forecast: "Potential audience of 150-200 thousand people, average order $45".
Rate your confidence in this forecast on a scale from 0 to 1, where:
- 0 = completely unsure, a pure guess
- 0.5 = moderate confidence, there are factors for and against
- 1 = absolutely sure, based on reliable data
Write only the number.
Result: The model will output a number (say, 0.6). But if you rephrase "rate your confidence" as "what is the probability this is correct," the number may jump to 0.9 or drop to 0.2: the method is unstable.
Option 2: P(True):
You gave a forecast: "The potential audience for farm-to-door grocery delivery in Chicago is 150-200 thousand people, average order $45".
Is this statement true? Answer True or False.
Result: The model will most likely say "True" with high probability (close to 1.0), even if the forecast is off: the method is overconfident.
Option 3: Combined check:
You gave a forecast: "Potential audience of 150-200 thousand people, average order $45".
1. Rate your confidence from 0 to 1.
2. List the factors that support this forecast.
3. List the factors that contradict it.
4. If you had to bet money on this forecast, how much out of $100 would you bet?
Result: The model will give several confidence indicators: a number, arguments, a "bet." If they diverge (for example, confidence 0.8 but "I'd bet $30"), that's a signal of ambiguity.
How to Choose a Method
Control knobs:
- How you phrase the confidence request: "confidence," "probability," and "certainty" give different results for VC. Test several variants.
- Scale: 0-1, 0-100%, and 0-10 all affect the answer. For stability, use 0-1.
- Context of the check: asking the model to rate its confidence right after the answer (in the same message) vs. in a new request can produce different results.
When to use what:
| Situation | Recommendation | Why |
|---|---|---|
| You need a stable estimate across phrasings | Avoid VC; use P(True) or Prob. (if you have API access) | VC swings between "confidence" and "probability"; P(True) is stable |
| You need sensitivity to changes in the answer | Use Prob. or VC | They react to changes in the answer; P(True) is too conservative |
| You're working in a regular chat without the API | VC or P(True) | The others require access to logits |
| A critical decision | Combine methods: VC + P(True) + reasoning | If the methods give different estimates, dig deeper |
| Bulk checks through the API | Prob. or Calib1 (if you've trained one) | Fast, no extra requests |
Practical tip: For important tasks, don't rely on a single method. Ask for two or three ways of estimating confidence and compare. If the model says "confidence 0.9" but answers the "True or False?" question with a probability of 0.5, that's a red flag.
Prompt Templates
Basic confidence check (VC)
{your_question}
After answering, rate your confidence on a scale from 0 to 1, where:
- 0 = completely unsure, a random guess
- 0.5 = moderate confidence
- 1 = absolutely sure, based on verified data
Response format:
Answer: {answer}
Confidence: {number from 0 to 1}
What to fill in: {your_question}: any question for the model.
The P(True) check
[First, get the model's answer to your question]
Is the following statement true: "{model_answer}"?
Answer only True or False, with no explanation.
What to fill in: {model_answer}: the answer the model gave in the first step.
Note: Through the API you can see the probability of the "True" token, and that's your confidence. In a regular chat the model will just say True/False, but you can treat that as 1/0.
Combined check (the most reliable)
{your_question}
After answering, run a self-check:
1. Confidence: Rate from 0 to 1 how sure you are of this answer.
2. Arguments FOR: List 2-3 facts that support your answer.
3. Arguments AGAINST: List 2-3 factors that could refute your answer or make it inaccurate.
4. Final estimate: If you had to bet money on this answer being correct, how much out of $100 would you bet?
Format:
Answer: {answer}
Confidence: {0-1}
Arguments FOR: {list}
Arguments AGAINST: {list}
Bet: ${amount}
What to fill in: {your_question}: any question.
Result: The model will give several independent indicators. If it says "Confidence: 0.9" but "Bet: $20," the model itself senses the answer is weak. If all indicators are high (0.8+, a bet of $80+, strong arguments FOR and weak AGAINST), you can trust it.
Quick start for tricky cases
If you don't want to dig into the methods, just copy this:
I want a reliable confidence estimate for your answer.
Question: {your_question}
After answering:
1. Give a numeric confidence estimate (0-1)
2. Explain why you gave that estimate
3. Answer True/False to the question "Is your answer true?"
4. Say how much out of $100 you'd bet on it being correct
If the estimates contradict each other, explain why.
The model will run a multi-method check on its own and show you where the discrepancies are.
Limitations
⚠️ Access to probabilities: Prob. and PS only work through an API with logit access. They're not available in the ChatGPT/Claude UI.
⚠️ VC instability: Verbalized Confidence can swing 0.5-0.9 for the same answer under different request phrasings ("confidence" vs. "probability" vs. "certainty"). Use a fixed phrasing or combine it with other methods.
⚠️ P(True) is conservative: It often outputs extreme values (close to 0 or 1) and is poor at distinguishing nuances of confidence between 0.3 and 0.7.
⚠️ The calibrator requires training: Calib1 is the most stable method, but it requires training on a dataset of correct/incorrect answers. Not available without coding.
⚠️ GPT-4o judge limitations: The study used GPT-4o to determine answer correctness and semantic equivalence. In your tasks, automatic checking may be inaccurate (especially for subjective questions or specialized domains).
⚠️ Confidence ≠ correctness: A model can be very confident in a wrong answer. All these methods measure the model's internal estimate, not actual correctness.
Resources
Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations Yuxi Xia, Dennis Ulmer, Terra Blevins, Yihong Liu, Hinrich Schütze, Benjamin Roth University of Vienna, University of Amsterdam, Northeastern University, LMU Munich
Related concepts: calibration (ECE, Brier score), discrimination (AUROC), Platt Scaling, P(True), linguistic confidence, consistency-based methods