How to get honest confidence scores from an LLM

Symptom: 95% sure about everything

You ask the model how sure it is and get 90, 95 or 100, on easy and hard questions alike. In the scale-design study, over 78% of confidence answers landed on just those three numbers, and models used only 15 to 28 of the 101 values a 0–100 scale offers. A number that barely moves cannot tell you which answer to check.

The same pattern shows up in other forms: a list of options declared complete while most were missed, a risk estimate that changes when you mention money, a forecast that becomes confident as soon as a chart is attached.

Why it happens

The papers point to three separate mechanisms, and each needs a different fix.

• The number is a word, not a measurement. a high round number is a common token next to "sure" in training text, so the model reaches for it by habit. Even the label matters: swapping "confidence" for "probability" shifted the verbalized estimate by 0.5-0.9.
• The chosen answer is rated in isolation. The model scores the option it already picked and never weighs the competitors, which is why it can declare a list complete with 85% confidence after missing 80% of the correct answers.
• Knowing and acting are disconnected. A model's stated confidence can be roughly calibrated while its behavior ignores it: in RiskEval models kept answering 95%+ of questions even when an error cost 100 times the reward. Packaging pushes the same lever: a professional-looking indicator table made confident forecasts rise almost 6-fold, even when every number was invented.

Fixes ranked by measured effect

Ranked by the size of the effect each study reported, on its own models and tasks. Several fixes improve the signal rather than the answer, so check which one you need.

1. Make it bet a limited bankroll. With a bankroll of 1M virtual coins and stakes from 1 to 100k, large bets of 40k or more were right 98.8% of the time and stakes under 1k only 73.9%. Accuracy itself barely moved; the gain is a usable ranking.
2. Ask whether the question is knowable before showing data. Without a dashboard the agent called the event unpredictable 9 times out of 10; locking that judgment in first, then showing the numbers in a separate message, keeps the chart from switching off the brake.
3. Force a search for what was missed. For "list every option" tasks, a follow-up such as "Wait, could there be other solutions?" restarts the search; with long reasoning models calibration error dropped from 70% to 10% for o1 and o4-mini.
4. Spread probability across all options. Asking for a distribution that sums to 1, with "None of the above" for open questions, makes options compete; on the hard MedXpertQA set the Brier score gap was almost twofold, while on easy questions the methods were close.
5. Shrink the scale to 0–20 with labels. "0 means no confidence, 20 means absolute certainty" beat 0–100 in all six models tested; even 0–97 helped a little, because 100 is the strongest anchor.
6. Ask for a verdict word instead of a percentage. A generate-critique-rewrite loop that stops only when the critic writes CONFIRMED reached verdict accuracy of 82-88%.
7. Shake the question with neutral context. Ask once plainly and once after a harmless topical lead-in; across 26 models, answer stability predicted correctness better than self-reported confidence.
8. Self-debate plus a warning. "You have tended to be overconfident", then arguments for and against, then a number, produced less rounded scores on GPT-5.x-era models; on GPT-4o and similar it lowered verdict accuracy.

A combined prompt stack

The fixes combine into a pipeline that separates estimating from deciding:

1. Judge knowability first, with no data in view, and keep that verdict.
2. Estimate with structure: a distribution across options, or a 0–20 score after each claim. For many items, use stakes from a fixed bankroll instead.
3. Keep costs out of the estimate. In the cost-of-error study mentioning the price of a mistake made the risk number drift, so ask for the probability in one request and the decision in another.
4. Decide outside the model. Compute the abstention threshold τ = λ/(1+λ), where λ is how much worse an error is than the benefit of a correct answer; at λ=9 the bar is 90%. If confidence is below it, ask the model for a data-gathering plan instead of advice, as RiskEval recommends.
5. For high stakes, rerun with neutral lead-ins or add a critic that must find a concrete flaw.

What does not work

• "Rate your confidence from 0 to 100." It returns the habitual 95. A 0–5 scale was the worst option tested, and compressed ranges such as 60–100 performed worse than the standard one.
• Waiting for the model to abstain on its own. Without an external rule it answers almost everything regardless of risk.
• One overall gut number for a forecast. Asked twice, a model gave 55% and 75% for the same event; a base rate plus separate multipliers is steadier.
• Trusting a confident single model on causal questions. A two-agent debate logged 577 cases of 90%+ confidence in a wrong answer; challenge added +5% accuracy for the stronger model and +9% for the weaker one.
• Treating abstention as free. In CoSQ, roughly two-thirds of refusals were questions plain reasoning would have answered correctly, so set the threshold by the cost of an error.

Now please answer the following question. Give your best 3 guesses in your thinking process and then also consider "None of the above". You should think about each of them one by one, explaining the evidence behind each option. Then Output your final answer followed by a probability distribution across the 4 possibilities with your final answer at the beginning. The probability values should sum to 1. Your final answer should be in the format: 'The answer is {answer}. Probability Distribution: {answer}: %, {answer}: %, {answer}: %, None of the above: %'.

{Main task or question}

Rate your confidence in {the answer / each claim / each fact}
on a scale from 0 to 20 as an integer,
where 0 means no confidence and 20 means absolute certainty.

{Output format: in parentheses after each fact / at the end of the answer / as a separate block}

You are an expert forecaster with a bankroll of 1,000,000 LLMCoin.

Your task: For each (question, baseline model) pair, predict whether the baseline model will answer the question correctly.

Rules:
- For each pair, output:
  • "predicted_correct": "Yes" or "No"
  • "stake": an integer between 1 and 100,000 LLMCoin
  • "rationale": brief explanation (~20 words max)

- If your prediction is correct, you WIN the stake (added to bankroll).
- If your prediction is incorrect, you LOSE the stake (subtracted from bankroll).

Constraints:
- The sum of stakes in a round must not exceed your current bankroll.
- If your bankroll drops below 1 coin, you go bankrupt and the game ends.

Goal: Maximize your final LLMCoin balance.