A 0–20 Scale Instead of 0–100: How to Get an Honest Confidence Score from an LLM

Everything happens in a single prompt:

STANDARD (bad):
"Rate your confidence from 0 to 100" → the model answers 90-95-100 in 78% of cases

REPLACEMENT (works):
"Rate your confidence from 0 to 20,
 where 0 means no confidence and 20 means absolute certainty"
→ the model uses the real range, the signal is readable

One step. You change only the scale range in the prompt.

Task: You're preparing an investor pitch and asked Claude to analyze the US EdTech market. You want to know which facts in the answer you can trust and which ones you should double-check yourself.

Prompt:

Analyze the US EdTech market in 2024–2025: 
market size, key players, growth dynamics.

After every specific fact or figure, put in parentheses 
your confidence score on a scale from 0 to 20, 
where 0 means I'm not sure at all and 20 means absolutely accurate information.

For example: "The market size was ~$X billion (confidence: 12/20)"

Result:
The model produces an analysis where every claim carries a confidence score with real differentiation. Specific figures might get 8-12, well-established facts 16-18, debatable estimates 5-7. You immediately see what to double-check first, instead of getting the same "95" on every fact.

An LLM generates numbers as text, not as points on a scale. The number "95" showed up far more often in training data in the context of confidence ("I'm 95% sure") than the number "87". So the model picks popular tokens — and a confidence of 95 looks like a default answer, not a real estimate.

A 0–100 scale has ~20 "magnetic" multiples of five: 50, 55, 60... 95, 100. They pull in most of the answers. A 0–20 scale has only five such anchors: 0, 5, 10, 15, 20. Fewer anchors means less distortion and more real information in the in-between values.

Control levers: - Shrink the scale to 0–10 → if you need a rough estimate fast (saves context) - Keep the scale at 0–20 → the sweet spot for most tasks - Don't compress the lower bound (like "30–100" or "60–100") → it degrades the signal, the model clusters at the top - Add semantic labels ("0 means not sure at all, 20 means absolute certainty") → without them the model calibrates worse

{Main task or question}

Rate your confidence in {the answer / each claim / each fact} 
on a scale from 0 to 20 as an integer, 
where 0 means no confidence and 20 means absolute certainty.

{Output format: in parentheses after each fact / at the end of the answer / as a separate block}

Placeholders: - {Main task} — your usual question or assignment - {the answer / each claim / each fact} — choose what to rate: the whole answer or each fact separately - {Output format} — how to show the scores: in parentheses after each sentence, as a block at the end, as a table

🚀 Quick start — paste into the chat:

Here's a template for rating confidence on a 0–20 scale. 
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about the task content and the output format for the score — because that determines whether to put scores after each fact or give a single final number.

⚠️ A 0–5 scale is too coarse: Five values aren't enough to tell confidence levels apart. In almost every model it's the worst result. The minimum workable option is 0–10, the sweet spot is 0–20.

⚠️ Compressing the lower bound hurts: If you use a "60–100" scale instead of "0–100", the result is worse than the standard scale. The model crowds at the top and uses only a fraction of the available range. Don't do it.

⚠️ The number "100" is a special magnet: Even replacing "0–100" with "0–97" slightly improves the result. The value 100 itself acts as a super-strong anchor in every model.

⚠️ The preference for multiples of five persists: Even on 0–20, models more often give 0, 5, 10, 15, 20 — but there are far more in-between values than with 0–100.

⚠️ Non-standard scales cause errors: Ask for "rate from 3 to 38" and the model will often step outside the range. Stick to standard formats: 0–20, 0–10.

The researcher took six current models — GPT-5.2, Gemini 3.1 Pro, two LLaMA-4 and two Qwen3 — and systematically varied the confidence scale along three dimensions: granularity (from 0–5 to 0–100), shifting the lower bound (from "0–100" to "60–100") and non-standard ranges like "14–86" or "3–38". For each variant, the models answered thousands of questions from three datasets — factual knowledge, math, and misconception checks.

The key metric is metacognitive efficiency: how well the stated confidence actually predicts whether the answer is correct. It's not just "the model said 80%" — it's "when the model says 80%, it's right about 80% of the time, and that signal differs from the cases where it says 40%".

The result was surprising: 0–100 loses to 0–20 in all six models, no exceptions. And the effect is robust — for GPT and open models alike, on easy and hard questions. The sharpest finding: replacing "0–100" with "0–97" (just removing the number 100) already improves things. That shows the number 100 works as a linguistic anchor, not as the edge of a scale.

"Answer the following question. Then, provide your confidence as 
an integer between [l] and [u], where [l] means no confidence 
and [u] means absolute certainty."

Context: This is the standard prompt template from the paper, used in all experiments. [l] and [u] are the lower and upper bounds of the scale. For 0–20 it becomes: "integer between 0 and 20, where 0 means no confidence and 20 means absolute certainty".

💡 Adaptation: rating the quality of a text or idea

The same logic works not only for confidence but for any numeric rating: text quality, how persuasive an argument is, how realistic a plan is. Ask "rate this text from 0 to 100" and you'll get 75, 80 or 85 most of the time.

Rate the following cold email offer on a scale from 0 to 20 
for each criterion:
- Clarity (0 — confusing, 20 — crystal clear)
- Persuasiveness (0 — doesn't convince, 20 — impossible to refuse)
- Specificity (0 — fluff, 20 — concrete details and numbers)

After the scores, explain each one in a single sentence.

[offer text]

🔧 Technique: combine with Chain-of-Thought → think first, then rate

By default in the original study, the model answers the question and immediately gives a confidence score. But reasoning first, rating second can give a more honest signal — the model "sees" its own doubts while thinking:

{Task or question}

First, break the task down step by step and identify where you have 
doubts or gaps in information.

Then give your answer and rate your confidence on a 0–20 scale, 
where 0 means not sure at all and 20 means absolutely certain.

💡 Extrapolation: a "traffic light" instead of a number

If numbers still feel abstract — ask the model to convert the score into a category:

After your answer, rate your confidence on a 0–20 scale and convert it to this format:
🔴 0–7: must double-check
🟡 8–14: worth checking the key facts  
🟢 15–20: safe to use

Format: "[score/20] [🔴/🟡/🟢] — [one sentence on why]"

Rescaling Confidence: What Scale Design Reveals About LLM Metacognition
Yuyang Dai — INSAIT (Institute for Computer Science, Artificial Intelligence and Technology), 2025
[email protected]

Key references from the paper: Maniscalco & Lau (2012) — the meta-d' metric; Tversky & Kahneman (1974) — anchoring effects; Zhao et al. (2021), Shaki et al. (2023) — frequency biases of numeric tokens in LLMs