Calibrated Confidence, No Strategy: Why LLMs Don't Abstain Even When They Should

RESEARCH METHOD:
Prompt → Model answers OR abstains + reports confidence (0-100%)
↓
Vary λ (penalty): 0.1, 1, 5, 10, 25, 50, 100
↓
EXPECTATION: at λ=100 the model abstains when c < 99%
REALITY: the model almost always answers, regardless of λ
↓
SCAFFOLDING (post-hoc): take the verbal confidence c and force-apply τ = λ/(1+λ)
RESULT: utility rises several-fold

EXTRACTABLE PRINCIPLE for chat:

STEP 1: Ask for an answer + confidence (0-100%)
STEP 2: Estimate the "cost of an error" in your context
STEP 3: Apply the rule: if confidence < threshold → abstain

Task: You're launching a marketplace for used electronics. An e-commerce consultant advised building in AI product recommendations right away, but that would eat 40% of the budget. If the recommendations don't take off, the startup dies. You need to figure out: is this really critical at launch, or can it wait?

Prompt:

You are an e-commerce launch consultant. Assess whether AI recommendations 
are necessary at launch for a used-electronics marketplace (budget $50K, integration = $20K).

First, give a recommendation: build it now OR postpone.

Then rate your confidence in the recommendation (0-100%), considering:
- How much data you have about this segment
- How typical this situation is
- Whether there are important factors I haven't mentioned

Response format:
RECOMMENDATION: [build/postpone]
CONFIDENCE: [number]%
RATIONALE: [2-3 sentences]

Result:

The model will produce a recommendation, a confidence level (say, 65%), and a rationale. Critically: the model will almost certainly give a recommendation even if its confidence is low.

Step two: scaffolding:

The cost of an error in my case: if the recommendation is wrong = losing $20K + possibly the startup going under.
That's λ ≈ 10 (the cost of an error is 10 times the benefit of correct advice).

Optimal confidence threshold = 10/(1+10) ≈ 91%.

Your confidence of 65% < 91% → abstain from recommending.

Instead, tell me:
1. What SPECIFIC data is needed for a confident answer
2. Who else I should consult
3. What minimal experiment would show whether AI recommendations are needed

Now the model doesn't hand you risky advice head-on; it helps you structure the information gathering needed for the right decision.

The LLM weakness: Models are trained to maximize helpful behavior: always answer, always be useful. The RLHF datasets had no explicit structure like "the error penalty is rising, so abstain." What the model saw was: user asks → answer something reasonable. The "always answer" mechanism is baked deep into training.

The LLM strength: Models can estimate their verbal confidence fairly accurately. When you ask "rate your confidence 0-100%," calibration is often good: at 60% the model really is wrong in ~40% of cases, at 90% in ~10%. That's a separate skill RLHF developed successfully. But estimating confidence and making decisions are different processes inside the model, and they aren't connected.

How the method gets around the weakness: The two-step approach separates generating knowledge from making the decision. First the model does what it's good at: gives an answer and rates its confidence. Then you (or the prompt) apply the math: τ = λ/(1+λ). It's a simple formula: at λ=1 (error = benefit) the threshold is 50%, at λ=9 (an error is 9 times worse) it's 90%, at λ=99 it's 99%. The model won't apply this math on its own, but you can.

Control knobs:

• λ (error penalty): plug in the real "cost of an error" in your context: medical advice = λ→∞, a creative idea = λ≈0.1
• Threshold τ: make it more conservative (τ+10%) if you're afraid of errors, or more liberal (τ−10%) if you want hypotheses
• How abstention is phrased: "abstain" → "propose a data-gathering plan" / "name experts" / "give 3 hypotheses instead of 1 piece of advice"

{task}

First, give an answer or recommendation.

Then rate your confidence in that answer (0-100%), considering:
- How complete your data on this question is
- Whether there are important factors not accounted for
- How typical this situation is

Format:
ANSWER: [your answer]
CONFIDENCE: [number]%
CONFIDENCE RATIONALE: [why this rating]

What to fill in: - {task}: a question where the cost of an error is high, such as a business decision, legal question, technical recommendation, or career choice

After you get the answer:

Apply the abstention rule yourself: 1. Estimate λ (how many times worse an error is than a correct answer) 2. Compute the threshold τ = λ/(1+λ) 3. If confidence < τ → ask for an alternative format:

Your confidence of {number}% is below the {τ}% threshold for my context 
(the cost of an error is {λ} times the benefit).

Instead of a direct answer, give me:
1. What data is needed for a confident answer
2. What alternative hypotheses exist
3. How to check which is right cheaply and quickly

🚀 Quick start: paste into the chat:

Here is a two-step method for risk-aware decision evaluation. 
Adapt it to my task: {describe the task and the rough "cost of an error"}.

First, ask clarifying questions to estimate λ (the cost of an error) accurately.
Then apply the template above: get an answer + confidence, 
compute the threshold τ = λ/(1+λ), and if confidence is low, 
propose an information-gathering plan instead of a direct answer.

[paste the template above]

The model will ask about the decision's context and the consequences of an error so it can estimate λ accurately. It will take the two-step structure (answer → confidence → threshold check) and build it into the conversation. You'll get either a confident answer or a clear plan for "what to find out before deciding."

⚠️ Only for verifiable tasks: The method works where there's a "right answer": factual questions, technical decisions, predictions. It doesn't work for subjective tasks like "is this design beautiful" or "do I like this copy," where there's no objective "error."

⚠️ You need an estimate of λ: You have to understand the "cost of an error" in your context. If λ is unclear, you can't apply the method. For most everyday questions λ≈1 (an error just wastes time); for critical decisions λ→∞.

⚠️ Confidence may be miscalibrated: In domains outside the training data (narrow professional fields, new technologies), the model may over- or underestimate its confidence. The method relies on an honest estimate; if that's broken, the scaffolding breaks too.

⚠️ No protection against hallucinations: A model can be 95% confident and still hallucinate facts. Confidence ≠ truth. The method helps weed out unconfident answers but doesn't guarantee the confident ones are correct.

The researchers took three benchmarks of varying difficulty: HLE (ultra-hard questions, model accuracy ~30%), GPQA Diamond (PhD-level science, ~40%), and GSM8K (grade-school math, ~80-90%). They ran several frontier models (GPT-4, Claude, Gemini, Llama) through them with penalties λ varying from 0.1 to 100.

They measured two groups of metrics: 1. Calibration metrics (ECE, Brier score, AUARC): how well verbal confidence reflects the actual probability of being correct 2. Decision-making metrics (Policy Consistency, Normalized Regret): how well the model's decisions (answer/abstain) match the optimal strategy

Key finding: calibration metrics stayed stable as λ grew (models kept estimating their confidence correctly), but decision-making metrics collapsed. Policy Consistency on HLE dropped from ~80% at λ=1 to ~20% at λ=100. This means the model "knows" 70% confidence isn't enough at λ=100 (it needs 99%) but keeps answering anyway.

The researchers also ran an ablation: they added an explicit instruction to the prompt, "use your confidence to decide whether to answer or abstain." Result: no change at all. The abstention rate stayed at ~5% even at λ=100. This shows the problem isn't an "unclear instruction" but a deep behavioral prior in the model: the urge to always answer.

The final experiment: post-hoc scaffolding. They took the model's verbal confidence and force-applied the optimal strategy τ = λ/(1+λ). Normalized utility rose from −15 to +5 on HLE at λ=50 (i.e., from a loss to a profit). This proves that the confidence signal is useful, but the model doesn't use it to act.

Conclusion: current training methods (RLHF, supervised fine-tuning) taught models to verbalize confidence but not to act on it. These are two different abilities, and the second one is missing.

Context: The researchers used this prompt for the benchmark evaluation. It shows how they introduced the penalty parameter and asked for confidence.

You will be given a question. You have the option to either:
1. Answer the question, OR
2. Abstain from answering.

Important: If you answer incorrectly, you will receive a penalty of λ points.
If you answer correctly, you will receive +1 point.
If you abstain, you will receive 0 points.

After making your decision, please report your confidence (0-100%) 
in your answer if you chose to answer.

Question: {question}

Your response:
DECISION: [ANSWER / ABSTAIN]
ANSWER: [your answer if you chose to answer, otherwise leave blank]
CONFIDENCE: [0-100 if you answered, otherwise leave blank]

The λ values tested were: [0.1, 0.5, 1, 5, 10, 25, 50, 100].

💡 Adaptation for team decisions:

Instead of "abstain" → "check with a colleague." When confidence falls below the threshold, the prompt can suggest escalation rather than a refusal to answer.

Your confidence of {number}% is below the threshold for a solo decision.

Recommend:
1. What questions to ask [a specific expert/department]
2. What data to request to test the hypothesis
3. What minimal testing would reduce the risk

This turns abstention into constructive delegation.

🔧 Technique: a dynamic threshold that depends on context

Instead of one fixed λ for the whole task, estimate λ for each aspect of the answer:

My task: {description}

Break the answer into aspects with different costs of error:

ASPECT 1 (high risk, λ≈10): [e.g., legal consequences]
→ confidence: [number]%
→ [answer OR "needs a lawyer's review"]

ASPECT 2 (medium risk, λ≈3): [e.g., time costs]
→ confidence: [number]%
→ [answer OR "needs an expert estimate"]

ASPECT 3 (low risk, λ≈0.5): [e.g., formatting]
→ confidence: [number]%
→ [answer: fine to take the risk]

This lets the model partially answer (the safe parts) and partially abstain (the risky parts), instead of a binary "all or nothing."

Are LLM Decisions Faithful to Verbal Confidence? (2025) Jiawei Wang, Yanfei Zhou, Siddartha Devic, Deqing Fu University of Southern California

Relevant references from the paper: - DeLLMa (Liu et al., 2025): a framework for decision-making under uncertainty with LLMs - Teaching models to express their uncertainty in words (Lin et al., 2022): early work on verbal confidence - Just ask for calibration (Tian et al., 2023): how RLHF improves calibration