PROBLEM (Short-CoT):

Request → Model generates 2-3 solutions → Stops
         ↓
         Confidence: 85%
         Actual recall: 15%
         ⚠️ Overconfidence: +70%

FIX 1 (Long-CoT models):

Request → Generation → Reflection 1 → Correction → Reflection 2 → ...
         ↓            ↓               ↓              ↓
         3 solutions  errors found    +2 solutions   confidence ↓

Result: Recall ↑ (up to 50%), Confidence ↓ (to 60%) → Better calibration

FIX 2 (Exploratory prompt for regular models):

First request → Answer with 2-3 solutions
↓
"Wait, there might be other solutions" → +3-5 new solutions

FIX 3 (Parallel exploration):

Run 5-10 independent generations → Collect all unique answers → 
Merge → Recall is significantly higher

Task: You're advising a startup launching a fast grocery delivery service in Denver. The marketing budget is $10,000 for the first 3 months. You need to find ALL possible channels for acquiring the first customers.

Prompt (basic — demonstrates the problem):

A list of all possible customer acquisition channels for a fast grocery 
delivery service in Denver. Budget: $10,000 for 3 months.

List ALL the options.

What happens with a regular model: The model will give 3-5 obvious channels (search ads, social media, door hangers) and stop. It will miss the non-standard options: partnerships with gyms, posts in neighborhood Facebook groups and on Nextdoor, pickup promos for residents of new apartment complexes, deals with property management companies.

Prompt (with an exploratory trigger):

A list of all possible customer acquisition channels for a fast grocery 
delivery service in Denver. Budget: $10,000 for 3 months.

After you give the first list, tell yourself: "Wait, are there any 
non-standard channels I haven't considered?" — and add them.

List ALL the options, including the non-obvious ones.

Result: The model will first give a standard list, then interrupt itself with the phrase from the prompt and add 5-10 non-standard channels. The final list will be 2-3 times more complete.

Prompt (for Long-CoT models — o1, Claude extended):

Task: find ALL possible customer acquisition channels for a fast grocery 
delivery service in Denver. Budget: $10,000 
for 3 months.

Requirements:
- Completeness matters more than response speed
- Consider both standard and non-standard options
- Account for the specifics of the US market and local quirks
- After the first list, revisit what you might have missed

Result: The model will show several rounds of thinking: first it generates the obvious channels, then it critiques itself ("missed offline touchpoints"), adds new ones, checks again ("didn't consider local community digital channels"), and adjusts. The final list will be as complete as possible, with a self-rated completeness below 100% — a sign of good calibration.

The LLM's weakness: The model generates text sequentially, step by step. When it finds the first working solution, its attention locks onto that path — cognitive rigidity. Alternative paths "fade out" in the model's deep layers (low attention entropy in layers 15-30). The model sincerely believes it has found everything and stops with high confidence.

The LLM's strength: The model is great at following structured instructions and can simulate reflection — reviewing its output from a different angle. If you explicitly ask "wait, is there more?" or build in a reflection loop, the model will resume the search and find new paths.

How the method gets around the weakness: - An exploratory prompt ("wait, are there others?") works as a hard reset of cognitive rigidity — the model leaves the first path and starts a new search - Long-CoT reflection creates multiple review cycles — the model can't "get stuck" for long; each round brings it back to "what else?" - Parallel generations (self-consistency) get around the problem statistically — each run "gets stuck" on its own path, and merging them covers the whole solution space

Control knobs:

Number of reflection rounds — Long-CoT models: - Fewer rounds (1-2) → faster, but lower recall - More rounds (5-7) → recall rises, overconfidence drops, but it costs more tokens

Exploratory triggers — regular models: - "Wait, is there more?" → baseline - "Critically review — what did you miss?" → stronger unlock - "Imagine you ABSOLUTELY must find 5 more options" → forces the search to continue

Parallel generations (n requests): - n=3-5 → minimal gain in recall - n=10-32 → significant gain, the optimum per the study - Merge strategy: Voting (all unique answers) beats Median Confidence (picking one)

Option 1: Exploratory prompt (for regular models)

{task_description}

Requirement: find ALL possible {what_we_need}.

Step 1: Draft a first list of options.

Step 2: Tell yourself: "Wait, can I find more options I missed?" 
Critically review and add non-obvious solutions.

Step 3: A final list with every option you found.

What to plug in: - {task_description} — the task's context and conditions - {what_we_need} — what exactly needs to be listed (channels, strategies, scenarios, options)

Option 2: Multi-round reflection (for Long-CoT models)

Task: {task_description}

Goal: find the MOST complete list of {what_we_need}.

Process:
1. Generate a first list of options
2. Critically assess — which categories/directions might you have missed?
3. Add the options you found and repeat the check
4. Keep going until you're confident you've covered >90% of the possibilities

Important: completeness matters more than speed. Show the intermediate rounds of thinking.

What to plug in: same as in option 1

🚀 Quick start — paste into the chat:

Here is a template for finding ALL possible solutions. Adapt it to my task: 
{your task}. Ask questions if you need clarification.

[paste the template above]

The LLM will ask about the specifics of the task, which options to look for, and whether there are constraints — so it can build them into the reflection loop. It will take the exploratory search structure from the template and adapt it to your context.

Option 3: Parallel exploration (self-consistency)

{task_description}

Find all possible {what_we_need}.

[Run this prompt 5-10 times with small variations in wording]

Then merge all the unique answers from the different runs into a single list.

Note: This option requires several requests to the model and manual merging. It suits mission-critical tasks where you need maximum completeness.

⚠️ Structured vs open-ended tasks: The study focused on tasks with an enumerable solution space (schedules, combinations). For purely creative tasks (write a story "in different ways"), the methods work, but completeness can't be measured — there's no ground truth for "all possible stories."

⚠️ Open models are still overconfident: Even Long-CoT mode in open models (Qwen, DeepSeek) shows 56-60% calibration error — the model still overestimates its completeness. Closed models (o1, o4-mini) calibrate better (ECE <10%), but not perfectly.

⚠️ The cost of Long-CoT: Models with extended thinking (o1, Claude extended) spend 5-10 times more tokens on reflection. For simple tasks with 2-3 solutions that's overkill — an exploratory prompt is enough.

⚠️ Temperature doesn't help: Raising the sampling temperature doesn't reduce overconfidence and doesn't improve recall. It adds variety at the word level, not at the level of reasoning paths. The model still "gets stuck" on the first direction it found.

Reasoning Overconfidence isn't a prompt bug but a fundamental property of the LLM architecture. The model isn't "too lazy" to keep searching — it sincerely doesn't realize there are more solutions. That explains a lot of situations:

• Why ChatGPT gives 3 ideas and says "those are all the options" when there are 15
• Why the model "forgets" whole categories of solutions
• Why asking again with different wording finds completely different answers

Cognitive rigidity in the core reasoning layers (layers 15-30) is a measurable effect: the researchers showed via attention entropy that the model literally "narrows" its attention onto one path. Long-CoT and exploratory prompts physically restore high entropy — the model "looks wide" again.

Practical takeaway: For tasks that need COMPLETENESS rather than a single solution, ALWAYS use search-widening techniques. Don't trust the model's first answer, even if it's confident.

The researchers built MuSoBench — a benchmark of two task types with a precisely known number of correct solutions. TimeTabling: build a conflict-free course schedule (100 tasks × 10 difficulty levels). SubsetSum: find every subset of numbers with a given sum (100 tasks × 7 levels). Crucially, for each task they algorithmically enumerated ALL possible solutions and verified them by hand — which let them measure Recall (completeness) precisely.

They compared Short-CoT (regular mode: Qwen, DeepSeek, GPT-4o-mini) with Long-CoT (extended thinking: Qwen thinking mode, DeepSeek-R1, o1-mini). For each answer, the model gave solutions + a verbal confidence estimate (0-100%). That measured the gap between confidence and actual completeness.

The main finding: Short-CoT clusters in the "10-20% recall, 80-90% confidence" zone — reasoning overconfidence was confirmed visually on the charts. Long-CoT shifts the points toward the diagonal (better calibration) and raises Recall to 50%+. Expected Calibration Error (ECE) drops from 78% to 18-60% depending on the model.

Why did it come out this way? The researchers measured three behavioral metrics: - CSR (Correct Solution Retention) — keeping correct solutions during review - ESC (Error Solution Correction) — fixing errors - NSD (New Solution Discovery) — finding new solutions

Short-CoT showed a critically low NSD (<0.3%) — the model practically does NOT find new solutions even when asked to review. Long-CoT raises NSD to 0.5-3% — 10× higher, but still weak. That's cognitive rigidity in action.

The surprising discovery: Raising the decoding temperature doesn't help! It increases word-level variety but doesn't unlock the search for new paths — Recall and ECE stay put. So the problem runs deeper: in the reasoning logic itself, not in token sampling.

The mechanism from the inside: They analyzed attention entropy in each layer of Qwen3-8B and found three phases: 1. Layers 0-10: entropy converges quickly (the same for all modes) 2. Layers 15-30 (core reasoning): Short-CoT shows low entropy — narrowly focused attention = cognitive rigidity. Long-CoT keeps high entropy — a broad search 3. Layers 35+: the trend reverses — Short-CoT ends with high entropy (uncertainty due to rigidity), Long-CoT with low (confident convergence after exploration)

This is the first physiological evidence of cognitive rigidity at the level of the model's internal activations.

🔧 Technique: A forced solution quota

The basic technique asks "find more" but doesn't guarantee a count. You can force a minimum:

{task}

Requirement: find AT LEAST {number} different {what_we_need}.

If after the first list you have fewer than {number} options, keep searching 
until you reach the required count. Non-obvious and non-standard 
options count too.

Effect: The model can't stop early — the quota forces it to keep generating even when it "feels like it's found everything." The risk: with too high a quota the model may start inventing irrelevant options.

🔧 Technique: Category decomposition

Instead of "find everything at once," break the solution space into categories:

{task}

Step 1: Define 5-7 categories/directions for {what_we_need}

Step 2: For EACH category, find 3-5 concrete options

Step 3: Assemble the final list from all categories

Effect: The model structures its search — it goes through each category separately, reducing the risk that an entire direction gets skipped. This gets around cognitive rigidity via forced context switching.

Example: Instead of "find all acquisition channels" → "1. Online ads 2. Offline touchpoints 3. Partnerships 4. Word of mouth 5. PR" → generate sub-options for each category.

🔧 Technique: Adversarial review

Add an opposing role that critiques the completeness:

{task}

Step 1: Make a list of {what_we_need}

Step 2: Now imagine you're a skeptical expert. Critique this list: 
which obvious directions are missing? Where are the gaps in coverage?

Step 3: Expand the list based on the critique

Step 4: Final version

Effect: Switching to the "critic" role unlocks a different perspective — the model looks not for new solutions but for gaps in coverage. It's a different kind of search that finds the blind zones.

Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Tasks

Authors: Qiguang Chen, Libo Qin, and colleagues
Institutions: Harbin Institute of Technology, Central South University, Tsinghua University

The research was supported by the National Natural Science Foundation of China (NSFC) and the CCF-Zhipu Large Model Innovation Fund

Key references from the paper: - Chain-of-Thought (CoT) prompting [Wei et al., 2022] - Long-CoT reasoning [Chen et al., 2025 - survey on long CoT] - Tree-of-Thought [Yao et al., 2023] - Calibration research [Lin et al., 2022 - TruthfulQA, CalibratedMath]