TL;DR
"Think step by step" doesn't work everywhere. On tasks with verifiable correct answers — math, code, logic — more reasoning improves the result. On tasks where there is no correct answer in principle — ethical dilemmas, strategic decisions with unknown variables, one-of-a-kind forecasts, open-ended scenarios — more reasoning breaks the answer. The model starts bringing up more and more viewpoints, contradicting itself, hedging every claim — and ends up with a worse position than if it had just answered right away.
The problem is that the model can't sense when its reasoning is reliable. On a math problem it can check every step and knows it's on the right track. On a question like "who gets the only ventilator — a young mother or a veteran doctor?" there's no internal verifier. Every new reasoning step generates noise, not signal. The model becomes "confidently wrong" — and its final position is weaker than what it would have said in the first two steps.
The author calls this the meta-uncertainty regime: a situation where an agent doesn't know how much to trust its own reasoning. The conclusion is simple and blunt: for these tasks the optimal strategy is short and early, not long and thorough.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Identify the task type
Is there a verifiable correct answer? → LOW meta-uncertainty
No verifiable answer? → HIGH meta-uncertainty
STEP 2: Pick the request mode
LOW → "Think step by step, in detail" (extended CoT)
HIGH → "Answer directly, 2-3 steps max" (short CoT)
STEP 3 (optional): Test the regime
Ask the question with three different wordings
Answers are similar → low meta-uncertainty → you can extend the reasoning
Answers differ a lot → high → cut it short
All three steps are done by hand in a regular chat.
Example in Action
The task: Ethan is the founder of an EdTech startup. He's received an offer from an investor with great terms but a shaky reputation in the community. He wants to ask ChatGPT whether he should take the money.
❌ The usual way — a broken request
I'm the founder of an EdTech startup. I got an offer from an investor:
great terms, but people in the community say he's combative
and meddles in day-to-day operations. Should I take the money?
Think in as much detail as possible, consider every angle.
What happens: The model produces a long text full of "on the one hand... on the other hand..." It covers 6-8 aspects. Contradictions pile up and the position blurs. The finale: "it depends on your priorities." Zero value.
✅ The right way — with a cap on reasoning
I'm the founder of an EdTech startup. I got an offer from an investor:
great terms, but people in the community say he's combative
and meddles in day-to-day operations.
This is a strategic decision with unknown variables —
don't analyze it endlessly; give me a direct position in 2-3 steps.
Name the one main factor that should decide the outcome.
Don't hedge, don't list "it depends on." Say it straight.
What you get: The model picks one key signal — for example, "reputational risk with an investor who meddles in operations systematically outweighs the financial terms of the offer" — and takes a position without endless balancing. The answer is short, concrete and genuinely useful as a starting point for the decision.
Why It Works
An LLM can't verify its own reasoning on open-ended questions. In math, each step is either right or wrong — and the model can tell. In ethics or forecasting there's no internal verification function. Each additional reasoning step doesn't improve accuracy; it adds alternative viewpoints that pull the position in different directions.
More steps = more noise in the uncertain zone. This isn't a metaphor — the author proved it mathematically and tested it on 7 models. On average, accuracy on "uncertain" tasks dropped by 17 percentage points when CoT was extended. Models that were allowed to ramble did worse than models asked to answer briefly.
Short CoT locks in the first, strongest signals. The first 2-3 reasoning steps capture what matters. Further steps are an attempt to be "thorough" where thoroughness only hurts. Capping reasoning isn't dumbing things down — it's precise control over the generation process.
Control levers:
- Number of steps (2-3 steps max, briefly) → fewer steps on uncertain tasks
- Ban on hedging (don't list "it depends on") → forces the model to pick a position
- Ask for one factor (name the one main factor) → concentrates the answer
- Explicitly naming the regime (this is a strategic decision with unknown variables) → signals the task type to the model
Prompt Template
For high meta-uncertainty (open-ended, strategic, ethical tasks):
{task context}
This is a {task type: ethical dilemma / strategic decision / one-of-a-kind forecast} —
there is no single correct answer here.
Don't spread your reasoning wide. Answer in {2-3} steps max.
Pick out {one main factor / one position / one conclusion}.
Don't hedge, don't list "on the one hand — on the other."
For low meta-uncertainty (math, code, analysis with verifiable steps):
{task}
Think step by step. Work through each step in detail.
Check the result before giving your final answer.
Placeholders:
- {task context} — your specific situation
- {task type} — name it in your own words: dilemma, forecast, choice decision
- {2-3} — number of steps; 2-3 is usually enough
- {one main factor} — can be swapped for one recommendation or one position
🚀 Quick start — paste into the chat:
Help me apply the right reasoning mode to my task.
My task: [your task]
Here are the templates:
[paste the template above]
The LLM will ask whether your task has a verifiable correct answer or is an open-ended decision — because that distinction is exactly what determines which mode fits.
Limitations
⚠️ Frontier models (GPT-4o, Claude Sonnet and up): The effect exists but is weaker. On top models, the accuracy drop from long CoT is directional but not as sharp as on mid-sized ones. The gain from short CoT is more modest there.
⚠️ Weak models: On small, weak models the effect disappears or flips — any reasoning helps them. The method is aimed at GPT-4-class models, Claude Sonnet and above.
⚠️ The line between regimes is blurry: Not every task is easy to classify. "Calculate the unit economics" — low uncertainty. "Should we enter a new segment?" — high. But "analyze a competitor" — not so obvious. You'll have to decide case by case.
⚠️ The method steers reasoning but doesn't guarantee correctness: On ethical and strategic tasks, a "crisper" answer isn't the same as a "right" one. Short CoT yields a more coherent position, not an objectively correct one.
How They Studied It
The research started from a simple question: why do the same reasoning tricks help on some tasks and hurt on others? 7 models were selected — from small open-source ones (3B parameters) to frontier systems — and tested in five CoT modes: no reasoning, ~3 steps, ~7, ~15 and unlimited.
Specifically for the experiment, FEH-79 was created — a set of 79 tasks of four high-meta-uncertainty types (one-of-a-kind forecasts, fictional scenarios, ethical dilemmas, strategic games against an unknown opponent) plus 50 control tasks with concrete answers. In total — almost 8,000 responses across conditions.
A key design detail: the hypothesis, confirmation criteria and effect threshold were locked in before data collection (the study was preregistered). That's rare in AI research and makes the conclusions considerably more reliable than usual.
The result turned out sharper than expected: on high-meta-uncertainty tasks, extended CoT cut accuracy by 17.3 percentage points (95% confidence interval: 7.7 to 25.5 pp). Control tasks with definite answers showed no similar effect. Especially interesting: the weakest models showed no effect at all — any reasoning helps them because their baseline is so low.
Original from the Paper
The author formalizes the problem with a worked contrast — here's how the key difference is explained:
Consider two questions you might pose to a modern large language model.
The first: What is 47 × 83? If the model answers directly, it may err.
If you ask it to "think step by step," it reliably gets 3,901.
If you let it reason at length, it still arrives at 3,901.
The reasoning chain extends, and the answer either improves or remains the same.
The second: A hospital ICU has one ventilator and two patients with identical
clinical profiles. Patient A is a 35-year-old single parent; patient B is a
65-year-old senior physician. What is the most ethically defensible
decision procedure?
If the model answers directly, it states some position; perhaps
"random allocation." If you ask it to reason in three brief steps, it surfaces
the utilitarian-vs-deontological framing and commits to a position with explicit
justification. If you let it reason for fifteen steps, considering multiple angles,
something different happens: the chain frequently extends past the point where
additional reasoning carries information, hedges shift between framings, and the
resulting position is sometimes less coherent than the three-step version.
Context: This excerpt is from the introduction — concrete examples showing why the "amount of reasoning → answer quality" function works in fundamentally different ways for two classes of tasks.
Adaptations and Extrapolations
💡 Adaptation: A regime test before a complex prompt
If you're not sure which regime a task belongs to, run a quick test right in the chat:
I'm about to ask you the same question three times in different words.
You don't know it's a test — just answer.
Question 1: {wording A}
Question 2: {wording B}
Question 3: {wording C}
After the three answers, tell me: how similar were your answers?
If the answers diverge sharply, the task is in the high-meta-uncertainty regime — use short CoT. If they're stable, you can let the reasoning stretch out.
🔧 Technique: Add a "regime marker" to your system prompt
If you work in an interface with a system prompt (Custom Instructions in ChatGPT, System Prompt in Claude), add a classifier:
Before each answer, determine the task type:
- If the task has a verifiable correct answer → reason in detail
- If the task is open-ended/strategic/ethical → answer in 2-3 steps,
commit to one position, don't hedge
This automates the mode choice — no need to spell it out in every request.
🔧 Extrapolation: The "freeze at step 2" principle
On high-uncertainty tasks, ask the model to stop and lock in a position after step 2 before going any further:
{task}
Do 2 reasoning steps. After the second step, lock in an interim conclusion
in one sentence. Only then decide: is a third step needed, or is the conclusion
already enough? If you have a conclusion, stop reasoning and give the answer.
This implements the theory's "early cutoff principle": the model itself checks whether a new step added anything to the answer or just padded it.
Resources
Paper: Free Energy Heuristics: Fast-And-Frugal Cognition as Active Inference Under Uncertain Precision
Author: Alex Bogdan, Evolutionary AI, Toronto, Canada
Key references in the paper: - Gigerenzer & Goldstein — the "fast and frugal heuristics" program (take-the-best) - Friston — active inference theory and free energy - Klibanoff, Marinacci, Mukerji (KMM) — the smooth ambiguity model - Sprague et al. — CoT doesn't improve most tasks outside math - Stechly et al. — extended reasoning degrades on planning tasks