Apply to my task
arXiv:2511.10621 ★ 74 Nov 13, 2025

SSR (Socratic Self-Refine): Step-by-Step Reasoning Checks Through Sub-Questions

Key result

The problem: LLMs are bad at checking their own reasoning. Ask "find the mistake" and you get "all good" or something vague. An error in step 3 breaks everything after it, but the model can't tell you exactly where it messed up. SSR lets you find weak spots in multi-step reasoning (financial models, math, logic problems) by checking each step individually. The trick: split the solution into sub-questions and have the model re-solve each one 3-5 times without knowing it's a check. The model thinks it's solving new problems. If the answers disagree, that step is unreliable. Disagreements pinpoint the error more precisely than "look at the whole solution".

SSR (Socratic Self-Refine): Step-by-Step Reasoning Checks Through Sub-Questions

SSR (Socratic Self-Refine): Step-by-Step Reasoning Checks Through Sub-Questions

02

TL;DR

SSR is a technique for improving LLM reasoning that breaks an answer into "sub-question → sub-answer" pairs and checks each step separately. Instead of looking at the whole answer at once, the model re-solves each sub-question several times and compares the results: if the answers disagree, that step is unreliable.

Ordinary self-checking ("look over your solution and find the mistakes") is a blunt instrument: the model sees a long chain of reasoning and either says "all good" or points to something vague. The problem is that a mistake in one step breaks everything after it, yet the model can't say exactly where it went wrong. It's like grading an essay as "pretty good overall" instead of checking each paragraph.

SSR solves this through decomposition: it splits the reasoning into a sequence of subtasks, checks confidence in each by re-solving it, finds the weakest step and fixes it surgically. There are three variants of the method: linear (checks on every iteration), adaptive (SSR kicks in only when ordinary self-checking found no errors), and with planning (first checks the overall plan, then the steps).

Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.

03

Method Blueprint

STEP 1: CoT → get the reasoning and the answer

STEP 2: Decomposition → split into pairs (sub-question₁, sub-answer₁), (sub-question₂, sub-answer₂)...

STEP 3: Verify each step:
        └─ Re-solve the sub-question M times (without the answer's context)
        └─ Compare with the original sub-answer
        └─ Confidence = share of matches

STEP 4: Find the step with the lowest confidence

STEP 5: Fix that step via majority voting → new answer

[Repeat STEPS 2-5 K times]

Each step runs as a separate LLM request.

04

Example in Practice

Task: You're building a financial model for an investor pitch. You need to work out the unit economics of a new food delivery service in Austin, from customer acquisition cost to break-even.

Prompt (decomposition after the first solution):

Here is my solution to the problem and the answer:

[paste the problem]

[paste your solution with the calculations]

Break this reasoning down into a sequence of subtasks. For each step, state:
- Sub-question: what exactly are we calculating or determining in this step?
- Sub-answer: what result did we get?

Format:
Step 1:
- Sub-question: [...]
- Sub-answer: [...]

Step 2:
...

Prompt (verifying a single step):

Context from previous steps:
- Customer acquisition cost (CAC): $12
- Average order value: $30
- Margin per order: 25%

Sub-question: How many orders does one customer need to place to pay back the cost of acquiring them?

Solve this sub-question using only the data above. Give only the numerical answer.

Result: The model will produce a decomposition into 5-7 steps (CAC → average order value → margin → LTV → payback period → break-even). Then for each step you'll get several independent solutions. If on the "margin calculation" step 3 out of 5 re-solves give $7.50 while the original gave $6.25, that's a red flag. The model will point to exactly that step as unreliable and suggest a fix.

05

Why It Works

The LLM weakness: A model is bad at finding its own mistakes when it looks at the whole answer at once. It tends to confirm its own conclusions: confirmation bias. Especially when the error happened in the middle of a long chain: everything looks logical because the later steps follow from the faulty one.

The LLM strength: A model is good at solving isolated problems with clear context. Give it only the sub-question and the necessary data, without its own previous answer, and it will solve it "from a clean slate". If the solutions disagree, that's a signal.

How the method exploits this: SSR turns checking from "look at your solution" into "re-solve each subtask independently". The model doesn't know it's checking itself; it's just solving sub-questions. Disagreements expose unreliable steps, and majority voting yields a more reliable answer.

Control levers:

  • Number of re-solves (M): more = a more accurate confidence estimate, but more expensive. 3-5 is enough for most tasks
  • Number of iterations (K): how many times to repeat the check-and-fix loop. 2-3 iterations is the sweet spot
  • Decomposition granularity: finer steps = more precise error localization, but more overhead. 5-7 steps is the balance
06

Prompt Template

Decomposition

Here is the problem and my solution:

Problem: {problem}

Solution: {solution}

Answer: {answer}

Break the reasoning down into a sequence of subtasks (no more than {max_steps} steps). For each step, state:
- Sub-question: what are we calculating or determining?
- Sub-answer: what result did we get?

Format:
Step 1:
- Sub-question: [...]
- Sub-answer: [...]

Step Verification

Context (results of previous steps):
{previous_steps_context}

Sub-question: {sub_question}

Solve this sub-question using only the data from the context. Give a brief answer.

Research digest

Key result

The problem: LLMs are bad at checking their own reasoning. Ask "find the mistake" and you get "all good" or something vague. An error in step 3 breaks everything after it, but the model can't tell you exactly where it messed up. SSR lets you find weak spots in multi-step reasoning (financial models, math, logic problems) by checking each step individually. The trick: split the solution into sub-questions and have the model re-solve each one 3-5 times without knowing it's a check. The model thinks it's solving new problems. If the answers disagree, that step is unreliable. Disagreements pinpoint the error more precisely than "look at the whole solution".

Principle

Don't let the model check the whole answer at once; it will confirm its own conclusions (confirmation bias). Split it into isolated subtasks and have it re-solve each one "from a clean slate". The process: get a solution via step-by-step reasoning → decompose it into (sub-question, sub-answer) pairs → for each sub-question the model gives 3-5 independent solutions → compare them with the original sub-answer → if most disagree, you've found the weak spot → fix it via majority vote. The neat part: the model doesn't know it's checking itself. It's just solving short problems without the context of its previous answer.

Why it works

An LLM tends to confirm its own conclusions when it sees the whole answer at once. If the error happened in step 3, steps 4-7 look logical because they follow from the faulty one. The model doesn't notice the problem. The LLM's strength: it's good at solving isolated problems with clear context. Give it only the sub-question plus the data it needs (without its own previous answer) and it solves it from scratch. Disagreement between fresh solutions and the original = a signal of unreliability. SSR turns fuzzy self-checking into a series of independent recalculations. Multiple re-solves (3-5 times) give a confidence estimate: if 4 out of 5 attempts produce a different answer, the step is clearly weak.

When to apply

Multi-step reasoning → specifically math, financial models, logic problems and legal analysis, especially when the cost of an error is high (an investor pitch, an important calculation). NOT a fit for: simple one-step problems (overkill), creative tasks with no right answer, or weak models like GPT-3.5 (it can make results worse). You need strong models: Claude Opus/Sonnet, GPT-4+.

Mini recipe

1. Get a solution via step-by-step reasoning (CoT): ask the model to show its thinking, not just the answer. 2. Decompose: `Break this reasoning down into a sequence of subtasks (5-7 steps). For each step, state: Sub-question — what are we calculating? Sub-answer — what result did we get?` 3. Check each step: for each sub-question, have the model re-solve it 3-5 times without the context of the original answer. Prompt: `Context: [data from previous steps]. Sub-question: [question]. Solve using only the context and give a brief answer.` 4. Find the weak spot: compare the original sub-answers with the new solutions. Where they disagree, that's the error. 5. Fix it surgically: `There's an error in step [N] of the solution. My answer: [X]. Correct answer (by majority voting): [Y]. Fix the solution and recalculate the following steps.`

Examples

[BAD]: `Here is my solution to the unit economics problem. Check whether I calculated it correctly.` (The model will say "looks logical" or point to something vague) [GOOD]: `Task: unit economics for a delivery service. My solution: [solution with 7 steps]. Let's apply SSR: 1) Break it into subtasks (CAC → average order value → margin → LTV → payback → break-even). 2) For each sub-question, give 5 independent solutions without the context of my answer. 3) Wherever they disagree, that's the problem. 4) Fix the weak step using the majority answer.` (The model finds that on the "margin calculation" step, 4 out of 5 re-solves give $7.50 versus the original $6.25, and fixes it surgically)

In plain words

The core idea of SSR (Socratic Self-Refine) is that models are disastrously bad at admitting their mistakes when they look at a text as a whole. When a model spits out a long wall of reasoning, it falls into confirmation bias: if it blurted out something dumb in step two, the next ten steps will flow logically from that blunder, and the model won't notice the catch. To fix this, the researchers made the AI work the Socratic way: chop a complex problem into tiny "question → answer" pairs and meticulously check each link in the chain in isolation from the final answer.

It's like auditing a complicated accounting report: instead of just rereading the bottom line, you take every single transaction and run it through the calculator three times. If you get "a hundred" twice and "a thousand" once, the step is unreliable and needs redoing. On paper everything looks neat, but SSR exposes the rot in the very foundation of the reasoning and keeps an error from snowballing into a full-blown disaster in the final answer.

In practice it works as a tough filter: first the model produces a standard chain of thought, and then Socratic Self-Refine kicks in. Each sub-question is re-solved several times, and if the results disagree, the model knows it's messing up. For example, when calculating unit economics it might recompute customer acquisition cost (CAC) three times, and if the numbers don't add up, it won't move on until it gets a stable result. This kills hallucinations in the cradle, because it's harder for a model to lie the same way three times in a row than to confidently spout nonsense once.

Although the method was tested on math and logic, the principle is universal for any complex analysis, from legal contracts to writing code. Anywhere a mistake mid-process turns the result into a pumpkin, SSR plays the role of a fussy quality inspector. It's a shift from plain "write me some text" to multi-layer verification, where every logical leap is backed by facts, not just by how nice the sentence sounds.

Bottom line: stop taking the model at its word when it produces long reasoning; it has almost certainly embellished reality somewhere. SSR turns AI from an overconfident storyteller into a meticulous analyst who questions every step. Whoever starts building these self-check loops into their prompts will get far fewer defects. Everyone else will keep wondering why their "smart" chatbot produces logically flawless nonsense.

AI lab assistant

Apply this study to your task

Describe what you're working on. AI takes the method and the measured results from this paper and adapts them to your case, or writes a ready prompt built on them.

Based on SSR (Socratic Self-Refine): Step-by-Step Reasoning Checks Through Sub-Questions arXiv:2511.10621

Sign in to apply this study New accounts get 50 free credits.

Get the week’s best papers by email

One email on Sundays with the highest-rated new techniques. Unsubscribe with one click.