Basic CoT-Guided mode (6 rounds in one chat):

ROUND 1 (Teacher): Greeting + task context

ROUND 2 (Student):
  (A) Summary of what I know: [...]
  (B) What's missing: [subtasks]
  (C) Question: [precise question]

ROUND 3 (Teacher): Answer WITHOUT revealing the final solution

ROUND 4 (Student): (A)→(B)→(C) next question
ROUND 5 (Teacher): Answer
ROUND 6 (Student): (A)→(B)→(C) next question

→ After each teacher answer: Pass@5 check

With Assessment:

Pre-assessment (math): inserted before Round 2 Mid-assessment (code): inserted before Round 4

ASSESSMENT:
The student generates 2 solution attempts
The teacher grades each one (correct/incorrect + why)
Feedback → into the dialogue history

Task: You need to figure out how many servers to rent to launch a SaaS product. You have traffic data, but you don't know the load formulas and aren't sure which parameters to account for (RAM, CPU, concurrent users).

Prompt:

You are a student learning to solve a problem by asking an expert questions.
The expert won't give you the final answer, but will answer your questions.

Problem: Calculate the number of servers for a SaaS product projected at 50,000
monthly active users, average session 15 minutes, 3 requests
per minute per user. Budget: $2,000/month.

Structure of each question:
(A) What I already know from previous answers
(B) Which subtasks/data I'm missing to solve it
(C) One precise question for the expert

Start the dialogue. The expert (me) will answer, but won't tell you the final number
of servers — you have to get there yourself in 5-6 questions.

---

CHECK (after 3 questions):
After your third question, give TWO solution attempts using different approaches.
I (the expert) will tell you whether each attempt is heading the right way.

Result:

The model will start with a question about load metrics (RPS, concurrent connections), then ask about server parameters (RAM/CPU per user), then about redundancy and load peaks. After round 3 it gives two attempts: (1) a calculation based on RPS, (2) a calculation based on concurrent users. You tell it which approach is more accurate. By round 5-6 the model delivers the final calculation with formulas and reasoning. The advantage: instead of hallucinated numbers or generic advice, the model deliberately extracts the structure of the solution through dialogue.

The LLM's weakness: A model tries to answer right away even when it lacks information or a solution structure. The result is confident but wrong answers (hallucinations) or shallow solutions with unchecked steps. In complex tasks (math/code) recalling facts isn't enough — you have to build the solution step by step, and the model tends to "jump" to a conclusion.

The LLM's strength: A model is good at generating structured questions when explicitly asked. It can tell "what I know" from "what I don't know" if you give it a format. It works well as a conversation partner that reflects and adjusts direction based on new information. In dialogue mode the model keeps the context and uses the answers to build the next steps.

How the method uses this: Instead of one long prompt, the method splits thinking into rounds. Each round = reflection (A: what I know) + planning (B: what's missing) + action (C: question). The (A)→(B)→(C) structure removes "jumping to conclusions" — the model can't answer right away; it has to assess the gaps first. The teacher doesn't reveal the solution, only answers questions — this forces the student to synthesize the answer itself from pieces of information rather than copy a ready-made one.

Timing of checks: Pre-assessment works in math because a mistake in the first step multiplies (start from a wrong number → everything after is wrong). Mid-assessment works in code because code is divergent: an early "this is wrong" can kill the right direction before it has turned into working code. A check in the middle lets the model explore the solution space first, then correct the chosen path.

Control levers:

• Number of rounds → 3-4 for simple tasks (saves tokens), 6-8 for complex ones
• Question structure (A)-(B)-(C) → drop it for a quick mode and keep only (C), but you'll lose quality
• Assessment position → round 1 for step-by-step tasks (math, processes), round 3-4 for exploratory ones (code, design, strategy)
• Teacher role → change it to "the expert never answers directly, only with leading hints" for even deeper learning
• Pass@k check → after each round, ask the model to "try solving it now" to see the progress

Basic (no assessment):

You are a student solving a problem through dialogue with an expert.

PROBLEM: {describe the problem}

RULES:
- The expert will answer your questions but will NOT give the final solution
- You must reach the answer yourself through a series of questions
- 5-6 rounds of questions in total

STRUCTURE OF EACH OF YOUR MESSAGES:
(A) What I know: [summary of known information from previous answers]
(B) What I'm missing: [specific gaps, subtasks, data]
(C) My question: [ONE precise question for the expert]

FINAL:
After 5-6 rounds, give the final solution with your reasoning.

Begin. I'm playing the expert.

Placeholders: - {describe the problem} — your specific task (a business calculation, a technical decision, a complex analysis)

With mid-assessment (for code/design):

[the same basic template as above]

ADDITIONALLY:
After the 3rd round of questions:
- Give TWO different solution attempts
- I (the expert) will tell you whether each one is heading the right way
- You continue, taking the feedback into account

Begin.

With pre-assessment (for math/processes):

[the same basic template]

ADDITIONALLY:
BEFORE your first question:
- Give TWO quick attempts at solving the problem (even if you're not sure)
- I (the expert) will tell you what's wrong with each
- Then start asking questions using the (A)-(B)-(C) structure

Start with the two attempts.

⚠️ Dialogue length: 6 rounds = a lot of tokens. For simple tasks it's overkill — a regular CoT prompt will get there faster.

⚠️ Teacher quality: If you're playing the expert yourself and don't know the answer, the method won't work. You need a real source of knowledge (you, documentation, an API).

⚠️ Not for facts: The method is for tasks where you need to build a solution (calculations, code, strategy). For "what's the capital of France" it's overkill — the model knows the facts, no dialogue needed.

⚠️ Depends on the (A)-(B)-(C) structure: If the model ignores the structure and asks everything at once, the effect disappears. Keep an eye on the first 1-2 rounds.

The researchers took weak models (Qwen-7B, Mistral-7B) as students and a strong model (Qwen-72B) as the teacher. They selected tasks where the student fails (Pass@5 < 0.5) but the teacher succeeds — that gave 2000 math problems (GSM8K) and 100 coding problems (HumanEval/OPC). They ran 6 rounds of student→teacher dialogue and measured the student's Pass@5 after each teacher answer (5 attempts to solve the task).

They compared 5 strategies: (1) Unguided — the student just asks whatever it wants, (2) CoT-guided — the student follows the (A)→(B)→(C) structure, (3) Pre-assessment — at the start the student gives 2 solution attempts + gets feedback, (4) Mid-assessment — the same but mid-dialogue, (5) Guided training — the student was trained via DPO to ask better questions (they generated several candidate questions, picked the best one by Pass@k, built preference pairs, fine-tuned).

The key discovery: Even an unguided dialogue (no structure at all) gives a +0.5-0.85 gain in Pass@k — the dialogue itself helps. CoT-guided adds another +7-18% in the early rounds. Pre-assessment is better for math (+24.7% peak), mid-assessment is better for code (+12.8% peak) — which confirmed the hypothesis about the nature of the tasks. Guided training (DPO) had the strongest effect: the models learned to ask questions 2-3 rounds better (a round-1 question after training resembles a round 3-4 question before training). That cut the number of rounds needed from 6 to 3.

LLM-as-Judge analysis: To understand why the methods work, the researchers had Qwen-72B rate progress toward the solution after each round (not final correctness, but "how far it has moved"). The chart showed that every method except unguided produces a steeper progress curve. They also checked the similarity of questions between unguided and guided students — the questions after training are shifted forward: guided round 1 = unguided round 2-3, guided round 2 = unguided round 3-5. That explains why trained models reach the goal in fewer rounds.

🔧 Technique: Simulation in one model → saving on the "teacher"

Instead of two windows (student + teacher), you can ask one model to play both roles in turn:

You play two roles, taking turns.

ROLE 1 — STUDENT:
[the (A)-(B)-(C) structure from the basic template]

ROLE 2 — EXPERT:
Answer the student's question WITHOUT revealing the final solution.
Give hints, facts, methods, but do NOT state the answer directly.

PROBLEM: {your problem}

Start the dialogue. Alternate roles: Student → Expert → Student → ...
5-6 rounds. At the end (Student only), give the solution.

Effect: In "student" mode the model is more critical of gaps; in "expert" mode it gives structured hints. A dialogue with itself = internal Chain-of-Thought with checks, but made explicit.

🔧 Technique: Three students instead of one → group learning

For very complex tasks (architecture decisions, strategy) you can run three students with different roles:

You play 4 roles:

OPTIMIST STUDENT: Looks for how the solution CAN work
CRITIC STUDENT: Looks for where the solution WILL BREAK
PRACTITIONER STUDENT: Looks for how to implement it IN REALITY
EXPERT: Answers the questions of all three

PROBLEM: {complex problem}

PROCESS (6 rounds):
Round 1: Optimist asks a question → Expert answers
Round 2: Critic asks a question → Expert answers
Round 3: Practitioner asks a question → Expert answers
Rounds 4-6: Repeat the cycle

FINAL: All three students give their solutions. The expert picks the best one
or combines them.

Begin. Label each role explicitly.

Effect: The different "students" explore different dimensions of the task in parallel. It's a version of multi-agent debate, but inside the Socratic Students framework.

🔧 Technique: Questions to documentation → replacing the teacher

If you have a long document (a technical API, a manual, a contract), you can use it as the teacher:

I have a document: [paste the text/link]

You are a student studying the document through questions.

PROCESS:
(A) What I've understood from the document so far
(B) What's unclear or missing
(C) Question: [precise question to the document]

After the question, you (as the expert) find the answer ONLY in the document and quote it.

TASK: Understand {specific aspect of the document}

5 rounds. At the end — a summary of your understanding.

Start as the Student.

Effect: Instead of "read and explain the whole document" (cognitive overload), the model extracts knowledge iteratively through targeted questions. It's active reading through dialogue.

🔧 Technique: Assessment as a fork → two parallel paths

After the mid-assessment, you can continue both attempts in parallel:

[basic template with mid-assessment]

AFTER THE MID-ASSESSMENT:
You gave two attempts. I told you what's right/wrong in each.

NOW:
Continue BOTH branches in parallel for 2 more rounds:

BRANCH A (attempt 1): Student asks a question → Expert answers
BRANCH B (attempt 2): Student asks a question → Expert answers

FINAL: Compare the final solutions from both branches. Pick the best one or
combine them.

Effect: Tree-of-thought inside Socratic Students. Two hypotheses are tested in parallel, and the final solution is a synthesis.

Socratic Students: Teaching Language Models to Learn by Asking Questions
Rajeev Bhatt Ambati, Tianyi Niu, Aashu Singh, Shlok Mishra, Shashank Srivastava, Snigdha Chaturvedi
UNC Chapel Hill, Meta