Socratic Skill Learning: a teacher coaches the agent mid-practice, but the agent writes the skill itself

STEP 0: Define the "family spec" → what stays fixed, what may vary (text, once)
STEP 1: Student works on the anchor task → drives the environment, writes the SKILL.md draft
STEP 2: Every ~10 actions the Teacher reviews the latest actions + current skill
        → lesson (explain / challenge / diagnose / check) OR silence
STEP 3: Student responds → argues, tests, edits the skill, records evidence
STEP 4: End of round → a fresh session solves the anchor task using only the skill → result goes back
STEP 5: 2–3 rounds → the final skill is tested on new variations of the task

Issue-Guided variant: the Teacher is called not on a timer, but when an open question
is opened, changes, or is reopened (an issue tree).

Steps 1–3 run in two long-lived sessions (Student and Teacher). The check in step 4 is always a new session.

Task: An Amazon seller reconciles the weekly settlement report against their QuickBooks books every week. Mismatches come from returns, fees, shipping charges, and adjustments. Right now a bookkeeper does it by hand. You need a Claude Code skill that works on next week's report and on a different seller account too. It must not overfit to a single export.

Prompt (for the Teacher, a separate chat with a strong model, every ~10 agent actions):

<Role>
You are the Teacher. You observe a Student who is building a skill for an agent.
You have read-only access. You do not write the skill and you do not solve the task for the Student.
</Role>

<FamilySpec>
Goal: reconcile the weekly Amazon settlement report with the QuickBooks export, list discrepancies with their causes.
Fixed: the final payout amount must match to the cent; every discrepancy has a cause or is marked "unexplained".
May vary: the week, number of SKUs, presence of returns, fees and reimbursements, date format, CSV encoding, column names.
</FamilySpec>

<Input>
Below are the Student's latest actions and the current SKILL.md:
{latest_actions}
{current_skill}
Open questions: {open_questions}
</Input>

<Actions>
Pick one: EXPLAIN / CHALLENGE / DIAGNOSE / SUGGEST_CHECK / STAY_SILENT.
</Actions>

<LessonFormat>
If you give a lesson, specify:
- learning_focus: what to investigate;
- acceptance_criterion: what observable result will close the question.
</LessonFormat>

<Rules>
- Do not provide ready-made skill text.
- Look for assumptions that hold only for this export (for example, "there are never any returns").
- If you see no problems, answer STAY_SILENT.
- Do not reopen closed questions without new evidence.
</Rules>

Result: The Teacher will either stay silent or give a short lesson like "check that the skill doesn't break when there are no returns." The lesson comes with a closing criterion: exactly what the agent needs to see. The Student will test the advice on the data, fix the skill, and note what confirmed the edit or why the advice was rejected. At the end you get a SKILL.md with conditions of applicability, not a step-by-step log of one week. It's worth running it in a clean chat on a different week.

The weakness. An agent that solved one task tends to write down what it did, not when it applies. An assumption from a single run turns into a "rule." An expert writing the skill alone can't see where the procedure actually breaks. A post-mortem fixes text that has already set, not the testing process.

The strength. A strong model is good at spotting weak points in someone else's reasoning: a fishy assumption, a skipped check. An executor inside the environment is good at quickly testing a hypothesis on real data. The evidence is produced during the process, and it can be written into the skill.

How the method exploits this. The Teacher asks a question, the Student answers with an experiment, and the skill stores the result of that experiment. That's why the text keeps the distinction between "not tested" and "disproven," plus the limits of applicability. In an example from the paper, a generic arithmetic check flagged four metrics, while a check of financial ratios, which the Student arrived at after a hint, found the one real error.

Levers: - Review frequency (once every N actions): less often is cheaper but gives worse results. In the paper, an interval of 5 costs more and isn't better; an interval of 20 is cheaper but weaker. - The right to stay silent: take it away and the Teacher starts butting in for no reason. As training went on, the share of reviews with a lesson dropped from roughly three quarters to about one sixth. - The question-closing criterion: defines what counts as evidence. A stricter criterion means a more reliable skill but longer work. - The family spec: the more precisely you describe what varies, the better the Student probes the boundaries. - Number of rounds: two or three in the paper. Each round ends with a test in a fresh session.

Two prompts: one for the Student (the agent doing the work) and one for the Teacher (a separate chat).

Student prompt (the agent's system prompt):

<Role>
You are the Student. You are the only one who works in the environment and writes the skill package
(SKILL.md and, if needed, scripts). The Teacher only gives advice.
You can ask it questions, test its advice, adapt it, or reject it.
</Role>

<FamilySpec>
Goal: {task_family_goal}
Fixed: {what_is_always_true}
May vary: {allowed_variations}
</FamilySpec>

<Task>
Anchor task for practice: {anchor_task}
</Task>

<LearningLogic>
1. Explore the anchor task. Decide for yourself what evidence to collect.
2. Do not write an assumption into the skill until you have tested it on data or assessed its limits.
3. When a Teacher lesson arrives: figure out what it teaches, test it in practice, then adapt or reject it.
4. Group file edits into revisions. For each one, specify:
   <Revision>
     <Summary>what changed</Summary>
     <Evidence>what confirms it</Evidence>
     <IssueAddressed>which question is closed, if any</IssueAddressed>
   </Revision>
5. Close a question only if: (a) a check passed after the lesson, or
   (b) you recorded the conditions under which the procedure applies, with evidence, or
   (c) you rejected the advice and explained why, with evidence.
6. In SKILL.md, write: when the procedure applies, which steps transfer, where adaptation is needed.
   Mark "not tested" and "disproven" separately.
</LearningLogic>

<Finish>
End the round when you believe the skill transfers to the variations in FamilySpec.
</Finish>

Teacher prompt (a separate chat):

<Role>
You are the Teacher. You review the Student's latest actions and the current skill.
Read-only access. You do not write the skill and you do not solve the task for the Student.
</Role>

<FamilySpec>
Goal: {task_family_goal}
Fixed: {what_is_always_true}
May vary: {allowed_variations}
</FamilySpec>

<Input>
Student's latest actions: {latest_actions}
Current SKILL.md: {current_skill}
Open questions: {open_questions}
Result of the last test in a clean session: {test_result}
</Input>

<Actions>
Pick one: EXPLAIN / CHALLENGE / DIAGNOSE / SUGGEST_CHECK / STAY_SILENT
</Actions>

<LessonFormat>
- learning_focus: what to investigate
- acceptance_criterion: what observable evidence will close the question
</LessonFormat>

<Rules>
- Do not provide ready-made skill text or a ready-made solution.
- Look for assumptions that hold only for the anchor task.
- If there are no problems, STAY_SILENT.
- Do not reopen closed questions without new evidence.
</Rules>

What to fill in: {task_family_goal} is the recurring task the skill solves. {what_is_always_true} and {allowed_variations} describe what stays fixed and what changes from case to case. {anchor_task} is one concrete task to practice on. The remaining fields you copy between chats: the agent's latest actions, the current skill, the open questions.

🚀 Quick start: paste this into a chat:

Here is the Socratic Skill Learning template (Student and Teacher prompts). Adapt it to my task: {describe the recurring task you need a skill for}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what in your task is always the same and what changes from case to case, because that spec is exactly what pushes the exploration beyond a single example. It will also ask about the anchor task and how you'll know the result is correct. Then it will assemble two ready-to-use prompts.

⚠️ The original is orchestrated in code: in the study, the every-N-actions review and the open-question tree are automated. You can reproduce the idea by hand (two chats, copying context), but the paper doesn't include the prompts themselves; they had to be assembled from the method description. The Issue-Guided version requires automation.

⚠️ You need a verifiable result: in the benchmark every task had hidden tests, and each round the skill was checked by a fresh session. If you have no way to tell whether the task is solved, the Student has nothing to close questions with.

⚠️ The Teacher was stronger than the Student: in the main experiments, the Teacher was a larger model from the same family. The paper didn't test what happens when they're evenly matched.

⚠️ The effect is moderate and uneven: there is a gain, but not a multiple. In domains where the model already does almost perfectly without a skill (office automation), there's no gain. Some methods even hurt in certain domains. For instance, expert authorship and post-mortem review dropped in finance and systems tasks.

⚠️ Cost: it's several rounds of practice plus reviews. Review tokens grow as context accumulates, and most late reviews end in silence anyway. Issue-Guided slashes the Teacher's spend, but for one of the models total spend went up because the Student worked more.

⚠️ Simply handing skill-writing to an expert doesn't help: if the Teacher just writes the skill itself, the benefit is close to zero. That doesn't necessarily mean expert-written skills are bad, but as a "strong model writes, weak model uses" strategy it didn't work.

The authors built the SkillTransferBench benchmark: 35 task families, 408 runnable cases, six domains (systems, services, data processing, scientific computing, finance, office). Each family has one "anchor" for practice and a public description of what stays fixed and what may vary. The remaining cases stay hidden for evaluation. The main sample is 22 families and 155 cases where the model without a skill isn't perfect. The skill was tested only in a fresh session: it has to help someone who never saw it being written.

They compared seven setups: no skill, expert writes the skill alone, post-mortem review, two external methods (Trace2Skill and SKILL-KD, which "distill" a skill from logs of past runs), and two SSL variants. Then they repeated the comparison on three model pairs. SSL with a fixed window scored about 80 out of 100 versus 72 with no skill and 76 for Trace2Skill. The share of fully solved tasks rose from a third to a half. Teacher lessons raised the probability that the Student immediately writes a file (by about 24 pp) and changes the skill within the next ten steps (by 39 pp).

The surprise: an expert writing the skill alone gave almost zero gain, and post-mortem review was weaker than interleaving on all three models, and even weaker than expert authorship. Asking and testing along the way mattered more than getting the "right text." Second finding: the need for lessons drops as training goes on (from 73% of reviews with a lesson in round one to 16% in round three), and 86% of late reviews end in silence. The version that calls the Teacher only for an open question delivered the same quality at about 62% lower Teacher cost.

Practical takeaway: don't ask a strong model to "write the skill for the agent." Let the agent test the hypotheses itself, and bring in the strong model as a sparring partner while the work is in progress.

💡 Context adaptation: a lightweight single-chat version: if you don't have two models, keep the role split but drop the automation. Ask the agent to write the skill as it works, and every 8–10 steps pause and paste an excerpt of its actions plus the draft into another chat with the Teacher prompt.

Work on the anchor task. After every {number_of_steps} steps, stop and show:
1) what you have already tested, 2) what you wrote into the skill, 3) which assumptions are still untested.
Wait for my reply before continuing.

🔧 Technique: add a closing criterion → get verifiability. Ask the Teacher to state in every lesson exactly what the Student must see for the question to count as closed. Then the agent doesn't just reply "noted"; it shows a run on data with no returns or on an empty export.

• Socratic Skill Learning: Interleaving Teaching and Practice for Transferable Agent Skills — Letian Zhu, Han Li, Yifan Yao et al., Nanjing University, Alibaba Group (corresponding author: Jiaheng Liu).
• Code, benchmark and trajectories: https://github.com/NJU-LINK/Socratic-Skill-Learning
• Works mentioned: Trace2Skill, SKILL-KD, SkillLearnBench, SkillsBench, Self-Refine, Reflexion.