⚠️ The paper's text cuts off in the middle of the STEWARD description. STEWARD's own results from the tables and figures aren't visible, only what's in the abstract and introduction. Everything about the diagnostics (models, tool selection, skills) is covered from the full text of sections 3–4.
TL;DR
An agent with a large tool catalog finds almost all the tools it needs, but drags along a pile of extra ones. It recalls nearly everything it needs while getting confused by similar and outdated options. The second finding: skills an agent accumulated on old tool versions stop helping over time, and in places do harm. The STEWARD method (an ecosystem manager) proposes keeping tools and skills as a linked pair and updating a skill whenever its tool changes.
The pain is familiar. A tool in your MCP setup gets updated: the old one stays, and next to it appears a new one that does the same job in three calls instead of eleven. Out of habit, the agent takes the old route. The task gets done, everything's green, but you're paying for the extra steps. Worse, a skill written for the old tool keeps “suggesting” the outdated path. The reason: task success hides missed opportunities. The agent doesn't compare its path with the best one; it just reaches the result the familiar way. When the researchers removed the extra and outdated tools, results improved noticeably. When they also handed over a ready-made action plan, there was no gain. The bottleneck is tool selection, not planning.
STEWARD works in three moves. It tracks the “skill ↔ tool” link. When a tool changes, it surgically updates the affected skills instead of regenerating everything from scratch. During a task, it gives the agent only the tools needed at that moment, along with their instructions. STEWARD itself is code, but its logic carries over to a manual process: audit your skills every time the tool set changes, and narrow the set to the task at hand.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint (STEWARD, based on the abstract and introduction)
STEP 1: Store skills together with links “this skill uses these tools”
and availability status → a skill library with dependencies
STEP 2: A tool changed/disappeared/got an equivalent → update ONLY the affected skills
(incrementally, without rebuilding everything) → an up-to-date library
STEP 3: During a task: show the agent a narrow tool set + relevant skills,
adjust the set along the way if something is clearly missing → a compact “view” of resources
This is code (a separate manager sitting on top of the agent). The steps run while the agent works, not in a single prompt.
Example in Practice
Task: You run marketing for an online store. In Claude Code you have MCP tools connected to HubSpot, Google Analytics, and Google Sheets. There's a weekly-leads-report skill the agent wrote for itself six months ago. The CRM connector's developers shipped an update. The old get_leads and get_lead_details are still there, but a new get_leads_with_details has appeared that does the same thing in a single call. The skill still works and the report still comes together. But you suspect the agent is firing off dozens of unnecessary calls and burning tokens.
Prompt:
Audit the weekly-leads-report skill after the tool update.
Here's the current skill:
[paste the contents of SKILL.md]
Here's the current list of HubSpot connector tools with descriptions:
[paste the list]
What changed in the connector:
- added get_leads_with_details (combines get_leads + get_lead_details)
- the date_from parameter in get_leads is now called created_from
Do the following in order:
1. Build a dependency map: which skill steps call which tools.
2. For each step, mark a status: “works as is” / “parameter outdated” /
“shorter path available” / “tool no longer exists”.
3. For steps with the “shorter path” status, count how many calls
there were and how many there will be.
4. Rewrite ONLY the affected steps. Leave the rest alone.
5. At the end, list the steps you couldn't verify without running them.
Result: The model will output a “skill step → tool → status” table. Next comes an estimate of call savings for the affected steps and rewritten versions of only those fragments. At the end you get a list of what needs to be verified with a real run. It won't propose replacing the whole skill, because item 4 explicitly forbids that.
Why It Works
The weakness. An agent picks tools by their descriptions. When the catalog has similar and outdated options and the descriptions are imprecise, it grabs what it needs plus extras. Skills written for an old version “hard-wire” the outdated path into the agent's habits. Nobody checks an old path that still works, so improvements go unnoticed.
The strength. Give the model exactly the tools that are current right now, and it uses them well. A ready-made plan on top added nothing. The model can plan; what it needs is a clean map of what's available. It's also good at comparing a skill's text against a tool's description and spotting mismatches.
How the method uses this. It doesn't rebuild knowledge; it links skills to tools and updates them surgically. Your levers: - Narrowing the set (only the tools for the task) → less confusion between similar options. - Banning “rewrite everything” → you keep the hard-won experience instead of losing it in a regeneration. - An explicit step status (“works / outdated / shorter path exists”) → you see where you're leaving value on the table, not just where things broke. - Audit frequency → run it after every connector update, not on a fixed schedule.
Prompt Template
The paper has no ready-made prompt: STEWARD is implemented in code. The template below is a manual transfer of the “link the skill to its tools and update surgically” logic. It's an adaptation of the principle, not a prompt from the paper.
Audit the skill {skill_name} after changes to the tools.
Skill:
{skill_text}
Current tools (names, parameters, descriptions):
{tool_list}
What changed:
{change_list}
In order:
1. Dependency map: which skill step calls which tools.
2. Status of each step: “works as is” / “parameter outdated” /
“shorter path available” / “tool no longer exists”.
3. For the “shorter path” steps, count the calls: before → after.
4. Rewrite only the affected steps; leave the rest alone.
5. A list of what can't be verified without running it.
Fill in: {skill_name} — the skill's name, {skill_text} — the file contents, {tool_list} — descriptions from your MCP server or connector, {change_list} — the update notes or whatever you noticed yourself.
🚀 Quick start — paste into the chat:
Here's a template for auditing a skill after tool changes. Adapt it to my situation: [describe which skills and tools you have].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask which tools are connected, what changed in them, and where your skills live. It needs this to build the “skill ↔ tool” map the method relies on. It'll take the pattern from the template and adapt it to your setup.
Limitations
⚠️ The main method is code: STEWARD is a separate manager on top of the agent. Without programming, you can only reproduce the idea: manual auditing and narrowing the set. The researchers didn't test this manual version.
⚠️ You don't have perfect data: in the diagnostics, “Oracle Tools” means the model was given only the correct, current tools in advance. In real work no such list exists; you have to put it together yourself.
⚠️ Synthetic environment: the services and tasks were generated, and the user is a model too. Small manual changes at your company may behave differently.
⚠️ Diagnostics on a single model: the controlled experiments and the comparison of skill methods ran on one base model. The overall picture is backed by a comparison of nine models, but details may differ.
⚠️ The text is truncated: STEWARD's results and the details of how it works aren't visible in the available excerpt. Don't draw conclusions about its exact gains.
How They Studied It
The researchers built 42 domains with “live” services (backed by a database and MCP interfaces) and 1,588 multi-step tasks. Then they changed the tool set six times without removing any capabilities: renamed and added parameters, split one tool into several, merged several into one. The old tool could either stay next to the new one, forcing the agent to choose, or disappear, forcing it to find a replacement. Some descriptions were deliberately made incomplete and outdated. For each version, dynamic programming computed the shortest path to the goal, and the agent's path was compared against it.
Nine strong models and agent frameworks reached no more than 42% success. The best one recalled about 90% of the needed tools, but its selection precision was around 20%: they find what they need but don't cut the extras. Next came diagnostics with escalating levels of help: a narrow domain, the task's tools, only current tools, a ready-made path. Success climbed up to the “only current tools” level. The ready-made path added nothing. So the problem isn't planning, it's choosing between similar and outdated options.
Then they compared four off-the-shelf skill-accumulation methods against an agent without skills. On the first version, all four helped. As versions changed, the advantage melted away. In one version, two methods dropped to zero on adapting to path degradation, while the skill-less agent held at about 10%. The strongest method turned out to be 2–3 times more expensive than the others in skill-maintenance costs. The practical takeaway: skills not linked to tools go stale, and the more you invest in improving them, the costlier they are to update.
Adaptations and Extrapolations
💡 Adaptation for the agent's instructions file: add a rule to CLAUDE.md or AGENTS.md that gets the agent back into the habit of checking against the current catalog. This is my extrapolation; the paper has no such rule.
Before applying any skill, check the tools it calls
against the current list of available tools.
If a tool from the skill is missing, renamed, or has a shorter
replacement, don't follow the skill blindly: complete the task the current way and at the end
report which step of the skill is outdated.
🔧 Technique: narrow the set → less confusion. If you can control which tools are connected, keep only the necessary ones for a given task or session. In the study, narrowing the tools down to the domain and to the task consistently improved success and efficiency. In a chat or a custom GPT, you do the same by listing them directly: “For this task, use only: A, B, C. Don't call the others.”
Resources
- Δτ-Bench: Pushing Agents Toward a Moving Capability Frontier (Preprint)
- Code: github.com/zjunlp/delta-tau-bench
- Dataset: huggingface.co/datasets/zjunlp/delta-tau-bench
- Authors: Shuofei Qiao, Xuanguang Pan, Yijun Chen, Wendi Feng, Bin Wu, Runnan Fang, Yuqi Zhu, Shumin Deng, Emine Yilmaz
- Universities: Zhejiang University, University College London
- Skill-accumulation methods compared: EvoSkill, AutoSkill, Trace2Skill, SkillOpt