Agent Skills: when a plugged-in skill helps an agent and when it gets in the way

Task: An Amazon seller asks an agent (Claude Code) to build a weekly report from three exports: sales, returns, ads. Five skills are connected. If you just write “use the skills,” the agent will jumble the order of steps, won’t re-check the numbers after an edit, and will break the final file.

Prompt:

Task: build a weekly report for the owner of an Amazon store
from the exports sales.csv, returns.csv, ads.csv. Output: report.xlsx and report.pdf.

Connected skills: csv-cleaner, xlsx-builder, chart-maker, pdf-report, number-check.
Use them according to the plan below.

STAGES AND GOALS
Stage 1. Data prep. Goal: tables with no duplicates and no empty SKUs. Skill: csv-cleaner.
Stage 2. Workbook assembly. Goal: report.xlsx with the sheets “Summary,” “Returns,” “Ads.” Skill: xlsx-builder.
Stage 3. Charts. Goal: PNG charts of revenue and returns by day. Skill: chart-maker.
Stage 4. PDF. Goal: a one-page summary for the owner. Skill: pdf-report.
Stage 5. Verification. Goal: the numbers in the xlsx and pdf match the source exports. Skill: number-check.

ARTIFACT DEPENDENCIES
clean_sales.csv, clean_returns.csv, clean_ads.csv ← csv-cleaner ← source exports
report.xlsx ← xlsx-builder ← clean_*.csv
charts/*.png ← chart-maker ← report.xlsx (“Summary” sheet)
report.pdf ← pdf-report ← report.xlsx + charts/*.png
check.md ← number-check ← report.xlsx, report.pdf, source exports

RULES
1. If a skill’s procedure doesn’t fit my data, you may change the method, but you may not
   change the result criteria: the numbers must match the exports.
2. If you fix any artifact, re-check every artifact that depends on it.
3. At the end, show in check.md which files you regenerated and why.

Result: The agent works through the five stages and creates the files in the specified order. If the verification turns up a discrepancy, it goes back to the source artifact and rebuilds the dependent files. At the end there’s a check.md listing what was regenerated. Without a dependency map, this chain of fixes usually gets lost.

The weakness. Searching for a skill by its description compares the topic, not the set of actions. A “working with Excel” skill matches any task involving Excel, but it may not be able to do the operation you need. Meanwhile the agent dutifully follows the procedure, even if it fits poorly with its model + harness combo. A weak model in an awkward harness gets bogged down in the skill’s steps and loses sight of the goal.

The strength. A model is good at breaking text into concrete actions and matching two lists. It holds a plan well when the plan is explicit, and it passes results along through files: intermediate artifacts make it possible to check the source and propagate fixes to dependent steps.

How the method exploits this. Breaking things into operations turns a fuzzy “on topic” into a verifiable “covers 4 of the 5 actions needed.” A Stage Plan sets the goal of each step, so the agent understands why it’s running the procedure and doesn’t lose the requirements. A DAG shows what depends on what, so after an edit the agent knows what to re-check.

Control knobs: - Depth of organization. For 3–4 skills, stages with goals are enough. A dependency map pays off at 5–6 skills. - Permission to change the method. The rule “you may change the method, not the result criteria” gives flexibility while keeping the agent from bending the result to fit a convenient procedure. - Re-check rule. Spell out which changes require re-verifying the final file. - Number of candidates. Compare 3–5 skills, not one.

Part A. Choosing a skill among candidates:

I’m choosing a skill for an agent. Compare the candidates not by topic but by operation coverage.

TASK:
{task_description_and_result_requirements}

CANDIDATES (full SKILL.md):
=== Skill 1: {name_1} ===
{skill_text_1}
=== Skill 2: {name_2} ===
{skill_text_2}
=== Skill 3: {name_3} ===
{skill_text_3}

Steps:
1. Break the task into atomic operations. An atomic operation is a basic action
   on a specific object with a verifiable result
   (e.g., “remove duplicates in sales.csv by SKU”).
2. For each skill, list the operations its procedure actually describes
   or supports with scripts.
3. Build a table: task operation × skill (covered / partially / not covered).
4. Flag operations where the skill’s procedure might get in the way
   (requires tools or conditions my agent doesn’t have: {my_agent_and_tools}).
5. Rank the candidates by coverage. Show how your ranking differs
   from a “by topic” ranking, and name the top candidate.

Part B. Organizing a set of skills (Stage Plan + Dependency DAG):

Task: {task_and_final_result}

Connected skills: {skill_list}. Use them according to the plan below.

STAGES AND GOALS
Stage 1. {stage_name}. Goal: {expected_output}. Skill: {skill}.
Stage 2. {stage_name}. Goal: {expected_output}. Skill: {skill}.
Stage N. ...

ARTIFACT DEPENDENCIES
{output_file} ← {skill} ← {input_files}
{output_file} ← {skill} ← {input_files}

RULES
1. If a skill’s procedure doesn’t fit my data, you may change the method,
   but you may not change the result criteria: {acceptance_criteria}.
2. If you fix any artifact, re-check every artifact that depends on it.
3. At the end, list what you regenerated and why.

Fill in: the task description with acceptance criteria, the skill texts, the agent’s tool list, the stages, and the file names.

🚀 Quick start — paste into a chat:

Here is a template for choosing a skill and organizing a set of skills. Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about the final result and acceptance criteria, the list of skills and what’s in each, and your agent and its tools. It needs this to break the task into operations and the skill set into stages and file dependencies. It will use the template as a structural model.

🔧 Technique: add an “Adaptation” block to SKILL.md → the agent doesn’t lose the requirements when it changes the method

The authors’ practices in the paper include two points: keep well-founded acceptance criteria when the method changes, and state which changes require re-verifying the final file. The available part of the text has no ready-made wording, so here’s my own version:

Copy

## If the procedure doesn’t fit
The method may be replaced. The result criteria may not be changed:
- {criterion_1}
- {criterion_2}
After replacing the method or editing an intermediate file, re-verify the final file: {what_to_check}.

💡 Adaptation for CLAUDE.md / AGENTS.md: move the “stages and artifact dependencies” into the project instructions file. The agent will get the dependency map every time, not just the skill descriptions.

⚠️ The paper’s text was incomplete: this breakdown relies on the abstract, introduction, related work, and methodology. There’s no results section, no full list of the authors’ 17 practices, and no verbatim wording of the organization instructions. The templates above are assembled from the description, not taken from the paper.

⚠️ The re-ranking effect is small: choosing by operation support raised the top pick by 4–6 percentage points. With 23 tasks, 4.35 points is exactly one task. The direction is clear, but treat the size of the effect as approximate.

⚠️ The authors’ practices weren’t tested by intervention: the 17 practices were derived from comparing real skills and runs. The authors themselves call them “a starting point for testing,” not proven rules.

⚠️ A DAG isn’t always needed: its added value over a Stage Plan is concentrated in tasks with five or six skills. For two or three, stages with goals are enough, and extra markup just bloats the prompt.

⚠️ One benchmark: SkillsBench (87 tasks, three models, three agent harnesses, three runs each), with automatically graded tasks. Subjective tasks (writing, creative work) may behave differently. Also, the agent was explicitly told “use the skills.” In real work it may never reach for a skill at all.

The team took 87 tasks from SkillsBench with automatic checks. They came with 232 skills. Each task was run with skills and without, three times each, in a clean environment. That way they compared three models and three agent harnesses, nine combos in total. An unpleasant picture emerged: on 36.78% of tasks, the same skills helped in one combo and hurt in another. Analyzing the trajectories (step-by-step logs of the agent’s actions) showed that the procedure sometimes becomes a burden.

Then, from a corpus of 37,596 skills (filtered down from 124,000 marketplace entries), they pulled the five most relevant alternatives for each of 23 single-skill tasks. Relevance was ranked in two stages: text and embedding search, then LLM selection. They ran every candidate and re-ranked them by how well their procedures cover the task’s atomic operations. That improved the top pick by 4.35–5.80 points. A lower-ranked candidate beat the top one in 23.19% of cases.

For skill sets (41 tasks, three or more skills), they took the same set and changed only the organization: none, an order, stages, artifact dependencies. Stages and the DAG won. The logs explained why: intermediate files make it possible to check the source and propagate fixes down the chain. The log analysis was done by an LLM, and two of the authors verified the conclusions by hand.

The surprise: how often a lower-ranked candidate wins. The practical takeaway: look at operation coverage, not rank.

• Paper: An Empirical Study of Agent Skills' Downstream Utility
• Authors: Yu Cheng, Dehai Zhao, Zhongxin Liu, Xiaoxue Ren (Zhejiang University); Qing Huang (Jiangxi Normal University); Zhenchang Xing (CSIRO's Data61)
• Benchmark: SkillsBench
• Skill sources: claude-plugins.dev, ClawHub
• Related work mentioned in the text: SkillRouter, SkillRet, SkVM, SkillDAG, AgentSkillOS, Graph of Skills