TL;DR
Researchers looked at what happens when the files inside an agent skill (a folder with SKILL.md, scripts and configs) contradict each other. Say the docs promise sorting and the script doesn't sort. Or one file pins library version 6 while another says 7.9. They found contradictions like these in two thirds of skills, picked 49 real cases and ran agents in three modes: no skill, the skill as is, and the skill with the contradiction minimally fixed.
The headline finding: the number of contradictions you find says almost nothing about the actual damage. The agent reads the skill in nearly 100% of runs, so it always runs into the bug. What differs is what it does next. A weak model takes the broken instruction and dutifully follows it all the way to a broken result. A strong one notices the mismatch and works around it or fixes it on the fly. As a result, fixing the skill helped one model a lot (Claude Haiku 4.5) and barely moved the needle for another (DeepSeek-V4-Flash).
The method boils down to this: audit the skill across three "boundaries". The first is environment vs. skill, the second is document vs. document, the third is document vs. code. Then make a minimal fix to just the conflicting spot. Defects that sit on the execution path get priority. The weaker the model, the more skill hygiene matters.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Read the whole skill (SKILL.md + scripts + configs) → a map: what's in it and what references what
STEP 2: Hunt for contradictions across three boundaries → list of findings
• Environment ↔ skill (versions, dependencies, paths, permissions)
• Document ↔ document (two parts of the description say different things)
• Document ↔ code (one thing promised, another delivered)
STEP 3: Drop the trivia (code does more than documented — usually harmless) → confirmed findings
STEP 4: Rank by reachability: will this spot be hit in a normal run? → priority
STEP 5: Minimally fix one spot (don't rewrite the skill) → "fixed" skill
STEP 6: Compare task results: no skill / original / fixed → decide whether the fix was worth it
Steps 1–4 fit into a single chat or a single agent session. Step 6 takes several runs.
Example in Action
Task: You run a small online store selling on Amazon and Etsy. You grabbed a Claude Code skill from a public repo: "weekly revenue report from marketplace exports". On it, a cheap model keeps producing reports with the weeks in the wrong order and choking on the pandas install. Before blaming the model, you check the skill itself.
Prompt:
You are an auditor of agent skills. Below is the "weekly-revenue-report" skill folder:
SKILL.md, README.md, scripts/build_report.py, requirements.txt.
Find contradictions across three boundaries:
1. Environment ↔ skill: Python/library versions, paths, permissions, external services.
2. Document ↔ document: SKILL.md, README and comments say different things.
3. Document ↔ code: the docs promise something the code doesn't do
(over-promised), or the code does the opposite of what's written.
If the code does more than documented — skip it, that's harmless.
For each finding: type, a quote from each file with line numbers,
why it's a conflict, will this spot be hit in a normal run
(yes / no / not sure), how to fix it minimally.
At the end — top 3 findings by likelihood of breaking the result.
[paste file contents]
Result: The model returns a structured list of findings. Each one gets a type (environment, doc-doc, doc-code), two quotes side by side, a "will it be hit in a run" call and a minimal-fix option. At the end comes a short prioritized top 3. Expect version conflicts there, plus spots where the text promises sorting or validation and the script has none. You'll have to toss some findings by hand as false positives.
Why It Works
The weakness: An agent reads a skill as an instruction, not as an argument. If the text says "sort by date", a weak model will do exactly that, even when the script sitting right next to it can't. It doesn't cross-check promises against reality; it just executes the text. So a bug in the skill becomes a bug in the output.
The strength: LLMs are good at spotting discrepancies between two pieces of text if you put them side by side and ask point-blank. Matching a README quote against code is easier for them than catching the bug "live" during execution. That's why an audit before the run beats hoping the agent will sort it out on its own.
How the method exploits this: You move the check from execution time to prep time. The three boundaries give the model a clear frame instead of a vague "find the bugs". The "will it be hit in a run" filter keeps you from drowning in a hundred harmless nitpicks. The paper shows that ranking by reachability matters more than the raw number of findings.
Control levers: - Agent model. Weak or cheap model: fixing the skill is a must. Strong one: you can test without fixes first. - The "harmless" filter. The original drops "code does more than documented". For skills that humans will read later, you can remove this filter. - Fix mode. "Minimal fix" keeps the skill stable. "Rewrite it all" risks introducing new discrepancies. - Check depth. Ask for "not sure" instead of a verdict, and the model won't invent conflicts.
Prompt Template
The paper doesn't include ready-made prompt text: the SkillAudit auditor is code with several LLM checks. The template below is built from its description and the paper's taxonomy, so it's a reconstruction.
<Role>
You are an auditor of skills for LLM agents. Your job is to find places where parts
of a skill contradict each other and assess whether they will break execution.
</Role>
<Input>
Skill: {skill_name}
Files:
{file_contents_with_names_and_line_numbers}
</Input>
<Process>
Step 1. Skill map. List the files, what each one does, and what it references.
Step 2. Search across three boundaries:
A. Environment: skill ↔ environment ({runtime_environment}). Versions, dependencies, paths, permissions.
B. Inter-Document: document ↔ document. Different versions, steps, constraints
in SKILL.md, README, comments.
C. Document-Code: document ↔ code. Flag two kinds:
• Over-promised: the docs promise it, the code doesn't do it
• Semantically-conflicting: both describe the behavior, but differently
Under-promised (code does more than documented) — don't include.
Step 3. Verify each finding: make sure both quotes exist and actually
conflict. Mark anything doubtful as "not sure"; don't make things up.
Step 4. Reachability: for each finding, say whether this spot will be hit
in a normal run of the task "{typical_task}": yes / no / not sure.
</Process>
<Output>
For each finding:
- Type (A / B / C-over / C-conflict)
- Quote 1 (file:line) and quote 2 (file:line)
- Why it's a conflict
- Reachability: yes / no / not sure
- Minimal fix: what to change in ONE place. Don't rewrite anything else.
At the end: top 3 findings by likelihood of breaking the result.
</Output>
What to fill in:
- {skill_name} — the folder or skill name.
- {file_contents_with_names_and_line_numbers} — the file texts. Line numbers let you verify the quotes.
- {runtime_environment} — for example, "Python 3.11, Linux, no internet".
- {typical_task} — what users usually ask this skill to do.
🚀 Quick start — paste into the chat:
Here's a skill-audit template. Adapt it to my skill: [describe what the skill does and what environment it runs in].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask about the runtime environment, the typical task and which files are included. You can't judge reachability without knowing what will actually run. It will take the structure from the template (three boundaries, the "harmless" filter, reachability, minimal fix) and assemble a ready prompt for your skill.
Limitations
⚠️ Experiment scale: The impact of fixes was measured on 49 tasks and three small models, with one target defect per task. Real skills contain many defects at once, and how they interact wasn't studied.
⚠️ No flagships: They tested Claude Haiku 4.5, Qwen3.6-35B (4-bit quantized) and DeepSeek-V4-Flash with reasoning off. On the strongest models the picture may be even milder, but the paper didn't show that. The authors ask for caution with head-to-head comparisons between models.
⚠️ The effect isn't uniform: For one model the fix gave a noticeable boost, for another almost nothing. There's no universal "fix the skill, get X% better".
⚠️ The audit itself is done by an LLM: Automated contradiction search produced false positives, so the authors confirmed findings by hand. Eyeball the auditor prompt's output.
⚠️ No ready-made prompt: The paper gives a taxonomy and a pipeline description, but not the prompt text. The template above is a reconstruction. You'll have to test its effectiveness yourself.
⚠️ Text was truncated: The version we received is missing some metrics (token costs, steps). Conclusions about execution efficiency aren't covered here.
How They Studied It
The question was simple: if a skill has a contradiction, is that bad? The researchers collected 575 skills: 488 from popular GitHub repos and 87 from the SkillBench test set. First they built a taxonomy of contradictions: environment, doc-doc, doc-code. Then an automated tool flagged candidates, and two authors manually confirmed 1182 real inconsistencies. They show up in 67% of GitHub skills and 64% of SkillBench skills. Even hand-written reference skills are no cleaner. SkillBench has the most "docs promise, code doesn't deliver" cases (45.6% of all findings).
Then came the main experiment. They turned 49 findings into executable tasks in Docker, with results checked by a program, no LLM judge. For each one they built a pair: the original skill and a minimally fixed one. Every task ran in three modes (no skill, original, fixed) on three models, three repeats each. That gave 441 trajectory pairs to analyze.
Here's the surprise. You'd think models gain differently because some read the skill and others don't. But the skill was read in 97% of runs. The difference shows up after contact with the bug. Haiku followed the bug into failure in 30% of runs, Qwen in 24%, DeepSeek in 12%. And DeepSeek recovered in 56% of cases where it noticed the bug. The practical takeaway: counting defects in a skill is pointless. Look at which model the skill runs on and whether the defect sits on the real execution path.
Original from the Paper
Contradiction taxonomy (layer definitions verbatim):
We first classify inconsistencies by the participants of the violated contract:
• Environmental layer: Skill vs. runtime environment.
• Inter-document layer: conflicts among descriptive documents.
• Document-code layer: documentation vs. implementation.
Within the Document-code layer, we further classify inconsistencies by the direction of deviation:
• Over-promised (Doc > Code). The documentation promises functionality, constraints, or behavior that the implementation does not provide.
• Semantically-conflicting (Doc ≠ Code). The documentation and implementation both specify the relevant behavior but contradict each other semantically.
• Under-promised (Doc < Code). The implementation provides capabilities that are not documented.
Context: This is the framework the SkillAudit tool used to find issues. The authors dropped under-promised cases by default as harmless.
Adaptations and Extrapolations
💡 Adaptation: minimal fix of a contradiction
The paper describes a repair rule: the fix is surgical, and the solution doesn't "leak" the task's answer. In a chat, this becomes a separate prompt for a second pass:
Here's a finding from the audit: {finding}.
Decide which of the two fragments is authoritative (go by what
actually works in {runtime_environment}). Fix ONLY this spot:
align the description or the code. If the promise can't be kept in our environment,
narrow the documentation instead of writing new code. Don't change anything else.
Show the diff.
🔧 Technique: add a "stop rule" to the instructions file → less blind following (extrapolation, not tested in the paper)
The paper showed that weak models blindly follow a broken instruction. It makes sense to try adding this to CLAUDE.md or AGENTS.md:
If the skill's instructions disagree with the actual behavior of the scripts, errors or
command output — don't keep following the instructions.
Name the discrepancy in one line and choose the option that is confirmed
by real output.
This is a hypothesis inspired by the finding that "strong models notice and work around bugs on their own". Test how well this phrase works on your own tasks.
Resources
- Paper title: Why Static Skill Inconsistencies Do Not Equate to Runtime Risks: A Behavioral Analysis of LLM Agent Execution
- Authors: Fengrui Fan, Huahao Wang, Jingyi Fang, Yutao Hu, Yueming Wu, Deqing Zou
- Affiliation: Huazhong University of Science and Technology, China
- Mentioned in the paper: SkillBench (a skill set with reference tasks), SkillAudit (a static audit tool), WildSkills (a set of 49 verifiable tasks)
- Models tested: Claude Haiku 4.5, Qwen3.6-35B-A3B, DeepSeek-V4-Flash