SkillNEO: Evolving Agent Skills, Where a Lower-Scoring Version Can Become the Base of a Better One

STEP 0: Original skill = root of the version tree. 3 agent runs → scored by the checker (verifier)
LOOP (up to 5 generations):
  STEP 1: ANALYST compares saved logs (runs of one version against each other, parent↔child, “siblings”)
         → what works (keep) + what is unsolved (fix) + an edit proposal backed by evidence
  STEP 2: EDITOR applies the edit to the whole package (instructions + scripts + resources) → new child version
  STEP 3: FILE CHECK: file references intact, scripts in place, contents valid (otherwise → back to the editor)
  STEP 4: RUNS: 6 children → each runs once → the top 3 run twice more
  STEP 5: TWO SEPARATE DECISIONS:
         • production: replace the active version only if the child beats it on the mean of 3 runs
         • refinement: the current generation’s children, even if weaker than their parents
  STOP: two generations in a row without replacing the active version (HOLD) or 5 generations
  REVISIT: after HOLD you can go back to an older version and refine it using the new logs

This is a loop of separate runs driven by orchestrator code. You can’t reproduce all of it in a regular chat. But the principles of analysis (step 1) and editing (step 2) carry over into prompts.

Where the method shines: you have an agent skill, a recurring routine task and a checkable result (the file reconciles or it doesn’t). Where it struggles: subjective quality like “looks good” or “sounds convincing.”

Task: an Amazon seller set up a Claude Code skill that reconciles the Amazon sales report against stock levels exported from QuickBooks. Version v1 reconciled in 1 run out of 3. In v2 a script for parsing the xlsx was added, and the result dropped to 0 out of 3. But the logs show the file is now read correctly; what fails is assembling the final table. You can’t throw v2 away, it’s half the fix. You feed the logs of both versions to the analyst.

Prompt:

<Role>Skill edit analyst. You don’t edit the skill; you find the causes of failures and propose one targeted edit.</Role>

<Input>
Parent skill v1: [paste SKILL.md and the v1 file list]
Child skill v2: [paste SKILL.md v2 and the parse_amazon.py script]
Logs of 3 v1 runs: [paste]
Logs of 3 v2 runs: [paste]
Success criterion: the final reconciliation table matches the Amazon report for every SKU, discrepancies exported to discrepancies.xlsx
Directions that already made things worse: [none]
</Input>

<Analysis>
1. Compare runs of the same version against each other: what is stable, what is random.
2. Compare v1 and v2: which change in the agent’s behavior is tied to which edit.
3. For each failed run, determine whether the agent never opened the skill, or opened it and made a mistake. In the first case, propose an edit to the skill description (when to use it). In the second, an edit to the procedure or the files.
4. List the suspected causes. Rank them by the number of failed runs each one explains. Check every cause against the successful runs.
5. Errors the agent recovered from on its own are not grounds for an edit.
</Analysis>

<Output>
- WORKS (keep): steps that are executed correctly, with evidence from the logs
- UNSOLVED: remaining failure causes, by rank
- EDIT: which parent version, what exactly to change, which runs show it
</Output>

<Constraints>
Do not propose directions from the “already made things worse” list.
One edit at a time. If it needs a new file, also specify the line in SKILL.md that calls it.
</Constraints>

Result: the analyst returns three blocks. The first lists what already works in v2 (most likely the xlsx parsing) and should be kept. The second gives the ranked causes of the remaining failures, with references to specific runs. The third is one targeted edit with the parent named. In your next message you hand the edit to the editor (template below), then run the agent 3 more times and compare the averages.

The weakness: the final score is a blunt metric. It doesn’t show which steps in the procedure are already right, and it cuts off versions where only part of the problem is fixed. If you demand improvement immediately, a chain of several linked edits snaps at the first one.

The LLM’s strength: a model is good at reading and cross-referencing long texts. Give it two logs of the same task and it will spot the step where they diverged. That tells you far more than the number “0.33.”

How the method exploits this: it stores versions together with their logs. That way every version, even a weak one, has both a starting point for an edit and evidence of what to fix. Two tricks from the paper are useful on their own: - The agent didn’t open the skill vs. opened it and messed up. These are different diseases. In the first case you fix the description so the agent knows when the skill is needed. In the second, you fix the procedure itself. - Ranking causes by the number of failures they explain, then checking them against successful runs. That way you don’t “fix” something that looks exactly the same in the successful runs.

Control knobs: - Runs per version (3 in the paper) → fewer for cheap tasks, more for “noisy” ones. - Children per generation (6) and top candidates for re-checking (3) → wider search or lower cost. - Stopping rule (2 in a row with no improvement, max 5 generations) → tune to your budget. - The “already made it worse” list in the analyst → keeps it from going in circles. - The criterion for replacing the production version → a strict one (only if the mean score is higher) protects against lucky flukes.

The paper puts the full prompt text in Appendix B, which isn’t in the part we reviewed. Below is a template assembled from the Analyst (Proposer) and Editor (Skill Editor) instructions described in Section 3.3. The analyst in the template is the same as in the example above, just with placeholders.

<Role>Skill edit analyst. You don’t edit the skill; you find the causes of failures and propose one targeted edit.</Role>

<Input>
Parent skill: {parent_skill}
Child skill (if any): {child_skill}
Parent run logs: {parent_logs}
Child run logs: {child_logs}
Success criterion: {success_criterion}
Directions that already made results worse: {failed_directions}
</Input>

<Analysis>
1. Compare runs of the same version against each other: what is stable, what is random.
2. Compare parent and child: which change in behavior is tied to which edit.
3. For each failure: did the agent not open the skill, or open it and make a mistake? The first → edit the skill description. The second → edit the procedure and files.
4. Rank causes by the number of failed runs they explain. Check each one against the successful runs.
5. Errors the agent recovered from on its own are not grounds for an edit.
</Analysis>

<Output>
WORKS (keep): ...
UNSOLVED (by rank): ...
EDIT: parent, what to change, evidence from the logs
</Output>

<Constraints>
Do not repeat directions from {failed_directions}.
One edit at a time.
</Constraints>

Second step, in a separate message or a different session:

<Role>Skill editor. You turn a proposed edit into a complete new version of the skill.</Role>

<Input>
Parent skill (in full): {parent_skill}
Proposed edit: {analyst_edit}
Another version you can borrow a procedure from (optional): {donor}
</Input>

<Rules>
1. The parent stays untouched. Output the new version in full.
2. Keep everything unrelated to the edit exactly as it is.
3. If the edit requires linked changes (a script + the line in SKILL.md that calls it), make them together.
4. You may rewrite, add, delete or reorganize files. You may carry over a procedure from the donor.
5. Before output, check: every referenced file exists, scripts are in place, the SKILL.md structure is intact.
</Rules>

🚀 Quick start — paste this into the chat:

Here is the SkillNEO template (edit analyst + skill editor). Adapt it to my task: [describe the task and the skill]. 
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what your success criterion looks like and how you check the result, which skill versions and logs you have, and which edits you’ve already tried. These fields matter because the method is built on comparing logs and on a list of failed directions. The model will take the structure from the template and fit it to your skill.

⚠️ You need an automatic checker: in the paper, every task has a verifier that outputs a score. Without a clear “reconciles / doesn’t reconcile” criterion, there’s nothing to compare versions by. The method doesn’t fit subjective results.

⚠️ You need code: the full loop is an orchestrator with a version tree, dozens of agent runs and automated file checks. In a regular chat you can only reproduce the log analysis and the editing, not the version selection itself.

⚠️ Development and final evaluation on the same tasks: the skills are optimized for specific tasks and inputs. The authors didn’t test transfer to new data. Part of the gain may be overfitting to the test.

⚠️ Only some skills improved: out of 86 tasks, modified skills came out for 41; the other 45 stayed as they were. For strong models the overall gain is noticeably more modest than for the weak one, and on some tasks results got worse.

⚠️ A skill doesn’t guarantee requirements are met: the paper has examples where, even after refinement, the agent never checked a critical condition. For instance, it didn’t verify that the emergency mode was ever triggered at all.

⚠️ One model ran the entire evolution: all roles (executor, analyst, editor) were played by GPT-5.4 Mini. What happens if those roles go to other models wasn’t tested. The appendix isn’t in the available text, so the full prompts and exact stopping rules aren’t visible.

SkillNEO: Non-Elitist Evolutionary Optimization of Agent Skills (preprint). Authors: Enze Ma, Chunyu Miao, Yankai Chen, Jie Yang, Liancheng Fang, Kening Zheng, Hanrong Zhang, Muyao Xu, Xinle Yu, Chengze Li, Wei-Chieh Huang, Ziwei Ren, Yunhao Du, Zhixiang Shen, Xue Liu, Philip S. Yu, Zhen Wang. Affiliations: University of Illinois at Chicago, Mohamed bin Zayed University of Artificial Intelligence, UC San Diego, UCLA, University of Texas at Arlington.

Benchmark: SkillsBench 1.1 (Li et al., 2026), 86 professional tasks across eight domains. Published trajectories: github.com/benchflow-ai/skillsbench-trajectories. Related methods: EvoSkill, MetaSkill-Evolve, BONSAI, DIVE, GEPA, Trace2Skill, SkillAudit.