TL;DR
A skill optimizer is automation that rewrites an agent's instructions (its skill, think CLAUDE.md). An editor model reads run logs, appends rules, and keeps whichever edits helped the agent solve more tasks on a small validation set. The researchers tested three such optimizers on long agentic tasks. The learned skills solve tasks no more often than one well-written hand-crafted starting skill (an average difference of 0.1 points), but they make the agent read roughly 16% more tokens. The main culprit: rules like "double-check" and "make one call at a time" turn into extra turns.
The gain an optimizer reports on its validation set is mostly an illusion. The set is tiny (up to 30 tasks), and agent success is noisy: the same skill solves a task one run and fails it the next. Pick the best of 4–8 candidates and you will "find" a gain even among candidates identical to the original. On fresh tasks it evaporates. Second problem: a solved/unsolved score hides the price. Every agent turn resends the whole conversation to the model, so an extra turn costs far more than an extra line of instructions. Third: the editor appends rules in batches (37 per run on average, up to 105), and nobody checks each rule on its own. Of 337 added rules, not one asked the agent to read less, and half asked it to verify something.
The authors propose four principles: accept an edit only if its gain beats what selection alone produces; price a rule by the turns it adds, not by its length; let the editor delete and merge rules; show the editor what past rules have cost.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
The authors' method itself (BiPA) is a diagnostic, not a prompt. What's useful to you are its conclusions, so the blueprint shows how to vet any edit to an agent's instructions:
STEP 1: Run the original skill several times → learn its own variance
STEP 2: Run the new skill the same number of times → accept the edit only if the difference exceeds that variance
STEP 3: For every new rule, ask → how many turns will it add?
STEP 4: Delete / merge rules → don't just keep appending
STEP 5: Show the editor what past rules cost → before it writes new ones
Steps 1–2 are done by hand or in your automation. Steps 3–5 can be done with a prompt in a regular chat (see the template below).
Example in Practice
Task: You run a small studio that builds WordPress and Shopify sites. A Claude Code agent edits the markup and scripts. After every failure you added a rule to CLAUDE.md: "check again," "run the build after every change," "read files one at a time." Six months later you have 90 lines. The agent has slowed down, and your plan's tokens run out by mid-month. Time for an audit.
Prompt:
You are an instruction editor for an AI agent. Below is my CLAUDE.md.
It has bloated: rules were added after every failure, and nobody ever reviewed them.
Context: the agent edits WordPress and Shopify sites (PHP, JS, markup).
Every agent turn resends the whole conversation to the model, so an extra turn
costs more than an extra line of instructions.
Audit it in 4 steps:
STEP 1. Put every rule into a category:
"verify" (re-checking), "one at a time" (one call per turn),
"read" (what and how much to read), "format", "boundaries" (what's off-limits), "other".
STEP 2. For each rule, estimate how many EXTRA AGENT TURNS it
generates: 0 / 1–2 / many. Judge "verify" and "one at a time" rules especially harshly.
STEP 3. Find duplicates and contradictions. Propose which rules to DELETE
and which to MERGE into one. One line of reasoning per decision.
STEP 4. Output a new version of the file. Keep exactly one verification rule:
"before claiming completion, run a fresh full check."
Keep rules that generate many turns only if they cover
a real failure from my notes.
Here are my notes on real failures: {failure_list}
Here is CLAUDE.md: {file_text}
Result: The model returns a table of rules with categories and turn estimates, then a delete/merge list with reasons, and a new, short version of the file. Verification gets boiled down to a single rule. Next, run the agent on 5–10 typical tasks several times each and compare token spend against the old version.
Why It Works
The weakness. An agent follows instructions literally. It turns "check the result" into an extra tool call, and "write one at a time" into a string of turns instead of one. Each turn resends the whole conversation, so in a long task an extra turn nearly doubles its own cost (the paper calls this "turn elasticity"). The instructions look short, but the bill keeps climbing.
Why the pile-up goes unnoticed. Edits are judged as a bundle on a small sample. There's no way to tell which of ten added rules helped and which just costs money. And when you pick the best of several noisy results, the winner almost always "wins" on luck.
The strength, and how the method uses it. Models are good at classifying and condensing text. So you can hand them the audit: sort rules into categories, find duplicates, merge. The starting skill in the study was exactly that kind: one general rule, "a fresh full check before claiming you're done." It held its own against everything the optimizers learned.
Control knobs: - Rule categories → add your own (e.g., "security") so you don't delete something you need. - Turn-cost scale → swap in "tokens" or "seconds" if that's what you track. - List of real failures → keep a rule only if it covers a failure on the list. That's your guard against "dead" rules. - Number of runs for comparison → the noisier the task, the more repeats you need.
Prompt Template
This applies principles 2–4 from the paper to a manual or chat-based audit. The authors did not test this prompt themselves.
You are an instruction editor for an AI agent. Shorten and clean up the instructions
without losing the useful rules.
What the agent does: {agent_task}
Real failures the rules were written for: {failure_list}
Instructions:
{instruction_text}
Do the following:
1. For each rule, give its category: verify / one at a time / read / format / boundaries / other.
2. Estimate how many extra agent turns it generates (0 / 1–2 / many).
Every turn resends the whole conversation, so price rules in turns, not length.
3. Propose DELETING rules not tied to a real failure,
and MERGING duplicates and overlapping rules.
4. Output a new version. It must be shorter than the original.
No more than one "verify"-type rule.
Response format: a table (rule | category | turns | decision | reason), then the new version.
Fill in: {agent_task} — one sentence on what the agent does; {failure_list} — the real incidents that made you add rules; {instruction_text} — your full CLAUDE.md, AGENTS.md, or skill.
Limitations
⚠️ Tested on long agentic tasks: terminal work, customer support, business workflows. For short one-off requests (Q&A), the conclusions about turn cost don't necessarily carry over.
⚠️ Agent without history compression: in the study, the full conversation was sent on every turn. If your tool compresses or caches history, extra turns are cheaper. Tokens in the report don't map one-to-one to dollars.
⚠️ The diagnostic needs logs and code: BiPA itself (selection replay, the turn graph) is out of reach without an engineer. Take the conclusions from the paper, not the tool.
⚠️ The principles weren't tested as an intervention: the authors derived four principles from their analysis, but the available part of the text doesn't show that an optimizer gets better with them. The template above is our application, not a validated method.
⚠️ Small samples: one of the benchmarks has only 18 test tasks, and the variance is large.
⚠️ "Harmful" rules can still be needed: the paper doesn't say "don't verify the result." It says verification rules pile up and get expensive. A single general rule works well.
How They Studied It
The researchers took three published skill optimizers (GEPA, SkillOpt, SkillAdam) and three agentic benchmarks: Terminal-Bench 2.1 (fixing terminal environments), τ³-bench (customer support with a simulated user), and AutomationBench (business workflows). The agent was two models in the OpenCode harness with no history compression. Every optimizer started from the same hand-written skill: a general rule, "a fresh full check before claiming you're done." That produced 16 learned skills. Each skill, plus the original and a "no skill" variant, was run on the test set three times.
The key idea: run the optimizer's acceptance rule on candidates that are exactly identical to the original skill. If the "unchanged" candidates show the same gain, the gain comes from selection itself. And that's what happened. Unmodified candidates reproduced the reported gains with probabilities from 0.25 to 0.94. The reported 3–17 points on the dev set didn't hold up on the test set: the average result was plus 0.1 points, and for the nine skills under DeepSeek it was actually minus 1.4. The skills won about as many tasks as they lost (57 vs. 62). In other words, they didn't expand what the agent could do; they just made its behavior more uniform.
Cost was studied separately. Each run was recorded as a graph of "turns ↔ the texts they read." All nine skills under DeepSeek read 8–43% more tokens and took 3–22% more turns. Among the 337 added rules, not one asked to read less, half asked to verify, and 18 asked for "one at a time." On AutomationBench, 56% of writes under the original skill were batched; under the learned skills, no more than 0.3%: writes started going out one by one. One more surprise: the cost travels with the skill to another model and another benchmark, but the benefit doesn't. The practical takeaway: rules are expensive, their benefit is questionable, and you have to measure it with noise in mind.
Original from the Paper
The full text of the original skill (verification-before-completion, Vincent, 2025) isn't in the available part of the paper. It's only described as asking for a fresh, full check before any claim of completion. Section skipped.
Adaptations and Extrapolations
💡 Adaptation for vetting any instruction edit: instead of "it got better on two runs," use a noise-aware protocol.
I changed my agent's instructions. Help me plan how to test it. The agent's task: {task}. Number of typical tasks to run: {count}. 1. Suggest how many times to run the OLD version to estimate its own variance. 2. Suggest a rule: the new version counts as better only if the difference exceeds that variance. 3. Suggest what to record besides "solved / not solved": number of turns, token spend.
🔧 Technique: add "turn cost" to the audit prompt → the agent stops spawning checks. Line for your instructions: "Before claiming you're done, run one fresh full check. Don't repeat it after every step." It replaces a scatter of "double-check" rules with one.
🔧 Technique: independent calls → one turn. If you need to write several files or take several independent actions, ask: "Make independent tool calls in a single turn." The paper shows that "one write per turn" is one of the main drivers of turn growth.
Resources
- Paper: Does Your Generalist Skill Optimizer Really Incentivize Agentic Tasks? Rethinking Skill Optimization via Bipartite Attribution
- Code: github.com/valleysprings/BiPA
- Authors: Jiawei Zhou, Bo Peng, Yuxuan Huang, Ming Zhang, Youbang Sun, Xiang Wang, Chaochao Lu
- Affiliations: Shanghai AI Laboratory, Fudan University, Shanghai Jiao Tong University, USTC, Tsinghua University
- Optimizers and benchmarks mentioned: GEPA, SkillOpt, SkillAdam; Terminal-Bench 2.1, τ³-bench, AutomationBench; OpenCode harness