TL;DR
SkillForgeBench is a benchmark that takes skills (files with step-by-step instructions for an agent) and turns each one into a set of atomic checks. Some checks look at the outcome (the file, the report, the edits). Others look at the process: which steps the agent took, what it verified along the way, whether it broke any prohibitions. A task counts as done only if every item passes at once.
Main finding: if you count the share of items passed, agents look almost equally good — weak and strong models don't differ that much. If you require "all items at once," the picture changes: the best model completes fewer than half of tasks cleanly, the weak one only a few percent. Process is the worst part: the agent often hands back a plausible result but skips mandatory checks and steps. If you only look at the final file, you can't see this.
Second finding: a skill in the prompt raises per-item pass rates for everyone by about the same amount. But on "all items at once," the skill helps top models a lot and barely lifts the weak ones. Following an external instruction takes the model's own knowledge.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Take a skill (an instruction for the agent) → extract its requirements
STEP 2: Run the task twice: with the skill and without → compare what the skill added
STEP 3: Turn the difference into a checklist: atomic "pass / fail" items
• OUTCOME items (what ends up in the final file)
• PROCESS items (which steps and checks happened during the work)
STEP 4: A judge (LLM) reads the agent's work log and the output → marks each item
STEP 5: Two scores:
• soft — share of items passed
• strict — the task counts only if ALL items passed
In the paper this is a pipeline of several prompts. What the reader can take is the idea itself: a two-part checklist and strict scoring.
Example
Task: You run sales analytics for an Amazon seller account. An agent in Claude Code follows a "monthly report" skill: it takes the export from Seller Central and builds a summary table for the business owner. The skill spells out the steps: make a copy of the source file, don't edit the original, reconcile the row total with the report total, flag returns separately, don't round to the nearest thousand. The report looks fine. But whether the agent did the reconciliation and whether it touched the original is unclear.
Prompt (for the judge — a separate chat or a separate agent):
You are a strict reviewer. You're given the agent's work log and the output files.
Check the work against the checklist. For each item answer PASS or FAIL
and quote the evidence from the log or file. No evidence — FAIL.
OUTCOME CHECKLIST:
R1. The output file has a "Summary" sheet with sales by SKU.
R2. Returns are broken out into a separate column.
R3. Amounts are not rounded to the nearest thousand.
PROCESS CHECKLIST:
P1. Before editing, the agent created a copy of the source export.
P2. The source file was not modified (no write commands to the original).
P3. The agent reconciled the row total with the summary total and printed the reconciliation result.
P4. If the reconciliation didn't match, the agent reported it instead of silently continuing.
At the end, output:
- a table "item — PASS/FAIL — evidence";
- soft score: how many of the 7 items passed;
- strict score: ACCEPTED only if all 7 passed, otherwise NOT ACCEPTED.
AGENT LOG:
{log}
OUTPUT FILES:
{files or their description}
Result: The judge will produce a table of seven items with quotes from the log, plus two scores. Most likely the outcome items will pass while gaps surface in the process — for instance, the total was never reconciled. The soft score will say "5 out of 7, not bad." The strict score will say "NOT ACCEPTED" and thereby expose the problem.
Why It Works
Weakness: the final file can't tell you whether the agent took the mandatory steps. A report can look great and still have been assembled without a reconciliation or a backup. The average pass rate masks failures even further: "6 out of 7" sounds good, but in real work one broken item often means the result can't be used.
Strength: an LLM reads logs well and checks them against short, unambiguous items. Especially when each item has an explicit "PASS / FAIL" condition. A judge does poorly on fuzzy wording like "the report is high quality."
How the method uses this: a checklist of atomic items turns a vague "done fine" into a set of verifiable facts. Splitting into "outcome" and "process" catches cases where the result is good but the path to it broke the rules. Strict scoring reflects the real cost of an error.
Control levers: - Atomic items. Split "reconciled and reported the error" into two items; otherwise it's unclear what exactly failed. - PASS condition. State explicitly what counts as evidence: a command in the log, a line in the file. - Soft and strict scores. Soft shows exactly where the agent is slipping. Strict shows whether the result can ship without human review. - Criticality. If some items are critical (security, money), run a separate strict score on just those.
Prompt Template
Step 1. Turning a skill or instruction into a checklist. The idea from the paper: run the task with the skill and without, compare, and take what the skill added.
Below are two transcripts of the same task being performed.
A — the agent worked from an instruction (a skill). B — without it.
Compare A and B. List the requirements the instruction added
and what B is missing. Split them into two groups:
OUTCOME — what must be in the output files.
PROCESS — which steps, checks and prohibitions must be followed during the work.
Rules for each item:
- one item = one checkable condition;
- wording with an explicit PASS/FAIL criterion;
- no evaluative words ("good," "high quality");
- drop cosmetics (formatting, style), keep what matters.
INSTRUCTION: {skill}
TASK: {task}
TRANSCRIPT A: {log_with_skill}
TRANSCRIPT B: {log_without_skill}
Step 2. Checking the agent's work — the prompt from the example above. Swap in your own checklist items.
What to fill in:
- {skill} — the text of your skill, CLAUDE.md file or standard operating procedure.
- {task} — the wording you gave the agent.
- {log_with_skill} and {log_without_skill} — the work histories (commands, tool calls, responses).
🚀 Quick start — paste into the chat:
Here's a template for checking an agent against a two-part checklist (outcome and process)
with strict and soft scoring. Adapt it to my task: {describe what your agent does}.
Ask me questions to build the checklist.
[paste the template above]
The LLM will ask which steps and prohibitions are mandatory, what counts as evidence in the log and which items are critical. That's because these answers are exactly what the atomic PASS/FAIL items are built from, and the method rests on them.
Limitations
⚠️ It's a benchmark, not a ready-made prompting technique: the paper measures agents and explains where they break. It has no ready-made prompt templates — the template above is assembled from the logic of the paper's rubrics.
⚠️ You need access to logs: checking the process is only possible if you have the agent's action history. In a regular chat without tools, half of the method is out of reach.
⚠️ The judge is an LLM too: it can make mistakes. In the paper the rubrics were additionally checked by two human experts. For important tasks, spot-check some verdicts by hand.
⚠️ A skill doesn't replace a strong model: weak models get noticeably less out of an external instruction. A long SOP won't make a weak agent reliable.
⚠️ One set of conditions: all models ran in the same agent environment, so the numbers may differ on your tool. The practical takeaway is a direction, not exact proportions. The paper's text in the source material cuts off during the breakdown by item category, so that part of the analysis isn't covered here.
How It Was Studied
The researchers collected over 50,000 public skills from open repositories and selected the procedurally complex ones, split evenly across five areas (technology, business, creative work, agent behavior, knowledge). For each skill they found real files online and genuine user phrasings from forums. The files were assembled into sandboxes: what the agent may edit, what it may only read, and what (the reference answers) is hidden from it. The result was 842 tasks.
The most interesting part is how the checks were written. If you simply ask a model to come up with criteria, you get fuzzy rules like "the result is correct." So the same agent was run twice — with the skill and without. Then they compared what exactly changed in the actions and in the files. That difference became the basis for the items. Next, two experts broke complex rules down into atomic ones, removed formatting noise and wrote explicit PASS/FAIL conditions. Quality was checked separately: the experts' ratings agreed almost every time.
Eight models were run in the same environment. Two results were surprising. First: on the soft score the models are close, on the strict score the gap is huge — compounding error: each item is likely to pass, but all of them together aren't. Second: the skill "levels the field" differently than expected. Strong models got far more out of it than weak ones. Practical insight: don't judge an agent by the final file or by the average percentage; look at the process and demand "all items at once."
Adaptations and Extrapolations
💡 Adaptation for your skill — self-checking during the work. The same "process separate from outcome" trick can go right into the agent's instruction. This wasn't tested in the paper; it's a logical extension.
At the end of the work, output a "Process report" block:
for each step in the instruction — done / not done / not applicable,
plus evidence (a command or a file). If a mandatory step wasn't done,
don't call the task complete — report exactly what was skipped.
🔧 Technique: two scores instead of one → you can see where the agent is slipping. First compute the soft score to see which items the agent fails most often. Then the strict one, to decide whether the result can ship without a manual check.
💡 A contrast run for debugging a skill. Run the same task with your skill and without it, then ask an LLM to compare the logs. If there's no difference, the skill isn't working — its items are redundant or too vague.
Resources
- SkillForgeBench: Evaluating Agent Reliability Grounded in Agent Skill Specifications
- Authors: Zhengxi Li (MBZUAI), Junyi Zhou, Qiyuan Zhang, Chen Ma (City University of Hong Kong), Chuhan Wu, Le Tian, Xiao Zhou, Yufei Wang (WeChat AI, Tencent), Fuyuan Lyu (McGill – Mila & Quebec AI Institute)
- Skill sources: SkillsMP, SkillHub, public repositories from Anthropic and OpenAI
- Agent runtime environment: OpenHands