Cortex: testing a coding agent's rules and skills by measurement, not by eyeballing

/harvest (end of session, never runs the agent):
  STEP 1: Did something go from "broken" to "working"? → save a TASK
         (broken commit + the request verbatim + a check script)
  STEP 2: Did the developer correct the agent? → one LESSON line per correction
  STEP 3: Check the check itself: fails on the broken version, passes on the fixed one,
         passes without the tests from the fix

/evolve (one cycle = one candidate):
  STEP 1: Find a recurring theme in the lessons
  STEP 2: Write ONE skill/rule + choose how it loads
         (tied to a folder / at every step)
  STEP 3: Cheap run: k times without the candidate, k times with it
  STEP 4: Strict run on all tasks
  STEP 5: A script decides, not the model: KEEP only if
         it helped + it fired (rule loaded / skill opened) + no task got worse

/prune (once a month and after a model switch):
  STEP 1: Remove items one at a time, run the tasks
  STEP 2: Delete only if things are no worse without the item
  STEP 3: An item that never fired — don't delete it (nothing to test with)

The commands are prompts the model follows. The exact steps (running, counting, recording) are handled by a small program called cortex.

Our illustration, not a case from the paper (the paper's lab does have a similar "money in integer cents" house rule).

Task: You run the backend for a grocery delivery service. In the shop/billing/ folder, money is handled only in cents as integers via the money.py module. Any float breaks rounding on refunds through Stripe. The agent (Claude Code) keeps writing price * 0.87. The tests pass, but the convention is broken, and you're fixing it for the third time. Instead of yet another line in CLAUDE.md, make a rule tied to the folder and test it by measurement.

Prompt (for Claude Code at the repository root):

Task: check whether a rule against float in money calculations helps.

1. Create the rule .claude/rules/money-in-cents.md with the field paths: ["shop/billing/**"].
   The rule text — no longer than 5 lines: store all amounts in cents (int),
   calculate only via shop/billing/money.py, float is forbidden.

2. Take 3 tasks from shop/billing/ (I'll list them below) and make a check script for each:
   exit 0 if the changed code has no float for money and the tests pass.
   Tasks:
   - add a "−13%" promo code to the cart
   - calculate 20% VAT on the receipt
   - implement a partial refund on an order

3. Run each task 5 times IN A CLEAN CLONE of the repository:
   - variant A: without the new rule
   - variant B: with the rule
   For each run, record: did the check pass, did the rule load
   (did the agent read a file from shop/billing/).

4. Output a table: task × variant → share of successful runs, plus a "rule loaded" mark.
   Verdict KEEP only if: B beats A overall, the rule loaded at least once,
   and no task in B got worse than in A.

Result: The agent will create the rule file and the check scripts, then do 30 runs in isolated copies. The response will contain a table of success rates for each task in both variants and marks for whether the rule loaded. At the end comes a KEEP or KILL verdict referencing the three conditions. If the rule never loaded, the honest verdict is: nothing to test, and it can't be kept. That narrows the problem down to checking the pattern in paths.

The LLM agent's weakness: the result on any single task is random, because the model samples every time (picks a continuation with some randomness). So "I saw it work" isn't proof. On top of that, the agent doesn't read everything. It opens a skill only if its short description convinces it the skill is needed. An instruction that never reached the context changes nothing, yet takes up space and creates an illusion of control.

The strength: the agent dutifully follows whatever is actually in its context. A rule tied to a folder gets placed in the context by Claude Code itself as soon as the agent touches a matching file. The agent doesn't have to decide whether to read it.

How the method uses this: it repeats runs and compares "task with text / task without text" pairs, so the luck of individual attempts averages out. It counts an effect only where the text reached the context. And the decision is made by a script, not by the model that wrote the candidate and wants to keep it.

Your levers: - Loading method (paths on a rule or skill) → the narrower the binding, the less extra context; for skills, file binding is what got them opened at all (28 of 66 chances vs 0 of 198 for every-step skills). A rule without paths always loads and "pays" at every step. - Number of repeats k → more repeats mean less noise but cost time and tokens. Use more for noisy tasks. - Tolerance for regression → in the original it's zero for a task the agent never failed. Loosen it if your tasks are unstable. - Item type → a rule for what the agent must always know in a given folder; a skill for a rare, large procedure.

The original texts of the /harvest, /evolve and /prune commands live in the repository (link under "Resources"); they aren't in the part of the paper covered here. Below is a reconstruction of the logic of one /evolve cycle based on the paper's description, so you can repeat it by hand in Claude Code.

<Role>
You are a cautious editor of an agent's instructions. You do NOT decide the text's fate yourself:
the decision is made by the strict conditions below, based on run results.
</Role>

<Inputs>
- Corrections log: {corrections_log}
- Saved tasks (request + broken commit + check script): {task_list}
- Current harness (CLAUDE.md, .claude/rules/, .claude/skills/): as-is in the repository
- Number of repeats per task: k = {repeat_count}
</Inputs>

<Procedure>
Step 1. theme = find ONE recurring problem in the log.
        If nothing recurs — answer BARREN and stop. Don't write anything.
Step 2. candidate = write ONE skill or rule against this problem.
        Choose the loading method:
          - the problem lives in one folder → rule/skill with paths: ["{folder}/**"]
          - the problem is everywhere → first ask whether this belongs in CLAUDE.md
Step 3. screen = for tasks related to the theme that don't always pass,
        run k times each: base (without the candidate) and cand (with the candidate),
        each run in a clean clone of the repository.
        For each run, record: did the check pass; did the candidate reach
        the context (rule loaded / skill opened).
Step 4. confirm = if the screen looks promising, run ALL tasks the same way.
</Procedure>

<Gates>
KEEP only if ALL three conditions hold:
  1. cand beats base overall on the tasks the candidate reached
  2. the candidate fired in at least one run (rule loaded or skill opened)
  3. no task got worse beyond the tolerance {tolerance}
     (for a task that base never failed, tolerance = 0)
If it fails only on condition 3 → rerun the failed tasks, then decide.
If the candidate never reached the context → RERUN/"not tested," not KEEP.
If the data can't be trusted (no load marks) → RERUN, decide nothing.
</Gates>

<Output>
Table: task × (base, cand) → share of successful runs + "reached context."
Verdict: KEEP / KILL / RERUN / BARREN — and which condition decided it.
If KEEP — put the file into the harness. If KILL — put it in .graveyard/ with the reason.
</Output>

What to plug in: {corrections_log} — lines like "date, task, what the agent did, what it should have done"; {task_list} — tasks with check scripts; {repeat_count} — the paper used 2 for the screen and 3 for the confirm run; {tolerance} — how much a task is allowed to regress.

🚀 Quick start — paste into a chat (ideally straight into Claude Code in your repository):

Here's a template for a rule-testing cycle for an agent (based on the Cortex method). Adapt it to my project: {describe the project}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask which corrections you keep repeating, which folders the problem lives in, how to automatically check that a task is solved, and how many repeats you're willing to run. That's because the method rests on three things: a recurring theme, a script-based check and file binding. Without them there's nothing to measure.

⚠️ Strong model — little benefit: on a stronger model that already followed nearly all the conventions, the gain from a harness built in one run was small. On a real open-source project where the agent was hardly ever corrected, Cortex found nothing to learn at all.

⚠️ Harm doesn't get caught: the agent didn't open harmful skills, and harmful rules landed on tasks that mostly failed even without them. The cycle issued no verdict even when a harmful rule broke a task that passed without it. The method guarantees that unused text won't stick around. That it will catch text that damages the work is not proven.

⚠️ You need a script-based check: without a script that unambiguously decides "task solved," the method doesn't work. Writing and creative work have no such check.

⚠️ Every-step skills almost never get opened: in the measurements, the agent never once opened a skill with no file binding. That doesn't mean skills are useless in general, but "drop in a skill and hope" is a weak strategy.

⚠️ Removal checks make mistakes: /prune gave the expected verdict on most items, but on planted items with known answers — only on half. It kept a useless always-on skill (one the agent never opened) because 2 tasks happened to pass less often when it was removed. Better to confirm removal decisions by hand.

⚠️ A narrow test: one small synthetic codebase, one main model (Claude Haiku 4.5), and a script playing the developer. Real teams with live sessions are the next step.

The author built a small synthetic repository with four "house rules." For example, read the time only through a custom clock module. Code that ignores this works but breaks the convention, and the model breaks it on its own. For each rule he made a family of tasks, plus a fifth family of ordinary bugs where there's nothing to learn (the control). A script played the developer: 10 rounds of sessions. Cortex was run from scratch four times, and each resulting harness was tested on 30 new scenarios the cycle had never seen. That way the gain can't be explained by memorizing the tasks.

Result on the new tasks: the Cortex harness beat the empty one (no skills or rules) by 55 points of success rate (confidence interval from +34 to +74; Claude Haiku 4.5 as the agent, averaged over the four runs on the 23 house-rule scenarios), while the control family barely changed. But the most instructive findings aren't about the gain. Of 18 kept items, 16 turned out to be rules. A skill tied to files was opened in 28 of the 66 cases where it was available. An every-step skill was never opened by the agent across 198 cases.

They also planted 20 useless and 5 harmful candidates. None got through, but the test is weaker than it looks: most were simply never opened. The same harness measured twice diverged by −3.3 points. That's a vivid lesson about noise: a single measurement can be mistaken for an effect. When one run's harness was moved from Claude Haiku 4.5 to the stronger Claude Sonnet 5, its gain dropped from +66.7 to +5.6 points (a single harness on a single stronger model).

🔧 Technique: tight folder binding → less context, and skills actually get opened

Any rule that concerns one area of the code should get a paths field. An unbound rule loads at every step and always costs context. A bound one loads only when the agent touches the relevant files. In the study, every kept item was bound, and the harness added nothing "at every step."

Copy

---
paths: ["shop/billing/**"]
---
Money — in cents only (int), calculate via money.py. float is forbidden.

Extrapolation: a manual corrections log (the /harvest idea). At the end of a work session, tell the agent:

The session is over. Go through our conversation and find every case where I corrected you.
For each one, output a single line: date | task | what you did | what you should have done.
If there were no corrections — say so, don't make anything up.
Append the lines to lessons.md, even if the same line is already there: repeats are a signal.

After a couple of weeks, the repeats in lessons.md show which rule to write first. It's a safeguard against "fixing from memory."

• Cortex: Measurement-Gated Evolution and Pruning of Coding-Agent Skills and Rules — Samuele Giampieri, independent researcher.
• Tool and lab code: https://github.com/samugit83/cortex (tag v1.0-eval — the system as in the paper; v1.0.5 — data and reproduction scripts).
• Claude Code documentation on rules, skills and CLAUDE.md; the AGENTS.md format.
• Recommendations on evaluating LLMs as an experiment (Miller): repeated runs and paired comparisons.
• Related: SWE-bench, SWE-rebench, τ-bench, Meta-Harness, SkillReducer.