/retro and Harness Self-Evolution: How an Agent Learns from Past Sessions and Why Its Notes Often Don't Work

STEP 1: The agent solves a task in the repository → a normal session
STEP 2: In the same session, run /retro → numbered proposals for the harness
STEP 3: A human filters → keeps only those grounded in a session event
STEP 4: The agent applies the edit → CLAUDE.md / hook / config (one edit = one file or line)
STEP 5: New task in the same repository, fresh session → check: did the edit load? was it read? did it change actions?
STEP 6: Decision → keep / rewrite / delete

Steps 1–2 happen in one session. Step 5 is separate fresh sessions. The study compared five sessions with the edit and five without.

Task: You run the backend of a delivery service on Django, with payments through Stripe. Claude Code has already completed the task "fix the promo-code discount calculation." Along the way it: - tried to run pytest, which isn't in the container; - discovered that tests run via make test; - checked three times that the Stripe integration tests were failing even before its changes.

Prompt (in the same session, after the task is done):

Run a retrospective of this session. Propose changes to the agent harness:
CLAUDE.md, hooks, configs, scripts. Don't apply anything until I approve.

Output a numbered list of proposals. For each, specify:
- a title;
- what exactly happened in the session (event: command, error, repetition);
- what you propose to change and in which file;
- how we'll know in a future session that the agent repeated the unnecessary action.

If a proposal isn't grounded in an event from this session, mark it
"general practice, not from the session."

Result: The model will produce 3–6 numbered proposals. Likely among them: a line in CLAUDE.md about make test and pytest being absent, a file of "known failing Stripe tests," a pre-commit linter hook. Each proposal will show which event it's based on. Then it's your call: take the make test line right away (it's a durable fact). Take the list of failing tests with caution and test it. Test the linter hook separately, because it may never fire at all.

The model's weakness. After a session, the agent tends to produce lots of proposals and consider every one of them useful. Some aren't driven by events but by general notions of "good practice": add a linter, a hook, CI. Others are written too narrowly: they describe exactly the failures it saw today. On top of that, the agent may read the file and still act the old way.

The model's strength. The agent is good at recounting its own session. It sees where it wasted time: the wrong test command, a repeated check, an extra search. And it immediately formulates candidate rules. It's a cheap way to get a list of hypotheses you'd otherwise spend hours compiling by hand.

How the method uses this. The retrospective supplies hypotheses, and testing in fresh sessions decides the fate of each. An edit works when its scope matches what the agent will run into next. A line saying "tests are run like this" applies to every session in the repository. A list saying "here are three failed tests" applies only to sessions where exactly those three fail.

Control knobs: - Requiring an event ("what happened in the session") filters out generic advice. - A repetition criterion ("how we'll know the agent did something unnecessary again") makes the edit testable. - One edit at a time lets you tell which one actually helped. In the study, a single line about CI linting tacked onto a useful edit nearly doubled session time. - The wording in the file itself matters. The phrase "any failure outside the list is your fault" made the agent check things it hadn't checked before.

The paper doesn't include the verbatim text of /retro. The skill lives in the mattpocock/skills repository. Below is a template based on the mechanics described in the paper: numbered proposals, human approval, grounding in the session. It's not a copy of the original.

Run a retrospective of the finished session. Propose changes to the agent harness:
{what_can_change}. Don't apply anything until I approve.

Output a numbered list of proposals. For each:
1. Title.
2. What happened in the session (a specific event: command, error, repetition).
3. What to change and in which file: {harness_files}.
4. Repetition criterion: which action in a future session will show that the problem
   is still there.

Mark a proposal "general practice" if it isn't grounded in a session event.

What to fill in: - {what_can_change} — for example, "CLAUDE.md, hooks, linter configs, scripts." - {harness_files} — which files the agent is allowed to touch. The paper showed: if a file is outside the allowed set, the edit doesn't get applied or has no effect.

🚀 Quick start — paste into the chat with your agent after a finished task:

Here is a retrospective template. Adapt it to my project: {describe_project_and_stack}.
Ask me questions to fill in the fields.

[paste the template above]

The agent will ask which files can be changed and how tests are run. That's so the proposals are tied to the project's real harness, not an abstract one.

⚠️ One model and one environment: Everything was tested on a single agent (Claude Code) and a single version of the skill. Results may differ on other agents.

⚠️ Environment problems only: The study covered running tests, known failures and linting. Edits about code navigation and style weren't examined.

⚠️ Small sample: Five sessions per variant, seven meaningful comparisons. The authors themselves say the "scope is what matters" explanation remains a hypothesis.

⚠️ No improvement in task success shown: The share of solved tasks barely changed. Session time often went up, especially when an edit added new instructions.

⚠️ Hand-picked conditions: The test tasks were chosen so that the problem was definitely present. On random tasks the benefit will be noticeably smaller.

⚠️ A human is required: Without filtering, the agent produces lots of "general practices." Some edits turned out to be ones the agent never even opened.

The authors took 30 tasks from SWE-bench Verified (real bugs from Django, SymPy, Astropy, Sphinx, pytest) and ran them through Claude Code. After each session they ran /retro and collected 125 proposals. Two different LLMs labeled what each proposal was based on: a session event or general notions. Then 24 proposals were selected for implementation. 12 edits could actually be tested: on other tasks from the same repository, five fresh sessions with the edit and five without.

The big surprise: almost all of the "executable checks" turned out to be dead. Of 38 such proposals (hooks, linters), only nine could even "fire" in the environment. The rest required missing tools or commit hooks, and there were no commits in the sessions. On top of that, 18 of 43 linting proposals came up simply because the repository had no linter. The skill itself treats the absence of a linter as a "finding."

Of the 12 tested edits, seven were meaningfully compared. Two completely eliminated the target action: the line about bin/test and a file with three failing tests in Astropy. One partially reduced it. Four didn't help: the file listed some failures, while the sessions ran into others. Practical takeaway: test an edit the way the agent will read it in the next session.

🔧 Technique: require a quote of the event → cut out "general practices"

For each proposal, give a verbatim quote from the session (a command or
an error) that produced it. No quote — don't include the proposal in the list.

The paper showed that some proposals aren't tied to events. A quote makes that tie mandatory.

🔧 Technique: separate facts from lists → don't bloat the notes

Split the proposals into two types:
A) durable facts about the repository (how to run tests, where things live);
B) lists of specific cases (failed tests, environment errors).
For type B, describe which failures will NOT be on this list.

Durable facts worked in the study. Lists of specific cases misfired.

🔧 Technique: "unknown" wording → don't breed suspicion

Instead of "any failure outside the list is caused by your change," write: "the list may be incomplete; if a failure is unfamiliar, first check whether it's related to your changes." In the study, the original hard-line wording made the agent run extra checks on unrelated tasks. This is my hypothesis inspired by their observation; the paper didn't test this swap.

Extrapolation: test an edit like a mini-test (my idea, not tested in the paper)

Here is a proposed edit for CLAUDE.md: {edit_text}.
Give me three tasks from this repository where the problem
it solves should show up, and one control example where it
shouldn't. For each session, define what counts as a repetition.

This carries the "session with edit / without edit" idea over to a manual check on a small set of tasks.

• Paper title: How Do Coding-Agent Harnesses Self-Evolve? An Empirical Study of Retrospective Adaptation.
• Authors: Christoph Treude (Singapore Management University), Sebastian Baltes (Heidelberg University).
• The /retro skill: mattpocock/skills repository (github.com/mattpocock/skills), commit b0618bc.
• Study data and prompts: doi.org/10.5281/zenodo.23653700.
• Related work cited in the paper: Galster et al. (tool configuration for agents), Lulla et al. (impact of AGENTS.md on speed), Zhang et al. ("Guardrails beat guidance"), Agentic Harness Engineering, ModularRSI, "Harness updating is not harness benefit".