TL;DR
The study shows that lessons an agent draws from its own failures (memory entries, skills, rules in its instructions) help it redo the same task, but barely help on new ones. Sometimes they even make things worse. This was tested on three types of “self-evolving” agents, ten models, and four environments.
The core problem: the agent is bad at understanding why it failed. 20–50% of updates contain at least one error. The agent logs a failing action as a best practice. It invents a cause that never happened. It drops a condition (“backtrack before a new search” turns into “always backtrack”). Or it writes down a harmful instruction. But even a correct lesson transfers poorly: it stays tied to the specific task that produced it.
What works noticeably better: examples of successful runs. Verifier feedback (“fail/success” plus the reason) and extra attempts help only a little. The takeaway for practitioners: don't let an agent's self-written “lessons from mistakes” into its instructions unchecked, and add the experience of its wins.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Study Blueprint
SETUP (CFOE: cold start, failures only)
Agent memory/skills/rules: empty
↓
LOOP over a stream of tasks the model has already failed three times:
Agent attempts the task → fails → agent writes its own “lesson”
Successes are not used. The verifier is hidden.
↓
CHECK:
(a) the same failed task → partially fixed
(b) new tasks → almost no gain, sometimes a drop
↓
DIAGNOSIS (manual review of lessons):
M1 false “success” · M2 wrong cause
M3 lost conditions · M4 harmful advice
↓
WHAT HELPS (weak → strong):
extra attempts during reflection → verifier feedback → successful runs
Example in Practice
The method itself isn't something you copy, but the finding applies directly to how you maintain an agent's instruction file. Strong fit: an agent doing coding or workflow automation keeps making mistakes, and you want to accumulate “lessons” in CLAUDE.md or AGENTS.md. Weak fit: one-off tasks where lessons never pile up.
Task: The team behind an online store selling on Amazon and Etsy uses Claude Code to pull reports. After every failure, the agent appends to a “Lessons” section in CLAUDE.md. A month later there are 40 items. Some sound pretty rigid: “always recalculate inventory from scratch,” “on an API error, don't retry the request.” The agent has started acting weird, and you need to figure out which lessons are doing damage.
Prompt:
You are a reviewer of an agent's instruction file. Below is a “Lessons” section the agent wrote itself after its failures. Check every item.
For each lesson, answer:
1. What exactly happened in the failure (from the log below)? Does the log confirm the cause the agent named? Quote the log or write “not confirmed.”
2. Is there a false “success” in the lesson: an action that caused the failure recorded as a good practice?
3. Were conditions lost: when is the rule correct and when is it not? If they were lost, restore the condition.
4. Could the lesson become harmful advice in a neighboring situation (for example, “never retry a request”)? Name that situation.
Verdict for each item: KEEP / REWRITE (give a new wording with a “when” condition) / DELETE.
Don't invent anything: if the log doesn't explain the cause, write “not confirmed.”
“Lessons” section:
{lessons_from_file}
Failure logs:
{logs}
Result: The model returns a table or list covering all 40 items: whether the cause is confirmed by the log, which of the four problems were found, and a verdict. Questionable lessons get a rewritten version with a “when” condition. Some lessons will be flagged “not confirmed.” Those should be deleted or checked by hand.
Why It Works
The model's weakness. The agent analyzes its failure from the run transcript alone, with no access to the real cause. It explains plausibly but doesn't verify. The made-up explanation becomes a rule and carries over into future runs. Worse, the lesson is recorded as a conclusion from a single case. The conditions under which it holds get lost.
The model's strength. Models are good at working from a concrete example. A successful run shows which actions add up to a working procedure. A failure only tells you what went wrong. You can't assemble “how to do it right” from that.
How to use it. Separate the roles. The agent can propose lessons. They get checked by a separate request that must cross-check against the log, or by a human. Bake the conditions of applicability into each lesson. Put an example of a successful run next to the lesson.
Control levers: - Requiring “a quote from the log or not confirmed” → fewer invented causes. - A mandatory “when the rule holds / when it doesn't” field → conditions don't get lost. - Checking a neighboring situation (item 4) → catches harmful “always/never” rules. - Add “cite a successful run where this rule worked” to the review → the lesson gets something to stand on.
Prompt Template
You are a reviewer of an agent's instructions. Check the section of “lessons” the agent recorded itself after its failures.
For each lesson:
1. Cause: does {log} confirm the cause named in the lesson? Give a quote or write “not confirmed.”
2. False success: was something that caused the failure recorded as a good practice?
3. Conditions: when is the rule correct and when is it not? If there are no conditions, add them.
4. Risk: in what neighboring situation would the lesson do harm?
5. Grounding: is there an example in {successful_runs} where the rule worked?
Verdict: KEEP / REWRITE (new wording “When …, do …”) / DELETE.
Don't infer anything beyond the logs.
Lessons: {lessons}
Failure logs: {log}
Successful runs: {successful_runs}
Fill in: {lessons} — the section from CLAUDE.md, AGENTS.md, or the agent's notes. {log} — excerpts from failed sessions. {successful_runs} — 1–3 successful runs of similar tasks. Without them, item 5 will return “no grounding.”
🚀 Quick start — paste into the chat:
Here's a template for reviewing an agent's “lessons.” Adapt it to my instruction file: [describe the agent and where the lessons are stored].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask which lessons and logs you have and whether you have any successful runs. That's so it can check causes against facts instead of the agent's guesses.
Limitations
⚠️ Not tested with prompts: the authors didn't test review prompts or rules for writing lessons. They demonstrated the problem and its causes. The template above is an extrapolation of their finding, not a result from the paper.
⚠️ Simulated environments: testing was done in ALFWorld (household tasks), τ²-Bench, WebShop, and WebArena. Real workflows and “Lessons” in
CLAUDE.mdwork differently.
⚠️ Harsh setup: they used tasks the model had failed three times and hid successes. In real life you have both failures and wins. Read the conclusions about the harm of failures as “failures alone aren't enough.”
⚠️ Needs adaptation: the self-evolving frameworks themselves (ReasoningBank, SkillOpt, Evo-Harness) are code. The value for readers lies in the conclusions about what to trust in an agent's notes.
⚠️ Strong models make mistakes too: stronger models make fewer errors, but the errors remain. You can't skip the review.
How They Studied It
The researchers built a harsh “cold start, failures only” setup. The agent starts with empty memory. For each model they assembled a stream of tasks it failed three times in a row. The agent learns only from failures; successes aren't used and verifier feedback is hidden. This way they tested three types of self-evolution: reasoning memory, skills, and the agent “harness” (runtime environment rules). They tested ten models in four environments, with streams of up to 800 tasks.
The result was unambiguous. Old failed tasks get partially solved. On new tasks there's almost no gain, and sometimes performance drops. A longer stream doesn't rescue things. The authors then manually reviewed the updates and identified four types of errors. They show up in 20–50% of updates. The surprise was something else: even when they kept only manually verified correct lessons, new tasks still didn't improve. The lesson stays tied to the task it came from.
Then they relaxed the conditions. Verifier hints and extra attempts during reflection gave a small boost. Successful runs produced substantially more transfer. The practical insight: a failure tells you what not to do, while a success shows you how to put it together.
Adaptations and Extrapolations
🔧 Technique: add a “when” field → fewer harmful generalizations
Require the agent to record each lesson in this format:
When {condition}, do {action}. Do not apply if {exception}. Source: {session/log}.
This targets M3 (lost conditions) and M4 (harmful “always/never” rules).
🔧 Technique: two requests instead of one → separate the author from the reviewer
The agent proposes a lesson. A separate request or a different model accepts it only if the cause is confirmed by the log. Same logic as the template, but the review happens apart from the author.
Extrapolation: “golden runs.” Keep 2–3 examples of successful execution of typical tasks in the repo and reference them from
CLAUDE.md: “To pull a report, followexamples/report_ok.md.” This idea comes from the paper's finding about the value of successful trajectories. The authors tested it in automated frameworks, not in instruction files.
Resources
- Self-Evolving Agents Can Repair Failures but Struggle to Generalize from Them — Weixiang Zhao, Yichen Zhang, et al., Harbin Institute of Technology (Harbin and Shenzhen), The Chinese University of Hong Kong.
- Frameworks analyzed: ReasoningBank, SkillOpt, Evo-Harness.
- Environments: ALFWorld, τ²-Bench-Verified, WebArena-Verified, WebShop.
- Reference to Trace2Skill — the approach the extra attempts were based on.
- The paper text was truncated at the “Related Work” section; the appendices were not available.