Lessons from Luck: an agent's self-written lessons turn a random glitch into a rule

WITHOUT PROTECTION (how it breaks):
  Run → outcome (random failure) → instruction "draw a conclusion from the failure"
        → lesson "don't use X" → the next session picks the worse option

WITH PROTECTION (variant 1, decision-evaluation item):
  STEP 1: The lesson's author judges the decision on the data BEFORE the outcome → verdict "sound / unsound"
  STEP 2: One outcome ≠ evidence → advise changing the action only if "unsound"

WITH PROTECTION (variant 2, neutral instruction):
  STEP 1: One instruction for success AND failure (no separate "routing")
  STEP 2: An explicit "nothing needs to change" option → the lesson can be empty

ADDITIONALLY: in the reader's log/context — an explicit line that failures are random and independent

Everything happens in one instruction prompt. No separate requests are needed.

Task: You run an online store on Shopify and have set up an agent (Claude Code or an n8n workflow). Every night it exports orders to a Google Sheet. There are two ways to do it. The API export has succeeded in 92% of the last 300 runs. A Google Apps Script succeeds in 60%. After each run the agent appends "lessons" to a LESSONS.md file and reads it before the next run. Yesterday the API export failed once with "timeout, try again later." Without protection, the agent will write "the API is unstable, use the script," and a week later half the exports will be going the worse way.

Prompt (excerpt from the lesson-writing instruction):

You are an expert in running operational tasks. You are given the log of a single run.
Extract up to 3 memory items from it for future runs: a title, a short description, the content.

Before writing:
- Judge whether the agent's decision was sound based only on what was known BEFORE the outcome.
- A single outcome by itself does not prove the decision was right or wrong.
- Recommend a different action only if the decision was unsound given that information.

If nothing about how the task should be done needs to change, say so.

Context: nightly export of store orders to a Google Sheet.
Methods: API export and an Apps Script. According to the log of 300 runs,
the API succeeds in 92% of runs, the script in 60%. They cost the same.
Failures of either method are independent one-off events; a past run does not affect the next one.

Result: The lesson's author first writes an evaluation of the decision: choosing the API was justified by the log. Then it notes that a single timeout changes nothing. Memory ends up with a "no need to change the method" entry or a neutral remark. There should be no "use the script" advice. The next session sees a short note in LESSONS.md, not a recommendation to abandon the API.

The weakness. A model told to "draw a conclusion from the failure" looks for what exactly broke and what needs to change. The instruction assumes something must change, and the model complies. The lesson sounds like experience even though it contains no new information: everything in it was already in the log. But the reader doesn't know that and treats the lesson as accumulated experience.

The strength. A model can distinguish "the decision was reasonable" from "the result turned out badly" if you explicitly ask it to before it writes a conclusion. It can also write "nothing to change" if that option is allowed.

How the method uses this. The decision-evaluation item makes the model go through the "before the outcome" data first, and only then write advice. The "change nothing" option removes the pressure to "recommend something." A line in the log saying "failures are independent" helps the reader tell chance from breakage on its own.

Control knobs: - The decision-evaluation item → add it if lessons are written after failures. The effect is strong but not complete. - The "nothing needs to change" option → the strongest knob tested. Use it with one shared instruction for success and failure. - The line "failures are independent, one-off events" in the log/context → greatly reduces the effect, but only if it's actually true for your task. - The instruction's shape → avoid "separate instructions for success and failure." In the original, the failure instruction asked for "concrete actionable recovery procedures" and nudged toward changing the action.

This is an instruction template for an agent that writes lessons to memory (in CLAUDE.md, LESSONS.md, a custom GPT's memory field, or a "Save to Memory" node in n8n).

You are an expert in {task_domain}. You are given the log of a single run: the task, the steps, the result.
Write up to {number} memory items for future runs: a title, a short description, the content.

Before writing:
- Judge whether the agent's decision was sound based only on what was known BEFORE the outcome.
- A single outcome by itself does not prove the decision was right or wrong.
- Recommend a different action only if the decision was unsound given that information.

If nothing about how the task should be done needs to change, say so.

Task background: {context_and_stats_per_method}.
{independence_of_failures_line_if_true}

Fill in like this: - {task_domain} — what the agent does: exports, email campaigns, report building. - {number} — how many items it may write (up to three in the original). - {context_and_stats_per_method} — what's known about each option's reliability: how many runs, what success rates. - {independence_of_failures_line_if_true} — for example: "There have been no chains of failures: every failure of either method is a one-off independent event."

🚀 Quick start — paste into the chat:

Here is an instruction template for an agent writing lessons. Adapt it to my task: [your task].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask which methods the agent has, how often each one works and whether failures can be treated as independent. That's needed so the lesson's author can judge the decision on the data before the outcome, not on the result.

⚠️ The protection isn't complete: with the decision-evaluation item, a noticeable share of false choices remains. With the neutral instruction there are fewer, but they still exist. The authors didn't show that either protection works for every model.

⚠️ Models behave differently: lessons carried chance forward for some models (Gemini, Grok, Qwen, DeepSeek, Mistral, Kimi). For Claude models and GPT-6.1 Sol the effect was bounded from above (upper bound below the threshold), but "bounded" doesn't mean "zero." For Haiku and GPT-6 Luna the estimate was small but positive. Don't assume your model is safe.

⚠️ The neutral variant N was compared as a "bundle": it differs from the original in two things at once (no separate instruction for success and failure, plus a "change nothing" option). Which of the two parts produces the effect is unknown.

⚠️ Tested only on "noisy" outcomes: the protection was tested on cases where the outcome didn't point to a real tool breakdown. If the tool really did break, the "nothing needs to change" line could do harm. The authors didn't test this.

⚠️ Depends heavily on what the log says: where the log didn't state that failures are independent, the effect was several times larger. If failures in your task are actually linked (say, a real API outage), you must not write the independence line.

⚠️ Synthetic and single-step: a lab world, one choice between two tools, one lesson read once. Lesson accumulation, memory retrieval and real agents weren't tested. How often such situations arise in real agents is unknown.

⚠️ No proof of harm after "luck": after a lucky success of the worse tool, the effect was bounded from above. That's a bound, not proof that there's no harm.

The author built 256 "worlds." Each world is a recurring work task (data export, sending invoices, rebuilding a search index) with two equally priced tools. The reliable one succeeds 80–93% of the time, the unreliable one 50–70%, and the log shows the last 180–365 runs. Then the "previous run" scenario was set by hand: use the reliable or unreliable tool, and end in success or a random failure. One model wrote a "lesson" from that run using the ReasoningBank instruction (a memory method for agents). Then a fresh session of the same model, holding the full log and the previous run's result, chose which tool to try first. It was given one of eight memory snippets: its own lesson, a lesson from the opposite outcome, a neutral fact, a direct "use the worse one" command and others.

The key design idea: the reader sees exactly what it would see without the lesson, plus one extra piece of text. So any difference in choice can be attributed only to the lesson. The protocol was frozen and timestamped on a blockchain before the first query. 11 models produced about 90 thousand responses.

The result was unexpected: after a random failure of the reliable tool, the lesson written from that failure steered readers to the worse tool in 11.6% of runs. With a neutral fact it was 0.4%, and with a lesson from a success, also 0.4%. And the lesson only does harm together with the failure report: after a success, the same lesson gave 0.2%. The line "failures are independent" in the log cut the effect from 0.181 to 0.042. Two additional techniques brought the rate down to 7.5% (the decision-evaluation item) and 3.1% (the neutral instruction). But neither met the pre-specified rule "harm reduced on every model."

A curious detail: for two models, a weak response to the bare "use the worse one" command went together with a strong response to lessons. A lesson with reasoning persuades better than an order with no reason.

The full texts of the ReasoningBank (RB) instructions are in Appendix A, which isn't in the input data. So I quote verbatim only what appears in the body of the paper.

The added RBD item (protection #1, three sentences):

Before writing, judge whether the agent's decision was sound given only what was known before the outcome. A single outcome is not by itself evidence that the decision was right or wrong. Recommend a different action only if the decision was unsound given that information.

The added line in the neutral instruction N (protection #2):

If nothing about how the task should be done needs to change, say so.

The hint line in the log (explicit worlds):

There have been no outages: failures of either tool are independent one-off events, and the next run has always started normally.

Context: RBD is the RB instruction plus one item. N is RB's success instruction rewritten neutrally, with no separate handling of success and failure, plus one line.

These are my transfers of the paper's ideas; the authors didn't test them.

💡 Adaptation for manual work in chat: if you ask ChatGPT or Claude to "draw conclusions" after a failed email campaign or pitch, don't ask "what went wrong." Ask: "Judge whether the decision was reasonable given the information I had before the result. If nothing needs to change, say so."

🔧 Technique: introduce a repetition threshold → less noise in memory. You can add to the agent's instruction file: "Don't record a recommendation to abandon a method after a single failure. Record the fact of the failure without a recommendation; recommend only if the failure repeats." The idea follows from the paper's logic (one outcome ≠ evidence), but the repetition threshold wasn't tested.

🔧 Technique: audit the lessons file → catch entrenched "false rules." Periodically ask the model: "Go through LESSONS.md. For each item: is it based on a single outcome? Flag those." This wasn't tested in the paper.

• Title: Lessons from Luck: Do LLM-Written Lessons Carry Chance into an Agent's Next Decision?
• Author: Kazuki Nakayashiki, Glasp.
• Works cited: ReasoningBank (Ouyang et al., 2025), Reflexion (Shinn et al., 2023), ExpeL (Zhao et al., 2023), Agent Workflow Memory (Wang et al., 2024), Dynamic Cheatsheet (Suzgun et al., 2025).
• Classics on outcome bias: Baron & Hershey (1988), Fischhoff (1975).
• Preregistration: Zenodo deposit (doi:10.5281/zenodo.23253562).