SynAct: rules for an agent built from its own forks in the road, so the expert only spends time on decisions that change the outcome

STEP 1 (collect): Run the task k times (k=8) with temperature > 0 → a set of trajectories
STEP 2 (forks): Group steps by identical environment state →
        for each fork: do different actions lead to different outcomes?
        (score = mutual information between action and outcome)
STEP 3 (select): Take the top-B forks with the highest score → "evidence blocks"
        (goal + two paths with different outcomes; if all failed, two different wrong outcomes)
STEP 4 (authoring): ONE request to the expert with all the blocks:
        diagnose recurring errors → count blocks per error → mark the fixable ones →
        a 2–4 line rule + antipattern
STEP 5 (apply): Run without the rule. Success → stop.
        Failure → the gate checks whether the antipattern is present → if yes, rule goes at the top of the prompt, one retry
STEP 6 (iterate): New runs with the rule in place → analyze the remaining failures →
        the expert revises the rule (keep / add / remove items)

In the paper, steps 1–2 and 6 are automated in code. Steps 3–5 are prompts.

Task: You run analytics at an online marketplace. An agent in Claude Code writes SQL queries against the orders database: "weekly revenue excluding returns." On the very same task, it sometimes gets it right and sometimes doesn't. You ran it 8 times and labeled the results (correct/incorrect). You want to turn the recurring mistakes into a rule for CLAUDE.md.

Prompt (the rule author, pasted into a strong model):

<Role>
You are a senior data analyst. You review the mistakes of a junior SQL agent 
and write a short rule for it.
</Role>

<Evidence>
Below are evidence blocks. Each contains: the task, the fork (same situation, 
two different actions), what happened next, and the outcome (correct / incorrect).

Block 1: task "weekly revenue excluding returns." Path A: JOIN orders with returns 
on order_id, then SUM(amount) → result doubled. Path B: first 
subtract returns in a subquery, then aggregate → result correct.
Block 2: task "average order value by city, paid orders only." Path A: filter 
status='paid' after GROUP BY → incorrect. Path B: filter in WHERE before grouping → correct.
Block 3: {next block}
...
</Evidence>

<Task>
1. DIAGNOSE: identify the recurring error types in the blocks.
2. COUNT: how many blocks each type explains.
3. MARK: flag the types that can be fixed with a fixed rule 
   (as opposed to knowledge that depends on the specific task).
4. WRITE: write one rule of 2–4 lines in the imperative mood 
   covering the dominant fixable types. 
   Then, in one sentence, state the ANTIPATTERN: 
   the family of errors this rule prevents.
</Task>

<OutputFormat>
Diagnosis: ...
Count: ...
Fixable: ...
Rule:
- ...
- ...
Antipattern: ...
</OutputFormat>

Result: The model will give a diagnosis (for example, errors in the order of filtering and aggregation, plus row duplication on JOIN). Then comes a count of blocks per type, a flag on the fixable ones, and a short 2–4 item rule plus a one-sentence antipattern. You copy both onward: the rule into the agent's instructions, the antipattern into the gate prompt (template below).

The LLM's weakness. An agent checking its own work is bad at telling a correct interpretation from a confidently wrong one. Retries and self-critique fix random slips, but not missing knowledge. And if you manually review every failure in full, the expert drowns in steps that didn't matter.

The LLM's strength. The model is good at generalizing "right/wrong" pairs into a rule and good at following a short imperative rule. It can also tell whether a specific type of error is present in a trajectory's text. That's exactly what the gate does.

How the method exploits this. The method doesn't ask "where did the agent go wrong"; it asks "where did the choice of step split the outcomes." The signal is computed from the runs themselves: no reference answer and no success label needed. It works even when all runs failed: if the agent arrives at different wrong outcomes, the fork is still visible. A plain "success/failure" signal is identical, and blind, in that situation. The gate leaves solved tasks untouched, so the rule can't break what already works.

Control knobs: - Number of runs k (8 in the paper) → more runs give a more stable score but cost more. Each action has to be seen at least twice, or the fork doesn't count. - Number of blocks B → more blocks doesn't mean more requests: the expert still gets a single request with the full set. - Rule format (2–4 lines + antipattern) → the shorter and more specific, the less harm to a weaker agent. - The gate → remove it to test the rule "always on." In the paper, this made results worse for weaker agents. - Iterations → repeat as long as new runs turn up new error types. The first iteration delivers most of the gain.

The template has two parts. Part 1 writes the rule. Part 2 is the gate that decides whether to apply the rule. The authors' original prompts aren't included in the available text. The template is assembled from their description of the stages: diagnosis → count → mark fixable → 2–4 line rule + antipattern.

Part 1: the rule author (strong model, one request):

<Role>
You are a {expert_role}. You review the mistakes of an agent that solves {task_type} 
and write a short rule for it.
</Role>

<Evidence>
Block 1: task: {goal}. Fork: {identical_situation}. 
  Path A: {action_A} → {what_happened_next} → outcome: {correct/incorrect}. 
  Path B: {action_B} → {what_happened_next} → outcome: {correct/incorrect}.
Block 2: ...
[If neither path succeeds, mark the block ALL-FAIL and give two DIFFERENT wrong outcomes]
</Evidence>

<Task>
1. DIAGNOSE: identify the recurring error types.
2. COUNT: how many blocks each type explains.
3. MARK: which types are fixable with a fixed rule.
4. WRITE: one rule of 2–4 lines in the imperative mood 
   covering the dominant fixable types + one ANTIPATTERN 
   (the family of errors the rule prevents).
</Task>

<OutputFormat>
Diagnosis: ...
Count: ...
Fixable: ...
Rule: ...
Antipattern: ...
</OutputFormat>

Part 2: the gate (runs only after a failure):

<Role>You are reviewing a failed attempt by an agent.</Role>

<Antipattern>{antipattern_from_part_1}</Antipattern>

<Trajectory>{log_of_failed_attempt}</Trajectory>

<Task>
Does the trajectory show signs of this antipattern? 
Answer YES or NO, plus one sentence on which step shows it.
You don't know the task's correct answer; don't try to guess it.
</Task>

If the answer is YES, add the rule to the top of the prompt and rerun the task once. If NO, leave the result as is.

What to fill in: {expert_role} is a domain role ("senior analyst," "DevOps engineer"). {task_type} is the agent's class of tasks. Take the blocks from your own runs: a successful and an unsuccessful path from the same situation.

🚀 Quick start: paste this into a chat:

Here's the SynAct template (rule author + gate). Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what the agent is and what tasks it does, what successful and unsuccessful runs look like, and at what point they diverge. It needs this for the evidence blocks: the rule is written from "fork → different outcomes" pairs. From the template, it will take the structure (diagnosis → count → rule → antipattern) and assemble a prompt for your task.

⚠️ The full version needs code: automatically finding forks requires running the task many times, hashing the environment state, and computing mutual information. Without a script, you're left with the manual version, where you find the forks in the logs yourself. That's a simplification, not what the authors tested.

⚠️ You need repeating states: the method works when the environment returns identical observations across runs (SQL, a shell, a game environment, a dialogue with tools). In free-form text, where the "state" is new every time, there's nothing to group.

⚠️ Each action must be observed at least twice: rare actions (once in ten runs) rarely make it into the score. The method sees frequent forks, not one-offs.

⚠️ The rule fixes environment conventions, not understanding of the task: that is, "how things are done here," not "what was actually asked." If the mistake is about the meaning of the task rather than syntax and order of operations, a fixed rule helps less.

⚠️ A rule without a gate does harm: constantly injecting rules into the prompt cost a weaker model up to 17.6 pp of accuracy. Conditional application is part of the method, not an optional add-on.

⚠️ The expert in the experiments is a model: Claude Opus played the expert. The authors are upfront about it: whether a human specialist would write the same rule hasn't been tested. The agents were small open models; the effect on strong agents is weaker (Qwen3-32B as the control).

⚠️ Measurement noise: baseline accuracy fluctuated by 3.4–8.5 pp between iterations. The authors treat differences of that size between adjacent iterations as indistinguishable. The test sets are small (40 to 150 tasks).

The idea was simple: if expert time is limited, find the agent decisions where it changes the result the most. The Amazon team took four benchmarks: SQL queries, crafting in a text game, the bash shell, and tool-using dialogues in a retail store. Four agents: Qwen3-32B, Gemma-3 12B and 4B, and Ministral-3 3B. Claude Opus played the expert. Each configuration was averaged over three repeats.

They compared against four methods that accumulate "experience" from the agent's own trajectories: ACE, ReasoningBank, AWM, and AWM with iterations. These were given the same expert. That way the comparison showed specifically where the expert's attention was directed, not whether there was an expert at all. Separately, they tested three ways of picking forks: by success, by action entropy (how varied the steps are), and at random.

Result: SynAct improved accuracy in all 16 cases (+11.7 pp on average) and beat the four baselines in 15 of 16. The surprise: a "success/failure" signal is blind wherever all runs failed. Mutual information between action and outcome doesn't lose those tasks, and for one SQL model it picked 1.9 times more forks (274 versus 146). For Qwen on TextCraft, 20 of the 49 selected tasks were exactly the "all failed" kind.

The same signal was used at agent run time for routing: when to call the expert and when to make do with self-checking. It beat three budget-matched routers in 15 of 16 cases. A single targeted correction from the expert turned out better than seven additional blind retries. The practical takeaway: seven identical retries don't replace one correction in the right place.

🔧 Technique: drop the automation → manual log analysis in a chat

If you don't have code, find the forks yourself. Run the task 5–8 times and paste two logs into a chat: a successful one and a failed one. Ask:

Here are two runs of the same task. Find the FIRST point where the agent, 
in the same situation, took different steps. Explain why exactly this step 
split the outcomes. Show 2–3 such forks.

Then assemble the blocks for Part 1 of the template. The authors didn't test this manual version; it's a simplification.

Extrapolation: conditional rules in CLAUDE.md

The gate idea carries over into agent instructions. Instead of an unconditional list of rules, write rules with a trigger condition:

Copy

## Rules for when things go wrong
- IF a query returned a suspiciously large total, CHECK for duplicates 
  after a JOIN (antipattern: aggregating over a non-unique join). 
  In that case, aggregate in a subquery first, then join.
- IF there are no errors, don't use this rule.

This isn't the paper's method; it's a transfer of the principle "apply the rule only when the antipattern matches." Test it on your own tasks.

• Paper: SynAct: Synergistic Active Feedback Learning for LLM Agents
• Authors: Ning Liu, Ian Li, Kristina Klinkner (Amazon)
• Benchmarks: InterCode-SQL (Spider), TextCraft, InterCode-Bash (nl2bash), τ-bench retail
• Compared against: ACE (Zhang et al., 2026), ReasoningBank (Ouyang et al., 2026), AWM (Wang et al., 2025)
• Related ideas: Reflexion (Shinn et al., 2023), Self-Refine (Madaan et al., 2023), LEAP (Choudhury & Sodhi, 2025)
• The paper text in the request was truncated: the appendices with the original prompts and the results of sections 5.2–5.4 are unavailable.