Randomized Block Curation (RBC): Test an Agent's Self-Written Memory with Random A/B Runs Instead of Trusting Its Own Judgment

STEP 0: The agent grows K=3 playbooks (tasks in different orders) → K lists of lessons
STEP 1: Lessons → 8 thematic blocks (clustering by meaning)
STEP 2: Practice tasks: each block is on with probability 1/2, independently
        → difference in mean success "with block" − "without block" = block effect
STEP 3: Keep blocks with effect > 0, strongest first, up to the budget (40% of tokens)
        → a "pruned" playbook for each of the K runs
STEP 4: Candidates: K pruned + K full + "no playbook"
        → run them on FRESH practice tasks → ship the winner

In the paper, steps 1–4 are done by code (clustering and regression). The agent and the lessons themselves stay unchanged.

The task: you run a Shopify store that also sells on Amazon and Etsy. For six months Claude Code has been fixing your order-processing bot, and after every task it appends its own "lessons" to CLAUDE.md. The file now holds 140 rules. Lately the agent has been acting weird: poking into files it has no business touching, asking about the obvious, breaking things that used to work. You need to figure out which rule blocks help and which hurt. Asking the agent is pointless: it thinks every one of its rules is useful.

Prompt (in a regular chat; it's the template below, filled in):

You are helping me run a manual experiment using the Randomized Block Curation method.

Context: I have a CLAUDE.md file with 140 rules that the agent wrote for itself after working on tasks. The agent is Claude Code, working in the repository of an order-processing bot for my Shopify/Amazon/Etsy store.

Step 1. I'll send you the rules. Split them into 8 thematic blocks (git workflow, tests, marketplace APIs, code style, task completion, etc.). Output each block's name and its rule numbers.
Step 2. Build a randomization table for 24 test runs: rows are runs, columns are blocks, each cell is 1 (block on) or 0 (block off), with probability 1/2, independently. For each run, add the assembled list of block files to paste into CLAUDE.md.
Step 3. I'll run 12 typical tasks from my backlog (each one twice, in different rows of the table) and send back a score for each run: 1 = task solved, 0 = not solved.
Step 4. For each block, compute: the mean result of runs where the block was on minus the mean result of runs where it was off. Compare within the same task.
Step 5. Keep the blocks with a positive difference, starting with the largest, until you reach 40% of the original size. If none are positive, recommend working without CLAUDE.md.
Step 6. Propose a test of three candidates — "pruned file," "full file," "empty file" — on 12 new tasks that were not used in steps 2–4.

Start with step 1. Here are the rules:
{paste the contents of CLAUDE.md}

Result: the model splits the rules into blocks and gives you a table of zeros and ones for 24 runs, with the block set assembled for each. After your runs it calculates the effect per block and suggests what to keep. Odds are some block (say, "completion rules") will turn out to have a negative effect, even though the agent considered it the most useful. At the end you get a plan for the final showdown: pruned vs. full vs. empty.

The weakness: the agent only sees its lessons in the runs where they were applied. It can't compare "with the rule" against "without the rule." So the "helpful" rating quietly turns into a "sounds reasonable" rating. A rule that touches every run collects "helpful" tags in every one of them, even if its effect is negative. Models are generally bad, and biased, at judging their own conclusions. On top of that, the outcome depends on the randomness of growth: one run's playbook helps, another run's playbook hurts.

The strength: random inclusion makes "with block" and "without block" runs comparable. The difference in success directly shows the block's effect on success, not on plausibility. All 8 blocks are measured in the same runs rather than one at a time, so you need 4.5 times fewer runs. Blocks (rather than individual lessons) are used because a single rule rarely moves success more than the noise does across a few hundred runs.

Control knobs: - Number of blocks (8) → matters little; 5–10 works. - Token budget (40%) → 20% and 60% did worse; start around 40%. - Number of growth runs K (3) → more playbook versions raise the odds of finding a good one; two runs with pruning equal three runs without it. - "No playbook" among the candidates → your insurance. If every version hurts, the empty one wins. - Fresh tasks for the final pick → don't reuse the runs you measured effects on, or the winner will be overrated.

In the paper, the method is implemented in code. Below is a manual version that keeps the logic: blocks → random inclusion → difference of means → budgeted selection → candidate test, including empty memory.

You are helping me run a Randomized Block Curation experiment on an agent's self-written memory.

<Context>
Agent: {agent_and_tasks}
Memory file (lessons the agent wrote itself): {memory_file}
Success criterion for a single run: {success_criterion}
Typical tasks for testing: {num_tasks} tasks from a real backlog
</Context>

<Steps>
STEP 1 — BLOCKS: split the lessons into {num_blocks} thematic blocks (default 8). Output: block name, lesson numbers, size in tokens.

STEP 2 — RANDOMIZATION: build a table for {num_runs} runs. Columns are blocks. Each cell is 1 (block on) or 0 (block off), with probability 1/2, independently of the other blocks. For each run, assemble the final list of blocks to paste into the memory file. Each task must appear in the table several times with different block sets.

STEP 3 — RUNS: I'll execute the runs and send the scores (1/0 or 0 to 1). Do not interpret anything until you have the results.

STEP 4 — EFFECT: for each block, compute the mean success of runs with the block minus the mean success of runs without it, comparing within the same task. Output a "block — effect" table in descending order.

STEP 5 — SELECTION: keep blocks with effect > 0, starting with the largest, until you reach {budget}% of the original size. If no block has a positive effect, the result is "empty memory."

STEP 6 — VALIDATION: if I have several versions of the memory file (growth runs), repeat steps 1–5 for each. Then propose candidates: pruned versions, full versions, empty memory. Describe how to run them on NEW tasks that were not used in steps 2–4, and how to pick the winner by mean success.
</Steps>

Do not judge lessons by how reasonable they sound. The only criterion is the effect on success from the table.
Start with STEP 1. Here is the memory file:
{memory_file}

What to fill in: - {agent_and_tasks} — which agent and what it does; - {memory_file} — the file's text (CLAUDE.md, AGENTS.md, a saved "cheat sheet"); - {success_criterion} — something you can check unambiguously: tests passed, order placed, answer matched the reference; - {num_blocks}, {num_runs}, {budget} — 8, 24–50 and 40 as a starting point.

🚀 Quick start — paste into a chat:

Here is a template for the Randomized Block Curation method. Adapt it to my task: [your task]. 
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what your memory looks like, how you measure a run's success, and how many tasks you can realistically run. The success criterion matters because without it you can't compute the "with block" vs. "without block" difference. Sample size determines how many blocks you can compare honestly.

⚠️ You need a verifiable success metric: the method measures effects by run outcome. If "good or bad" is an eyeball call that people argue about, the difference of means drowns in noise.

⚠️ You need lots of runs: the paper's estimates rested on hundreds to thousands of runs. A manual version with 24 runs gives a rough read on the sign of the effect, not precise numbers. Conclusions about blocks with small effects are unreliable.

⚠️ In the paper, the method is code: clustering, randomization and regression adjusted for task difficulty are done by a program. The manual version above is a simplification. You can build it with an LLM's help, but it's noisier.

⚠️ Playbooks that get rewritten wholesale: in Dynamic Cheatsheet the lessons are less separable, and pruning gave no gain (61.1 versus 61.3 for the full version). The method only cuts when the blocks are genuinely separable.

⚠️ Most of the gain comes from selection, not pruning: the bulk of the effect comes from two things: growing several versions and comparing them against "empty" memory on fresh tasks. Block pruning adds roughly one more point on ACE (67.2 versus 66.1) and saves tokens. If you don't have the bandwidth for the full method, start with a simple A/B: "memory vs. no memory."

⚠️ Block interactions: a block's effect is averaged over all combinations. A block that only works when paired with another will be underrated. The final test of whole playbooks guards against this.

⚠️ Tested on agentic benchmarks: AppWorld, WebShop, ALFWorld — with automatic scoring. Transfer to office and creative tasks wasn't tested.

The team started with the question "does this even need fixing?" They grew three playbooks for each of 11 "model + benchmark" combinations in the ACE system. The models were open-weight: Qwen and Gemma. A playbook counted as harmful if it trailed the no-playbook run by more than 5 points; a repeat measurement moved the score by at most 5, so a gap that size isn't noise. The result was brutal: on one "model + benchmark" pair the playbook lifted success from 74.9 to 97.5, while on another every playbook hurt, one of them by 69.6 points. The same process on the same tasks in a different order produced 82.5, 58.9 and 30.0 against 78.4 with no memory.

Next question: what if the agent prunes the harmful lessons itself, based on its own "helpful/harmful" tags? No luck: in three out of four cases the "pruned by tags" playbook still hurt. The correlation between the agent's tags and the measured block effects was just 0.21. The most harmful blocks carried more "helpful" tags than almost all the others. That's the core discovery: the agent rates plausibility, not usefulness.

Then RBC was compared with six approaches: "no playbook," "full playbook," selection by the agent's tags, selection by similarity to the task (as in ReasoningBank), random selection, and a strong competitor, Best-Full, which picks the best whole playbook or empty memory based on practice tasks. All methods got the same playbooks and tasks with the same token budget. RBC's constants were tuned on four settings without touching their test tasks and then frozen for the other fifteen. Result across 19 settings: 66.9 for RBC versus 55.5 for "full" and 54.1 for "no playbook." On ACE, RBC never fell more than 0.8 points below "no playbook," while every other method fell at least 5.4.

A fun detail: five growth runs with best-of selection catch up to RBC on three, but ship 27% more tokens. So pruning can't be replaced by simply "I'll grow more." Another practical finding: on Dynamic Cheatsheet, where lessons are fused into a single text, pruning didn't help, and RBC picked up on that from the practice tasks and simply returned the whole playbook.

💡 Adaptation for an agent rules file (CLAUDE.md, AGENTS.md): same principle, no regression. Take one file, split it into 4–5 blocks, run the same set of tasks in 8–10 variants ("everything," "without block 1," "without block 2," etc.), and compare. It's crude, but it exposes the troublemaker block. The paper didn't do it this way (it used random inclusion), but the logic is the same: judge a block by its effect, not by how it sounds.

Here is my agent instructions file: {file}
Split it into 5 thematic blocks. Make a plan of 10 test runs: 
variant 0 — all blocks, variants 1–5 — all blocks except one, variant 6 — empty file.
For each variant, list which blocks to include. Then I'll send you the results 
on {num_tasks} identical tasks, and you'll tell me which blocks are hurting.

🔧 Technique: flag the "suspicious" rules → speeds up the hunt for troublemakers. In the paper, the harmful rule was one that touches every run (the task-completion call). In practice: check the rule blocks that fire on every task first, because they collect the most "false confirmations."

From the rules file, list the rules that apply to EVERY task (completion, answer format, 
general prohibitions). For each one, explain what behavior it pushes and in which cases 
it could backfire. We'll test these blocks first.

Extrapolation: the "memory is a decision, not a default" rule. Any notes file the agent maintains itself (ChatGPT memory, project notes, "lessons") is worth keeping in two variants and periodically comparing "with memory" vs. "without" on five to ten real tasks. If there's no difference, keep the empty variant: it's cheaper in tokens and doesn't risk sneaking in a harmful rule. The paper's findings on memory (RBC) weren't directly tested on this practice; it's a transfer of the logic.

• Paper: Reliable Memory from Unstable Growth: Curating Self-Written Agent Playbooks with Randomized Experiments
• Authors: Hoang Pham, Cong-Duy Nguyen, Dong Le, Ponhvoan Srey, Anh Tuan Luu — Nanyang Technological University and VinUniversity
• Code: anonymous repository anonymous.4open.science/r/rbc (cited in the paper)
• Related systems: ACE (Zhang et al., 2026), Dynamic Cheatsheet (Suzgun et al., 2026), ReasoningBank (Ouyang et al., 2026), RSEA / Best-Full (Nguyen et al., 2026)
• Benchmarks: AppWorld, WebShop, ALFWorld
• Related ideas: ContextCite (Cohen-Wang et al., 2024), the Banzhaf value (Banzhaf, 1965)