Curating AGENTS.md: an agent's instruction file needs a size ceiling, not just endless appending

STEP 1: New notes → into a separate candidate list, NOT into the file
STEP 2: Admission to the file → only after repeated independent cases (in theory the optimum is two)
STEP 3: Score each rule → how often it's needed × benefit/harm if followed − cost in tokens
STEP 4: Removal → by value, not by "the agent ignores it"; harmful rules go first
STEP 5: Size ceiling + small batches of edits → test on real tasks

Steps 1–4 can be done in a single chat: upload the file and transcripts, get a table of verdicts. Step 5 is your manual control.

Task: A developer maintains the Python backend of an online store with payments through Stripe. For six months, after every Claude Code session, they appended notes to CLAUDE.md. Now the file has 280 lines. Among them: "always run pytest", "write NumPy-style docstrings", "use conda" (the project moved to uv long ago), "don't commit to main", "don't use the phrase "Dear valued customer" in customer emails" and dozens more style nitpicks. The agent keeps forgetting about pytest and "don't commit to main".

Prompt:

You are the editor of an instruction file for a coding agent. Below is my project's CLAUDE.md (a Python backend for an online store, payments through Stripe, environment: uv) and three transcript excerpts where the agent made mistakes.

<file>
[paste the 280 lines of CLAUDE.md]
</file>

<failures>
[paste 3 cases: what the agent did wrong]
</failures>

For EACH rule, fill in a table row:
rule | approximate tokens | how often it's relevant (often / rarely / almost never) | what happens if the agent follows it (benefit / neutral / harm) | verdict (keep / delete / merge with another / move out of the always-loaded file)

Editing rules:
1. Don't propose deleting a rule ONLY because the agent ignores it. First assess what breaking it costs.
2. Rules that cause harm when followed (outdated, contradicting the project) get deleted first.
3. Target size: no more than 60 lines. If there are more important rules, name which ones pushed out the rest and why.
4. Don't add new rules. If the failures reveal a gap, put it in a separate "candidates" list and note how many times the gap showed up.
5. At the end, give the final file and a list of 5 test tasks I can use to compare the old and new versions.

Result: The model will produce a table of verdicts for each rule and flag the harmful ones, such as the outdated "use conda". Next comes a compressed file within the limit and a separate candidate list with repeat counts. At the end you get the test tasks. The model can't decide for you what really matters in your project. Skim the table yourself.

The weakness. The model has limited "attention": the more rules in the prompt, the worse each one gets followed. The file's tokens are paid for in every session. That's why the rule "add every note that seems reasonable on its own to the file" breaks down. While the file is small, you don't see it. When there are many rules, some crowd out others.

The strength. The model is good at classifying and comparing: "needed often or rarely", "harmful or not", "does it duplicate another rule". The prompt above leans on exactly that. It forces rules to be scored by value, not by whether they get followed.

The workaround. The session log only shows what was loaded: whether a rule was followed and whether it caused harm. What was missing only shows up once the absence of a rule has already caused a failure. That's why you need to separate "the agent ignores it" from "the rule is useless", and only adopt new rules after repeated cases.

Control knobs: - Line limit → the lower the limit, the tougher the selection. For a small project, 20–60 lines is plenty. - Admission threshold → the number of repeated cases before a rule gets added. Under their assumptions, the authors got minimal losses with two observations. For critical rules (security, money) you can use one. - Verdict criterion → add "cost of an error" so rare but expensive rules don't get crowded out by frequent trivia. - Edit batches → change 5–10 rules at a time and test. That way you can see which edit helped.

This is my own assembly based on the paper's principles. The paper has no ready-made prompt.

You are the editor of an instruction file for an agent ({agent_name}). Below are the file and examples of failures.

<file>
{file_contents}
</file>

<failures>
{failure_cases_from_transcripts}
</failures>

Project context: {project_and_environment_description}

For EACH rule, fill in a table row:
rule | tokens | relevance (often / rarely / almost never) | effect when followed (benefit / neutral / harm) | verdict (keep / delete / merge / move out)

Editing rules:
1. Don't delete a rule just because the agent ignores it. First assess the cost of breaking it.
2. Delete harmful and outdated rules first.
3. Target size: no more than {line_limit} lines. If it doesn't fit, explain what got pushed out and why.
4. Don't add new rules. Put gaps revealed by failures into a "candidates" list with repeat counts.
5. A rule is admitted to the file only if the problem showed up at least {repeat_threshold} times.
6. At the end, give the final file and {number} test tasks for comparing the old and new versions.

Fill in: the whole file, 3–10 failure cases from real transcripts, a short project description, the line limit and the repeat threshold (2 by default).

⚠️ The experiment is far from real files: The optimum and the quality drop were measured on pools of 200 to 3,200 instructions. Real AGENTS.md files have dozens of lines. For small files, the authors themselves cite studies where no drop in rule compliance is seen.

⚠️ Artificial instructions: The rules were taken from real files, but a model rewrote them into checkable rules about words in the answer. The tasks are texts in various formats, not a coding agent working in a repository.

⚠️ Narrow testing: Only 60 tasks and two models from the same family. Don't carry the specific optimum numbers over to your own case.

⚠️ The theory rests on assumptions: The main assumption is that each added instruction doesn't increase compliance with the others. The "two repeats before adding" number holds within the model and wasn't tested on live agents.

⚠️ Some conclusions require engineering: The algorithm of trial-and-error over batches of rules and value scoring assumes code and manual review. By hand you can apply the principles, but not the full algorithm.

⚠️ Effect size in the realistic range isn't shown: The multi-fold drop was observed when the entire pool of thousands of instructions was loaded.

The researchers first framed the instruction file as an assortment-selection problem: each rule has a cost in tokens, a chance of being needed and a benefit when followed (sometimes negative). The model's attention is shared and gets "diluted" as the file grows. From this they mathematically derived that the useful file size is bounded and doesn't grow along with the number of accumulated ideas. They also showed that "adding everything that's useful on its own" can be arbitrarily worse than the optimum. Separately, they analyzed what can be learned from logs: you can see rule compliance and harm, but not the size of the benefit. So deletion based on non-compliance can't tell the valuable from the useless.

For testing, they assembled a pool of 3,200 rules from real repository files. Claude Sonnet rewrote 2,560 of them into automatically checkable rules about words in the answer. The remaining 640 were left as is and contributed nothing to the scoring. The pool and files were scaled up in steps (files from 5 rules to the whole pool), on 60 ManyIFEval tasks in six formats. Rules went into the system prompt, the task into the user prompt, for 2,880 runs each for Claude Haiku 4.5 and Sonnet.

The result: with a pool of 3,200, both models worked best with a file of 200 rules. Haiku followed about 5 relevant rules, Sonnet about 24. When the whole pool was loaded, these numbers fell to 0.6 and 4.0. Surprisingly, as the pool grew, the best file size stayed small. The practical takeaway: a growing list of ideas doesn't justify a growing file.

Below are my own ideas, not from the paper.

🔧 Technique: a rule for an agent that writes its own memory → admission by repetition

If your agent appends lessons to a memory file, add this to its instructions:

Write new end-of-session notes to candidates.md, not to AGENTS.md. 
For each one, note how many different sessions the problem showed up in.
Move a note into AGENTS.md only when it has come up at least twice in different sessions 
and you can name exactly what will break without it.
When moving it, name an existing rule that can be deleted so the file doesn't grow.

This follows directly from the findings about two signals and the size limit. The "delete one when adding one" part is my own addition.

Extrapolation: test before deleting. Before removing an "ignored" rule, run 3–5 tasks where it matters, with and without it. If the agent starts making mistakes without it, the rule needs to be strengthened or moved higher, not deleted.

• Curating Always-Loaded Context for LLM Agents: A Capacitated Assortment Model with Censored Feedback — Zexuan Liu (University of Illinois at Urbana-Champaign), Yuning Yang (Georgia Institute of Technology), Tiancheng Zhao (Saint Louis University)
• ManyIFEval (Harada et al., 2025) — tasks with many instructions
• Gloaguen et al. (2026) — context files and rising request costs
• Chakrabarti (2026) — file growth across 1,867 repositories
• McMillan (2026), Zhang et al. (2026a) — studies showing no compliance drop in small ranges