Rule Replay: Checking Whether Your AGENTS.md Rules Still Matter, and Auditing Stale Documentation

STEP 1: Failure log → one row per recurring failure:
        date, category, how it was found, fix, fix TYPE (rule / code check), did it recur
STEP 2: Choose the fix type → “agent reads” (text rule) or “code blocks” (check/hook)
STEP 3: Once a month → audit: a read-only agent cross-checks docs against the file tree and git
STEP 4: Rule replay → 2 repo copies (with the rule / without), one task, 3 runs on each
STEP 5: No failure even without the rule → rule is a deletion candidate; failure → keep it, ideally turn it into a check

Steps 1–3 and 5 are done with regular chats and an agent. Step 4 is manual: two folders, the same request, no code.

Task: You're building a WhatsApp booking bot for “Sharp & Bearded,” a barbershop chain in Austin. The repository has a 400-line CLAUDE.md accumulated over six months. Some rules appeared after the old model's mistakes: “don't quote a price without checking price.md,” “always check the latest CHANGELOG entry, not the first one,” “task estimate: 1–5 minutes.” You've switched to a new model and want to figure out which rules you can throw out.

Prompt (run in two copies of the repository; in B the rule “check the latest CHANGELOG entry” is deleted):

Task: add a bot reply to the question “how much is a haircut with a beard trim?” 
Use the repository documentation as your guide. If the docs contain 
contradictions, report them as a separate item at the end.

Run it 3 times on copy A (rule present) and 3 times on copy B (rule absent). Use the same commit message in both copies so the condition can't be guessed from git log.

Result: You'll get 6 answers and compare them on a single criterion: did the agent take the price from the stale CHANGELOG entry? If the price is correct in all 6 answers, the rule is probably redundant on this model and can go. If the error shows up in copy B, keep the rule and consider replacing it with a check, for example a script that compares price.md against the bot's text.

The weakness. A text rule is just advice. The model reads it and may ignore it, and after a model upgrade it may become unnecessary. Nobody finds out: the rule is still sitting in the file, looking like it's protecting you. On top of that, documentation goes stale not because of the model but because you keep changing the project. A text rule saying “update the counters” doesn't stop that.

The strength. Comparing “with the rule vs. without” turns “it's probably needed” into a simple experiment. LLMs are good at reading lots of files and cross-checking claims against each other. That's why the audit finds stale spots with high precision. Code is good at one thing: it checks without any “please” and fires every single time.

How the method combines them. For staleness, you set up a code check (narrow but reliable) and add an LLM audit (broad, but finds about half). For behavioral rules, you run a replay and remove whatever changes nothing.

Control knobs: - Number of runs (3 in the study) → increase it for important rules so you don't mistake chance for a pattern. - Model → repeat the replay on the old and the new model. That shows it was specifically the upgrade that made the rule unnecessary. - Fix type → if the rule matters, replace the text with a check that code executes. - Audit setup → add an “independent” check (a different model or a second run): one pass catches about half.

The paper has no verbatim prompt: the audit ran on a “fixed prompt” whose text isn't given. Below are templates built from the described mechanics.

Template 1. Monthly documentation audit (the agent only reads):

Role: documentation auditor. You work read-only: don't edit anything.

Task: find claims in the documents {folders_or_files} 
that are outdated relative to the current state of the repository.

Verify against reality:
- file paths: do they exist in the file tree
- numbers and counters ({what_we_count}): do they match the actual count
- schedules and dates: do they match {source_of_truth_location}
- names of scripts, functions, identifiers: have they been renamed (check git log)
- descriptions of script behavior: do they match the code

For each finding, output one line:
[file:line] → quoted claim → what's actually true → evidence (file or commit)

Don't include findings you can't back up with a file or git history. 
At the end: how many files you checked and what you did NOT check.

Template 2. Rule replay (run in two copies of the repository):

Copy A: the repository with the rule {rule}.
Copy B: the same repository, rule deleted.
Task that used to cause the failure: {task}.
Failure signal: {what_counts_as_error}.

Run the task 3 times in each copy, in a fresh session. 
For each run, record: failure / no failure, and what exactly the agent did.

What to fill in: {folders_or_files} — the documents you're responsible for. {what_we_count} — number of skills, commands, pricing plans. {rule} — a specific rule from your instructions file. {what_counts_as_error} — a measurable signal like “price not taken from price.md” or “em dash used.”

🚀 Quick start — paste this into your agent's chat (Claude Code, Cursor):

Here's a documentation audit template. Adapt it to my repository: [describe the project]. 
Ask me questions to fill in the fields.

[paste the template above]

The agent will ask where the documentation lives, what in it can be checked against reality (numbers, paths, dates), and where the source of truth is. That way the audit compares docs against verifiable facts instead of “rating the quality of the writing.”

⚠️ One repository, one author: all results come from a single personal repository with three runs per condition. The author himself calls the work preliminary. It's a signal, not proof.

⚠️ “Didn't recur” ≠ “not needed”: three clean runs don't guarantee the failure won't show up on other tasks. Delete rules carefully: a rule may be guarding a rare edge case.

⚠️ The audit catches only half: a single LLM audit pass found roughly half of the known stale spots. Don't mistake “audit is clean” for “docs are up to date.”

⚠️ Code checks break too: a hook blocked a harmless edit because it quoted a dangerous command. A regex let git -C path reset --hard slip through. The secret scanner didn't look inside the changelog, where the agent had written an identifier. Background sync clashed with the agent's work.

⚠️ A text rule doesn't always change behavior: the time-estimate rule didn't fix Sonnet 4.5's inflated estimates. If you need a guarantee, you need a check, not a request.

⚠️ A size limit doesn't cure bloat: when the author capped the size of the main file, the rules simply moved into skills, and the total volume grew.

⚠️ Incomplete text: in the available version of the paper, part of the results (Figure 4 onward) is cut off. The audit and replay numbers come from the abstract and the methods section.

The idea was simple: instead of taking a snapshot of other people's repositories, the author spent six weeks watching one of his own, where three agentic tools edited a shared rule set every day. Every agent failure was logged without exception, along with its cause and fix. The result was a log of 118 failures. It was labeled in two LLM passes (Sonnet, then a “blind” Opus), and the author resolved disagreements himself.

The largest class was stale documentation (48 of 118), although 35 of those were found in a single day of audits. To test the audit, the author took 47 real stale claims and planted 20 artificial errors. He hid the edit traces from git so the agent couldn't “read” the answers off the diff. The audit was run three times each on Haiku, Sonnet, and Opus.

Then came the replay: five past failures were reproduced in two throwaway clones (with the fix and without). The result was a surprise: on the current model, in three of the four completed tasks the failure didn't happen either with or without the fix. Only the “forgot to update the counter” failure came back, and the code check stopped it in every run. On older models, an em dash slipped through once, and the check rejected it. Practical takeaway: code checks work predictably, text rules can turn into dead weight, and the absence of a repeat failure still has to be verified.

🔧 Technique: tag every rule → reviews become cheap

In the paper, each lesson was logged together with its cause. You can go further and put a tag right in the instructions file, so later you know what to re-check:

- Don't quote a price without checking price.md. 
  [reason: bot quoted an old rate on Mar 12 | model: Sonnet 4.5 | re-check after model change]

When you switch models, ask the agent to list every rule carrying this tag and run a replay on them.

Extrapolation: rule → a code check the agent writes itself

If a failure keeps recurring and the rule doesn't help, replace it with a script (an idea from the paper's related work: turning instructions into checks raised compliance from 67% to 88%):

The rule in AGENTS.md isn't helping: the agent still {failure_description}.
Write a small pre-commit check script that 
catches this failure by the signal {signal}, and demonstrate it on two 
examples: one where it fires and one where it doesn't. 
Suggest what to delete from AGENTS.md once the check is in place.

• Lessons Outlive Their Failures: A Case Study of Rules for AI Coding Agents — Hassan Hassanzadeh Aliabadi, independent researcher, October 2026
• Works cited: Chakrabarti (growth of instruction files across 1867 repositories), Cai et al. (survey of 99 practitioners), Gloaguen et al. and Lulla et al. (impact of AGENTS.md on outcomes), Zhang et al. (explicit prohibitions as the only useful rules), ContextCov, AgentSpec, Kang et al. (harness-level forgetting), Vasilopoulos (codified context)