TokenEconomizer: Cutting Coding Agents' Token Bills with "Repository Experience"

BEFORE EACH TASK the controller looks at the repository's history and decides:

1. MICRO-SKILLS:   compress past traces → an LLM proposes one-line facts
                   → a fact is enabled if confirmed across several tasks
                     touching DIFFERENT files and it was actually needed
                   → a fact is retired if contradicted even once (penalty > reward)
2. TOOL PRUNING:   a tool is removed if it went uncalled for N tasks in a row
                   OR an LLM judge deems it irrelevant to the task
                   → every so often, restore everything and re-check
3. GUARD:          turn budget = high percentile of turns on SUCCESSFUL past tasks × margin
                   → cheap rule-based check after every call
                   → on suspicion, a judge reviews a digest: stuck or making progress?
                   → stuck: redirect ("change your approach") → repeat: stop
                   → hard stop as a safety net

All of this is a separate, code-based controller. You can't reproduce it with a single chat prompt, but you can reproduce its principles.

Task: A team is building the backend for a quick-commerce delivery service in Django, something along the lines of Gopuff or Instacart. They use Claude Code. On every task the agent burns its first turns on the same routine: figuring out how to run the tests (docker compose exec web pytest), trying pip install where everything is already installed, and hunting for where the fixtures live. On top of that, six MCP servers are connected, and only two are actually needed. The monthly bill has gone up, and nobody can say why.

Prompt (a manual audit of recent sessions; paste the logs or session exports below):

You are a cost auditor for a coding agent. Below are the logs of the agent's 10 most recent sessions
in the "delivery-backend" repository (Django, docker compose, pytest).

<Task>
Find where the agent wastes turns and propose edits to its instructions and toolset.
</Task>

<Step_1_repeats>
List the actions that repeat in at least half of the sessions
(looking up the test command, unnecessary installs, searching for the same files).
For each one, write ONE line-long fact for CLAUDE.md.
For each fact, state:
- how to verify the fact is true (a command or a signal in the log);
- in how many sessions it was confirmed and in which files (the files must not be the same across all sessions);
- in how many sessions the agent actually lost turns because of it.
Mark any fact that turned out wrong even once as "CONTRADICTED — do not include".
</Step_1_repeats>

<Step_2_tools>
Build a table: tool / times called / in how many sessions.
Propose disabling the ones that were never called. For borderline ones, write
which task might still need them.
</Step_2_tools>

<Step_3_spinning>
Find sessions where the agent kept repeating the same error for a long time.
For each: at which turn it became clear it was stuck, how the stuck session
differed from a successful one, and what redirect phrase
should have been given to it ("change your approach: ...").
</Step_3_spinning>

<Format>
Return: (a) a block of lines for CLAUDE.md, (b) a list of tools to disable,
(c) a "when to stop the agent" rule. Do not invent facts that are not in the logs.
</Format>

[paste session logs here]

Result: The model returns three blocks. The first is a short list of verifiable facts for CLAUDE.md, each marked with how many sessions confirmed it. The second is a table of tool calls plus a list of candidates to disable. The third is a set of "agent is stuck" signals with a ready-made redirect phrase. You apply the edits by hand and compare the bill over the next 5–10 tasks.

Weakness: the agent remembers nothing between sessions, and every turn it takes is billed together with the whole accumulated history and the schemas of every loaded tool. So everything the agent re-figures-out at the start of a task, and every unnecessary tool, gets paid for on every single step. "Universal" accelerators don't help, because every project has its own quirks: what compresses output nicely in Django gets in the way in SymPy.

Strength: LLMs are good at compressing traces into short facts and at telling "stuck" apart from "hard but moving," as long as you give them a compact digest. A one-line fact in the instructions costs pennies but saves turns on every future task. An unnecessary tool is the easiest thing to cut: it's billed on every turn of every task. That's why in the ablations tool pruning made the biggest contribution (–13…–18%), micro-skills came close (–12…–14%), and the guard contributed noticeably less (–5…–7%). But no single technique matched the combined result on its own.

Control knobs: - Fact confirmation threshold → stricter means fewer facts make it into the instructions, but all of them are reliable. Require confirmation on tasks with different files to weed out "single-task facts." - Penalty asymmetry → one contradiction outweighs one confirmation. A stale fact gets paid for on every task and is dangerous, while a fact that was crossed out by mistake can always be proposed again. - Number of tasks without a tool call → fewer means more aggressive pruning. Always keep the core tools (shell, read, write, edit). - Re-checking the toolset → every so often, restore the full set and see whether anything turned out to be needed. - Turn budget margin → base it on successful past tasks in this repository. There's no single limit for all projects: in Matplotlib, failures run 45 turns versus 20 for successes; in Sphinx, 36 versus 31. - Redirect first, then stop → start by asking the agent to change its approach, and only cut it off if it repeats.

The available part of the paper doesn't include the controller's actual prompts. Below is a manual port of the method's principles into chat: the micro-skill "proposer" role from the first meta-strategy. The structure keeps the original's logic: candidate, detector, confirmation, retirement.

<Role>
You are a micro-skill proposer for a coding agent in the {repository} repository.
Micro-skill = ONE line of operational fact about the project + a way to verify it.
</Role>

<Evidence>
{recent_session_logs_or_digests}
</Evidence>

<Logic>
For each recurring action in Evidence:
  1. Draft a candidate: a one-line instruction for {instructions_file}.
  2. Come up with detectors:
     - scope: what signal in the log shows the situation arose at all;
     - contradicts: what signal shows the fact is WRONG;
     - need: what signal shows the agent lost turns on this
       or used this fact.
  3. Check every session: exercised? confirmed / contradicts? needed?
  4. Status:
     - ACTIVE — confirmed in at least {min_tasks} sessions with NON-OVERLAPPING
       file sets AND needed in a share of sessions no lower than {need_threshold};
     - WITHHELD — confirmed, but rarely needed (don't insert, don't delete);
     - RETIRED — has a contradiction that outweighs the confirmations
       (one contradiction weighs more than one confirmation); never propose again;
     - CANDIDATE — not enough data.
</Logic>

<Output>
Table: candidate | status | confirmations | contradictions | needed in how many sessions.
Then an "ACTIVE" block — ready-to-use lines for {instructions_file}.
Do not add facts that are not in Evidence.
</Output>

What to fill in: - {repository} — the project name and stack. - {recent_session_logs_or_digests} — commands, test runs, names of changed files, and error messages from 5–10 sessions. - {instructions_file} — CLAUDE.md, AGENTS.md, or the equivalent. - {min_tasks} — for example, 3. - {need_threshold} — for example, 50%.

🚀 Quick start — paste into the chat:

Here is a "micro-skill proposer" template for a coding agent. Adapt it to my project: [description of the repository and agent].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about your stack and how you run tests, which agent and which instructions file you use, and what session logs you have. It needs this to word the detectors correctly: what in your log shows "the situation arose," "the fact is false," and "time was wasted." Then it will tailor the template to your log format.

⚠️ The full method is code: the controller monitors agent events in real time, computes percentiles, and runs the judge on the fly. In a regular chat you can only reproduce the manual version: log audit, instruction edits, disabling tools. The authors did not measure the effect of the manual version.

⚠️ You need a success signal: the turn budget is calculated from successful past tasks. Without tests or some other clear "task solved" criterion, the loop guard is flying blind.

⚠️ The first tasks cost more: the controller pays for its own LLM calls and knows nothing about the project yet. In almost every repository, costs went up at first, and savings only kicked in around the sixth task. For one-off tasks in a new repository, you won't come out ahead.

⚠️ Facts go stale fast: if a fact in the instructions becomes false, you pay for it on every turn and send the agent down the wrong path. You need a retirement mechanism, and for the manual version, a regular review of CLAUDE.md.

⚠️ Outside coding, the gains are smaller: on sequential web tasks and security audits, savings were 6–21%, depending on how the tasks were structured. Quality stayed comparable, but in one of the five "agent–model" pairings, Pass@1 on SWE-Bench dropped by 1 pp.

⚠️ Third-party plugins applied blindly can hurt: command-output compression (RTK) and the repository map (RepoMap) made work more expensive in a sizable share of cases. Don't install them "because everyone does" without measuring on your own project.

⚠️ Lowering effort to save money is risky: it cuts the bill the most, but drops the share of solved tasks by 2–10 pp.

The authors didn't start by proposing a method; they first figured out where the money was leaking. They took 60 SWE-Bench tasks from six repositories and ran three popular "accelerators": command-output compression (RTK), semantic navigation (Serena), and Aider's repository map. None of them saved money consistently: in Django all three helped, in SymPy all three made work more expensive. That showed there are no fixed recipes and the policy has to be learned from a specific repository's history. Then the authors looked at untouched traces and saw three leaks: repeated actions (12–28% of calls; after the fifth task, about 30% of calls repeat the past), dead tools (16 of 23 never called), and expensive failures (17% of tasks accounted for 32% of costs).

Next they built the controller and tested it on five "agent–model" pairings: Claude Code with two models, Copilot CLI, and Codex with two models. That's 277 tasks from six repositories, three runs each, with all of the controller's own costs included in the bill. Tasks were run in chronological order so the controller learned only from "the past." The result: cost dropped by 18–31%, and Pass@1 rose in four of the five pairings. There were savings in every one of the 30 "repository × pairing" combinations (5–40%), whereas RTK made work more expensive in 12 combinations and RepoMap in 15. Transferring to web development (102 tasks) and security auditing (200 tasks) yielded 16–21% and 6–20%.

There were two surprises. The biggest contribution came not from clever "memory" but from boring tool pruning, because it acts on every turn of every task. The guard contributed the least: nearly half of repeated errors happen in runs that go on to finish successfully, so "repeated an error" doesn't yet mean "time to stop." The practical takeaway: first strip out the excess, then write down recurring facts, and only then worry about stopping.

🔧 Technique: removing a tool from the set → check what would replace it

Before disabling anything, ask the model to check whether you'll lose something you need:

For each tool in the to-disable table, name a task from my
backlog below where it would be irreplaceable. If there is no such task, say "safe to disable".
Backlog: {upcoming_tasks_list}

This is a manual analog of the method's "relevance judge": the decision takes into account what you're about to work on.

🔧 Technique: a redirect phrase → "change your approach" instead of canceling

When you see the agent banging its head against the same error, give it a summary, not just "stop":

Stop. Here's what you did over the last few steps: {command_list}. The error isn't changing:
{error_text}. Name 2 hypotheses for why it isn't working, pick the one that differs
from your previous attempts, and test it with a single command.

This is the manual version of "redirect, and stop on a repeat." The math is simple: changing approach is cheaper than paying for another lap of the same actions.

Extrapolation: a "history-based turn limit" rule in CLAUDE.md. The authors compute the budget from successful tasks in their own repository. The manual version: write the typical number of turns for your tasks into the instructions, plus a rule: "if you're on the third lap with the same error, stop and describe what you tried." This is my own extension; it was not tested in the paper.

• Title: TokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents
• Authors: Aaron J. Li, Weijia Xu, Chandan Singh, Zixiao Chen, Jeevana Priya Inala, Charlotte Siska, Acar Tamersoy, Bin Yu, Ion Stoica, Michel Galley, Jianfeng Gao
• Affiliations: UC Berkeley, Microsoft Research, Microsoft Security
• Project page: https://tokeneconomizer.github.io
• Code: https://github.com/microsoft/TokenEconomizer
• Related tools: Rust Token Killer (RTK), Serena, Aider RepoMap
• Benchmarks: SWE-Bench, Web-Bench, CredData, ReposVul

Some appendices and tables in the provided text were truncated, so the controller's prompt texts and the micro-skill examples (Appendix I) are not reproduced here.