Memory Tax: an agent's fancy memory comes with a "tax" — it loses facts that a plain search over the raw history would find

STEP 1: The raw history stays in storage as the primary source (throw nothing away)
STEP 2: Query → search both the memory (notes, digests) and the raw history
        → results are merged and ranked together → answer
STEP 3: Successful queries and the evidence found are appended to memory
        → the next similar question finds the answer right away

In the paper these steps are implemented in code on top of existing memory systems. The template below turns them into an instruction for the agent. The original is the paper's mechanics; the template is a transfer of the principle.

Task: You run a sales team and are setting up an agent wired into HubSpot. It "remembers" clients: agreements, objections, promised discounts. Right now it keeps short notes on each client and answers from them. A rep asks: "What did we promise Vector Inc. on the timeline?" The agent confidently says "two weeks." But the email said "two weeks after the down payment, provided the mockup is approved by the 15th." The note dropped the conditions.

Prompt (system instruction for the agent):

You are a sales team assistant. You have two sources about clients:
(A) NOTES — short entries you keep yourself;
(B) ARCHIVE — verbatim emails, calls and comments in HubSpot.

Rules:
1. For any question about a client, search BOTH sources, not just the notes.
2. Notes are a hint about where to look. The archive is the primary source.
3. If a note and the archive disagree, trust the archive and flag the discrepancy.
4. In your answer, give a verbatim quote from the archive and its date. If there is no quote, say "not found in the archive" instead of paraphrasing from memory.
5. Record commitments, numbers, deadlines and conditions in your notes verbatim, with the date and a link to the email. Do not paraphrase or round.
6. When you find an answer the notes didn't give you right away, append a line to the notes: "Question: … → answer found: [source, date]".

Question: What did we promise Vector Inc. on the timeline, and under what conditions?

Result: The agent checks both the notes and the archive, quotes the email verbatim with its date, and spells out the conditions: down payment, mockup approval. If the note said otherwise, it flags the discrepancy. At the end it appends a "question → where the answer was found" line to the notes.

The weakness. An agent's memory is a retelling of a retelling. When compressing, the model decides what matters before it knows the future question. Conditions, caveats and numbers look like "details" and fall out. What's lost can't be recovered: it simply isn't in the note. Some facts aren't lost but become unreachable: search compares the question to an entry, and the entry has already been reworded, diluted or split up.

The strength. A model answers well when it's given exact chunks of the source. It confidently assembles an answer from verbatim quotes and cross-checks two sources.

How the method uses this. Notes stay a fast "where to look" index, while the raw history provides the accuracy. The tax doesn't disappear, but it becomes reversible: if memory fails, the road back to the primary source is open.

Control knobs: - Rule 3 ("trust the archive") → swap for "ask a human when they disagree" if mistakes are expensive. - Rule 4 (verbatim quote) → relax it for routine questions so answers don't bloat. - Rule 5 (what to record verbatim) → extend the list for your field: legal wording, versions, prices. - Rule 6 (recording the search path) → drop it if the notes grow too fast.

You are {agent_role}. You have two sources of information about {memory_subject}:
(A) NOTES — short entries you keep yourself;
(B) ARCHIVE — {archive_contents}: verbatim source materials.

Rules:
1. For any question, search BOTH sources, not just the notes.
2. Notes are an index of where to look. The archive is the primary source.
3. If a note and the archive disagree, trust the archive and explicitly flag the discrepancy.
4. In your answer, give a verbatim quote from the archive and {reference: date/source/number}.
   If the archive has no confirmation, say so; do not paraphrase from memory.
5. Record critical data ({critical_items}) in the notes verbatim, with a link to the source. Do not paraphrase or round.
6. If the answer was found only in the archive, append a line to the notes:
   "Question: … → found: [source]".

Question: {question}

What to fill in: {agent_role} — who the agent is; {memory_subject} — clients, a project, a codebase; {archive_contents} — emails, logs, chats, documents; {critical_items} — what must never be lost: deadlines, amounts, conditions, decisions.

🚀 Quick start — paste into the chat:

Here is an instruction template for an agent with dual memory (notes + raw archive). Adapt it to my task: [your task]. 
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what's in your archive, which data is critical and how to format source references. That's exactly what determines what the agent records verbatim and how it quotes.

⚠️ Benchmarks favor plain search: most tests are "recall what was said" questions. That's where RAG is strong. For tasks that require generalizing habits or piecing together a picture from dozens of episodes, a digest may help. The authors show that memory sometimes solves what RAG doesn't.

⚠️ Domain-dependent: for agent experience (web tasks), compressed strategies sometimes beat plain search. On tool calls, many methods lost. The tax is everywhere, but its size varies.

⚠️ Needs tuning: in the paper, the changes were made in the memory systems' code. An instruction to the agent is a transfer of the principle that the authors didn't test directly. Check it on your own questions.

⚠️ Cost: dual search burns more tokens per query, and a long archive has to be searchable. If the archive is huge and unindexed, a chat agent won't cope.

⚠️ The paper is truncated: the numerical effects of the proposed improvements and some of the conclusions aren't visible in the text.

The team took twelve open memory systems from four families: flat stores, hierarchical, graph-based and learned policies. All were run on three models (a small one, a mid-size one and a recent one from OpenAI) and three conversational-memory benchmarks. The baseline was ordinary RAG over the raw conversation, with one and two rounds of retrieval. Retrieval token budgets were equalized so nobody won just by getting "more context."

The surprise was how strong plain RAG turned out to be. Many elaborate, expensive systems did worse. So the authors compared answers one by one (a Venn diagram) and saw that memory both helps and hurts: it solves some questions and loses others.

Then they hunted for the causes of the tax in three ways. Fidelity: they fed the model only the gold memory entries and checked how many facts were lost during construction. Accessibility: they counted how many retrieval rounds it took to pull up the evidence and how question-to-entry similarity dropped. Robustness: they built memory with one model and answered with another. It turned out that swapping the answering model breaks memory more than it breaks RAG.

Another result: even a primitive rare-word search scores over 50% on some popular benchmarks. So the tests are "retrieval-friendly," and the tax is especially visible on them. The same conclusions were then checked on agent experience and on video world models: a similar picture. Finally, the authors showed two simple tricks reduce the tax: a path back to the source, and recording past queries in memory.

🔧 Technique: ban paraphrasing in notes → fewer losses

Add this to the note-keeping rules:

Write notes in the format: [date] [source] "verbatim quote" + your tag.
Do not replace the quote with a paraphrase. The tag goes in a separate field.

That way a note stays both an index and a primary source, and paraphrasing doesn't wipe out the conditions.

Extrapolation: auditing memory in ChatGPT or Claude. The paper measures losses from compression. You can check your assistant's "memory" by hand:

Here is the list of what you remember about me: {memory_export}.
Here are the original chats for {period}: {texts}.
For each memory entry:
1. find a supporting quote in the chats;
2. rate it: fully preserved / conditions partly lost / no confirmation in the chats;
3. for partial ones, write what exactly was lost.
Output a table and a list of entries that need fixing.

This isn't the paper's method, but a transfer of its fidelity idea to a manual check.

• Memory Tax: Dissecting the Net Value of Agent Memory — a study of the net value of agent memory.
• Benchmarks: LongMemEval-s, PersonaMem-v2, MINT-Eval; agent experience — ALFWorld, WebShop, Mind2Web, BigCodeBench, AppWorld, τ²-Bench.
• Memory systems compared: A-MEM, SimpleMem, MemOS, MemoryOS, LightMem, OpenViking, HippoRAG2, PlugMem, MAGMA, Mem-T, Memory-R2.
• Authors and universities are not listed in the provided text.