TL;DR
Response-side instruction is a single rule in the system prompt of an agent with persistent memory: don’t name, imply or ask to confirm the user’s personal details until they bring them up themselves. Memory stays on. The agent simply stops saying out loud what it knows about the person.
The problem is a disclosure loop. The agent pulls a past fact from memory and slips it into a reply as a premise: “Since you work shifts, I’d suggest…”. The user goes along with it or doesn’t correct it: correcting means revealing even more, and people usually don’t push back on premises. The session gets written to memory, and the fact sticks, even if the agent only guessed it. Next session the same thing happens, and the gap versus a memoryless agent grows from session to session. The user can’t break out of the loop on their own. A correction reveals the attribute, refusing to discuss the topic nudges the model’s guesses toward “sensitive” values, and wrong facts get stuck in the record.
The instruction targets the one link the developer controls — the agent’s reply. Leakage drops to the level of a memoryless agent, usefulness doesn’t change, and most of the benefit of memory (continuity with past sessions) stays. Filtering and anonymizing what the user says or what’s stored in memory cuts leakage too, but eats quality.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
THE LOOP (what we break):
memory → agent IMPLIES a fact in its reply → user goes along / doesn’t correct
→ session summary records the fact → memory → ...
THE METHOD:
STEP 1: Add a rule to the agent’s system prompt:
don’t state, don’t imply, don’t ask to confirm
personal attributes unless the user raised them
→ one prompt, no extra calls
STEP 2: Memory stays on: facts shape the content of the reply but aren’t voiced in it
→ fewer mentions than even with no memory at all
STEP 3: Check: a sample of conversations, labeled “has a mention / doesn’t”
→ share of replies with mentions before and after
Example in Practice
Task: You’re building a support bot for an apartment rental service, something like Zillow Rentals or Apartments.com. The bot remembers past sessions. In January, a customer named Emma wrote: “My husband and I and our two kids are looking for a place in Denver, budget up to $2,500 a month.” In March she comes back with a question about the security deposit. Right now the bot replies: “For your family with two kids and a $2,500 budget, here are some 3-bedrooms…”. Emma nods along: “Yeah, yeah.” Facts about her family, its makeup and her income are now locked in. If the bot got something wrong (the kids are grown, the budget went up), correcting it is awkward, because she’d have to share details.
Prompt (system prompt excerpt):
You are the support assistant for the apartment rental service “HomeFinder Rentals.”
You have access to memory: summaries of past conversations with this customer.
Privacy rule:
Don’t state, imply or ask to confirm the customer’s personal details
(family, children, income and budget, health, age, employer, city of residence,
citizenship, marital status) until the customer brings the topic up in the current conversation.
What this means in practice:
- Don’t write “for your family with kids,” “with your budget,” “as you mentioned before.”
- Don’t phrase things as if the fact is known (“since you work shifts…”).
- Don’t ask “you’re still in Denver, right?”. Ask open questions:
“Which city are you interested in?”, “Which features matter to you?”.
- If the customer stated a fact in this conversation — use it freely.
Answer the substance of the question and keep continuity with the past:
don’t make the customer repeat things already settled about the issue itself (e.g., application status).
Result: The bot answers the deposit question without “for your family” or “with your budget.” If it needs parameters, it asks openly instead of suggesting the answer. It remembers and mentions the application status and the course of the earlier conversation, because those aren’t personal attributes. On a sample of 30–50 conversations, replies contain noticeably fewer mentions of personal data than without the rule.
Why It Works
The weakness. A model with memory tries to be coherent and personal, so it slips known facts into replies, often as a premise rather than a question. The user doesn’t object, because in everyday speech we let premises slide. The summary records the reaction, and the fact, even a wrong one, goes into memory. Each pass makes the next one more likely. That’s why simply removing the mentions has an effect roughly twice as large as a single turn would predict.
The strength. The model follows a direct “what not to say” rule well and can build a substantive reply without voicing anything personal. Without memory it actually asks open questions more often. That’s exactly what to copy: use memory for content, but not for claims about the person.
How the method uses this. The user won’t break out of the loop alone, and input filters and record anonymization also strip out what makes memory useful. A rule on the reply side cuts the cycle at the one link where you can intervene cheaply, and memory keeps working.
Control knobs: - The attribute list. Narrow it to truly sensitive ones (health, income, family) if you don’t want to lose personalization, or widen it to everything personal. - The “until the customer brings it up” condition. It lifts the ban once the person raises the topic, so replies don’t go tone-deaf. - An alternative to mentioning. Explicitly require an open question instead of a statement. - Allowed memory. Explicitly list what may be remembered out loud (application status, reply-format preferences) to keep continuity.
Prompt Template
The paper describes the instruction: the agent “does not state, imply or ask to confirm the user’s attributes unless the user raised them.” The exact wording is in Appendix N, which isn’t in the available text, so the template below is our own build based on that description.
You have access to memory: summaries of past conversations with the user.
Privacy rule:
Don’t state, imply or ask to confirm the user’s personal details
({attribute_list}) until the user brings the topic up in the current conversation.
What that looks like:
- Don’t name a fact about the user from memory (“as you said,” “for your {situation}”).
- Don’t build the reply as if the fact is known (a premise).
- Don’t ask “you’re still {fact}, right?”. Ask an open question.
- If the user stated a fact in this conversation — use it freely.
Keep continuity on {allowed_memory}: remember and mention it without restrictions.
What to fill in:
- {attribute_list} — what you consider personal: age, income, family, health, employer, city, citizenship.
- {situation} — an example of the typical “hint” the bot often drops.
- {allowed_memory} — what’s safe to remember out loud: application status, reply format, task stages.
🚀 Quick start — paste into the chat:
Here’s a privacy-rule template for an agent with memory. Adapt it to my product: [describe the bot and what it remembers about customers].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask which customer data counts as personal and what may be remembered out loud. These two lists define the method’s boundary: what’s off-limits to voice and what stays for continuity. It will plug them into the template and write a ready-to-use system prompt block.
Limitations
⚠️ Simulations, not real people: the users were played by a model, not humans. The check with real people was a pilot of eight participants, and it only confirmed the direction of the effect.
⚠️ The instruction was tested on two models: the main effect was observed on four support models, but the rule itself was tested only on two open models. For other models, check on your own conversations.
⚠️ This doesn’t protect the record: the rule damps the growth of leakage from memory but doesn’t bring it to zero. If an attacker has access to already accumulated memory, the rule won’t help.
⚠️ A small cost in continuity: usefulness didn’t drop, but continuity with past sessions dipped slightly. Restrict the list too much and you’ll lose more personalization.
⚠️ Memory-enabled agents only: in a chat without memory, where there’s nothing to carry between sessions, the rule is pointless.
⚠️ Models without reasoning and a limited horizon: the experiment ran with reasoning turned off and over 80 sessions. How the rule behaves over a very long history and with reasoning on is unknown.
How They Studied It
The researchers created 300 simulated users, each with 24 personal attributes (age, income, education, health, citizenship and others) drawn from US census data. Each user held 80 sessions with a support agent. In one condition the agent saw memory (summaries of past sessions); in the other it didn’t. The users’ requests were identical, and the difference arose only from how the conversations unfolded. Every five sessions, a model tried to guess all 24 attributes from the accumulated record. The share of correct guesses was called leakage.
The result: memory raises both quality and leakage, and the gap grows with the number of sessions. To prove the cause was the dialogue and not just accumulated time, the authors forked runs and continued them with and without memory from the same history. The memory branch learned more about the user, in places twice as much. There’s no difference in the first session or in the first turn of later sessions (the control points). It appears only after the agent’s reply: more than a third of memory-enabled replies carried a value from the past, and users mostly confirmed it.
To check that the loop reinforces itself, the authors edited the agent’s replies and removed the mentions. Disclosure dropped 1.6–2.3 times more than the single-turn effect predicted. Wrong values, meanwhile, got locked into the record. Trying to decline the topic didn’t help: after a refusal, the model leaned harder toward “sensitive” values.
Then they compared defenses. Idealized filters on the user’s turns and record anonymization lowered leakage but ate continuity. A single instruction to the agent brought leakage back to the no-memory level (0.58 → 0.53) with unchanged usefulness (4.2 → 4.3), while continuity fell from 4.5 to 4.2. The number of personal-data mentions dropped from 0.54 to 0.12, below even the no-memory level (0.31). The instruction turned out to be the cheapest and most effective intervention.
Adaptations and Extrapolations
💡 Adaptation for a personal assistant (ChatGPT Memory, Claude Memory, a custom GPT): you can put the same rule in Custom Instructions. Useful if you work with client data and don’t want the assistant “recalling” it in replies or locking in guesses.
Use memory for the content of your replies, but don’t say personal facts about me out loud and don’t build replies as if I’ve already confirmed them. If you need a fact — ask openly. If I mentioned a fact myself in this chat — use it freely.
🔧 Technique: “statement” → “open question.” In the paper, an open question about an attribute doesn’t count as a mention, and memoryless agents ask exactly those more often. Ask the bot to rephrase its hints as questions:
Instead of “X would suit your family with kids,” write “What kind of place do you need? Here are options X, Y, Z.”The effect: the customer chooses what to reveal, and the record keeps only what they decided to say.
Resources
- Paper: Disclosure Begets Disclosure: A Self-Reinforcing Privacy Loop in LLM Agents with Persistent Memory
- Authors: Masahiro Kaneko (MBZUAI, Third Intelligence), Hiromi Arai (RIKEN AIP), Timothy Baldwin (MBZUAI)
- Referenced: MemoryBank (Zhong et al., 2024) as the memory scheme; adversarial anonymization by Staab et al. (2025) as a comparison defense; Staab et al. (2024) as work on inferring attributes from text