Evidence Balance in Memory: Why Compressing an Agent Debate Breaks the Final Decision

The paper doesn't propose a single method. It's a set of tested rules. Below is the handoff pipeline that follows from them.

STEP 1: Agents discuss the task → full transcript
STEP 2: A separate "summarizer" call compresses the discussion → a list of facts
        WITHOUT author names or "N agents said this" counters,
        with facts for AND against each option
STEP 3: The final agent gets only: task + options + summary
        (not the whole transcript)

Steps 2 and 3 are separate requests. In the study, the decision was made by a separate call that saw only the description, the options and the memory.

Task: An Amazon seller is choosing a packaging supplier for a fragile product. Three analyst agents researched the candidates (pricing, lead times, reviews). They argued it out, and now you need a summary for the final agent, which will name the winner. The method shines where the decision rests on a pile of facts pointing in different directions. That's exactly the kind of task it fits.

Prompt for the "summarizer":

You are preparing a summary of a discussion for someone who will make the final decision
and did not see the debate. The choice: a packaging supplier for a fragile product sold on Amazon.
Options: "PackPro" (Chicago), "BoxWest" (Denver), "EcoBox" (Austin).

Below is the discussion between three analysts:
{transcript}

Summary rules:
1. List ALL distinct facts about the options, one per line.
2. For each fact, state which option it supports and which it counts against.
3. Keep inconvenient facts on equal footing with convenient ones. Don't drop anything
   just because it seems "off-topic" or "only one person mentioned it."
4. Do NOT say which analyst said what. Do NOT write "everyone agreed"
   or "the majority view." One fact, one line, no count of supporters.
5. If the analysts disagree, record both claims and the conditions under which each holds.
6. End with a table: option | facts for | facts against.

Result: The model returns a numbered list of facts with no author names. Each fact is marked with whom it helps or hurts. At the end comes a balance table for the three suppliers. The final agent gets a neutral picture, not "the majority opinion."

The weakness. The decider model doesn't weigh facts one by one; it effectively tallies up which side outweighs the other. So any summary that distorts the for-vs-against ratio shifts the answer. Selecting "by rarity" only works when the rare facts happen to be the useful ones. Selecting "by relevance" lets traps through, because a trap is relevant to the criteria too. Even a perfect relevance filter lost to rarity by 23–31 points in the classic scenario (useful facts held by individual agents, traps shared by all).

The strength. The model is good at tallying the balance when it's right in front of it. In the paper, accuracy rose almost in lockstep with the net margin in favor of the correct option: about 10–13% at a margin of −2 versus 96% at +4 and above. Once the balance was assembled honestly, the selection method added almost nothing.

How the method uses this. Keep facts on both sides and remove authorship tags. Then the model has no reason to give extra weight to a "mentioned many times" fact. Only the content remains.

Control levers: - Authorship tags → remove them to kill the "majority" effect. If you need traceability, keep it somewhere separate, not in the lines the decider reads. - Rule 3 (inconvenient facts) → strengthen it for high-stakes decisions: hiring, procurement, legal conclusions. - The for/against table → drop it for simple tasks, keep it for contested ones. - Summary length limit → the stricter it is, the higher the risk of losing a fact you need. For critical decisions, don't cut.

The paper's main text doesn't include the exact summary prompt. The template below is built from its description of an "evidence-preserving summary" (keep every piece of evidence along with its conditions, uncertainty and disagreements), plus the rule it found about authorship tags.

<Role>
You write a neutral summary of a discussion for someone who will make
the final decision and did not see the debate.
</Role>

<Task>
Decision: {what_we_are_choosing}
Options: {list_of_options}
Criteria: {comma_separated_criteria}
</Task>

<Discussion>
{discussion_transcript}
</Discussion>

<SummaryRules>
1. List each distinct fact on its own line.
2. For each fact: which option it helps, which it hurts.
3. Keep conditions, uncertainty and disagreements: if opinions differ,
   record both versions.
4. Don't drop a fact because it seems "secondary" or "only one person said it."
5. Don't name authors and don't write "everyone agrees" or "most people think."
6. Facts first, then a table: option | for | against.
   Make no conclusions or recommendations.
</SummaryRules>

What to fill in: {what_we_are_choosing} is the essence of the decision, {list_of_options} is 2–5 options, {criteria} is what you compare on, {discussion_transcript} is the text of the debate or the agents' notes.

🚀 Quick start — paste into the chat:

Here is a template for a neutral summary that hands the outcome of a discussion to a final decider.
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what decision is being made, what the options and criteria are, and where the discussion comes from. That's needed so the "facts for and against each option" rule latches onto your real options. It will take the rest from the template.

⚠️ Synthetic benchmark: the discussions in the main tests were scripted, with each agent taking turns stating its facts. The correct answer was derived by a rigid rule: each "for" counts +1, each "against" −1. In real decisions, facts don't carry equal weight.

⚠️ Natural discussions: when the agents argued freely, no selection policy beat the others. The full transcript gave lower accuracy than having no notes at all. So part of the evidence gets lost before the summary even happens: nobody brings it up. A good summary can't fix that.

⚠️ A "by the rules" summary isn't a cure-all: the evidence-preserving summary in the classic configuration scored only 51–59% versus 88% for rarity-based selection. The authors did not directly test the balance and tag rules inside the summarizer prompt.

⚠️ Small and not-so-recent models: the main tests ran on two open 7–8B-parameter models and two DeepSeek models. Flagship models may behave more gracefully.

⚠️ The paper text is incomplete: part of the section on authorship tags is cut off. The numbers come from the abstract and the main body.

The authors spotted a problem in a well-known benchmark: there, rare facts coincide with useful ones, so "keep what's rare" looks like genius. They built CrossProfile: 60 decision templates, each with 12 facts (5 supporting the correct option, 3 "traps," 4 irrelevant). Each group of facts was made, in turn, either shared by all four agents or private to one. That produced 8 configurations and 480 tasks with the fact text held constant.

Next they compared compression policies: "rare first," "relevant first" (including with a perfect relevance "oracle"), "first mentioned," random selection and summarization. They ran it on Qwen2.5-7B, Llama-3.1-8B and two DeepSeek models. On average, rarity was no better than random: it wins only when traps are widely shared and loses when they're held by just a few. Relevance lost 20–32 points in the classic case even with a perfect filter. The surprise: the full set of facts (no compression) scored only about 55% in that case, even though the balance favored the correct answer. The culprit turned out to be the "4 agents said this" tags. When they were removed or randomly shuffled while keeping the same line order, accuracy rose by 13–24 points. That separated the tag effect from the effect of position in the text.

The main practical takeaway: don't look at the selection policy, look at what for/against balance the decider receives.

🔧 Technique: ask for an explicit balance count → protection against hidden skew

This is my own adaptation; it's not in the paper. Add to the final agent's prompt:

Before choosing, count the facts "for" and "against" each option
from the summary and show the difference. Then choose and explain what tipped the scale.

The goal: if the summary is skewed, the skew becomes visible instead of dissolving into the answer.

• Paper: Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions
• Authors: Anouar Imel (UESTC), Imad Charradi (Xidian University), Abderrahman Sghir (Nanjing University of Aeronautics and Astronautics), Khalil El Khammar (Nanjing University of Information Science and Technology)
• CrossProfile benchmark (60 templates, 480 tasks); related benchmark HiddenBench (Li, Naito, Shirado 2026); the classic: Stasser & Titus 1985 on the common-knowledge effect