TL;DR
SmoothScore is a way to check a conclusion in a report against its sources. For each conclusion, one pass hunts the sources for supporting quotes, while another, the "skeptic," hunts for limiting and contradicting ones. The quotes are grouped by aspect (which side of the claim they speak to), and the conclusion gets a score from 1 to 5: how well its strength and scope match the evidence.
The headline finding: deep research agents write conclusions more confidently than their own sources allow. You read the report: "approach A is more promising than approach B." You open the cited paper, and it says "A wins on only one benchmark and loses on another." The report is smooth and confident, so the problem stays hidden. The authors call this smoothing: diverse and conflicting sources collapse into one confident verdict. Each quote is accurate on its own, but the conclusion is broader and harder than they are.
The method splits the work into roles and steps. First, the report's key conclusions are picked out. Then come a search "for" and a search "against," run separately. Next, the quotes are sorted by aspect, and the conclusion is scored on a 1–5 scale. If you then rewrite the report based on the quotes you found, it comes out better than rewriting "blind" or with the full documents.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Extract the report's key conclusions (verdicts, not facts) → a list preserving strength and caveats
STEP 2a: Advocate agent: quotes supporting the conclusion → quote + aspect + how it relates + strength
STEP 2b: Skeptic agent: quotes with caveats, conditions, counterexamples, alternatives → same format
STEP 3: Identify aspects (which sides of the conclusion have data) → merged list, no duplicates
STEP 4: Sort the "for" and "against" quotes by aspect → "aspect: for / against" table
STEP 5: The judge scores 1–5 → score + what the conclusion leaves out
(optional) STEP 6: Rewrite the report based on the evidence table
In a chat, steps 1–5 can fit into one long prompt or be split into 2–3 requests. The skeptic works better as a separate request.
Example
Task: The founder of an online data-analytics school in Austin asked ChatGPT Deep Research: "Which is more promising for attracting students: Instagram Ads or YouTube Ads, and why?" The report came back confident, with a verdict in the first paragraph. Her quarterly budget is $15,000, and she doesn't want to sink it into a conclusion that rests on a single case study. She gathered the 6 cited sources and runs the check.
A comparative "which is more promising and why" question is exactly the form where, according to the paper, smoothing shows up the most. The question demands a winner up front.
Prompt:
<Role>
You are an auditor of conclusions in research reports. Your job is to check
whether a conclusion claims more than the sources allow.
</Role>
<Inputs>
Question: Which is more promising for attracting students to an online school: Instagram Ads
or YouTube Ads, and why?
Report: [report text]
Sources: [source 1] ... [source 6, full texts]
</Inputs>
<Steps>
Step 1. Pull 3–5 key verdict-style conclusions from the report (where the report makes
a choice or a generalization). Keep the wording, strength and caveats as in the report.
For EACH conclusion, perform steps 2–5.
Step 2a — Advocate Agent. Find quotes in the sources that support the conclusion.
Include caveats that themselves support conditions stated in the conclusion.
Step 2b — Skeptic Agent. Separately, find quotes that limit the conclusion,
qualify its conditions or contradict it, plus quotes about alternatives succeeding
(to see the scope).
Format for each quote: verbatim quote | source | aspect | how it affects the conclusion | strength (weak/medium/strong)
Step 3. Identify the aspects — sides of the conclusion that have data (e.g.,
"cost per acquisition," "lead quality," "niche size"). Merge duplicates.
Step 4. Sort the quotes by aspect. Assign a quote to an aspect based only
on its topic, not on whether it's for or against the conclusion. In each aspect,
show "for" and "against" side by side.
Step 5. Score the conclusion on the scale (see below) and explain what the conclusion leaves out
or overstates — conditions, scope, strength.
</Steps>
<Rubric>
1 — the conclusion accurately reflects the evidence
2 — minor omissions, the gist doesn't change
3 — noticeable smoothing: conditions or scope are omitted, which changes the main takeaway
4 — heavy smoothing: the conclusion is significantly stronger and broader than the evidence
5 — the conclusion is not supported by the sources
</Rubric>
<Output>
For each conclusion: score → aspect table (for / against) →
"what's left out" → a rewording of the conclusion that the sources actually support.
</Output>
Result: For each of the 3–5 conclusions, the model returns a block with "for" and "against" quotes. The quotes are grouped by aspect, such as cost per click or audience. After the table comes a 1–5 score with an explanation of what the conclusion ignores, plus a more careful rewording. If the report said "Instagram Ads are clearly more effective," the check will likely surface the conditions under which that holds (niche, ad format) and the data where YouTube wins. Which specific numbers turn up depends on your sources.
Why It Works
Weakness. When a model writes a report, it is gathering material and crafting a polished conclusion at the same time. A "which is better and why" question nudges it to pick a winner and justify the choice. Caveats that spoil the verdict get dropped during compression. Per the paper, a model can be accurate in every single quote and still overreach in the summary: glue separate facts into a stronger claim, stretch a result from one benchmark to everything, write "proves" instead of "suggests."
Strength. Models handle a narrow task with a clear role well: "find everything in this text that contradicts this claim." When it doesn't have to write coherent prose, only search and quote, it's easier for the model not to smooth things over. Two separate passes are essential. If you ask for "for and against" in one go, the "against" search comes out limp.
How the method gets around the weakness. It separates the search from the verdict. The advocate and the skeptic collect quotes separately. Then the quotes are grouped by aspect, so "for" and "against" sit side by side. The judge doesn't have to compare apples and oranges: it sees that on the "cost per acquisition" aspect some sources say one thing and others say another.
Control knobs: - Number of conclusions (3–5) → increase for long reports, decrease if you need to save tokens. - Skeptic → give it a tough role ("your job is to break this conclusion") so it digs deeper. The authors also have it collect wins for the alternatives: that reveals the scope. - Scale → keep the "3 and above = needs rework" threshold. - Sources → full texts, not just the quotes in the report. The agent may have cherry-picked only the convenient bits. - Step 6 (rewriting) → feed the model the evidence table, not the raw documents.
Prompt Template
<Role>
You are an auditor of conclusions in research reports. Your job is to check
whether a conclusion claims more than the sources allow.
</Role>
<Inputs>
Question the report was answering: {question}
Report: {report_text}
Sources (full texts): {sources}
</Inputs>
<Steps>
Step 1. Pull {number_of_conclusions} key verdict-style conclusions from the report
(where the report makes a choice, comparison or generalization). Keep the wording,
strength and caveats as in the report.
For EACH conclusion, perform steps 2–5.
Step 2a — Advocate Agent. Find quotes in the sources that support the conclusion.
Include caveats that support conditions stated in the conclusion itself.
Step 2b — Skeptic Agent. Separately, find quotes that limit the conclusion,
qualify its conditions or contradict it, plus quotes about alternative
approaches succeeding (to see the conclusion's scope).
Format for each quote: verbatim quote | source | aspect | how it affects the conclusion | strength
Step 3. Identify the aspects — sides of the conclusion that have data. Merge duplicates.
Step 4. Sort the quotes by aspect, going only by each quote's topic,
not by whether it's for or against the conclusion. In each aspect, show "for" and "against" side by side.
Step 5. Score the conclusion on the scale and explain what it leaves out or overstates:
conditions, scope, strength of wording.
</Steps>
<Rubric>
1 — accurately reflects the evidence
2 — minor omissions, the gist doesn't change
3 — noticeable smoothing: conditions or scope omitted, the main takeaway changes
4 — heavy: the conclusion is significantly stronger and broader than the evidence
5 — not supported by the sources
</Rubric>
<Output>
For each conclusion: score → aspect table (for / against) → what's left out →
a wording the sources actually support.
</Output>
What to fill in: {question} — the original request to the agent; you need it to see whether it pushed toward picking a winner. {report_text} — the full report. {sources} — full texts of the cited documents, not summaries. {number_of_conclusions} — usually 3–5.
The 1–5 scale is reconstructed from the description in the paper: the rubric itself is in the authors' appendix. The meaning of the levels (3 — substantial smoothing, 4 — heavy, 5 — going beyond the evidence) is taken from there.
🚀 Quick start — paste into the chat:
Here is a SmoothScore template for auditing a report's conclusions. Adapt it to my task: [your task].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask for your question and report, what sources you have and how many conclusions to check, because without the full source texts the skeptic has nothing to search. It will take the "advocate — skeptic — aspects — score" structure from the template and adapt it to your task.
Limitations
⚠️ You need the full source texts: the check only covers documents the report cited. If the agent never found an important source at all, the method won't catch it.
⚠️ The judge isn't perfect: the judge model agrees with humans as well as two humans agree with each other, but in absolute terms that agreement is moderate. Of the conclusions the judge scored 3 or higher, a human confirmed the problem in only about 60% of cases. That means a third of the flags may be false. Eyeball the scores, especially for important decisions.
⚠️ The report improvement was measured indirectly: the rewriting experiment was scored with quality rubrics, not with the smoothing score itself. It was run on two models and one dataset. For Luna, rewriting without context gave no gain at all. You have to supply the evidence.
⚠️ Manual effort and tokens: a pass per conclusion over several long documents is a lot of text. In a chat with a large report, you'll have to stick to 3–5 key conclusions.
⚠️ The scale is inherently subjective: the line between "minor omissions" and "changes the gist" is debatable; even two humans agree on the exact score only about 58% of the time. Treat the scores as a "look closer" signal.
⚠️ "Just tell the model not to smooth" wasn't tested: the authors didn't check whether an instruction like "don't overstate the strength of conclusions" in the original prompt helps. That's only a hypothesis.
How It Was Studied
The team assembled four task sets for long research reports and hand-picked questions that require synthesis across many sources. Seven systems wrote reports on them: GPT agents with web search, Claude Sonnet and Opus, Gemini Deep Research and two open agents. Key conclusions were extracted from each report, roughly 100–450 per system and dataset. The pipeline described above was run on each conclusion, and an LLM judge gave a 1–5 score.
Smoothing turned up everywhere. In every system and every dataset, at least about a quarter of the conclusions scored 3 or higher. Gemini was the worst: 85–93% of conclusions scored 3+, and 22–35% scored 4+. That's several times more often than the other closed systems. The GPT agents did best: 1–4% severe cases. The open agents showed 7–23% heavy smoothing, even though they gathered fewer sources.
Surprisingly: "more documents and a longer report" don't explain the problem. Across systems there's a correlation, but within a single system it nearly vanishes. So a long report doesn't guarantee smoothing by itself; it's down to the system's style. Another finding: the GPT Luna agent opened only 19% of the pages it cited. Takeaway: reports are largely built from snippets and headlines, not full texts.
The human check worked like this: two annotators independently scored 73 conclusions with the same rubric, without knowing which system wrote them. The judge differed from an annotator no more than the annotators differed from each other (exact agreement around 56–58%, within one point 93–96%).
In the rewriting experiment, two agents got their own drafts and rewrote them in three variants: no context, with the full cited documents, and with the evidence table from the pipeline. For Sonnet 5, the rubric score rose from 64.4 by +12.0 with the evidence table, versus +9.7 with the full documents, even though the table is five times shorter (65K vs. 324K tokens). Without context the gain was 1.1, not statistically significant. Practical takeaway: not just "rewrite it better," but "rewrite it taking these for-and-against quotes into account".
Adaptations and Extrapolations
These are my own variants; the authors didn't test them.
🔧 Technique: add a check for three patterns → a quick audit without the full pipeline
If a full run takes too long, give the model the report and this list from the paper:
Find conclusions in the report where: 1. Several facts are glued into a stronger claim that appears in none of the sources (unwarranted composition). 2. A result from specific conditions is presented as a general law (scope expansion). 3. Wording like "proves," "the best approach," "strictly guarantees" is used (premature closure). For each, give the quote from the report and what exactly is overstated.Effect: quickly filter out the conclusions worth running through the full pipeline.
Extrapolation: reframe the comparative question before launching the agent. The paper shows that "which is better and why" questions provoke smoothing. You can replace them with "under what conditions is approach A better, when is B better, and which data contradict each other." This is untested, but it follows logically from the observation.
Don't pick a winner. For {A} and {B}, describe for each criterion under what conditions each one wins, and separately list the data that contradict each other. State conclusions with caveats.
Resources
- Quantifying Claim Smoothing by LLM Research Agents (Preprint, Under Review)
- Authors: Tiasa Singha Roy, Manya Wadhwa, Greg Durrett — New York University
- Code and data: https://github.com/tiasa2/SmoothScore
- Agent prompts are in Appendix B of the paper; the aspect approach builds on Kargupta et al., 2025
- Datasets used: ResearcherBench, ScholarQA-CS2, DeepResearch Bench II, ResearchRubrics