Multi-Layered Review (MLR): understand the document in layers first, then hunt for errors

STEP 0: Split the document → main text (up to ~10 pages) + appendices
STEP 1: Appendices → summarization (separate request, small model) → key experimental and implementation details
STEP 2: Literature search (optional, with web search) → report on context and novelty
STEP 3: Pass 1 → a brief structured outline of the document's ideas
STEP 4: Pass 2 → close reading following the outline + notes + critique
STEP 5: Pass 3 → synthesis: notes + appendix summary + literature report → final review

The steps run as separate requests. The original PDF isn't converted to text: formulas and figures stay intact, because the model reads the file as is.

The method shines where you need to find internal contradictions in a long document. It's not a fit for "like/don't like" judgments or for short texts.

Task: The founder of a SaaS startup in Austin is preparing a business plan for a VC fund. It's 28 pages of main text plus an appendix with the financial model. The founder is worried an investor will spot a discrepancy in the first five minutes: the executive summary says revenue is $1.2M, but the table says $950K. Or CAC is one number in the text and another in the appendix. A straight summary won't catch things like that.

Prompt (pass 1, in a chat with the PDF of the main section uploaded):

You are an analyst at a venture fund. Read the entire business plan.
Do NOT evaluate or critique yet. Build a structured outline:
1. Main claims (3–5): what exactly the authors assert.
2. Key numbers the claims rely on
   (revenue, CAC, LTV, growth, market size), with page numbers.
3. Assumptions the model rests on.
4. How the document supports each main claim.
Output only the outline. No conclusions or recommendations.

Prompt (pass 2, in the same chat):

Now reread the document closely, using your outline.
For each main claim and each key number:
- find every place it appears in the document;
- check whether those places agree with each other;
- flag any contradiction: what's said in one place, what's said in another, and exactly where;
- flag claims that aren't backed by data.
Add to your notes. Don't write the final assessment yet.

Prompt (pass 3):

Here are your notes, a summary of the appendix with the financial model,
and a report on the market and comparable companies: {paste}.
Put together a final assessment for the founder:
1. Contradictions (in order of importance to an investor), with quotes and pages.
2. Unsupported claims.
3. Which "weak spots" are already addressed by the appendix (leave these out of the critique).
4. Questions an investor will ask.

Result: The model will produce three layers. First, an outline with numbers and pages. Then notes cross-checking the same metrics across different places. Finally, a list of contradictions and investor questions, with the issues already resolved by the appendix removed. The specific findings depend on the document.

The weakness. A model that writes a review right away generates a plausible review from its overall impression. It doesn't cross-check details from different parts of a long text. The authors note that comprehension degrades with input length, even when the context window can hold it.

The strength. The model is good at local tasks: summarizing, outlining, pulling out numbers. It's also good at searching and matching when it's told exactly what to look for.

How the method exploits this. The first pass creates a "map": claims, numbers, assumptions. On the second pass the model has the map and checks claims against one another. The third pass just assembles what's ready. The appendix and the literature come in already compressed, so they don't bloat the input or trigger false objections.

Control knobs: - Number of passes: two is enough for short documents; for long ones you can add a section-by-section pass. - What to capture in the outline (numbers, definitions, assumptions): adjust to your document type; for a contract, it's deadlines, amounts, parties. - The literature step: turn it off if you don't need the brief. The authors themselves disabled it where search could give away the answer. - No evaluation on pass 1: remove this if you want preliminary comments right away.

<Role>
You are a meticulous reviewer of a {document_type} document.
Your main job is to find contradictions and unsupported claims.
</Role>

<Context>
Appendices (summary): {appendix_summary}
Context brief (comparables, market, standards): {context_brief}
</Context>

<ThreePassReview>
Pass 1: OUTLINE (no evaluation):
  List the main claims, the key {what_to_cross_check}, the assumptions,
  and how each claim is supported. Give locations in the document.

Pass 2: CLOSE READING:
  Using the outline, reread the document.
  For each claim and metric, find every mention.
  Cross-check them. Flag contradictions (what's said here / what's said there)
  and claims without evidence. Add to your notes.

Pass 3: SYNTHESIS:
  Combine the notes, <Context>, and your conclusions.
  Don't criticize anything already explained in the appendices or the brief.
  Deliver: {output_format}.
</ThreePassReview>

What to fill in. {document_type}: business plan, contract, report, paper. {what_to_cross_check}: numbers, deadlines, definitions, obligations. {output_format}: a list of contradictions with quotes, questions, a risk assessment. A separate request for the appendices: "Briefly summarize the key details (methodology, assumptions, calculations) in 10–15 bullet points."

🚀 Quick start: paste this into a chat:

Here's a three-pass document review template (MLR). Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what type of document you have and what needs cross-checking in it (numbers, deadlines, definitions). It'll also ask whether there are appendices that should be summarized separately. It needs this so the first-pass outline targets the things that can actually contradict each other.

This is a reconstruction based on the method description. The authors put the verbatim prompts in an appendix, and they're not in the provided text.

⚠️ Document length: The authors capped the main text at roughly 10 pages and observed that comprehension drops with input length. For bulky documents, you'll need to split by section.

⚠️ Real errors are harder than synthetic ones: All systems did better on artificially inserted errors than on real ones. The model misses genuine contradictions more often. A human needs to check the result.

⚠️ No guarantee on the severity of findings: The farther a contradiction is from the main claim, the less often it's found. The system often misses small discrepancies in the details.

⚠️ Vulnerable to manipulation: The authors confirm that automated reviewers are still vulnerable to adversarial insertions in the document. If an interested party wrote the document, don't rely on the model's verdict.

⚠️ Debatable comments: Document authors most often disagreed specifically with the "weaknesses" that were found. Use the output as a list of questions, not a verdict.

⚠️ Search can "leak" the answer: Web search sometimes surfaces information about the paper's retraction. For an honest check, turn it off.

The researchers wanted to test whether reviewer systems can find errors, not just write text that sounds human. To do that, they took 257 papers from five major conferences (ACL, AISTATS, CVPR, ICML, NeurIPS). They inserted artificial contradictions into each paper, resulting in 1164 test cases. At first they asked the model to alter random sentences. It turned out that sometimes it corrupted a claim from the abstract and sometimes a minor implementation detail. So they asked Gemini 2.5 Pro to build a claim graph of each paper. The distance from the main claim became the measure of an error's severity. That worked: the farther the error, the less often it was found.

An LLM judge, o3, checked the findings. It achieved 99.9% accuracy on papers without errors and 86.8% sensitivity on those where MLR actually found a contradiction. Each evaluation was repeated 10 times. A small audit of 50 insertions showed that 34% looked awkward in context, and 8% read as "obviously AI-written." The authors believe this makes the task easier, yet weak systems still did poorly. On real errors from retracted papers, all systems performed worse than on synthetic ones.

MLR was compared with LLM-Review, MARG, the AI Scientist reviewer, and GAR. The authors write that the gain comes from both the choice of base model and the system design. The text doesn't let you separate those contributions. The practical takeaway: layered reading and moving the appendices into a separate request are clear, usable techniques. But their effect can't be separated from the strength of the model itself.

🔧 Technique: add an "is this already covered by the appendix" check → fewer false objections

In pass 3, add a line: "For each comment, check whether the appendix summary already answers it. If it does, cross it out." This reproduces the purpose of the separate appendix agent.

Extrapolation: reviewing a contract before signing:

Pass 1: list the parties, subject matter, deadlines, amounts, penalties, and definitions of terms, with clause numbers.
Pass 2: find every mention of these items, including in exhibits and specifications, and cross-check them.
Flag where a payment deadline, amount, or definition differs.
Pass 3: a list of discrepancies by importance + questions for your lawyer.

This doesn't replace a lawyer. The method simply gathers every spot where the wording needs clarifying.

• Paper: Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review (TMLR, 2026), OpenReview
• Authors: Rachel S.Y. Teo, Yutaro Yamada, Shashank Kotyan, Yuki Imajuku, Tarin Clanuwat (Sakana AI, National University of Singapore)
• Source of the three-pass idea: Keshav (2007), "How to Read a Paper"
• Compared against: LLM-Review (Liang et al.), MARG, AgentReview, AI Scientist Reviewer, GAR
• Additional tests: WithdrarXiv-Check (retracted papers)