Splitting a Review Across Agents: It Catches More Errors, but Also Raises More False Alarms

BASELINE:  1 request → all 6 requirements → PASS/FAIL + quote for each

TEAM:      request A → requirements 1–2 ┐
           request B → requirements 3–4 ├→ merge the flags
           request C → requirements 5–6 ┘
           (in parallel, no communication, same document and same wording)

TEAM WITH CONTEXT (scoped):
           same, but each request gets an added block
           "The other requirements are checked by other reviewers: {list}"

THREE RUNS:  1 prompt × 3 runs → flag if FAIL in at least one

Each step is a separate request. The agents don't talk to each other.

The task: You're the finance manager of a regional coffee shop chain. Ahead of the bank's credit committee meeting, you need to check an internal memo for an expansion loan for internal contradictions. The document is 1,100 words, with six rules. Errors may be disguised. For example: "income is $37,000, but the calculation uses $31,700 because one-time bonuses were deducted." Every false alarm costs an hour of manual checking. A missed error costs you a dressing-down from the committee chair.

The method works best for checking internal contradictions against a checklist. So that's exactly what we take, with the requests in the "team with context" setup.

Prompt (request A, pair 1):

You are reviewing a credit memo for compliance with requirements.

YOUR REQUIREMENTS:
R1. Every mention of monthly revenue used in the repayment-capacity
    calculation must match the figure in the "Financials" section.
R2. Every mention of the approved credit limit must state the same
    amount, and any stated change to the limit must lead to it.

FOR CONTEXT (DO NOT CHECK, other reviewers judge these):
R3. All arithmetic totals involving revenue must be correct.
R4. All limit multiples must follow from the limit and revenue.
R5. All statements about payment history must match the history table.
R6. All stated business tenures must be possible given the owner's age.

Working rules:
- Evaluate each of YOUR requirements once.
- In one to three sentences, say what you compared with what.
- Once you've given a verdict, don't return to that requirement.
- Mark FAIL only if your own requirement is violated.

Response format (JSON), for each of your requirements:
{"requirement": "R1", "reasoning": "...", "verdict": "PASS|FAIL", "evidence": "quote from the document"}

DOCUMENT:
{memo_text}

Requests B (R3–R4) and C (R5–R6) are built the same way. Only the "YOUR REQUIREMENTS" block and the "FOR CONTEXT" block change.

Result: Each request returns JSON with a verdict and a quote for its two requirements. You merge the flags from the three answers into one report. Expect more disguised contradictions caught than with a single request covering six rules. There will also be slightly more false positives, but noticeably fewer than with a team lacking the "for context" block. Token usage will be about 2.3 times higher.

The weakness. When a single request has six requirements, the model can "lose" some of the rules or check them superficially. Hidden defects with a plausible explanation slip right past. That's why splitting really does catch more. But the weakness has a flip side. An agent with two rules doesn't know there are others nearby. It sees a suspicious phrase and writes FAIL, even though it's someone else's territory. A typical case from the paper: the R1 agent flagged a sentence about an income adjustment. That phrase was being checked under R3, which the agent couldn't see.

The strength. The model works well with an explicit boundary of responsibility. If you tell it the rest is checked by others, it stops trying to "cover everything."

How the method uses this. Splitting raises sensitivity. The "for context" block restores information about the boundaries and removes some of the false alarms. The paper is upfront about the cost. Shifting the trigger threshold is not the same as a smarter review.

Control levers: - Group size (2 requirements per request, or 1, or 3) → the smaller, the more tokens. The paper tested only one split: by meaning, in pairs of 2. - The "for context" block → remove it to raise sensitivity and accept more false alarms. Keep it to cut them down. - The "decide once" instruction → in the paper it was added to every request. The team's results depend on it, so test on your own sample. - The rule for merging runs → "any flag" gives the most catches. "Majority" gives fewer false alarms and misses more.

You are reviewing a document for compliance with requirements.

YOUR REQUIREMENTS:
{requirements_for_this_request}

FOR CONTEXT (DO NOT CHECK, other reviewers judge these):
{other_requirements}

Working rules:
- Evaluate each of YOUR requirements once.
- In one to three sentences, say what you compared with what.
- Once you've given a verdict, don't return to that requirement.
- Mark FAIL only if your own requirement is violated.

Response format (JSON), for each of your requirements:
{"requirement": "...", "reasoning": "...", "verdict": "PASS|FAIL", "evidence": "quote from the document"}

DOCUMENT:
{document_text}

What to fill in: - {requirements_for_this_request} — 2 rules from your checklist. Best grouped by meaning, as in the paper. - {other_requirements} — all the other rules verbatim, in the same wording. For a plain baseline without context, drop this block. - {document_text} — the entire document, in every request.

Run 3 requests with different pairs. Merge the flags.

🚀 Quick start — paste into a chat:

Here is a template for checking a document against a checklist, split across several
requests. Adapt it to my task: [your task].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what kind of document you have, how many requirements are on the checklist, and how to split them into pairs. It will also ask which error is costlier: a miss or a false alarm. That's needed to choose the group size and the flag-merging rule. Then it will assemble ready-to-use requests A, B, and C.

⚠️ No universal win: when both errors are weighted equally, the team is barely different from a single agent. There's a gain only if a miss costs more than a false alarm.

⚠️ Strong models are at the ceiling: with the strongest model tested (Claude Sonnet), the single agent already found practically everything. The team added nothing and brought false alarms.

⚠️ Synthetic data: defects were present in 37% of checks; in reality they occur far less often. On real documents, false alarms will cost more than they do here.

⚠️ Internal contradictions only: only errors within the document were tested. Omissions, cross-checking against external facts, and judgment calls weren't.

⚠️ One split, one run: pairs by meaning, 2 requirements each, and each variant was run once. There's no guarantee a different split gives the same result.

⚠️ The mechanism of the "for context" block isn't proven: it may work as information about boundaries, or it may simply make the model more cautious. The paper doesn't tease these apart. On some models the block slightly lowered the detection rate for arithmetic errors.

⚠️ Cost: 2.2–2.4 times more tokens than a single request. Three runs cost even more.

The researcher built 300 synthetic credit memos of ~1,100 words each, generated by a program rather than an LLM. The correct answer is known for each of the 1,800 "cells" (memo × requirement). The labels were cross-checked by three sources: the generator, a separate program, and a stronger model. Defects came in three types: an explicit contradiction, a disguised one (with its own explanation), and an arithmetic one.

Four setups were compared on four models (two Qwen3.5 and two Claude): a single agent, a team of three, a team with context, and three runs of a single agent. False alarms were measured separately from misses. Many earlier studies measured just "right/wrong," which makes it impossible to tell which kind of error a setup prefers. The main hypotheses were fixed in advance and tested with a correction for multiple comparisons.

Two results were surprising. First, the simplest trick, "run it three times and flag on any hit," gave a better balance than the team, even though the team is usually considered the more sophisticated design. Second, the team with context was added only after the author read through Sonnet's false alarms. Most of them happened when an agent touched someone else's rule. The practical takeaway: before splitting work among agents, think about what each one doesn't know about its neighbors.

🔧 Technique: the rule for merging runs → tune the "miss ↔ false alarm" balance to your task

Run the same prompt 3 times. If a miss is costlier (compliance, security, legal risk), flag on any FAIL. If a false alarm is costlier (say, a manual check takes an hour), flag on two out of three. In the paper, "any of the three" gave the most catches, while "majority" gave fewer false alarms.

🔧 Technique: the same template in CLAUDE.md or a subagent description

If you have reviewer subagents (for example, in Claude Code), add a block to each one's instructions: "Other subagents check: {list}. Don't flag violations in their area." This is a direct carry-over of the paper's trick to agent work. The paper itself tested it only on parallel calls without tools.

• Paper: The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task
• Author: Pei Liu, School of Engineering and Applied Science, George Washington University (Washington, D.C., USA)
• References mentioned: Schluntz & Zhang 2024 (Anthropic's practical guide, the "one concern — one call" rule), Fox et al. 2026 (the "moves the point along the curve or lifts the curve" framing), He et al. 2026, Liu et al. 2026, Yu et al. 2026, Wang et al. 2023 (self-consistency).
• Data: the synthetic memos are built on the Yeh & Lien 2009 dataset (credit cards, Taiwan).