TL;DR
Ask an LLM to "review" a document and deliver a verdict (accept, revise, reject), and the verdict comes out nearly random. It depends on the model and on one sentence at the top of the prompt. Models mostly write "major revision needed." But the comments inside the review do carry signal, especially the ones raised independently by several different models.
The core pain: you paste the same text into three chats and get three different verdicts. One model says "accept" 15% of the time, another says "revise" 100% of the time. Write "this is the second version after revisions," and "accept" jumps from 15% to 68% for the exact same text. The reason: the verdict doesn't follow from the comments. The researchers planted a serious error in papers, and the models often spotted it, yet the verdict barely changed. The model first takes a "safe" position ("needs revision," to look rigorous) and then writes comments to fit it.
Practical takeaway: don't ask for a verdict and don't hint at context. Ask for a list of comments, run the document through 2–3 different models, and look for what repeats. Comments that every model raised almost never turn out to be nitpicks. Comments raised by only one model more often turn out to be trivia.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Same neutral review prompt → into 2–4 different models (separate chats)
→ summary + major comments + minor comments (+ a verdict if needed, but we don't trust it)
STEP 2: All reviews → into a "judge" chat: group identical comments,
note who raised them, split into "substantive" / "nitpick"
→ a list of comments with the number of models that raised each
STEP 3: A human reads first the comments raised by all or most models
→ the human decides, not the model
Steps 1 and 2 are separate requests. Step 1 runs in several chats.
Example
Task: The founder of a healthy meal delivery service in Austin is preparing a pitch for a Shark Tank-style investment show. She needs to find the weak spots in her unit economics before she steps on stage. Asking "rate my pitch, is it worth investing in" won't work: she'd get a verdict that depends on the model and its mood. So she asks for comments only, doesn't mention that the pitch has "already been revised," and runs the text through three chats.
Prompt (step 1, identical for ChatGPT, Claude, Gemini):
You are an expert investor evaluating a startup pitch for an investment show.
Below is the full text of the pitch. Write the review you'd hand to the show's host.
Review structure:
SUMMARY: <2–3 sentences: what the founder claims>
MAJOR COMMENTS:
- <a substantial problem that must be fixed before going on stage, or "None.">
MINOR COMMENTS:
- <small stuff, wording, delivery>
Don't give an overall verdict. Comments only.
PITCH:
"GreenBox," Austin. A subscription for ready-made lunches: $12.90 per meal,
the average customer orders 12 meals a month. Cost of goods $8.20, delivery $2.40,
customer acquisition $85 via paid social and influencers. Retention: 4 months.
We're asking $500K for 10% to open kitchens in Dallas and Houston...
Prompt (step 2, in a fourth chat):
Below are three independent reviews of the same pitch (Review A, B, C).
1. List all major comments and group the ones that are the same in substance, even if
they're worded differently. For each group, note which reviews
raised it.
2. Label each group: "substantive" (a real problem that could affect
the investor's decision) or "nitpick" (technically correct, but minor or already
addressed in the pitch itself).
3. Sort: comments raised by all three reviews first.
Result: After step 1 you have three reviews in the same structure: summary, major and minor comments, no verdict. After step 2, a consolidated list. At the top are comments all three models raised (for example, the customer acquisition payback math). At the bottom are one-off comments, which are more likely to be nitpicks. The founder decides for herself what to fix.
Why It Works
The weakness. LLMs are bad at delivering an honest verdict. They gravitate toward a "safe" answer: "major revision needed" sounds rigorous but requires no conviction. And the decision isn't built from the comments: a planted serious error barely changed the verdict, even when the model found it. Any context hint ("this is the second version") moves the verdict a lot, even though the text is the same.
The strength. The model is good at generating comments on a text, and they're varied. When several different models independently arrive at the same comment, it's very likely real. A one-off comment is more often a nitpick.
How the method uses this. The verdict is taken out of the process. The framing is kept neutral so it doesn't push the model toward leniency. Several models act as independent experts, and their overlap works as a nitpick filter.
Control knobs: - Number of models (2–4): more models means a more reliable filter, but more time. - Verdict line: removed means no false anchor. Kept means read it as a weak signal. - Framing ("first draft" / "after revisions"): neutral by default. Don't mention that the document has already been reviewed or published if you want a tough review. - Consensus threshold: "all models" is what matters most; "two out of three" is a workable compromise.
Prompt Template
This is an adaptation of the study's prompt. The authors kept the verdict line, and it proved unreliable (see adaptations).
Reviewer (step 1):
You are a reviewer for {publication_or_venue}. Below is the full text of the submitted
material. Write the review you would send to the editor.
Review structure:
SUMMARY: <2–3 sentences: what the material claims or shows>
MAJOR COMMENTS:
- <a substantial problem that must be fixed before publication, or "None.">
MINOR COMMENTS:
- <small stuff, wording, delivery>
End your response with exactly one line, with no other text on it:
VERDICT: <accept|minor_revision|major_revision|reject>
MATERIAL:
{text}
Consolidating judge (step 2). The authors used a judge like this; the full text of its prompt is in the appendix. Below is a template based on its description in the paper:
Below is a piece of material and four anonymous reviews of it (Review A, B, C, D).
1. List all major comments from all reviews. Group the ones that are the same
in substance into one cluster, even if worded differently.
For each cluster, write down the letters of the reviews that raised it.
2. Label each cluster: "substantive" (a real, well-founded problem
that could affect the conclusions or the publication decision)
or "nitpick" (technically correct, but minor, pedantic, or already addressed).
3. Rate the constructiveness of each review on a 1–5 scale, regardless of
how harsh its verdict was.
4. Name the single best review and explain why in one sentence.
MATERIAL:
{text}
REVIEWS:
{review_A}
{review_B}
{review_C}
{review_D}
What to fill in: {publication_or_venue} — who the review is for (editor, investor, client); {text} — the whole document; {review_X} — the review texts, shuffled and with model names removed.
🚀 Quick start — paste into a chat:
Here's a reviewer template and a consolidating-judge template. Adapt them to my task:
{your task}. Ask me questions to fill in the fields.
[paste the templates above]
The LLM will ask what the document is and who the review is for, because the reviewer role and the "substantive" criteria depend on that. It will swap "publication reviewer" for a fitting role and adjust the comment structure.
Limitations
⚠️ Don't trust the verdict: the model's decision ("accept / revise") depends on the model itself and on hints at the start of the prompt. Don't use it as a quality score.
⚠️ Models miss planted errors: even the best model caught only some of the planted serious errors, and weaker ones caught fewer. No comment doesn't mean no problem.
⚠️ No expert verification: the researchers didn't check whether the comments were right on the merits. "Substantive" vs. "nitpick" was decided by other LLM judges. Model consensus is a useful indicator, not proof.
⚠️ Text only: the models saw the full text of the papers without figures or appendices. How they handle images and tables-as-images is unknown.
⚠️ Mid-priced models: they tested mid-priced models, not the most powerful ones. The patterns are probably general, but the specific numbers for flagship models may differ.
⚠️ Stability isn't guaranteed: the authors themselves note that behavior may change with new model versions, so you should periodically re-test on your own samples.
How They Tested It
The researchers took 200 papers published in 2026 across 20 journals, from Nature, Science, and Cell to Scientific Reports. The underlying idea: a paper published in a top journal is a "presumably good" sample, and a reasonable reviewer shouldn't be demanding major revisions across the board. Four LLMs (two closed, two open-weight) wrote reviews blind: they weren't told the paper had already been published. That made 800 reviews in total.
The results cut both ways. Three quarters of the reviews demanded major revision or rejection. But the spread across models was huge: one model demanded revision 100% of the time, another only 17.5%. All four gave the same verdict on just 10.5% of papers, and it was always "major revision." Journal prestige made no difference: Nature didn't get gentler verdicts than Scientific Reports.
Then they changed one sentence in the prompt: "you are reviewing the second version after revisions." Verdicts softened for every model, and for the most lenient one, "accept" rose from 15.5% to 68.5%.
Third experiment: another model planted one serious error in each paper, such as an incorrect statistical analysis. The share of errors caught depended on the model: from 17% to 60.5%. Yet the verdict barely changed, even when the error was found: in 84% of cases it stayed the same. From this the authors conclude that the verdict is poorly tied to the comments.
In the fourth experiment, two judge models blindly compared the four reviews of each paper. Comments raised by all four reviewers turned out to be nitpicks in 1–20% of cases, while comments from a single reviewer were nitpicks in 12–32%. Model agreement correlates with relevance.
Adaptations and Extrapolations
🔧 Technique: drop the verdict line → catch less noise
In the original, the review ended with a VERDICT: ... line. If you remove it and add "Don't give an overall verdict. Comments only," the model won't have to take a "safe" position that the comments then get bent to fit. This is an adaptation of the finding about the weak link between verdict and comments; the authors didn't test this version.
Extrapolation: a "planted error" check for your own workflow. The authors tested whether the model catches a known error. The same trick works as a test before you trust LLM review of contracts, code, or reports:
Take this report and carefully introduce one serious, plausible error
into the calculations (for example, the wrong base for a percentage or a swapped time period).
Return the full text with the error and separately tell me exactly what you changed.
Then give the modified text to another chat for review and see whether it finds the error. If it doesn't, that model with that prompt can miss it in real work too.
Resources
- Paper: Major Revision, Minor Accountability: LLMs as Reviewers of Published Articles in Top-Tier Journals
- Authors: Bastien Le Guellec, Aghiles Hamroun, Raphael Bentegeac (CHU Lille, France); Su-Hwan Kim (Technical University of Munich)
- Mentioned: the AAAI-26 experiment with AI reviews, NEJM AI (fast track with LLMs), work by Kim and colleagues on expert evaluation of LLM reviews of Nature-family papers