The Lenient Reviewer Effect: An LLM Finds Fewer Errors After It Has Just Checked and Fixed Something

If a language model has just found and fixed an error in one text, then on the next text, a completely different one with no connection to the first, it becomes less suspicious and reports problems less often. The effect lives in the history of a single conversation: the mere fact of "I checked → found → fixed" sitting in the context changes how the model reviews the next piece of text, even though the tasks don't overlap at all.

Logic suggests the opposite: since the model just caught a real error, it should stay on guard. Instead, the more "negative" the previous episode was (the model actually found and fixed a bug rather than just saying "all good"), the more lenient it is on the next item. The false-alarm rate drops by 3–11 percentage points across virtually every model and instruction wording tested.

The cause isn't that the model got better at telling right from wrong (its ability to discriminate doesn't change at all). What shifts is the threshold at which it decides to flag a problem. After a "check-and-fix" cycle, the model simply speaks up less often when it's unsure; it becomes more tolerant of questionable spots.

How the Effect Works

CONVERSATION (one chat):
Step 1: The model checks text A → finds an error → "fixes" it
       (this episode stays in the chat history)
   ↓
Step 2 (same chat): The model checks text B — a DIFFERENT one, unrelated to A
   ↓
RESULT: the model finds fewer problems in text B
than if it had checked it "from a clean slate" in a new chat

Example in Practice

Task: A team lead asks Claude/ChatGPT, in one chat, to review five pull requests in a row for bugs.

Prompt (the way people usually do it):

Check this code for errors: [code 1]

[the model finds an error and suggests a fix]

Thanks, now check this code: [code 2]

[the model reviews code 2 in the SAME chat]

Result: the review of code 2 (and of everything after it in this chat) will be noticeably softer than the first. The model doesn't get dumber (its ability to tell a bug from a non-bug doesn't change), but it reports bugs less often. The real danger: the team lead gets a "green light" on a PR with a bug because the model already "spent" its vigilance on the first case.

The same goes for grading several essays in a row, proofreading several chapters of a book, auditing several contracts, reviewing several resumes: any workflow where you ask an AI to check a series of similar, independent items in one conversation.

Why It Works

An LLM isn't a calculator with a fixed "error / not an error" threshold. The decision to "flag the problem or stay quiet" depends on the entire context of the conversation, even if the new task has nothing to do with the previous one. Models are good at picking up the overall tone and "mood" of a conversation; that's a strength that helps keep a dialogue coherent. But that same sensitivity to context becomes a weakness in the reviewer role: having closed one case ("found it, fixed it"), the model implicitly slips into "everything's fine" mode and carries that mood over to the next, completely different review.

An important detail from the study: an explicit instruction to "be more lenient" affected the model less than this hidden context effect. That means conversation history can override direct commands, a lever few people think about when building a chain of requests.

Control lever: if consistent strictness at every step is critical for you, don't let "check-and-fix" history pile up in the context.

Practical Recommendation (Instead of a Prompt Template)

Since this isn't a step-by-step technique but a warning about model behavior, here's a working trick to soften the effect:

Before reviewing a new, independent text:

"Evaluate the following text from scratch, as if it were the only document in this conversation.
Ignore the results of previous reviews in this chat —
treat every point as if you're seeing it for the first time.

{text to review}"

But the most reliable fix is a habit, not a technique: open a new chat for each independent review instead of building a "checked one, now checking the next" chain in a single conversation. That removes the effect completely, because it lives entirely in the message history.

🚀 Quick start — if you still work with several reviews in one chat:

I need one assistant to check several independent texts/code snippets for errors.
Help me write a prompt for each review so that the results
of earlier reviews in this chat don't affect the strictness of the next one.
Here's an example task: {your task}

Limitations

⚠️ This is a threshold shift, not better reasoning: the model doesn't get better at telling correct from incorrect; it just flags problems less often. If its original judgments were well calibrated, the same effect will lead to missed real errors, not just fewer false alarms.

⚠️ Whether the effect helps is an accident of the specific task: in this study the softening turned out to be handy, because 82% of the filtered-out "false alarms" were the model's own baseless complaints (it was nitpicking for no reason). In your task, the model's original objections may be far more justified, and then the softening will hurt.

⚠️ Tested only on step-by-step math solutions: the mechanism (conversation history shifting the threshold) is most likely universal, but the study offers no direct evidence for reviewing code, texts or contracts.

How They Studied It

The researchers took ProcessBench, an existing set of step-by-step math solutions labeled by humans as correct or containing an error. Three models (Qwen3.6-27B, Qwen3.6-35B-A3B, Ministral-3-14B) were asked to verify solutions that were definitely correct, so any "found error" was a false alarm by definition.

Before the main check, a completed "check → fix" episode on a different, entirely unrelated text was mixed into the context, and the results were compared with several control groups: the same text length without an audit, an audit without a fix, and an audit with an "empty" continuation instead of a real fix. Five different instruction wordings were tested to make sure the effect wasn't a fluke of one phrase.

The key design trick: they measured not just the false-alarm rate but also the ability to distinguish genuinely wrong solutions from correct ones (via signal detection analysis from perception psychology). That let them separate two explanations: "the model got smarter" versus "the model just stopped complaining." It turned out to be the latter: discrimination ability didn't change on any model, while the threshold shifted on all of them. A manual review of 50 false alarms showed that 82% of them were genuinely unfounded nitpicks by the model, so in this experiment the softening turned out to be more of a useful side effect than a bug.

Resources

"Prior Audit–Repair Context Shifts LLM Verifier Thresholds Toward Leniency", Parsa Mazaheri (UC Santa Cruz), Kasra Mazaheri (MIT). Code and artifacts: github.com/parsa-mz/crtitxer. Related work: Temkit (2026) on ratings shifting toward the polarity of the preceding conversation, Zhou et al. (2026) on controlling verifier strictness on the same ProcessBench benchmark.