LLM code review accuracy: how to stop false alarms and missed bugs

Symptom: the reviewer is wrong in both directions

Ask a model to review code and two things go wrong, often in the same session. The first is false alarms: the model flags clean code, proposes "optimizations" that change nothing, or rewrites a file to fix one line. The second is the opposite: rubber-stamped bugs. The model reads a buggy function, assumes the author meant it, and approves it or even writes a test that locks the bug in.

The numbers are large. In a backtest-auditing benchmark, models without a warning flagged 93.8% of clean code as violating rules (clean-control calibration study). In an efficiency-review test across 9 models, every run on already-optimal code produced a fake improvement. On the other side, coding agents given already-fixed GitHub issues changed working code in 35-65% of cases (the action-bias study), and LLM-as-a-judge evaluations of code found judges that approve incorrect solutions, especially logic bugs that look right on the page.

Treat this as one problem with two faces: the reviewer's decision threshold is set by the prompt, not by the code.

Why it happens

• The request presupposes a finding. "Review and improve" or "optimize this" implies something is wrong. Models were trained mostly on tasks where success means producing a change, so "nothing to do" does not register as a completed task. The action-bias authors report that more reasoning effort barely changed this; only the framing did.
• Safeguards look like the problem. Code that prevents data leakage contains the same words as code that causes it. Without a base-rate hint, the model treats the cure as the disease.
• No criteria means guessing. Told to "check edge cases", the model checks what it thinks usually matters, not what this author forgot. Specification Grounding measured a 33% false-alarm rate for free-form edge-case review.
• The implementation anchors the reviewer. Once the model sees code, it explains it: if it is written that way, it must be intended. The spec-instead-of-code study (318 buggy Java methods, 11 LLMs) showed buggy code shifts the model's own preferences toward bug-confirming tests.
• Chat history moves the threshold. After one find-and-fix cycle, the model flags problems less often on the next, unrelated item; the false-alarm rate dropped by 3-11 percentage points in the lenient-reviewer experiment (tested on step-by-step math solutions).

Fixes ranked by measured effect

1. Give an explicit checklist of rules. Replace "find problems" with "check each of these N rules". In Specification Grounding (18 Python tasks, Claude, GPT and Gemini) false alarms went from 33% to zero; a spec as a plain paragraph caught 27/30 bugs, as an enumerated checklist 30/30, while the plan alone caught 2/30. Content of the spec does most of the work; 5-10 rules is the suggested range.
2. Say that clean items exist. One sentence, "some of these examples are fully correct, with no violations", took false accusations on clean code from 93.8% to zero while detection of real errors stayed at 100% (clean-aware prompting, four models, 96 backtests). The same study found aggressive personas such as "skeptical reviewer" raised false accusations with no gain in detection; "auditor" balanced better.
3. Make "no change" a successful outcome. The ABSTAIN-OR-FIX line "either address the issue or abstain ... if the issue is already resolved" raised correct abstentions from 60% to 88% for coding agents (action bias). A plain "reproduce first" instruction without that permission did not help.
4. Add a confidence bar and a named exit. "Only suggest an edit if you are more than 90% sure ... otherwise output ALREADY_OPTIMAL" cut fake optimizations from 100% to 56%, with 100% correct edits kept on genuinely slow code (Penalty Prompt). Pricier model versions calibrated worse than their light siblings there.
5. Write the spec before showing the code. Ask for the intended behavior from the name and context only, then check the implementation against that text in a separate prompt, so the bug cannot anchor the reviewer (spec instead of code). For tests, contract-first test generation extracts pre- and postconditions, marks each as tested or not, and writes tests only for gaps; it still caught the intended bug only about 60-80% of the time.
6. Route, then check simple propositions. Mixture-of-Prompts first decides which problem types are even possible, then answers yes/no propositions combined by explicit logic, instead of hunting for every bug type at once.
7. One review per fresh chat. Because a prior fix lowers vigilance, review independent pull requests in separate conversations or ask the model to evaluate "from scratch, as if this were the only document" (lenient-reviewer effect).
8. Constrain the fix itself. When the review turns into a patch, "preserve as much original code as possible, change only what's necessary" cut unnecessary edits by almost a third in the preservation-prompt study, where one model had deleted 5 lines and added 60 for a one-line bug.

A combined prompt stack

These fixes target different parts of the decision, so they stack. A practical order:

1. Criteria first: an enumerated list of rules or bug categories with a status per rule (met / partial / violated) and a quote.
2. Base rate: state that some items are correct and that protective patterns are not violations.
3. Two valid outcomes: problems found, or "ready, no changes needed" with a reason; ban cosmetic edits.
4. Evidence bar: every finding needs the exact line and a specific fix; optimizations need a complexity argument or the named exit.
5. Isolation: for high-stakes logic, derive the spec in one prompt and check the code in another, and start a fresh chat per review.

To see what the model decided without telling you, AssumptionMiner asks it to list the 3-4 silent decisions it made per task (validation, error handling, storage, security) with a severity level.

What does not work

• Asking for more thinking. Higher reasoning effort barely moved action bias (action bias).
• Running the same vague review more times. Eight free-form edge-case runs combined still fell short of one pass through explicit rules, and doubling the test budget added +4 pp (Specification Grounding).
• Letting the author grade itself. Self-review inflates scores and misses logic bugs; use it as a second opinion, not a gate (LLM-as-a-judge for code).
• Trusting stated confidence. More accurate answers often came with lower self-reported confidence (clean-control benchmark); suggested fixes were irrelevant in 12.5% of detections. Measure or run the code.
• Overcorrecting. Permission to abstain can turn into inaction on genuinely weak code, and the confidence-threshold prompt still bluffed on roughly half of optimal samples. Keep the criteria explicit so "no change" has to be justified.

Review {item to check} for these types of issues: {list of categories}.

Important: some of these examples/items are completely correct, with no violations.
Don't mistake normal or protective elements for a real problem.
Only flag things that genuinely create a risk or error.

For each issue you find, provide:
1. The type of issue
2. The exact quote/location where it occurs
3. A specific fix (not general advice — exactly what to change)

Here's the {item to check}:
{text/code/data}

Resolve the following GitHub issue in the provided codebase.
First, reproduce the issue. Then either address the issue or
abstain from modifying the code if the issue is already resolved:
{issue_description}

Check {what we're reviewing} against the following list of requirements.

Requirements:
1. {requirement_1}
2. {requirement_2}
3. {requirement_3}
[... all requirements]

For each requirement, in turn:
- Status: met / partial / violated
- Specific explanation: what exactly is wrong (if not met)
- A quote from the material demonstrating the issue (if applicable)

At the end — a short list of edits: what exactly to rewrite and how.

Here's the material to review:
{text / plan / document}