TL;DR
When you ask an LLM to find mistakes in something — code, a contract, a piece of writing — the model starts nitpicking even the parts that are completely fine. It's primed to hunt for problems, not to confirm that everything checks out. Adding one phrase to the prompt — "some of these examples are fully correct, with no violations" — sharply cuts down false accusations without losing the ability to catch real errors.
Researchers tested how an LLM audits financial backtests (testing trading strategies against historical data) for methodological errors — like "look-ahead bias" or unrealistic transaction costs. Without a warning, the model falsely flagged 93.8% of clean, error-free code — meaning out of 100 correct examples, it found a "problem" in almost every single one. The reason is simple: safeguards in the code (correct data splitting, realistic costs) are textually similar to the patterns the model is scanning for as red flags — it confuses the cure with the disease.
Adding the phrase "some of these examples are clean, with no violations" (clean-aware) dropped false positives to zero while keeping the real-error detection rate at 100%. Bonus: an explicit list of error types (a taxonomy) in the prompt also helped the model catch "hidden" issues that weren't explicitly named — issues the model missed entirely without that list.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Give the model an explicit list of problem categories it should look for → a closed taxonomy
STEP 2: Add a warning "some of these examples are clean/correct — don't mistake them for a problem" → cuts false positives
(both steps go into a single prompt)
Example in Action
Scenario: You're the founder of a small company. You just got a commercial lease agreement for an office. You want to run it through ChatGPT to check for hidden traps, but you don't want the model to "panic" and flag every standard clause as risky — otherwise you'll end up arguing with your landlord over nothing.
Prompt:
Review this lease agreement for risks to the tenant. Specifically look for these
types of issues: hidden rent increases, a one-sided right for the landlord to
terminate the lease, penalties with no upper cap, no right to sublease, unclear
terms for deposit return.
Important: the lease may contain completely normal, standard clauses (for example,
ordinary payment deadlines or mutual termination terms) — don't mistake these for
risks. Only flag things that genuinely put the tenant at a disadvantage.
For each issue you find, provide: the type of risk, the exact quote of the clause,
and a specific explanation of why it's a risk.
Here's the lease text:
{lease text}
Result: The model will go through the lease clause by clause, flagging specific risks with a quote and explanation. Compared to a prompt without the warning, the number of "panic" flags on standard clauses will be noticeably lower, while the detection rate for real issues stays the same.
Why This Works
When an LLM is asked to "find an error," it drifts toward confirming the task — it's trained to be helpful and to find exactly what it was asked to look for. Without a limiter, it cranks up its sensitivity and latches onto safeguards that merely resemble a red flag: for example, the word "leakage" (data leakage) sitting next to a line of code that actually prevents that leakage.
The model picks up well on explicit context framing and base rates (how often an error actually shows up). If you flat-out tell it: "not everything is an error, some of these examples are clean" — the model recalibrates its internal suspicion threshold and stops hammering every pattern that merely looks similar.
Levers you can pull: - List of problem categories → the more precise and explicit the list, the better the model catches "hidden" issue types that aren't explicitly named. - The "clean examples" phrase → the main calibration lever. Without it — a flood of false positives; with it — accuracy jumps sharply. - Role/persona → in the study, the "skeptical reviewer" and "risk manager" personas produced more false accusations with no gain in detection rate. The "auditor" and "quant reviewer" personas struck the best balance. Be careful with aggressive personas if you care about precision rather than flag count. - Don't trust the model's stated confidence as a quality signal — in the study, better-calibrated (more accurate) answers were often paired with lower self-reported confidence, not higher.
Prompt Template
Review {item to check} for these types of issues: {list of categories}.
Important: some of these examples/items are completely correct, with no violations.
Don't mistake normal or protective elements for a real problem.
Only flag things that genuinely create a risk or error.
For each issue you find, provide:
1. The type of issue
2. The exact quote/location where it occurs
3. A specific fix (not general advice — exactly what to change)
Here's the {item to check}:
{text/code/data}
Fill in: {item to check} — what you're reviewing (a contract, code, a resume, a financial report, marketing copy for legal risk); {list of categories} — the specific error types that matter for your task; {text/code/data} — the actual material to review.
🚀 Quick Start — paste this into the chat:
Here's a prompt template for error-checking with built-in protection against false
positives. Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask you which specific problem categories matter and what counts as "normal" in your material — because without that, it can't calibrate its suspicion threshold to your specific task.
Limitations
⚠️ Small sample size: the study used only 6 examples per error category — confidence intervals are wide, and some findings are more of a trend than a precise number.
⚠️ Synthetic examples: the "clean/broken" pairs were artificially constructed from a single template — in real, complex documents (long contracts, large codebases) the effect may be weaker.
⚠️ The fix-quality problem remains: even with a clean-aware prompt, the model can correctly locate a problem but suggest an irrelevant fix — this happened in as many as 12.5% of detections. Double-check any proposed fix separately.
⚠️ Single-turn, no dialogue: every check in the study was a single call with no follow-up questions; in a real multi-turn chat, calibration might behave differently.
How the Study Was Done
The researchers assembled 96 examples of financial strategies (backtests): half contained a real methodological error (look-ahead data leakage, unrealistic costs, and so on), and the other half were "clean" versions of the same strategies, deliberately written with safeguards (correct sample splitting, realistic costs) that resemble suspicious patterns but are actually correct.
They tested four models (DeepSeek, GPT-4o-mini, Gemini, GPT-4.1-mini) under three prompting conditions: no hints, with a list of error types, and with an explicit warning that clean examples exist. An automated scorer checked not just whether the model "found the error," but also whether it located the error in the right spot and whether the proposed fix was relevant.
The surprising result: three of the models reached the same 100% recall (real-error detection rate) — indistinguishable by that metric alone. But the number of false accusations on clean examples varied dramatically between models — from 0% to 79%. The main practical takeaway: a simple "found it or didn't" report masks real differences in review quality — you need to look at detection rate and false-positive rate together.
Resources
Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark — Makar Ulesov, Vladislav Smirnov, Omar Ibrahim (Mohamed bin Zayed University of Artificial Intelligence), Arsenii Bobovnikov (University of Pittsburgh).