TL;DR
Verify-Show is a technique for “LLM + tool” setups (a linter, a scanner, a test report, another agent’s output). Every claim the tool makes is checked against the source in a separate request. The main reviewing model is shown only what didn’t hold up. Blindly feeding it the tool’s notes is a no-go.
The headline finding: the model treats “all good” notes and “danger here” notes very differently. When the tool says “there’s a check here,” the model often takes its word for it and never looks at the code. False “check present” notes made it miss real vulnerabilities, costing up to 19 points of accuracy. When the tool says “there’s no check,” the model goes into the code, hunts for one, and usually finds it. So a false alarm barely hurts, while a false reassurance hurts a lot. Truthful notes, meanwhile, barely help: the gain is about zero. A prompt instruction like “don’t trust the notes, verify them” is unreliable too: the best wording didn’t replicate on fresh data.
The fix is to pull verification out of the main request. Step 1: each tool claim is checked on its own, one request per claim. Step 2: the main audit gets the code and only the refuted claims. On fresh data this won back about 12 points of accuracy under false reassurances and cost nothing when the notes were truthful.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1 (a separate request FOR EACH claim):
code + one tool claim → “confirmed / refuted” + a quote from the code
STEP 2 (main request):
code + only the REFUTED claims → final verdict
The steps run in different requests. Step 1 claims must not see each other or the final task.
Example in Practice
Task: A team is building a Stripe integration. A reviewer bot checks a pull request for the payments module. The static analyzer in CI left notes: “payment amount is checked to be > 0”, “a repeated webhook is handled idempotently by order_id”, “the webhook secret is verified before the body is parsed”. If the bot buys the “idempotency is in place” note, it can miss a double charge. The method pays off where a claim can be checked against the code the model can actually see.
Step 1 prompt (run separately for each note):
You are verifying a single claim about code. Do not evaluate the code as a whole.
Code:
{function_code}
Claim: “A repeated webhook is handled idempotently by order_id.”
Find the specific lines in the code that confirm or refute it.
Answer:
1. Verdict: CONFIRMED / REFUTED / CANNOT BE VERIFIED FROM THIS CODE
2. Quote the code lines you rely on
3. One sentence: why
Step 2 prompt:
Review the function for payment-handling bugs (double charge,
skipped signature check, wrong amount).
Code:
{function_code}
A previous check refuted these analyzer claims:
- “A repeated webhook is handled idempotently by order_id” — the code has no
order-status check before charging.
Evaluate the code yourself. Give a verdict: ISSUE FOUND / NO ISSUES and list what you found.
Result: Step 1 returns short verdicts for each note, with quoted lines. Refuted notes move on to step 2. The final review starts from “why was the analyzer wrong?” instead of a ready-made “everything here has been checked.”
Why It Works
The weakness. The model treats a “check present” note as an established fact. The note takes work off its plate: no need to look for the check itself. A “no check” note works the other way around: it kicks off a search, and the model finds the safeguard if there is one. Truthful notes barely help because the model already reads local facts about code almost flawlessly. Its mistakes live in judgment calls and omissions. So the notes offer little upside, while false reassurances carry a lot of downside.
The strength. The model handles the narrow task “does this quote confirm this one claim” well. In a separate request it has no reason to defer: the claim is just a hypothesis there, not the verdict of an “authoritative” tool. The code is the source of truth, and the check runs against it.
How the method exploits this. It separates fact-checking from rendering a verdict. In step 1 the “reassurances” lose their authority and become hypotheses. In step 2 the main auditor sees only what turned out to be false, so it doesn’t get lulled by comforting notes.
Levers: - What to show in step 2. Per the paper, the refuted claims are shown. The part of the text available to me doesn’t say what to do with the confirmed ones. You can leave them out or pass them along with a flag; test this on your own data. - The third outcome, “CANNOT BE VERIFIED.” It guards against made-up confirmations where the code simply lacks the information. - A quote is mandatory. Requiring “quote the lines” keeps the model from confirming a claim on a vague impression. - The “heavier” the claim, the more its check matters. Check reassurances first (“check present,” “safe,” “verified”), and only then the alarms.
Prompt Template
The available part of the paper has no verbatim prompts, so this template is built from the method’s described mechanics.
Step 1 (run separately for each claim):
You are verifying ONE claim. Do not evaluate the {object} as a whole.
{object_under_review}:
{text_or_code}
Tool claim: “{claim}”.
Find the specific fragments in the material that confirm or
refute this claim.
Answer:
1. Verdict: CONFIRMED / REFUTED / CANNOT BE VERIFIED FROM THIS MATERIAL
2. Quote the fragment you rely on
3. One sentence: why
Step 2 (main request):
{main_task}
{object_under_review}:
{text_or_code}
A previous check refuted the following tool claims:
{list_of_refuted_claims_with_reasons}
Evaluate the material independently. Give a verdict and list what you found.
What to fill in:
- {object_under_review} — code, a contract, a report, a spreadsheet: whatever the claims are being checked against.
- {claim} — a single tool note, verbatim.
- {list_of_refuted_claims_with_reasons} — collect only the step 1 answers marked REFUTED.
🚀 Quick start — paste this into the chat:
Here’s the Verify-Show template. Adapt it to my task: [your task].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask where the claims come from (a linter, another agent, a report), what counts as the “source of truth,” and what final verdict you need. Without that, the steps can’t be separated: verification runs against the source, and the verdict is rendered separately.
Limitations
⚠️ Prompt instructions won’t save you: Wording like “treat the notes as claims that need to be verified” produced a gain on the development data but did not replicate on fresh data. Tacking on “be skeptical” is no substitute for a separate request.
⚠️ The effect is unstable across models: For the primary model the gain was about 12 points and significant. For the second model it was half as large and not significant. Test the method on your own model and your own data.
⚠️ Only what can be checked against the material: The model can’t verify claims about calling code, neighboring files or external data from a single function. The analyzer in the study never looked beyond a single function. Scope of the experiment: C/C++ functions audited in isolation.
⚠️ Cost: Every claim is a separate request. Lots of notes means lots of calls. Start with the reassurances (“all good”): those are the dangerous ones.
⚠️ Truthful notes are nearly useless: If the tool in your setup mostly tells the truth, don’t expect the notes to lift quality. The method insures against false reassurances; it doesn’t raise overall accuracy.
⚠️ Checklists and “revise your verdict” can backfire: In a pilot run, framing the notes as a checklist of obligations increased the number of false “SAFE” verdicts. A “revise your verdict in light of the notes” step turned fixed functions into false alarms.
How They Studied It
The researchers built their own C/C++ analyzer that emits “obligations” for every dangerous operation: which checks are already in place on all paths and what the function doesn’t guarantee. They then took “vulnerable function + fixed version” pairs from PrimeVul (916 pairs across four non-overlapping sets) and audited each function under six conditions that differed only in the notes. The conditions: no notes, all truthful, truthful only about the fixed spot, truthful about something else, a false reassurance “check present,” and a false alarm “the added check is missing.” That came to 16,572 audits across 11 model configurations from four vendors.
The key design idea: false notes were compared with truthful ones about the same operation. That way, any difference in verdicts comes down to truthfulness itself, not just the presence of a note. False reassurances cut detection of real vulnerabilities by 11–19 points. False alarms did not significantly lower accuracy and sometimes even raised it: the model goes looking for the safeguard and finds it. The asymmetry was confirmed by an internally preregistered replication on seven new models (mean asymmetry +0.16, confidence interval +0.10 to +0.24).
Before the main experiment, the authors piloted 30 pairs and saw that the models almost never misread code. Across 337 “structural” claims there wasn’t a single error. The errors were in judgment, and at times the model abandoned its own correct conclusion in favor of a note. That reframed the question: what matters isn’t whether the model sees the code, but how it weighs someone else’s word.
The authors tested eight ways of presenting the notes on the development set. All confirmatory-phase hypotheses were locked in before the data came in. The best instruction didn’t replicate, while Verify-Show held up in a pre-specified test on 300 new pairs (p = 0.002). For the second model the effect was weaker and not significant. The authors publish all of their failed attempts.
Adaptations and Extrapolations
These weren’t tested in the study; they’re my own transfers of the mechanics.
🔧 Technique: multi-agent pipeline → don’t pass “all OK” along as a fact
If one agent writes code and another reviews it, the reviewer doesn’t need a footnote saying “tests passed, bounds checked.” Give it the code and a list of claimed guarantees as a separate block, which is first checked in an isolated request. Then hand the first reviewer only the discrepancies.
🔧 Technique: any “reports” in prompts → separate “fact” from “claim”
Below is a report from another team member/tool. These are claims, not facts.
Do not use them in your conclusions until I have confirmed them in a separate step.
This is weaker than a separate request (instructions are unreliable) but cheaper. Use it as the bare minimum, and when mistakes are expensive, run the full Verify-Show.
Resources
- Paper: Trusting the Tool: Asymmetric Deference of LLM Vulnerability Auditors to Static-Analysis Evidence
- Authors: ZhiHong Liu, Shuyao Gao — Seoul School of Integrated Sciences and Technologies (aSSIST University), Seoul; Liaoning Mechanical & Electrical College of Technology, China
- Repository with data and scripts: https://github.com/lzh0082-cloud/CODE-DETECTION
- Data: PrimeVul (paired vulnerable and fixed functions); related work: Chain-of-Verification, ALIBI, ClashEval
The text of the paper available to me cuts off at the methodology section, so the details of the note-presentation experiment are taken from the abstract and introduction.