TL;DR
LLMs are better at checking individual claims than at producing the right answer outright. PVD exploits this: one role (the prover) defends the answer with 3–5 atomic, checkable claims, and a second role (the verifier) attacks the weakest of them. If the answer survives all that and doesn't change, it is more likely to be correct than the rest (with a capable verifier).
The core problem: when should you trust an AI's answer? The model produces confident text both when it's right and when it's wrong. There's no internal signal saying "this answer is reliable, that one isn't." Simple self-critique gave little signal in the paper: a Reflexion-style baseline separated right from wrong answers by only +4.0 points, not statistically significant.
PVD solves this with structure: it separates the defender role from the skeptic role and watches one specific signal — did the answer change under pressure. If not, that's ANC (Accept + No Change): high confidence. If it changed or didn't hold up — low.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1 — PROVER (one request or one role in the chat):
→ Picks an answer
→ States 1 claim ("Answer X is correct because...")
→ Breaks it into 3–5 atomic sub-claims (each independently checkable)
→ Adds a paragraph justifying the chain
STEP 2 — VERIFIER (next request or a role switch):
→ Reads everything
→ Issues a verdict:
ACCEPT — all sub-claims are specific, checkable, consistent → stop
REJECT — a factual error or contradiction found → stop
CHALLENGE — attacks the single weakest sub-claim → the prover responds
STEP 3 — LOOP (until a final verdict; the paper capped it at 12 challenges, 6 in the single-call version):
→ The prover responds to the attack
→ If it changes the answer → log a "change"
→ Verifier again: ACCEPT / REJECT / CHALLENGE
OUTCOME:
ANC (high confidence) = ACCEPT + the answer never changed
Not ANC (low confidence) = there was a REJECT / the answer changed / never reached ACCEPT
The paper tested two setups through the API: two models (prover and verifier) and a single call where one model plays both roles. Running it in a single chat or across two browser windows is our adaptation.
Example in Action
Our illustration, not from the paper (the paper used multiple-choice science questions).
The task: You run a small single-member LLC. Your accountant says you can save on payroll taxes by moving part of the work to an independent contractor. Claude gave an answer. But you don't know — is it true, or a confident hallucination?
Prompt (Phase 1 — PROVER):
You are a tax attorney acting as the prover. Your job: defend the answer to the question
through structured argumentation.
Question: Can a single-member LLC legally reduce its tax burden
by paying a former W-2 employee as a 1099 independent contractor
for the same work they did on staff?
Your conclusion: [give your answer, YES or NO, with a short thesis]
Then provide:
1. CLAIM: one sentence on why your answer is correct
2. SUB-CLAIMS: exactly 4 atomic facts, each independently checkable
(citing a specific section of the Internal Revenue Code, IRS guidance or ruling)
3. JUSTIFICATION: a paragraph on how the sub-claims support each other
Prompt (Phase 2 — VERIFIER, new request or role switch):
You are an IRS auditor acting as the verifier. Your job: check the prover's
argument and issue a strict verdict.
[paste the prover's answer]
Check every sub-claim. Find the ONE weakest —
where the citation is imprecise, the fact is doubtful or the argument doesn't prove the thesis.
Issue a verdict — strictly one of three:
ACCEPT — all 4 sub-claims are specific, accurate, logically connected
REJECT — there's a factual error or a direct contradiction → explain exactly what
CHALLENGE — name ONE weak sub-claim and ask a specific probing question
The result:
The model shows a structured chain: the prover's claim → the verifier's attack → the prover's response to the attack. If three rounds go by and the answer ("NO, this is worker misclassification under the IRS common-law control test") hasn't changed, that's an ANC signal: you can trust the answer more than average. If along the way the prover switched positions or the verifier issued a REJECT with a specific justification, that's a red flag: the topic needs to be checked with a real attorney.
Why It Works
LLMs are bad at knowing when they're wrong. The model generates text from patterns — the confident tone is the same for a correct answer and for a hallucination. Head-on self-critique gave little signal in the paper (the Reflexion baseline: +4.0 points, not significant); our reading is that the same model with the same context tends to rephrase the same thing.
But LLMs are good at checking small steps. Judging one specific claim — "does this contradict section 3509?" — is far easier than judging the whole answer at once. That's an asymmetry: defending a full answer is hard, checking a single sub-fact is easy. That's exactly what PVD exploits.
A changed answer = a signal of uncertainty. If the prover changed its answer under pressure, it found a weakness in its own argument. ANC (the answer survived unchanged) is the opposite: the model found no loophole to retreat through. It's not a guarantee of being right, but with a capable verifier it was a strong filter: on GPQA Diamond, accepted-unchanged answers were about 30 points more accurate than the rest for Claude Sonnet 4.6 + Haiku 4.5.
Control knobs (our suggestions, except where the paper is cited): - Number of sub-claims → the paper's prover gives 3–5 - Number of rounds → the paper capped challenges at 12 (6 in single-call mode); most runs ended within about 2–3 rounds - Verifier strictness → add "you are a skeptic; don't accept any sub-claim without a precise citation" → presumably a tougher filter, less coverage (not tested) - Two models vs. one → on GPQA, Sonnet 4.6 + Haiku 4.5 separated right from wrong better (+32.0 points) than Sonnet 4.6 playing both roles in one call (+13.6). The paper does not show that a verifier from a different model family is sharper in general
Prompt Template
=== PHASE 1: PROVER ===
You are an expert prover in {field}.
Question: {question}
Structured defense of the answer:
ANSWER: [one clear statement]
MAIN THESIS: [one sentence — why the answer is correct]
SUB-CLAIMS (exactly {number 3-5}, each independently checkable):
1. [a specific fact + source / citation / mechanism]
2. [a specific fact + source / citation / mechanism]
3. [a specific fact + source / citation / mechanism]
LOGIC: [a paragraph — how the sub-claims together prove the thesis]
=== PHASE 2: VERIFIER ===
You are a strict verifier in {field}.
Read the prover's argument below and issue a verdict.
[paste the output of phase 1]
Find the ONE weakest sub-claim — imprecise, unproven
or not logically supporting the thesis.
Issue STRICTLY one verdict:
ACCEPT — all sub-claims are specific, accurate, consistent
REJECT — there's a factual error or contradiction [explain]
CHALLENGE — name the weak sub-claim and ask the prover one precise question
=== PHASE 3 (if CHALLENGE): PROVER'S RESPONSE ===
[paste the verifier's challenge]
Prover: respond to the challenge. If you found an error, say so directly
and correct the answer. If you're confident, defend the sub-claim more rigorously.
[repeat phase 2 — up to {number 3-5} rounds]
=== OUTCOME ===
Record:
- Final verdict: ACCEPT / REJECT
- Did the answer change? YES / NO
- If ACCEPT + NO → ANC: high confidence in the answer
- Otherwise → low confidence: the answer needs further verification
Placeholders:
- {field} — the topic of the question: tax law, medicine, marketing
- {question} — the specific question you need a reliable answer to
- {number 3-5} — the number of sub-claims (the paper uses 3–5) and your cap on rounds (the paper allowed up to 12 challenges)
The template is our adaptation of the paper's protocol, not its original prompts.
🚀 Quick start — paste into the chat:
Here is a PVD (Prover-Verifier Deliberation) template. Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask: which field (to set up the roles), the specific question (to know what to defend), and how strict a check you need (to pick the number of rounds) — because without those it can't properly formulate the sub-claims and build the chain.
Why It Works — the Mechanics
The weakness: An LLM generates text from patterns; it doesn't "think" in the literal sense. Self-assessment ("are you sure you're right?") lands in the same session with the same context — the model reproduces the original pattern, changing the wording but not the position.
The strength: LLMs are good at local comparison — checking whether one specific sub-claim contradicts another or a known fact. That's easier than a global judgment of correctness.
How the method connects them: By breaking the answer into atomic sub-claims, PVD reduces the verifier's job to simple local checks. It's like checking each step of a proof separately instead of evaluating the whole proof at once. Plus there are separate histories: the verifier sees only the prover's structured output, not its internal reasoning. That removes the effect of "contamination" by a shared context.
Adaptations and Extrapolations
🔧 Two browser windows instead of one chat — sharper criticism
Run Claude in one tab as the prover and ChatGPT as the verifier (or vice versa). This is our idea: the paper did test cross-family pairs (e.g. Gemini 3.1 Pro + GPT-5.5-pro), but it does not show that a verifier from another family challenges more often.
🔧 No citations = a reliability warning
Add this to the prover's prompt: "Each sub-claim must contain a specific citation (a section of law, a study, a date, a name). A sub-claim without a citation is invalid." If the prover can't provide citations, that alone signals the answer is unreliable — before verification even starts.
🔧 The ANC principle works without the full protocol
The quick version: ask it to answer the question → then ask it to "attack your own answer and find three weak spots" → then "correct the answer in light of the attack." If the corrected answer matches the original, confidence goes up. If it changed, the answer was unstable. This shortcut is our simplification and was not tested in the paper; note that the paper's Reflexion-style self-critique gave little signal.
Limitations
⚠️ The verifier has to know the subject. If the question is at the edge of the model's competence, the verifier won't notice the prover's mistakes and will ACCEPT hallucinations. On such topics the ANC signal loses its meaning — the study showed this directly on HLE (Humanity's Last Exam): with Claude Sonnet 4.6 + Haiku 4.5, accepted answers were 7.1 points less accurate than the rest.
⚠️ Not for subjective judgments. "Which slogan is better" or "rate this text" — the sub-claims can't be checked as facts. The paper tested only English multiple-choice questions (GPQA Diamond, HLE) and warns the results may not transfer to open-ended answers. Using it for law, medicine or business facts is our extrapolation.
⚠️ It costs more tokens. One cycle is at least 3 requests. With 3–5 rounds and two models, that's 6–15 requests per question. Overkill for simple tasks.
⚠️ ANC is a signal, not a guarantee. A confident hallucination sometimes passes the check. With a capable verifier, ANC raises the probability of a correct answer statistically; it doesn't give a 100% guarantee.
How It Was Studied
The NYU team framed the problem unconventionally: not "how do we make AI more accurate" but "how do we know when AI is accurate". They took two hard benchmarks — GPQA Diamond (graduate-level questions in biology, physics and chemistry) and Humanity's Last Exam (the frontier of academic knowledge) — and tested several prover–verifier pairs: Claude Sonnet 4.6 + Haiku 4.5, GPT-5.4 + GPT-5.4-mini, Gemini 3.1 Pro + Gemini Flash-Lite, Gemini 3.1 Pro + GPT-5.5-pro and others. The headline pair (Sonnet 4.6 + Haiku 4.5) gave a gap of about 30 points on GPQA.
An interesting design detail: they deliberately tested a degraded version — a single call where the model plays both roles itself. That's close to what an ordinary user can do in a chat (our observation). With Claude Sonnet 4.6 on GPQA, that version reached an HC-Prec of 83.9% on the 63% of questions it accepted unchanged, versus 70.3% on the rest (overall accuracy 78.8%). So the gap between the ANC subset and the remaining questions was 13.6 percentage points with a single model, and 34.8 pp with the best pair (GPT-5.4 + GPT-5.4-mini: 97.6% precision, but on only 43% of questions). A clear separation of "knows" from "guesses" — for strong pairs on this benchmark.
A surprising finding: on Humanity's Last Exam the ANC signal can invert — accepted answers turn out less accurate than the rest. This happened for Sonnet 4.6 + Haiku 4.5 on HLE overall (−7.1 points; the authors say Haiku lacks the domain knowledge to raise non-trivial challenges) and for GPT-5.5 + Gemini 3.1 Pro on the small engineering subset (−24.3 points, 25 questions), even though that pair was strongly positive on HLE overall (+27.9). The authors describe it as "a practical failure mode when the verifier operates outside its effective region."
Resources
Paper: Trust but Verify: Prover-Verifier Deliberation for Selective LLM Prediction (arXiv preprint 2605.25133, May 2026)
Authors: João Sedoc, Baotong Zhang (New York University), Dean Foster
Code and Claude Code skill: github.com/jsedoc/prover-verifier-deliberation
Related methods from the paper: Chain-of-Thought (Wei et al.), Self-Consistency (Wang et al.), Reflexion (Shinn et al.), Multi-agent Debate (Du et al.), Universal Self-Consistency (Chen et al.), Neural Interactive Proofs (Hammond & Adam-Day)