TL;DR
Plausibility capture is a finding about LLM judges: attach a coherent “here’s why” explanation to an answer, and the judge goes softer and accepts the answer more often. The judge is no longer asking only “does the answer match the image?” but also “does the reasoning sound sensible?” It works even when the image flatly contradicts the answer.
The catch: a plain explanation throws the judge off almost as much as a deliberately persuasive pitch loaded with pressure and appeals to authority. A judge that confidently wrote “definitely incorrect” with no explanation flips its verdict to “correct” in a third of cases once the explanation is there. Overall accuracy barely moves: the judge starts accepting both right and wrong answers more often, and the two kinds of errors mask each other. So you won’t notice a thing until you separately count how many wrong answers slipped through. The root cause is that coherent text arguing for the answer competes with the image, and the judge trusts the text.
Practical takeaway: don’t show the judge the explanation alongside the answer if what you need to verify is the answer itself. Prompts like “be careful” or “solve it yourself first” help only partially, and not on every model.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
PROBLEM: [image + question + answer + EXPLANATION] → judge goes soft, lets errors through
SOLUTION (basic, one prompt):
STEP 1: Give the judge only the image → question → answer (NO explanation)
STEP 2: Judge rates on a 4-point scale → verdict
OPTIONAL (partial protection, model-dependent):
+ “Solve the question yourself before grading” (self-check)
+ “Decide conservatively: when in doubt, reject”
Example in Action
Task: A marketer at a Shopify store has an AI analyst read Google Analytics screenshots and answer questions about the charts. A second AI acts as the judge and checks the answers. If the analyst’s answer always comes with a slick “because we ran a Friday promo and traffic spiked,” the judge will start letting mistakes through. So the judge gets only the question, the screenshot and the final answer. The explanation stays with the analyst.
Prompt (for the judge):
You are checking an answer to a question about an image. The image is the only source of truth.
[IMAGE: screenshot of a 30-day conversion rate chart from Google Analytics]
Question: On which day was the conversion rate highest, and what was it?
Answer to check: On the 14th, about 4.2%.
How to work:
1. First find the answer to the question from the image yourself, without relying on the answer above.
2. Compare your result with the answer to check.
3. Decide conservatively: if you cannot confirm the answer from the image, do not accept it.
Choose exactly one verdict:
(A) definitely correct
(B) probably correct
(C) probably incorrect
(D) definitely incorrect
Output: your own answer, the discrepancies, and the verdict as a single letter.
Result: The model first gives its own reading of the chart, then compares it with the analyst’s answer and lists the discrepancies. It ends with a single verdict letter. The judge never sees the analyst’s explanation, so the explanation can’t sway it. Keep in mind that prompt-level protection is partial: on some models the effect is weak.
Why It Works
The LLM weakness. A judge model reads all the text in the request as one context. A coherent explanation in favor of the answer acts as an argument “for”: the acceptance threshold drops. Meanwhile, the ability to tell right from wrong barely changes. What shifts is the bar, not the eyesight. That’s why overall accuracy hardly drops while the share of missed errors climbs.
What the model can do. The model genuinely can check an answer against an image. Replace the image with a blank background and discrimination gets noticeably worse for almost every judge. So visual verification works, as long as text arguing for the answer doesn’t get in the way.
How the method uses this. The simplest move is to remove the source of pressure: don’t pass the explanation. Everything else is a weak safety net. Asking the model to solve the task first forces it to form its own opinion before reading someone else’s. A conservative instruction pushes the bar back up. Neither one guarantees the result.
Control levers: - Explanation in the judge’s input → remove it, this is the main lever. - “Solve it yourself first” → add it for tasks with a verifiable answer, especially with images and tables. - “Decide conservatively” → add it when missing an error costs more than rejecting a correct answer. - A four-step scale → lets you spot the shift: watch “definitely incorrect” turn into “definitely correct.” - Option order → shuffle it in bulk checks so the answer’s position doesn’t matter.
Prompt Template
You are checking an answer to a question about {material_type}. {source_of_truth} is the only source of truth.
{material: image, chart, screenshot, document}
Question: {question}
Answer to check: {answer}
How to work:
1. First find the answer to the question yourself, without relying on the answer above.
2. Compare your result with the answer to check.
3. Decide conservatively: if you cannot confirm the answer from the material, do not accept it.
Choose exactly one verdict:
(A) definitely correct
(B) probably correct
(C) probably incorrect
(D) definitely incorrect
Output: your own answer, the discrepancies, and the verdict as a single letter.
What to fill in:
- {material_type} — an image, chart, screenshot, table.
- {source_of_truth} — what exactly counts as the reference.
- {material} — the material itself, placed before the answer (in the paper the image comes before the answer).
- {question} and {answer} — without the analyst’s commentary.
Lines 1 and 3 are the two defenses the study tested separately (self-check and the conservative instruction). They give a partial, model-dependent effect. The main defense is keeping the explanation out of the input.
🚀 Quick start — paste into a chat:
Here is a prompt template for an LLM judge that checks an answer without the explanation. Adapt it to my task: [your verification task].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what material is being checked, what counts as the reference and what kind of verdict you need. That way the judge relies on a concrete source of truth rather than on plausibility.
Limitations
⚠️ Visual tasks only: The whole study is about questions on images: medical scans, charts, hallucinations. Carrying it over to text and tables makes sense, but the authors didn’t test it.
⚠️ Protection is partial: The two prompt defenses that worked best were the conservative instruction and “solve it yourself first.” But the effect is limited and model-dependent: it helps noticeably on some models and barely at all on others. The available text gives no numbers for these defenses.
⚠️ Strong models aren’t immune: Even judges that solve the task correctly almost every time on their own still fall for explanations, just less. Two of the six closed models shifted in the opposite direction on average, but one of them still caved on a separate dataset.
⚠️ Hidden in overall accuracy: If you only look at the overall share of correct verdicts, you won’t see the shift. Count separately how many wrong answers got through.
⚠️ Eloquence doesn’t matter: Hardening your defenses “against persuasiveness” is pointless. What throws the judge off isn’t rhetoric, it’s the mere presence of coherent text supporting the answer.
How They Studied It
The idea was simple: freeze everything except the explanation. The team built the JUPE benchmark of 250 image-based questions: medicine, charts and hallucination-provoking questions. Each question came with a ready-made answer (right or wrong) and different kinds of explanations. There was a plain “post-hoc” justification, a chain of thought (CoT) and 41 persuasion techniques (threats, appeals to authority, “information density” and others). All of this was compared against the answer with no explanation.
There were 14 judges from five families: eight open models from 2B to 32B parameters and six closed ones. On top of that, they added two controls: an “empty” explanation with no content and an explanation taken from a different question. Both pushed judges toward rejection, not acceptance. So it isn’t the mere presence of text that matters, it’s that the text supports a specific answer.
The result was surprising. A plain explanation cut the rejection of wrong answers by 12.7 pp, chain of thought by 13.8 pp, and persuasion techniques by 12.5 pp, i.e. no better than plain. The correlation between “how rhetorical the text is” and “how much it throws the judge off” came out at zero. Using signal detection theory, they showed that it’s the acceptance threshold that moves, while the ability to tell right from wrong barely changes. The effect held with a different generator model, on other data (object-grounding questions on images), on compressed and noisy images, and across different rating scales. A red box around the object named in the answer amplified the effect further.
Adaptations and Extrapolations
🔧 Technique: remove the explanation from the input → the judge doesn’t fall for someone else’s reasoning
If your pipeline currently hands the judge “answer + explanation,” split them. The judge gets only the answer and the material. This is the most direct consequence of the result, and it needs no extra prompting.
Extrapolation (my assumption, not tested in the paper): two passes. The first judge sees only the answer, the second sees the answer with the explanation. If the verdicts disagree, that’s your cue for a manual check:
Pass A (no explanation): [template above] → verdict A
Pass B (with explanation): same template + “Author’s explanation: {explanation}” → verdict B
If verdict A is stricter than verdict B → flag the answer as “possibly accepted because of the explanation” and send it for manual review.
The idea builds on the paper’s finding that the explanation shifts the bar, but the authors did not test the two-pass approach itself.
Resources
- A Plausible Explanation Is Enough to Mislead Multimodal Judges — Anglin Cai (Northwestern University), Zhiqi Wang, Dongwon Lee, Yuchen Yang (The Pennsylvania State University), Xiaozhi Yang (University of Pennsylvania), Yurun Lin (University of Washington).
- JUPE (Judge Under Plausible Explanation) — a benchmark of 250 questions, 14 judges.
- Taxonomy of persuasion techniques: Zeng et al., 2024; “information density”: Hackenburg et al., 2025.
- Question sources: VQA-RAD, PMC-VQA, ChartQA, ChartQAPro, HaloQuest, PreHal, GQA.