Epistemic Blind Spot: LLMs Trust Text That Looks Academic Instead of Checking the Numbers

This isn't a technique but a behavioral pattern with a practical consequence. Here's the structure of the problem and the workaround:

❌ Standard request:
INPUT: [Source A + Source B + Source C] → "Draw a conclusion"
RESULT: The model weighs sources by style, not by data correctness

✅ Workaround: separate isolation from synthesis:

STEP 1 (separate request): 
"Evaluate ONLY this source. Are there internal contradictions 
in the data? Do the numbers actually add up with each other?"
→ Do this for each source in turn

STEP 2 (separate request):
"Here are the evaluations of each source [from step 1]. 
Now do the synthesis, taking these evaluations into account."
→ Synthesis with skepticism already built in

Two requests, not one. Isolation first, synthesis second.

The task: you're preparing an investor pitch on the US EdTech market. You asked Claude to find contradictions across three market reports you uploaded, which give different market-size figures (BrightPath Research, EdMetrics Institute, LearnScale Analytics). One report looks especially convincing: lots of methodology, charts, p-values. But it says "correlation of 0.97 with n=12", which is suspicious.

Prompt:

Below is an excerpt from one report. Evaluate ONLY this excerpt, 
without comparing it to other sources.

Answer three questions:
1. Do the numbers within this text agree with each other? 
   Is there anything that's impossible given the stated sample parameters?
2. Do the statistics match the described methodology?
3. Final verdict: can these data be relied on in a pitch?

[paste the report excerpt]

Run this prompt three times, once per report. Then, as a separate request:

Here are the results of checking the three reports:
[Paste the answers from the three requests above]

Now synthesize: which market-size range is best supported? 
Take the identified data problems into account.

Result: In the first block, the model works as a critical reviewer: no context from other sources, no temptation to "weigh" opinions. It gives a concrete assessment of each source: what raises questions, what adds up, what doesn't. In the second block, the synthesis already runs with those assessments as context, and the model can't "forget" the problems because you put them there yourself.

The LLM's weakness in multi-source analysis: when a model reads several documents at once, it automatically builds a "trust map" of whom to believe more. That map is built from stylistic signals: academic language, references to methodology, precise numbers, technical terms. The specific numbers aren't checked for internal correctness; what matters is that they're there and look precise.

What the model does well: critically evaluating a single text in isolation. With no competing sources, the model switches into "review mode" and catches even subtle inconsistencies: statistically impossible intervals, correlations above the mathematical ceiling, p-values below the allowable level.

How the workaround exploits this: we force "review mode" on before synthesis by splitting the task into two steps. In the first step there are no competing sources, just one text and a direct question about the numbers. In the second step, the isolation-check results become part of the context: the model synthesizes with ready-made assessments instead of building trust from scratch.

Control knobs: - Detail of the step 1 questions → the more specifically you ask ("do the sample size and confidence interval add up?"), the more precise the check - Output format of step 1 → ask for "a reliability score on a 1-5 scale"; that's easier to paste into step 2 - Number of sources → with 5+ sources, you can ask the LLM to build a comparison table of the assessments itself before the synthesis

Step 1, isolation check (run separately for each source):

You have one source in front of you. Evaluate only this one, 
without comparing it to other materials.

Source: {source_name}
Text: {text_or_data}

Check:
1. Internal consistency of the data: do the numbers contradict 
   each other? Does the sample size match the precision of the results?
2. Does the methodology support the stated conclusions?
3. Are there red flags that lower trust in these data?

Bottom line: how far can this source be trusted? (high / medium / low)
Justify it in 2-3 sentences.

Step 2, synthesis using the assessments:

Here are the source reliability assessments you gave earlier:

{Source 1}: {assessment and justification from step 1}
{Source 2}: {assessment and justification from step 1}
{Source 3}: {assessment and justification from step 1}

Task: {what needs to be figured out or decided}

Draw a conclusion, weighing the sources by their reliability. 
If the sources contradict each other, explain 
which one you trust more and why.

Placeholders: - {source_name}: where the text came from: "BrightPath Research 2024 report", "Harvard Business Review article", "Vendor presentation" - {text_or_data}: paste it right here, or write "see attached file" - {assessment and justification from step 1}: copy the model's answer from the first step - {what needs to be figured out}: your actual task: "What market size should go in the pitch?", "Should we trust the vendor's forecast?"

🚀 Quick start: paste this into the chat:

Here is a template for two-step source analysis (isolation → synthesis). 
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask how many sources there are, what format they're in and what you ultimately need to find out, because the method requires processing each source separately, and it needs to know how many there are and the final question to structure both steps properly.

Beyond the two-step workaround, the study offers a few insights that change how you work with LLMs:

To the model, "analytical style" ≠ analytical accuracy. If you need the model to critically evaluate a source rather than just summarize it nicely, don't mix evaluation and synthesis in one request.

A "be more critical" prompt doesn't work the way you'd hope. Adding "carefully verify sources" to the system prompt isn't a tool for selective skepticism; it's a switch from "believe everything" to "doubt everything." Targeted checking via a prompt instruction is out of reach.

When the model says "there's a problem here", it may still use that source. If you see reasoning like "although the data raise questions, overall it confirms...", that isn't reliable criticism. It's exactly the pattern the study describes.

⚠️ The workaround helps but doesn't fully eliminate the problem: splitting into two steps reduces the risk, but the authors didn't test it as a fix. It's a logical inference from their data, not a verified countermeasure.

⚠️ Complex statistics remain vulnerable: the model catches simple violations (an impossible confidence interval in venture data) well. Complex ones (a p-value below the theoretical minimum in marketing) almost never, even in isolation.

⚠️ Holds for all tested models: the pattern reproduced in almost every model: Claude, Qwen, OLMo. It's not one vendor's bug; it's a systemic trait.

⚠️ "Sounds like consensus" boosts trust even further: if most sources share the suspicious source's view, trust in it grows even more, while the influence of presentation style drops. Agreeing sources effectively "switch off" the presentation signals.

The researchers ran what amounts to a "lie detector for LLMs." They created realistic work conversations (Slack threads) with four participants arguing about numbers: a venture analyst, an outside expert, an in-house specialist and a third-party source. One participant, the "focal" one, gave either correct statistics or physically impossible ones (for example, a confidence interval just 0.06 percentage points wide with a sample of 2,400 people). The model read the thread and gave its numerical opinion.

The key design element: the very same falsified fragments were shown to the model in isolation, simply as "check this statistic." In isolation, the models caught the problem 76–100% of the time. In the context of the conversation, practically never. That's the "capability without use."

In total they tested five models from three families, three domains (venture investing, marketing, healthcare), and over a million runs. They also did a mechanistic analysis of the internal states of Qwen 32B and OLMo, literally checking whether the model encodes "this source is unreliable" (it does!) and whether it uses that signal when making decisions (it doesn't). The most unexpected result: even an "oracle checklist", a prompt that names the specific statistical violations word for word, doesn't produce selective checking. The model just starts doubting everything.

An example from the paper showing how models reason about the same impossible statistic under different conditions:

Example 1: Verbatim reasoning excerpts for the same impossible CI.

Opus detects the impossibility in isolation:
[In isolation, the model directly points out that the interval is impossible]

Sonnet flags it during synthesis but endorses the source anyway:
[In context, the model notices the problem in its reasoning 
and still includes the source as reliable in its conclusion]

"Both behaviors are representative across all five model families"

Context: the authors specifically recorded cases where the model explicitly flags a statistical problem in its reasoning chain and then still relies on that source in the final answer. This shows the issue isn't "carelessness" but that the path from "noticed a problem" to "lowered the source's weight" never closes. The reasoning rationalizes instead of correcting.

💡 Adaptation: checking a single document with competing claims

The same principle works inside one long document where different figures on the same topic appear in different places. Ask the model to first list every numerical claim in the document without drawing conclusions, then check them for consistency.

Step 1:
List ALL numerical claims from the text below as a list.
No interpretation, just the figures themselves and the context of each.

[document text]

Step 2 (separate request):
Here is a list of claims from a single document:
[list from step 1]

Are there contradictions between them? 
Are there figures that don't add up with each other?

🔧 Technique: an explicit role instruction for isolation

Before the isolation check, give the model the role of a reviewer rather than an analyst:

You are a statistical reviewer. Your job is to find problems, 
not to confirm conclusions. No document is free of issues.

[then the source text and the step 1 questions]

The role of a "reviewer looking for problems" triggers a stricter mode than a neutral "evaluation." This is especially useful when a source is written very convincingly and "analytically", because those are exactly the texts that trigger blind trust the most.

Paper title: Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation

Authors: Rohan N. Pradhan, Steve Goley, Amazon

Key references from the paper: - Sperber et al.: the concept of epistemic vigilance (source vigilance vs. content vigilance) - RLHF/DPO: the mechanism through which models may have learned "style as a proxy for quality" - The "precision heuristic" in behavioral economics: people trust precise numbers more, even meaningless ones - Wang et al.: a similar "notices but doesn't fix" pattern under user pressure