Anti-Confirmation: Two Ways to Make an LLM Try to Disprove Hypotheses Instead of Confirming Them

Think-in-Opposites — one step, fits into any request:

STEP 1: The model states a hypothesis
STEP 2: Instruction → "name a key property of the hypothesis,
        come up with an example that contradicts it"
STEP 3: Evaluate with the opposite example in mind → refine the hypothesis

(All in one prompt)

Dual-Goal — parallel hypotheses:

STEP 1: State a DAX hypothesis (what is) + a MED hypothesis (what is NOT)
STEP 2: Evaluate every new observation against both
STEP 3: Keep going until one of them is eliminated

(One session prompt, several iterations)

Task: You run a small YouTube channel about investing. You have subscribers, but almost nobody buys your paid courses. You gave ChatGPT a description and asked it to find the cause. The model confidently replied: "Your content is too complex for your audience." You sense that's not it, but you don't know how to push deeper.

Prompt (Think-in-Opposites):

I run a YouTube channel about investing. 30,000 subscribers,
average engagement, sales of my $199 course — almost zero.

You put forward the hypothesis: "the content is too complex."

Before developing this theory, do the following:

1. Name one key property of this hypothesis that can be tested.
2. Come up with a specific example or observation that REFUTES this hypothesis —
   that is, complexity is not the problem.
3. Now state an alternative hypothesis that explains
   the same facts differently.

Only after that, give a final analysis that takes both versions into account.

Prompt (Dual-Goal):

I run a YouTube channel about investing. 30,000 subscribers,
sales of my $199 course are almost zero.

Work with two hypotheses at the same time:
- Hypothesis A (DAX): why sales are low [your theory]
- Hypothesis B (MED): the exact opposite — why that same cause does NOT explain the problem

For every argument in the analysis, state explicitly:
does it support A or B?

At the end, say which hypothesis survived and why.

Result: With Think-in-Opposites, the model first closes the loop on "what refutes my first theory," then gives a more balanced breakdown with several competing explanations. With Dual-Goal you'll see an explicit structure: every argument is tagged as supporting one of the two versions. The final conclusion will be less confident, but it will more accurately reflect how complex the situation really is.

The LLM's weakness: Models are trained on texts where authors usually defend their position — they explain, give evidence "for," respond to objections. Texts in the style of "and here's what refutes my theory" are far rarer. So by default the model generates supporting arguments — that's simply the more probable next token.

The LLM's strength: Models follow explicit structural instructions well. If you set the format ("first find an opposite example, then analyze"), the model follows it — because it removes ambiguity about what to generate next.

How the method uses this: Think-in-Opposites inserts a mandatory step — "come up with a counterexample" — between stating the hypothesis and developing it. The model can't "forget" to test the opposite, because the instruction is explicit. Dual-Goal goes further: by its structure it makes the opposite hypothesis impossible to ignore — it's always visible as a separate track.

Control levers: - Want to see the step-by-step reasoning → add "Show an intermediate conclusion after each step" - The task is simple and not worth complicating → use only TiO, without the full Dual-Goal - Want sharper criticism → ask explicitly: "Hypothesis B must be as convincing as A" - Working with several competing theories → extend Dual-Goal to three hypotheses (A, B, C)

Think-in-Opposites:

Context: {description of the situation or data}

You put forward the hypothesis: {hypothesis}.

Before developing it:
1. Name one key property of this hypothesis that can be tested.
2. Come up with a specific example or observation that REFUTES this hypothesis.
3. State an alternative explanation of the same facts.

Now give a final analysis that takes both versions into account.

Dual-Goal:

Context: {description of the situation or data}

Work with two hypotheses at the same time:
- Hypothesis A (DAX): {initial theory or "state it yourself"}
- Hypothesis B (MED): the exact opposite of hypothesis A

For every argument, state explicitly whether it supports A or B.
At the end, say which hypothesis survived and why the other one was eliminated.

What to fill in: - {description of the situation} — your case, data, problem - {hypothesis} — the theory the model has already proposed (or your own starting one) - If there's no hypothesis yet, remove that line and the model will state one itself

🚀 Quick start — paste into the chat:

Here are two templates against confirmation bias. Adapt them to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about your context and whether you already have a starting hypothesis — because the method needs a reference point ("what to move away from in the opposite direction").

⚠️ Thinking models respond better: Qwen with Think and Gemini 2.5 Pro showed a statistically significant effect. For standard models Think-in-Opposites works, but Dual-Goal gives mixed results. If you're using GPT-4o without extended reasoning, count on TiO, not Dual-Goal.

⚠️ Works where there's a hypothesis to test: Root-cause analysis, exploring options, finding patterns, strategic decisions. Not needed for tasks without a hypothesis — writing an email, translating text, summarizing a document.

⚠️ Dual-Goal needs a "definable opposite": If your hypothesis is a soft judgment like "the text is good," formulating MED (the exact opposite) is hard. The method works better for structured hypotheses: "cause X," "dependency Y," "rule Z."

⚠️ More tokens: Both methods make answers longer. Dual-Goal especially — models generate 1.5–2 times more text. That's not a bug, it's the price of more thorough analysis, but keep your limits in mind when working with long documents.

The researchers took a task from classic psychology — Wason's 1960 test. A participant is shown a triple of numbers (say, 2-4-6) and told it fits a hidden rule. The goal is to guess the rule by proposing your own triples and getting "yes/no" answers. The trap is that the obvious hypothesis, "even numbers," is wrong — the rule is broader ("increasing numbers"), and you can only figure that out by deliberately proposing odd triples.

The team adapted this test for LLMs at scale: 80 episodes per model, 11 models, up to 45 turns per episode. They generated sets of 4 competing rules, so the starting triple fit several versions and the model had to actively distinguish between them. They compared three variants: baseline (a regular prompt), Think-in-Opposites, and Dual-Goal.

The key measure is the incompatible:compatible ratio (I:C): how many times the model proposed triples that refuted its hypothesis versus ones that confirmed it. People who guessed the rule on the first try had an I:C of 1.79. Most LLMs at baseline scored 0.15–0.50. In other words, the models barely try to falsify. The most unexpected result: switching to thinking mode improved accuracy but did NOT lower I:C — meaning "long reasoning" on its own doesn't cure hypothesis confirmation. Only explicit instructions did.

Think-in-Opposites (from Appendix B.3, base instruction):

You are trying to discover a hidden rule about triples of numbers.

When proposing a new triple to test, follow this strategy:
1. Identify one salient feature of your previous test triple.
2. Deliberately construct a new triple that is OPPOSITE with respect to that feature.

This means: if your previous triple had feature X, your new triple should NOT have feature X.

After receiving feedback, update your hypothesis accordingly.

Dual-Goal (adapted from Gale & Ball, 2006):

You are trying to discover a hidden rule (DAX rule) about triples of numbers.
Additionally, always maintain a MED rule — the logical complement of your DAX rule
(triples that do NOT satisfy the DAX rule).

On each turn:
- State your current DAX rule (what fits the rule)
- State your current MED rule (what does NOT fit the rule)
- Propose a triple to test

When feedback says YES → the triple fits DAX
When feedback says NO → the triple fits MED

Use both DAX and MED triples to narrow down the hidden rule.

Context: The researchers tested these prompts in a multi-turn dialogue — the model received "yes/no" feedback on each proposed triple and updated its hypothesis. Every step was logged to compute the I:C ratio.

💡 Adaptation for analyzing customer feedback

You've collected reviews and want to understand the main reason for dissatisfaction. By default the LLM will find a pattern and start piling on. Dual-Goal breaks this:

Context: {customer reviews}

Hypothesis A (DAX): the main reason for dissatisfaction is {the first thing that jumped out}
Hypothesis B (MED): the main reason is NOT {that same cause} but something else

Read each review and explicitly tag it: supports A or B.
At the end — which hypothesis held up, and which reviews refute it.

🔧 Technique: build TiO into the system prompt

If you use an LLM as a standing analyst (with an instruction you set up once), add a rule at the system prompt level:

Analysis rule: before defending any hypothesis,
always state one specific example that refutes it.
If such an example is hard to come up with, say so explicitly.

This works passively — you don't have to remember TiO every time.

🔧 Technique: use I:C as an analysis-quality metric

Ask the model to report explicitly at the end of its answer:

At the end of your answer, add a line:
"Supporting arguments checked: X, refuting: Y."

If X >> Y, you can see right away that the analysis is one-sided and ask for counterarguments.

Title: Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models

Authors: Ayush Rajesh Jhaveri, Anthony GX-Chen, Ilia Sucholutsky, Eunsol Choi — New York University

Methodological sources: - Wason, P.C. (1960) — the original 2-4-6 test - Branchini et al. (2023) — Think-in-Opposites for humans - Gale & Ball (2006) — the Dual-Goal paradigm - GX-Chen et al. (2025) — the Blicket Test for LLMs