DBA (Disagreement-Based Abstention): catching unreliable answers when two prompts disagree

STEP 1: Direct question → Answer A
   "What year did Airbnb hold its IPO?"

STEP 2: Same question via decomposition → Answer B
   "First: what is Airbnb?
    Then: when did that company go public?"

STEP 3: Compare A and B
   If the answers match → trust it
   If the answers differ → don't trust it (the model is unsure)

All three steps are separate chat requests. You do the comparison yourself (or ask the model to compare).

Task: You're preparing an investor pitch for a marketplace. You need to mention when your main competitor went public, and the fact has to be accurate because the audience will check it.

Prompt 1 (direct):

In what year did Airbnb hold its IPO on a US stock exchange?

Prompt 2 (decomposition):

Answer step by step:
1. What kind of company is Airbnb? (country, industry)
2. Has this company gone public?
3. If so, in what year and on which exchange?

Answer each step separately, then give a final answer.

Result: If the model says "2020, NASDAQ" both times, the fact is solid and you can use it. If the first prompt says "2020" and the second says "2019", or hedges ("the exact date is unknown"), keep it out of the deck and check it with a search. Disagreement = red flag.

An LLM generates text probabilistically, but knowledge is stable, a hallucination is random. If the model really memorized a fact from its training data ("Airbnb IPO 2020"), it will produce that fact no matter how the question is phrased. That's robust knowledge, like a memory you'd tell the same way regardless of how someone asks.

If the model doesn't know, it starts building a plausible story out of patterns: "big home-sharing marketplace → probably had an IPO → probably in the 2020s → I'll say 2020." That's a shaky guess, and with a different prompt structure the patterns play out differently. Decomposition changes the order of generation, so if the fact isn't in memory, the improvisations in the two modes will drift apart.

Control knobs:

• Type of decomposition: split into 2 steps (fast) or 5 (more reliable, but slower). The more steps, the more hallucinations diverge.
• How you phrase the steps: split chronologically ("first the company's founding, then the IPO") or logically ("first the business model, then the financials"). Different splits = different reasoning paths = a stronger check.
• How you compare answers: strictly (exact match) or loosely (same substance, different words). Strict comparison = more abstentions, but higher accuracy on the answers you keep.

Direct request:

{question}

Decomposed request:

Answer the question step by step:

1. {sub_question_1}
2. {sub_question_2}
3. {sub_question_3}

After answering each step, give a final answer to the original question: {question}

Placeholders: - {question}: your original question - {sub_question_1, 2, 3...}: the question broken into stages (first identify the entity, then ask for the fact about it)

How to split: If the question is about an entity + one of its properties ("When did X do Y?"), split it: first "what is X?", then "what did X do?", then "when did that happen?".

🚀 Quick start: paste into your chat:

I have a question: {your question}

Help me break it into 3-4 sub-questions for a step-by-step check.
Goal: if the answer to the direct question and the step-by-step answer match, the fact is reliable.

The model will suggest a decomposition. Then run both versions (direct + step-by-step) and compare the final answers.

⚠️ Cost: You need at least 2 requests instead of one (double the tokens and time). With a 4-5 step decomposition you may need 3-4 separate requests.

⚠️ Consistent errors: If the model doesn't know the fact but has learned a sticky stereotype ("every big tech company lists on the NYSE"), it will give the same wrong answer in both modes. The method won't catch errors like that.

⚠️ Easy questions: For obvious facts ("the capital of France") both modes always agree, so the method tells you nothing new. DBA pays off on complex multi-step questions where the model has room to hallucinate.

⚠️ Requires decomposition: You need to be able to break the question into sub-questions. For some tasks (creative work, opinions) decomposition makes no sense: the method is only for factual questions with a verifiable answer.

The team tested the idea on six benchmarks (Bamboogle, MuSiQue, HotpotQA and others) and nine models from 7B up to frontier (GPT-5.1, Gemini-2.5). For every question they built a manually verified decomposition, a gold standard guaranteeing that the direct and step-by-step versions are semantically identical. That removed planning errors from the picture, leaving only knowledge and execution errors.

They compared three modes: Direct (the plain question), Assistive (all sub-questions listed at once, the model answers in a single request), and Incremental (sub-questions asked one by one, each as a separate request). They measured accuracy (match with the correct answer) and consistency (match between modes).

Main finding: For small models (up to 70B), decomposition boosts accuracy a lot: +20-27% on hard datasets. But for frontier models the gain vanishes or turns negative (−3% to −8%). Meanwhile the correlation between accuracy and consistency stays high (up to r=0.98) for every model. In other words, decomposition stopped being a crutch for accuracy and became a diagnostic tool.

They introduced a metric called the Reliability Multiplier (RM): how many times more often the correct answer appears when the modes agree versus when they disagree. Small models get RM = 1-2× (close to chance), mid-size models 2-10×, and Gemini Pro on Mintaka exceeds 50×: correct answers occur about 50 times as often among agreeing cases as among disagreeing ones. That's a ratio of counts, not accuracy or odds, and it holds for Gemini Pro only. Still a strong signal for filtering.

They pitted DBA against three baseline error-detection methods: AYS ("Are you sure?"), IC-IDK (an instruction to "say 'I don't know'"), and Self-consistency (voting over several answers sampled with temperature). DBA beat every baseline on F1-score in 16 of 18 model-dataset pairs. On Llama-70B and Qwen-8B, F1 reached 0.88-0.94, far above AYS (0.51-0.76). AUROC was always above 0.5, which shows the method genuinely separates right from wrong answers instead of just abstaining more often.

Why DBA wins: The baselines suffer from overconfidence: the model hallucinates, then answers "Yes" to "Are you sure?". DBA doesn't rely on the model's self-assessment; it tests internal stability with a behavioral experiment. Disagreement = the model can't reproduce the same answer twice = the knowledge isn't there.

🔧 Technique: three modes instead of two → even more reliable

The paper compares Direct with one kind of decomposition (Assistive or Incremental). You can combine all three:

1. Direct question → Answer A
2. All sub-questions at once in one prompt → Answer B
3. Sub-questions one by one, step by step → Answer C

If A = B = C → maximum confidence
If even one differs → don't trust it

This gives you a double check: Assistive and Incremental are different ways to decompose, so if the fact is solid, all three converge. If the model is improvising, at least one of the three will drift. The price is three requests instead of two, but for mission-critical facts it's worth it.

🔧 Technique: swap decomposition for rephrasing → faster to adapt

Splitting into sub-questions requires understanding the structure of the task. You can apply the DBA principle with rephrasing instead of decomposition:

Prompt 1: "What year did Airbnb go public?"

Prompt 2: "Let me rephrase: when did Airbnb, Inc. first
list its shares on a public stock exchange? Give the year and the name of the exchange."

Compare the answers: if they differ, the fact is unreliable.

Same mechanics: stable knowledge survives changes in wording, a hallucination doesn't. Rephrasing is easier than decomposition but gives a similar verification effect.

🔧 Technique: use DBA as a pre-check before RAG → save on retrieval

In a RAG (retrieval-augmented generation) pipeline you can first check whether you need search at all:

1. Ask the question closed-book, in two ways
2. If the answers match → the model knows, no search needed
3. If the answers differ → run RAG

Payoff: RAG burns tokens and time on retrieval plus document processing. If the model confidently knows the answer (DBA agreement), why search? You save on easy questions and stay accurate on hard ones.