TL;DR
This is a falsification protocol. The model builds a table: hypotheses (candidates) in the rows, evidence in the columns. In every cell it writes “supports,” “contradicts,” or “irrelevant” and names the piece of evidence. A hypothesis with more contradictions than supports gets thrown out. The model may only draw a conclusion once at least half of the hypotheses are eliminated. The answer comes out as a single line in a fixed format.
The headline finding: on “pick one of a fixed set of options” tasks, model strength sets the ceiling, but output discipline sets the floor. Two years of new model generations barely moved the needle (from −0.03 to +0.08 on a 0–1 scale). Meanwhile, “discipline” failures persist even in top models: reasoning that never finishes, broken answer formats, results that swing wildly between runs on the same question. One model swung by as much as 0.39 of the whole scale, landing near zero one run and near the max the next. The root cause: out of habit, the model jumps to an answer based on surface similarity instead of checking hypotheses one by one.
The protocol puts the model on rails: hypotheses are listed, each one is checked, the answer fits on one line. It rescues “undisciplined” models: DeepSeek-V4-Flash gained +0.27, entirely because its reasoning finally finished. Kimi K3’s run-to-run spread dropped from 0.218 to 0.007. But the protocol hurts models that are already disciplined: Qwen3 lost 0.09–0.12. So switch it on selectively.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Hypotheses (candidates) → closed list, nothing outside it allowed
STEP 2: “Candidate × evidence” matrix → each cell: supports / contradicts / irrelevant
+ for “contradicts,” the evidence must be named (“contradicts E3”)
STEP 3: Row verdict → a hypothesis is compatible if supports outnumber contradictions
and the key evidence is confirmed; otherwise it is incompatible
STEP 4: Sufficiency check → are ≥ ½ of hypotheses eliminated? No → re-examine the evidence
STEP 5: Pick → the compatible hypothesis with the most supports
→ one line: [Verdict]: X
BANNED: picking by position in the list; copying a ready-made answer from an example
You can do all of this in a single prompt, in one pass.
In the full version of the paper, the protocol sits inside five stages: hypotheses → verification → causal chain → action → backward check. The template below covers all five.
Example
Task: An Amazon seller notices that orders for their “Oversized Hoodie” listing dropped 35% in a week. They don’t want to guess. They want a verifiable breakdown of the causes they can show their business partner.
Prompt:
<Role>
You are a marketplace analyst. You don’t guess; you test hypotheses against evidence.
</Role>
<Task>
Orders for the “Oversized Hoodie” listing (Amazon) dropped 35% over 7 days. Find the cause.
</Task>
<Candidates>
H1: A competitor cut their price
H2: The listing slipped in search because its rating dropped
H3: Key sizes (M, L) sold out
H4: Fees or shipping costs went up, so we raised our price
H5: Seasonal dip in demand
Do not add any other hypotheses.
</Candidates>
<Evidence>
E1: Listing rating went 4.8 → 4.7 this week
E2: Inventory: M — 0, L — 0, S and XL — 40 units each
E3: Our price went up from $24.99 to $27.99 (4 days ago)
E4: Top 3 competitors: prices unchanged
E5: Searches for “oversized hoodie” in the category: −5% vs. last week
E6: Rank for the main keyword: 12 → 31
KEY: E2
</Evidence>
<ReasoningLogic>
1. Build a matrix: rows H1–H5, columns E1–E6. Each cell gets exactly one of: support / oppose / irrelevant.
2. For every oppose, you must name the evidence: “oppose — contradicts E4.”
3. A hypothesis is compatible if support outnumbers oppose and the key evidence does not contradict it.
4. If fewer than half of the hypotheses are incompatible, write “Insufficient: re-examine evidence” and recheck the most contested cells.
5. Pick the compatible hypothesis with the most support. Ignore list order and examples.
</ReasoningLogic>
<OutputFormat>
First the matrix, then the row verdicts, then one line:
[Main cause]: Hx
</OutputFormat>
Result: The model returns a 5×6 table with a mark in every cell and an evidence reference for every “contradicts.” Next comes a verdict for each hypothesis (compatible or not), the “at least half eliminated” check, and the final line in the required format. You can eyeball every conclusion cell by cell.
Why It Works
The weakness. Models tend to answer by association: they see “orders dropped” and immediately blurt out “competitors.” Hypotheses never get enumerated, and the answer can’t be checked. Asking the model to “critique itself” without outside structure barely helps: self-checking with nothing to anchor it doesn’t improve reasoning and sometimes makes it worse. Even an explicit “fix the mistake” instruction works less than half the time.
The strength. Models are good at filling cells for small, local questions: “Does evidence E3 support hypothesis H1 or contradict it?” They handle a bite-sized task like that far more reliably than “find the cause.”
How the protocol connects the two. It breaks a big task into cells, forces the model to name the contradicting evidence (“contradicts E3”), and blocks any conclusion until half the hypotheses are gone. The one-line format makes the answer checkable. The paper shows that the effect comes from structure, not knowledge: when all injected domain knowledge was stripped out of the protocol, the result didn’t change.
Control knobs: - The “at least half” threshold → tighten it (two-thirds) for high-stakes decisions, loosen it for quick ones. - Key evidence (KEY) → set your own. It’s the “without this, the hypothesis is dead” fact. - Closed hypothesis list → let the model add one hypothesis if you need a wider search. This weakens the protocol. - One-line format → per the ablations, this layer is “the most expensive” on strong models. Drop it if you read answers yourself and don’t need machine checking. - Bans (“don’t pick by order,” “don’t copy the example”) → always keep them; they guard against laziness.
Prompt Template
<Role>
You are an analyst. You don’t guess; you test hypotheses against evidence.
</Role>
<Task>
{task: what needs to be determined or chosen}
</Task>
<Candidates>
{H1: hypothesis}
{H2: hypothesis}
{H3: hypothesis}
Do not add any other hypotheses.
</Candidates>
<Evidence>
{E1: evidence}
{E2: evidence}
{E3: evidence}
KEY: {number of the key evidence}
</Evidence>
<ReasoningLogic>
1. Build a matrix: rows are hypotheses, columns are evidence. Each cell gets exactly one of: support / oppose / irrelevant.
2. Justify every oppose by citing a specific piece of evidence: “oppose — contradicts E_.”
3. A hypothesis is compatible if support outnumbers oppose and the key evidence doesn’t refute it. Otherwise it is incompatible.
4. Sufficiency check: at least half of the hypotheses must be incompatible. If fewer, write “Insufficient” and recheck the contested cells.
5. Pick the compatible hypothesis with the most support. List order and examples must not influence the choice.
6. If a recommendation is needed, state the action that follows directly from the chosen hypothesis. Then check it against bans and risks: {constraints}.
</ReasoningLogic>
<OutputFormat>
Matrix → row verdicts → sufficiency check → one line:
[{verdict label}]: {hypothesis}
</OutputFormat>
What to fill in:
- {task}: a single question (diagnosis, picking a cause, picking an option).
- Hypotheses: 3–7 of them; if you don’t know them, have the model generate a list first and sign off on it.
- Evidence: facts, numbers, observations, ideally numbered.
- {constraints}: what must not be recommended (budget, legal limits, risks).
🚀 Quick start — paste into the chat:
Here is a template for the “candidates × evidence” falsification protocol. Adapt it to my task: [your task].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask which hypotheses are on the table, what facts you have, and which piece of evidence is key. Without a closed list of hypotheses and numbered evidence there’s no matrix, and the matrix is what the whole method rests on. The model will take the pattern from the template and build a ready-to-use prompt for your task.
Limitations
⚠️ The protocol hurts disciplined models: on models that already reason carefully and finish their answers, quality drops on average (roughly 0.09–0.12 on a 0–1 scale). Test your model without the protocol first. Turn it on if answers keep getting cut off, breaking format, or swinging between runs.
⚠️ Only for tasks with enumerable hypotheses: the method needs a closed list of candidates and a set of evidence. It’s a poor fit for open-ended, creative, or “come up with something” tasks.
⚠️ The protocol doesn’t raise the ceiling: it removes discipline failures. If the model doesn’t know the domain or fills cells wrong, the matrix will record the mistake neatly but won’t fix it.
⚠️ The same model grades the cells: errors in individual cells can happen. The protocol makes them visible but doesn’t remove them. In the paper, the format is checked by code; in a chat, you check the cells by eye.
⚠️ One domain: the main test was diagnosis in traditional Chinese medicine, plus a separate check on math. Transferring it to business tasks makes sense, but the paper didn’t measure that directly.
⚠️ More tokens, longer answers: the matrix takes up space, which is overkill for simple questions.
How They Studied It
The researchers used the public TCMEval-SDT benchmark: 200 clinical cases where the task is to identify a syndrome from a set of signs. Each of five base models was run with and without the protocol, up to three repeats, with paired statistical tests. First they showed that two years of model progress delivered almost nothing in “bare” mode. What did show up were discipline failures: reasoning that never finished, broken formats, huge spread between repeats.
Then they tested the hypothesis “the protocol helps where discipline is missing and hurts where it’s already there.” It held up. DeepSeek-V4-Flash gained +0.27, and the breakdown showed it came entirely from rescuing unfinished reasoning. Kimi K3’s spread dropped from 0.218 to 0.007. On Qwen3-4B and Qwen3-32B (with the same token budget and controlled temperature), the protocol cost 0.09–0.12.
Ablations (removing parts of the protocol one at a time) turned up two interesting results. First, on strong models most of the cost comes from the strict-format layer, while the causal-chain-plus-format combo pulls in different directions under different conditions. Second, removing all injected knowledge didn’t change the score: the effect comes from structure, not knowledge. Finally, the protocol was adapted to math problems with planted false premises. It caught them 89.8% of the time with 6.8% false alarms and brought accuracy back to the no-misinformation level. Caveat: that was on a single model with little headroom for improvement.
The practical takeaway: don’t bolt structure onto everything. First see where your model’s discipline breaks down, and add guardrails exactly there.
Adaptations and Extrapolations
🔧 Technique: apply it to premise checking → protection against false inputs
The paper shows a two-stage version of the protocol for tasks with planted misinformation: the “candidates” are the task’s premises, and the “evidence” is a set of checks (numbers, relationships, internal consistency). False premises get rewritten as reminders, and only then does solving begin.
Before solving the problem:
1. List every premise in the problem statement (P1, P2, ...).
2. Check each one on three points: are the numbers right? are the relationships right? does it contradict other premises? Cells: ok / fail / n/a; for fail, say what doesn’t add up.
3. Rewrite every premise marked fail as a reminder: “Reminder: actually ...”.
4. Solve the problem using only the verified and corrected premises.
🔧 Technique: add a “key evidence” block → faster elimination
If you set
KEY: E_in<Evidence>, any hypothesis the key evidence contradicts is automatically incompatible. Handy when there’s a fact that makes a hypothesis impossible (say, zero inventory in the warehouse for the “demand dropped” hypothesis).
These are my adaptations; they don’t appear in the paper in this form. The first builds on the two-stage scheme described in the paper, the second on the “key evidence” rule.
Resources
- Paper: Output Discipline Sets the Floor: A Falsification-Guided, Model-Agnostic Agent Protocol for Auditable Hypothesis-Driven Decision Making
- Authors: Jing Liu, Yongxing Qi, Ping Yao, Ting Wu (Hangzhou Innovation Institute, Beihang University; 32007 Unit, Beijing)
- Benchmark: TCMEval-SDT (syndrome identification in traditional Chinese medicine, 200 cases)
- Related work: research on how false premises propagate through a chain of thought (10–72% accuracy loss); self-correction without external feedback; MedAgents, MDAgents (multi-agent approaches for comparison)
- This breakdown is based on the available part of the paper (abstract, introduction, framework and protocol description); detailed result tables and appendices were not included in the material.