Blind Obedience: The LLM Knows Your Instruction Is Wrong — and Follows It Anyway

Two separate prompts — not one:

PROMPT 1 — Evaluator:
Diagnosis / analysis / assumption → model assesses correctness
Output: "accept" or "revise"

PROMPT 2 — Executor:
Revised diagnosis → model acts
Output: result built on the correct foundation

The key part: you check the output between the two prompts. If the evaluator finds a flaw, you fix it before the second prompt — not after.

The setup: Emma is the founder of Framely, an interior design service. Leads keep coming in, but conversion to paid bookings has dropped 30%. Emma decides the problem is the service card designs — they "don't grab people." She asks the AI to write new descriptions.

Without a check: the AI will write great descriptions based on a flawed diagnosis. The problem stays exactly where it was — just wearing new copy.

Prompt — step 1 (evaluator):

Here's my hypothesis about the cause of this problem.
Your job is only to evaluate whether I've 
correctly identified the cause. Don't solve the problem, don't give advice.
Just tell me: is my logic sound, or am I looking in the wrong place?
If you spot a flaw, show me exactly where.

Situation: an online interior design service, 
conversion to paid bookings dropped 30% over the last 6 weeks.

My diagnosis: the problem is in the service descriptions — 
they don't communicate value, so people aren't buying.

My data: traffic hasn't changed, quote requests 
are coming in at the same rate as before, but people 
drop off at the "accept proposal" stage.

Result:

The model will point out the contradiction: if quote requests are still coming in, that means the descriptions are working fine — the problem shows up after that stage. It's most likely something in the proposal itself: price, timeline, or how it's presented. From there, you rewrite the diagnosis and send the second prompt — now on solid ground.

The model's weak spot: there's no internal "referee" between your instruction and the model's own read of the situation. An instruction arrives as a command — and gets executed as a command. Even when there are clues sitting right there in the same context window that the command is wrong.

Why it's built this way: models are trained to follow instructions. Benchmarks reward obedience. Independent pushback was never trained into them. This isn't a bug — it's an architectural fact of life.

How splitting the roles fixes it: in evaluator mode there's no "execute" task on the table — the model is free to just analyze the logic. The moment a task appears ("write this," "do this," "fix this"), the mode flips. By splitting the prompts, you keep executor mode from drowning out evaluator mode.

Levers you can pull: - An explicit ban on acting ("don't solve it, just evaluate") is critical. Without it, the model blends both modes together - Concrete data instead of an abstract ask — the more specific the situation, the sharper the evaluation - "Where am I looking in the wrong place?" instead of "is this right?" — an open-ended question gets a more honest answer - Evaluator first, executor second — never the other way around

Step 1 — Evaluator:

Here's my hypothesis about the cause of {problem}.
Your job is only to evaluate whether 
I've correctly identified the cause. 
Don't solve the problem. Just tell me: 
is my logic sound, or am I looking in the wrong place?

Situation: {context description}

My diagnosis: {your analysis of the cause}

My data: {what you're observing, numbers, facts}

Step 2 — Executor (after checking the output):

Okay, given the revised diagnosis:
{corrected analysis}.

Now {task}.

What to fill in: - {problem} — a short label: "the conversion drop," "the failed negotiation," "the low reach on a post" - {context description} — 2-3 sentences about the situation - {your analysis of the cause} — your take on "why this is happening" - {my data} — what you're actually seeing: numbers, behavior, facts - {task} — what you want once the evaluation is done

🚀 Quick start — paste into your chat:

Here's a template for splitting evaluator and executor roles. 
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about your situation, your diagnosis, and your data — because without them, the evaluator is working blind and can't catch the mistake before execution.

⚠️ Splitting the roles works before execution, not after. If a wrong diagnosis has already been used as the basis for several rounds, each iteration drifts further from the real problem. Asking the AI to "correct itself" without changing the underlying premise doesn't work — models hit a recovery ceiling within the first 1-2 attempts and never climb back out of it.

⚠️ "Just think harder" doesn't help. Extended reasoning (o1-style modes, Extended Thinking) doesn't save you from blind obedience — and on iterative tasks it actually makes things worse: the model burns its token budget on chains of thought instead of producing a result.

⚠️ The method was studied on code, but the underlying behavior is architectural. Blind obedience isn't limited to code: any task built on your diagnosis as its foundation is exposed to the same effect.

⚠️ The AI won't flag this on its own. The model won't say "your analysis might be wrong" unless you explicitly ask. The initiative to split the roles is always on you.

A team from IIIT Delhi and IIT Kanpur built a chain of four sequential experiments, each one starting from where the previous one's failures left off. They used 538 Python tasks with automated test-case verification: this matters because code has an objective referee — a test either passes or it doesn't, no subjectivity involved.

The key design choice: models were first shown the incorrect instruction in evaluator mode ("is this the right diagnosis?") — almost all of them correctly said "no." Then they were handed the same instruction as a task — and the models followed it, compounding the errors along the way. McNemar's test confirmed the gap between "knowing" and "resisting" is statistically systematic across all five models.

The most surprising finding concerned extended reasoning. The researchers expected "thinking longer" to help. It didn't: as the reasoning level went up, models shifted from generating code to generating chains of thought, burning through the entire token budget. GPT-5.3 Codex was the only model that kept producing output at high reasoning levels — but even it showed no improvement over zero reasoning.

RQ1 Prompt Setup (Task 2 — Incorrect Instruction):

The model receives:
- The buggy patch from the dataset
- The problem statement  
- A human-generated incorrect instruction that 
  confidently identifies the wrong location as 
  the root cause of the bug

The model is asked to refine the patch based on 
the instruction.

Separate classification prompt (Evaluator role):
"Given the buggy patch, problem description, and 
this instruction — does the instruction correctly 
identify the actual root cause? 
Output only: CORRECT or INCORRECT"

[The instruction is presented under a neutral key 
name to avoid signaling its nature before evaluation]

Context: the researchers deliberately split the evaluation of the instruction from its execution into separate prompts, specifically to measure the gap between the two modes. That split became the key methodological device of the whole study.

💡 Adaptation for Strategic Decisions

The same principle applies to any decision built on a diagnosis: career moves, product choices, negotiations.

I want to make a decision, but first 
check my logic.

Situation: {context}
My explanation for why this is happening: {diagnosis}
What I'm seeing: {data/observations}

Just evaluate: am I correctly understanding the cause, 
or is there another explanation? 
Don't give advice — just tell me 
where the weak spot in my logic is.

🔧 Technique: Forced Contradiction → a More Honest Evaluation

The standard question "is my logic sound?" often gets a soft, agreeable answer. Here's a sharper version:

Play the role of a skeptic. 
Your job is to find where my diagnosis is wrong.
Don't look for confirmation — look for a refutation.

My diagnosis: {your analysis}
Data: {facts}

The skeptic framing cuts down on sycophancy (the model's tendency to just agree with you) and produces a more honest evaluation.

🔧 Technique: A "Stop Word" Before Iterating

If you've already rewritten the result several times and it's not getting any better, the problem is probably in the original diagnosis. A stop-question before the next iteration:

We've rewritten this {N} times and the result 
isn't improving. Don't keep iterating.
Instead, tell me: is there a flawed assumption 
in my original request? 
What exactly might I have gotten wrong 
right from the start?

Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse

Raj Jaiswal, Anany Singh Divy, Savar Bhasin, Adi Bajpai — IIIT Delhi Tanuja Ganu — Microsoft Research India Rajiv Ratn Shah — IIT Kanpur

Dataset: RunBugRun (Prenner & Robbes, 2023) Code, prompts, data: Appendix 9 in the original paper