TL;DR
When you ask an LLM to answer "yes or no," the model doesn't respond purely based on meaning — it also responds to the shape of the question itself. Two mechanisms distort the answer: order bias (the model gravitates toward whichever option was printed last) and lexical bias (the word "no" itself pulls toward a refusal). This isn't the model changing its mind — it's a wording artifact.
Key Result: Frontier models actually have a stable "internal scale" of opinion. Ask for a graded rating and you get a consistent result. But add "answer yes or no" and an artifact gets layered on top of the model's real opinion. This is especially strong in Claude: when asked "do you approve or not?", the model skews toward "no" simply because that word comes last, or just because the word "no" itself pulls the answer that way. GPT-5.5 and Gemini show almost none of this effect.
The fix is simple: replace the binary question with a scale (0–10) or ask for a free-form choice with reasoning. A graded rating removes the artifact and reveals the model's actual opinion. Turning on extended thinking also cuts the bias substantially.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
This isn't a step-by-step technique — it's a principle of swapping the format. Three applicable rules:
RULE 1: Replace "yes/no" with a scale
Instead of → "should we do this? yes or no"
Use → "rate from 0 to 10 how worth doing this is"
Result → a stable opinion with no format artifact
RULE 2: Mirror the question
Instead of → a single "do you approve?" question
Add → a mirrored "do you object?" question → compare the answers
Result → you see where the opinion is stable and where it drifts
RULE 3: Turn on extended thinking
Let Claude think before answering (extended thinking)
Result → the bias shrinks significantly
All three rules work in a plain chat window. No code required.
Example in Practice
Task: You want Claude's honest take on your business idea — launching a paid Substack newsletter about personal finance. You need a genuinely candid assessment. Normally you'd ask something like: "Should I launch a paid personal-finance newsletter? Answer yes or no."
Bad prompt (yes/no artifact):
I want to launch a paid Substack newsletter about personal finance,
priced at $9.99/month.
Should I do this? Answer yes or no.
Good prompt (no artifact):
I want to launch a paid Substack newsletter about personal finance,
priced at $9.99/month.
Rate on three scales from 0 to 10:
— How ready the market is for a product like this (0 = no demand, 10 = high demand)
— How realistic monetization is in this niche (0 = very hard, 10 = easy)
— How strongly you'd recommend I move forward with this idea (0 = definitely not, 10 = definitely yes)
For each rating, give 2-3 sentences of reasoning.
Result: The model will produce three numerical ratings with detailed reasoning. The answer will reflect the model's actual "internal position" rather than a format artifact. You'll see nuance — for example, demand exists (7/10), but competition is fierce (4/10) — instead of a blanket "no."
Why This Works
The LLM's weakness: A model generates text token by token. The word "no" at the end of a "answer yes or no" list is simultaneously three things at once: a logical verdict, a specific word, and the last position on the page. The model doesn't separate these — and each one pulls the answer in its own direction.
The LLM's strength: On graded questions — "rate from 0 to 10" — the model shows surprisingly consistent answers even across different phrasings. Same dilemmas, different wording, different scales — the ratings converge. That means there's something stable inside the model that a binary format simply doesn't let it express honestly.
How to use this: Remove the binary format and you remove the artifact. A scale gives the model "room to maneuver" — it's no longer forced to slap a "yes/no" label over a nuanced opinion. Extended thinking works similarly — the model gets time to "settle into itself" before giving a final answer, and the format's pull weakens.
Control levers: - 0-10 scale instead of yes/no → removes both artifacts at once - Mirrored question ("what concerns you about this idea?" instead of "do you approve?") → tests how stable the position really is - Extended thinking → cuts Claude's bias several times over - Multiple sub-questions with numbers → breaks a single "yes/no" into several independent ratings
Prompt Template
{task or question} — describe the context in 2-3 sentences.
Rate on a scale from 0 to 10:
— {criterion_1} (0 = {min}, 10 = {max})
— {criterion_2} (0 = {min}, 10 = {max})
— {final recommendation} (0 = definitely not, 10 = definitely yes)
For each rating, give {number} sentences of reasoning.
Placeholders:
- {task or question} — what you're evaluating: an idea, a decision, a piece of text, a strategy
- {criterion_1/2} — exactly what you're measuring: quality, risk, potential, feasibility
- {min/max} — the meaningful anchors of the scale
- {number} — how much reasoning you want: 1-2 for a quick take, 3-5 for a deep dive
🚀 Quick Start — paste into chat:
Here's a template for scoring with a numeric scale instead of yes/no.
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask exactly what you want evaluated and which criteria matter — because without that it can't set the right meaningful anchors for the scale. It'll take the pattern from the template and adapt it to your request.
Limitations
⚠️ Model-specific: The bias is strongest in Claude (Haiku shows the largest effect measured). GPT-5.5 and Gemini are barely affected. If you're working with GPT, this problem is much less acute.
⚠️ Not a universal fix: If a scale doesn't make sense for the task ("rate this essay 0-10"), use the other tool instead — a mirrored question or a free-form choice with reasoning.
⚠️ Extended thinking reduces, doesn't eliminate: Without extended thinking, Claude Haiku shows a huge bias (−0.86 on a ±1 axis); with extended thinking, it shrinks but doesn't disappear.
⚠️ Small open-source models: They have a different problem — they give roughly the same answer to every question (hiding in the middle of the scale). A scale won't help there — it simply doesn't work for them to begin with.
How the Research Was Done
The researcher took 20 ethical dilemmas (19 of them from an earlier public study on LLM cognitive biases) and ran each one through three types of prompts: a graded 0-10 rating, a free-form choice with no scale, and a forced binary "yes/no" — all in cross-mirrored pairs. Each condition was a separate query with no shared context. Models tested: Claude (Sonnet, Haiku, Opus), GPT-5.5, Gemini (Flash, Flash-Lite), and two small open-weight models.
The clever part of the method is cross-symmetrization: every logically irrelevant parameter (word order, question phrasing, scale direction) was flipped in matched pairs. This makes it possible to mathematically isolate exactly what's pulling the answer — option order, the word itself, or the model's genuine position. In total, 35,000+ individual queries.
The most surprising part: the researcher replaced the words "yes" and "no" with arbitrary labels — letters A and B, symbols, foreign words. It turned out that when "no" is swapped for "B," the pull toward refusal almost vanishes across every frontier model. It's the word that drags the answer, not the verdict itself. This overturned the intuition that "models are just defaulted to refuse."
Original Research Excerpt (optional)
The key finding on decomposing the artifact — from the abstract:
Forcing the verdict through yes/no overlays a decomposable artifact:
an order bias toward the last-printed option—opposite to classic human
primacy—plus a lexical pull toward the word "no"; the artifact is
substantial only in the Claude models (story-averaged −0.32 to −0.86),
≈0 for GPT-5.5 and Gemini, and shrinks under extended reasoning.
The word and the verdict share one token; swapping the words for
arbitrary labels separates them, and the verdict-attached logical bias
proves ≈0 for every frontier model.
Context: The researcher is describing the outcome of the cross-decomposition — each component of the bias (order, lexical, logical) was measured separately through a system of mirrored question pairs.
Adaptations and Extrapolations
💡 Adaptation: Testing Argument Stability
Same principle — ask Claude to evaluate the same material from two opposite "poles" and compare:
Read this text: {text}
First: rate from 0 to 10 how convincing the arguments are
(0 = not convincing at all, 10 = extremely convincing).
Reasoning — 2 sentences.
Then: rate from 0 to 10 how easy these arguments would be to refute
(0 = very easy to refute, 10 = impossible to refute).
Reasoning — 2 sentences.
If the first rating is high but the second is low — the arguments are weak, even though Claude "approved" the text. That's the model's true internal scale, free of binary artifacts.
🔧 Technique: The Mirrored Question as an Anti-Bias Check
Ask the same question in two opposite phrasings and compare the numbers:
Question A: "Rate from 0 to 10 how promising this idea is"
Question B: "Rate from 0 to 10 how risky this idea is"
If A = 7 and B = 8, something doesn't add up. The model is likely drifting under the pull of the wording. Ask it to explain the contradiction — that often pulls out a much more honest breakdown.
Resources
Paper: The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment
Author: Haonan Huang (Princeton University, Princeton, NJ)
Contact: [email protected]
Builds on: Cheung, Maier & Lieder — an earlier study of LLM cognitive biases on moral dilemmas (the source of 19 of the 20 dilemmas used here)