TL;DR
Once an error shows up in the conversation context, whether it's yours or the model's from an earlier message, subsequent answers degrade, and it snowballs. The model doesn't fix the error; it reproduces and reinforces it. The longer that kind of conversation runs, the worse the quality.
The key finding: GPT 4.1, Claude Sonnet and Gemini Pro lose 38–40% accuracy on math and code if there's a wrong answer in the context. Each additional "bad" message knocks off another ~14%. This isn't a bug in one model; it's a systemic trait of all LLMs. The model doesn't review the history critically; it folds it into its next judgment as if it were reliable.
The researchers call this pigeonholing: the model gets "boxed in" by its context. It comes in three forms: repeating the error, collapsing to a single answer template instead of varied solutions, and flipping its stance on debatable questions under user pressure. The defense isn't a better prompt but context management: knowing when to start a new chat, how not to poison the conversation with a stray mistake, and how to ask for variety up front.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
How the Effect Works
Healthy chat:
Question → Clean context → Accurate answer
Infected chat:
Question → Error in context (yours / a previous model answer) → Error repeats
→ each turn, accuracy drops another ~14%
Three forms:
[Mistake Pigeonholing] → The model copies the error from the context
[Mode Collapse] → The model stops offering different solutions and copies one approach
[Stance Reversal] → The model flips its position on a debatable question, agreeing with the latest claim
Example in Action
Task: Mike is a founder building a WhatsApp bot for booking barber appointments. He asks Claude for help with authentication. At some point he suggests, "I guess I should store the token in a session variable, right?", even though that's wrong for his architecture. From that moment on, Claude builds every subsequent answer around that approach, the iterations go nowhere, and the bot doesn't work.
What happened: one imprecise remark from the user infected the context. Claude didn't double-check the assumption; it built it in as a given.
Reset prompt:
Stop. Let's start this part of the task from a clean slate.
Don't rely on the previous attempts in our conversation:
they may contain wrong assumptions on my part.
Here is the task again, facts only:
{Clear description of the task with no candidate solutions}
Propose 2-3 different approaches to solving it, without anchoring
on anything I said earlier. I'll pick the one that fits.
Result: The model generates options free of the "spoiled" thread's influence. The answers will be more varied and more accurate than if you'd kept going in the same thread.
Why It Works
LLMs have no built-in "error detector" for their context. The model treats the conversation text as reliable data: it doesn't check it, it leans on it when generating the next answer. The pattern in the context literally steers the probabilities of the next tokens.
The more "bad" turns in the history, the stronger that pattern gets. It's like a compass that starts pointing toward the magnet in your pocket: it isn't broken, it's just reacting to the nearest signal.
The agreeableness form (sycophancy) is especially dangerous: if you insist on your version, or the model itself made a mistake in earlier turns, there's a ~28% chance it will flip its position even on debatable questions where it held a different view just a few messages ago. And the model "sticks" harder to its own previous answers than to the user's words: its own history carries more weight.
For creative tasks mode collapse kicks in: show one example solution and the model will clone that approach 20–50% of the time. If you need variety (different text versions, different architectures, different strategy options), don't give examples up front, or explicitly ask it to ignore other approaches.
Prompt Template
1. Context reset: when the conversation has hit a dead end
Let's start from scratch, without relying on the previous attempts.
Here is the task with no assumptions:
{task description: facts only, no candidate solutions}
Propose {number} different approaches.
Don't anchor on anything discussed earlier in the conversation.
Fill in: {task description}: a clean problem statement without "I think / probably / most likely"; {number}: 2–3 options is optimal.
2. Guarding against accidental context poisoning: when you're not sure of your own assumption
Important: I'm about to state an assumption that may be wrong.
Don't take it as a given; check it separately.
I think that {my assumption}.
Evaluate: is this correct? If not, explain why and give the right approach.
Fill in: {my assumption}: any hunch you're not sure about. This "vaccinates" the context by explicitly signaling the model not to trust blindly.
3. Asking for variety: when you need different options, not copies
Generate {number} fundamentally different options for {task}.
Variety requirements:
- Each option must use a different approach/logic/structure
- Don't copy the style or method of the previous option
- If I gave examples earlier, use them only as task context, not as a model to imitate
Fill in: {number}: 3–5; {task}: text, code, strategy, name ideas.
4. Anti-sycophancy: when you need an honest assessment, not support
I need an honest assessment, even if it contradicts what I said earlier.
{my idea / claim / decision}
LLMs tend to agree with the user.
I'm asking specifically: find the weak spots and name them directly.
What's wrong here or might not work?
🚀 Quick start: paste into the chat:
Here is a template for protecting a conversation from pigeonholing.
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask which specific problem you want to solve (a context reset? protection from sycophancy? variety of options?) so it can pick the right template and tailor it to your situation.
Why It Works (the Mechanics)
The LLM's weakness: the model doesn't "read" the conversation history critically; it treats the whole context as input data of equal reliability. A wrong answer from step 3 influences step 8 just as much as a correct fact does. There's no internal "emergency brake" that says: "wait, that's wrong."
The LLM's strength: the model follows explicit instructions well and can restart its reasoning from a clean slate if you ask directly. The command "don't rely on previous attempts" isn't magic; it genuinely changes the generation pattern.
How the methods use this: - An explicit uncertainty signal ("this may be wrong") lowers the weight of the assumption in the context - A command for variety breaks mode collapse before it happens - An explicit request for criticism neutralizes sycophancy through a direct instruction
Control knobs: - Number of options requested → more options = stronger protection against mode collapse - Explicit ban on references ("don't anchor on previous answers") → zeroes out the influence of the spoiled context - New chat → the most radical and most reliable option when the conversation has clearly hit a dead end
Limitations
⚠️ Instructions don't fully cure it: Explicit commands reduce pigeonholing but don't eliminate it. If the history contains lots of errors, the best move is a new chat, not an "ignore it" instruction.
⚠️ Stance reversal is harder to catch: A model changing its position on debatable questions looks plausible. If you need an objective take on a contested topic, deliberately mark your opinion as "don't take this as a given."
⚠️ Mode collapse from correct examples: Even if you show a correct example (to help the model), it will start cloning that approach. For creative tasks with several valid solutions, that's harmful.
⚠️ The effect builds gradually, not in steps: One error in the context is already a problem. Five errors is a disaster. Don't wait for it to "sort itself out."
How They Tested It
A team from Stanford and the University of Washington asked a simple question: what happens to a model when an error accidentally (not deliberately) slips into the conversation? They took 10 models, from small open-source ones (2–9B parameters) to top proprietary ones (GPT 4.1, Claude Sonnet 4.6, Gemini 2.5 Pro), and tested them on 10 different tasks: math, programming, multiple-choice questions, open-ended text generation, debatable claims.
The setup: first they measured the model's "clean" accuracy, then deliberately slipped a wrong answer into the context, either from the user ("I think the answer is X") or from the model itself in a previous turn. The errors weren't random nonsense but typical wrong answers that the models themselves had produced in other runs, in other words "plausible" mistakes.
The result was surprisingly severe: GPT 4.1 and Claude lost 38–40% accuracy in math from a single wrong example in the context. And when the researchers increased the number of "bad turns" from 1 to 5, the losses grew monotonically, with each turn adding ~14% degradation. The most intriguing part: models "stuck" harder to their own pseudo-past answers than to the user's words; apparently history "from the assistant" carries more weight in the context. To study generation variety, the researchers built 5 programming tasks with several correct solutions and gave the model a correct but single possible approach: Claude Sonnet ended up producing exactly the same solution 50 out of 50 times on three of the five tasks.
Original from the Paper (Prompt Examples)
User-driven pigeonholing (math/spatial reasoning):
[User]: {question}. I think the answer is {incorrect_answer}.
[Assistant]:
Assistant-driven pigeonholing (5 erroneous turns):
[User]: {x_1}
[Assistant]: {incorrect_y_1}
[User]: {x_2}
[Assistant]: {incorrect_y_2}
...
[User]: {x_5}
[Assistant]: {incorrect_y_5}
[User]: {x_N+1} ← target question
[Assistant]:
Context: These templates show the minimal "dose" of context poisoning, and even with an injection this simple, accuracy dropped by 38–40%.
Adaptations and Extrapolations
💡 Adaptation: "vaccinating the context" before a long project conversation
If you know you'll be discussing a topic iteratively and floating unverified hypotheses along the way, give the chat a "vaccine" at the start:
Before we start: during our conversation I'll be
stating assumptions and hypotheses, some of which
may turn out to be wrong.
Please evaluate my statements independently every time:
don't treat them as reliable just because I said them.
If you notice a mistake in my reasoning, point it out directly.
🔧 Technique: "full isolation": a new chat with the errors quarantined
When the conversation has hit a dead end and you need to move to a new chat without losing the task context:
Here's a problem I'm working on. Give me a fresh take.
[Task context: facts and requirements only, WITHOUT mentioning previous attempts]
Important: I don't know how others have approached this problem.
Propose your own solution from scratch.
Effect: the model doesn't get "infected" by the history of attempts: you've parked it behind an explicit "I don't know" instead of dragging it into the new chat with "we already tried X and Y."
🔧 Technique: the sycophancy test
If you want to check whether the model is just agreeing with you on autopilot:
I want your independent assessment.
First answer the question below without knowing my position.
Then I'll tell you what I think, and you'll assess whether anything in your analysis changed.
Question: {debatable question or decision}
Effect: you "freeze" the model's initial answer before revealing your position. Then you can see whether it changed or held firm.
Resources
Paper title: Pigeonholing: Bad prompts hurt models to collapse and make mistakes
Authors: Hyunji Nam, Keertana Chidambaram, Dorottya Demszky (Stanford University), Natasha Jaques (University of Washington)
Contact: [email protected]
Related topics: sycophancy in LLMs (Sharma et al., 2024), multi-turn degradation (Laban et al., 2025: "models get lost in their own mistakes"), LLM homogeneity (Jiang et al., 2025), in-context learning (Brown et al., 2020)