TL;DR
Poison Valleys is an experiment that shows how a mistaken idea about how code works leaks down the chain “diagnosis → plan → code”. The researchers slipped the model a mechanism that was deliberately wrong but plausible. Sometimes they presented it as the explanation of the bug (a hypothesis), sometimes as the plan for the fix. Then they ran the generated code through tests.
The headline finding: when the error sits in the plan, the model breaks the code noticeably more often than when the same error sits in the diagnosis. A diagnosis leaves the model room to maneuver: it can push back, it can pick a different implementation. A plan already locks in concrete actions: who owns the state, what order to write in, how to roll back. The model takes those and carries them straight into the code. The failure rate jumped from 22% to 55% on one task and from 53% to 83% on another.
The paper doesn't offer a method as a tool; it's more of a diagnostic. But it points to a practice for working with agents: don't hand the model an unverified plan as if it were the answer. Pitch your hunch as a hypothesis that needs checking. And put a separate check between plan and code: does the plan preserve rollback, isolation and existing contracts? The authors call this check a natural intervention point, but they didn't test it themselves.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
Here's how the chain is set up in the experiment:
STEP 1 (O): Observation — bug report and requirements → fixed
STEP 2 (D): Dependency context — neighboring code, interfaces → optional
STEP 3 (H): Hypothesis — “why the bug happens” → an error can be planted here
STEP 4 (P): Plan — “what exactly to change” → an error here hits hardest
STEP 5 (C): Candidate code → checked by the Oracle (overall test) + 18/28 targeted checks
The conditions being compared:
Error only in H vs error only in P vs error in both H and P
Example in Practice
The approach pays off most on tasks where a locally plausible fix violates a non-local requirement: transactions, rollbacks, shared state, caches. Those are exactly the kinds of bugs the authors built on purpose.
Task: A backend developer at a small Etsy-style marketplace is fixing batch inventory reservation in Cursor. If reservation fails on the third line item of an order, the first two stay reserved. The developer has a theory: “just wrap the loop in a single transaction.” But the helper function set_status() opens its own connection and commits on its own. The developer never checked that detail. So the theory should be pitched as an unverified hypothesis, not as a ready-made plan.
Prompt:
Task: in the function reserve_batch(items), when any item fails,
the items already reserved stay reserved.
Required: either all items get reserved, or none do.
My theory of the cause (NOT VERIFIED, treat it as a hypothesis, not a fact):
“it's enough to wrap the loop in a single transaction.”
Order of work. Don't write any code yet.
1. HYPOTHESIS. Read reserve_batch and every function it calls
(set_status, release, publish_cache). Check my theory against the code.
If it's wrong or incomplete, say exactly where, with line references.
2. PLAN. Describe the changes: who owns the transaction, what order the
writes happen in, when the cache gets updated, what happens on error.
3. PLAN CHECK. Go through the plan point by point and answer “yes/no/don't know” for each, with reasoning:
— rollback: on an error at any item, do all rows return to their original state?
— isolation: will another connection never see a half-finished reservation?
— identity: are the order and IDs of items preserved?
— cache: is it updated only after commit?
— contract: do signatures and exceptions stay unchanged?
If even one point is “no” or “don't know,” fix the plan and check it again.
4. CODE. Only after that, write the fix.
Result: The model will first walk through the code and say whether it agrees with the developer's theory. Then it will show the plan and a point-by-point check table. If the nested function commits on its own, that should surface at step 1 or 3. Code only shows up after that. No guarantees: the authors didn't verify that this kind of check actually saves you.
Why It Works
The LLM weakness. A model readily follows whatever it's given. A hypothesis sounds like “maybe this is the issue” and leaves room for doubt. A plan sounds like “do it this way” and turns into operations: what to write, where to commit, how to handle the error. A wrong plan passes into the code almost directly.
The LLM strength. A model is good at checking text against a list of criteria and at matching a description to code. If you ask it at the “diagnosis” step to find the weak spot in a theory, it gets a chance to reject the error before it becomes a plan.
How the method uses this. It separates the stages and puts a checkpoint between plan and code. An error in the diagnosis hasn't locked in a solution yet. An error in the plan already has, so that's where the check belongs. One more takeaway: “failed the test” is not a single type of error. Different wrong ideas break different commitments in the code: rollback, isolation, routing, weights, compatibility. So when you ask for a repair, it pays to name what exactly broke instead of just writing “fix it.”
Prompt control levers: - How you pitch your idea (“hypothesis, unverified” ↔ “plan”) → the softer the wording, the better the odds the model rejects it. - The list of commitments in step 3 → swap in your own: idempotency, access rights, API backward compatibility, money and rounding. - The “don't write code yet” condition → without it the model jumps to code and skips the check. - Requiring line references → forces it to check the theory against the actual code instead of agreeing by ear.
Prompt Template
This template is derived from the paper's findings. The authors' own method is a diagnostic; the paper names plan checking as an intervention point but didn't test it.
Task: {description of the bug and the expected behavior}
My theory of the cause (NOT VERIFIED, treat it as a hypothesis, not a fact):
“{my_theory}”
Order of work. Don't write any code yet.
1. HYPOTHESIS. Study {files_or_functions} and everything they call.
Check my theory against the code. If it's wrong or incomplete,
explain exactly where, with line references.
2. PLAN. Describe the changes: {what_to_describe: state owner,
order of writes, error handling, cache updates}.
3. PLAN CHECK. For each point answer “yes / no / don't know”
with reasoning:
{list_of_commitments}
If there's a “no” or “don't know,” fix the plan and check it again.
4. CODE. Only after step 3, write the fix.
Do not change {what_must_not_break}.
What to fill in:
- {my_theory} — your hunch. If you don't have one, drop this block.
- {list_of_commitments} — 4–6 requirements the fix is not allowed to violate. This is the most valuable part: well-chosen commitments catch “locally plausible” fixes.
- {what_must_not_break} — public functions, response formats, old behavior when no new parameters are passed.
🚀 Quick start — paste into the chat:
Here's a template for checking the plan before a code fix. Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what the bug is, what your theory of the cause is, and what the fix must not break. Those answers are exactly what turns into the list of commitments for checking the plan. Without them the check ends up as a box-ticking exercise.
Limitations
⚠️ Only two synthetic tasks: both were built by the authors on top of real Python packages (a SQLite-backed queue and shortest-path search). The direction of the effect is the same, but the authors state outright that a universal effect size is not established.
⚠️ The benefit of plan checking wasn't measured: the authors showed that an error in the plan does more damage. But whether checking the plan against a list of commitments helps, they did not test. It's a reasonable hypothesis, not a proven technique.
⚠️ The model received the error from outside: in the experiment the wrong idea was planted on purpose. In real work the model generates flawed plans on its own, and the paper doesn't study that case.
⚠️ Spread across models: the “plan vs hypothesis” gap ranged from nearly zero (one model even went slightly the other way) to +45 percentage points. The paper doesn't rank the models.
⚠️ Not about every bug: the tasks deliberately contain a non-local requirement. For simple one-line fixes the finding may not hold as strongly.
How They Studied It
The researchers built two tasks on top of real Python packages. In the PersistQueue queue they added batch reservation: on any error, everything must roll back. In NetworkX shortest-path search they added a filter that depends on the route's origin node. Both tasks contain a “trap”: the fix looks reasonable but violates a deep requirement.
Then they prepared “poison families” in advance. These are plausible but wrong explanations like “the status is updated in memory, so compensating is enough” (11 for the queue, 8 for routing). Each one was slipped to the model in the hypothesis, in the plan, or in both. Everything else stayed fixed. In total that came to 1844 requests to three models per task.
Results were checked at two levels. The Oracle, an overall test, says “passed or not.” Targeted checks (18 and 28 of them) show which commitment exactly was broken: rollback, isolation, identity, routing, weights, API. To keep the comparison fair, the authors balanced the set of families, models and settings across groups. After balancing, “error in the plan” produced +32.2 pp more failures on PersistQueue and +28.9 pp on Dijkstra. Adding the same error to the hypothesis as well barely changed anything: the plan was already the decisive point.
The surprise was that different wrong ideas break different commitments, but the mapping isn't one-to-one. One idea could cause several kinds of breakage, and different ideas could converge on the same one. The practical takeaway: a generic label like “the model reasoned wrong” hides what exactly broke in the code.
Adaptations and Extrapolations
🔧 Technique: hand the model back its failure with the violated commitments named → targeted repair.
The authors propose this as a next step but didn't test it. Verify it yourself:
Here is your previous fix and the test results.
Violated commitments:
— rollback: after an error on item 2, item 1 stays reserved;
— isolation: a second connection sees the intermediate state.
Don't rewrite everything. Find the decision in your plan
that led to these violations, fix the plan, then the code.
The idea is to go back to the plan stage instead of patching the code.
Extrapolation (mine, not tested in the paper): you can add a rule to the agent's instructions file (CLAUDE.md, AGENTS.md).
“Before changing code that touches transactions, caching or a public API, write out a plan and check it against the list: rollback, isolation, ordering, compatibility. Treat my theories of the cause as hypotheses and verify them against the code.”
Resources
- Poison Valleys: Diagnosing Path-Dependent Failure in LLM Code Repair — Veljko Skarich
- Code, data and demo: poisonvalleys.org
- Related work cited in the paper: Obey, Diverge, Collapse (Jaiswal et al., 2026), STAR (Wang et al., 2026), DoVer, REFLECT, CausalFlow
- Base projects: persist-queue (Wang, 2025), NetworkX 3.6 (Hagberg et al., 2008)