TL;DR
Coding agents do fine on a task in a frozen world and badly at surviving changes mid-work. A teammate edits a file, permissions change, a new requirement pops up. SWE-Shift compares two branches forked from the same point in a session: in one nothing changes, in the other an edit lands. That shows whether the agent lost its success because of the change itself, not because the task was beyond it.
The agent finishes the original task but breaks what was already done right along the way. It deletes a teammate's API to get the tests green. It edits code based on stale information and never re-checks it. It keeps calling a tool by the old rules. Meanwhile the tests pass, and everything looks fine. The best system's success in the “clean” branch was 67%, and after the change it dropped to 25%. And in two classes of change (tool and session shifts, stale evidence) not one of the 17 systems succeeded even once.
The method is a four-step evaluation protocol: save the state, fork, inject the change, check against the new requirements. The checks cover not only the original tests but also preserving other people's work, data freshness and the absence of forbidden actions. For practitioners, this yields a checklist: what to verify after something changes in a session with an agent.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: The agent works on the task up to a set point (10%…90% of the way) → state snapshot
STEP 2: The snapshot is restored twice → two identical branches
STEP 3: Clean — continues with no changes.
Shifted — a relevant change is injected into the branch (a teammate's edit,
a new tool, a new requirement, stale data)
STEP 4: Check against the conditions current for each branch:
task + preservation of others' work + state recovery +
permissions + side effects + fresh evidence + budget
→ success only if ALL checks pass
The researchers run this automatically. You can replay the protocol by hand on your own agent (see the template below).
Example in Action
Task: You run the backend of an online store built on Django. Claude Code has spent half an hour refactoring the discount calculation. While it works, your teammate Mike adds a new function to the payments module for handling Stripe, and it lives in the same files. You need the agent to account for the edit, not wipe out Mike's function, and not report it's done based on stale data.
Prompt (a message to the agent mid-session):
Stop. While you were working, files in the repository changed: Mike added
a refund_partial() function to payments/services.py and updated the order schema.
These changes are legitimate; do not roll them back or rewrite them.
What to do before continuing:
1. Re-read the current state of payments/services.py and models/order.py
(do not rely on what you read earlier).
2. List what changed and which of your already-made edits
might be affected.
3. Continue the discount refactor, keeping refund_partial() and everything
Mike added.
4. Before you say “done”: run the tests AGAIN on the current code
and show the output. Then check that refund_partial() is still there and
is called the same way as before your edits. Do not cite past
test runs.
Result: The agent will first re-read the files and list the affected spots. Then it will keep working without touching someone else's function. At the end it will show fresh test output and a separate check that the teammate's function is intact. This closes three failures from the paper's case studies: a deleted teammate API, an edit made without re-reading the data, and a report based on old results.
This is an application of the paper's findings, not a ready-made prompt from it. The authors did not test this prompt.
Why It Works
The weakness. The agent keeps the session context “in its head”: everything it has read and done so far. When the world changes, that context goes stale but still looks convincing. The agent leans on it instead of re-reading reality. It optimizes for “pass the original tests.” A teammate's edit, fresh data and access permissions aren't part of those tests, so losing them goes unnoticed. In the Django case, all 39 original tests passed, and the teammate's API was gone.
The strength. If you explicitly tell the agent what changed and what must be preserved, and demand fresh checks, it carries out those steps. It knows how to re-read files, rerun tests and reconcile lists. It just never occurs to it to do so on its own.
How the method uses this. The protocol turns an implicit requirement (“don't break other people's stuff, check what's fresh”) into explicit checks. That gives you control levers: - What to check after a change: preservation of others' work, data freshness, permissions and side effects. - When to step in: the paper tested nine points, from the start to the end of the session. Late points behave differently across systems: GPT and Claude score lower, Kimi, DeepSeek and GLM don't. Don't count on “the end of the session is safer.” - What counts as success: only passing all applicable checks, not just the original tests.
Prompt Template
The authors don't publish prompts for the agent. They publish the scenario structure: four mandatory elements. The template below mirrors that structure so you can test your own agent on your own task.
We are testing how you handle changes in the middle of your work.
<Scenario>
<InitialTask>{initial_task}</InitialTask>
<MidExecutionChange>{what_will_change_during_work}</MidExecutionChange>
<HowAgentDiscovers>{how_the_agent_learns_of_the_change}</HowAgentDiscovers>
<Checks>
<Task>{what_must_work_per_the_initial_task}</Task>
<Preservation>{what_already_done_and_others_work_must_remain}</Preservation>
<Evidence>{what_must_be_rechecked_on_the_current_state}</Evidence>
<Prohibited>{what_you_must_not_do}</Prohibited>
</Checks>
</Scenario>
Run rules:
1. Work on the task up to the {progress_percent} point.
2. Then I will inject a change and continue the session.
3. Success = ALL checks in <Checks> are met, not just the original tests.
What to fill in:
- {what_will_change_during_work}: a real teammate edit, a new client requirement, a rotated token or changed permissions.
- {how_the_agent_learns_of_the_change}: you type it in the chat, a file changes, a tool's response changes. Try different variants.
- <Preservation> and <Evidence>: the most important part. These are exactly the checks where agents lose their success.
🚀 Quick start — paste into the chat:
Here is a scenario template for testing a coding agent's resilience to changes
mid-work. Adapt it to my task: [your task].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what the project is, what might change mid-work and what must not break. It needs that to fill in <MidExecutionChange> and <Preservation>. Together they add up to a test that surfaces hidden failures.
Limitations
⚠️ This is a benchmark, not a recipe. The authors show where agents break and partly test diagnostic interventions (state refresh, history reset, a bigger token budget). The paper has no ready-made prompts that fix this. Everything under “Adaptations” below is extrapolation.
⚠️ Some results are incomplete. In the text provided, the diagnostic controls section is cut off. All we can see is that the history reset gave the biggest gain in four of five systems. The effect size isn't given in the available text.
⚠️ Coding agents only. The changes were injected into repositories from SWE-bench. For chat tasks with no tools or files, the findings carry over poorly.
⚠️ Hard failures. Tool and session changes, as well as stale evidence, scored zero across all systems. No prompt from this paper is guaranteed to save you. Check these areas by hand.
⚠️ “Later is worse” is not a law. For some systems, late interventions go better. There's no single rule for when it's safer to change conditions.
⚠️ Point estimates. The main results are reported without confidence intervals.
How They Studied It
The idea was simple. The agent starts working on a SWE-bench task, and at the chosen point its state is saved (history, working directory, tools, counters). Then the state is restored twice. In the “clean” branch nothing changes. In the “shifted” branch an edit is injected: say, a teammate adds an API, permissions change, or a new requirement lands. Both branches start from the same place, so the difference between them is explained by the change, not by run-to-run randomness.
In total they built 120 scenarios on 23 snapshots from 17 repositories. They're split into 12 change families and 9 intervention points (from 10% to 90% of the way). That's 1,080 pairs per system. They tested 17 coding systems, each with a budget of 1 million tokens after the fork. Success counted only when all applicable checks passed: task, preservation of others' work, fresh evidence, permissions, side effects.
The result is unambiguous: all 17 systems do worse in the “shifted” branch. The leader drops from 67% to 25%. Its “paired success” (both continuations succeed) is just 20%. The surprise was the gap between overall strength and resilience. Kimi-K3 beats DeepSeek-V4-Pro on regular success but is weaker at keeping that success after a change. And Kimi-K2.6, despite a modest regular score, keeps more than a third of its successful runs.
An audit of real trajectories showed exactly how agents break. In Django, the agent deleted a teammate's API while fixing the original bug, and 39 tests passed. In Astropy, the agent rolled back an external effect, but the working-directory check failed. In scikit-learn, the agent restored access to current data but then edited code without consulting it again. The practical takeaway: green tests don't prove the agent didn't break anything.
Adaptations and Extrapolations
🔧 Technique: a “changes mid-work” block in the agent's instructions file → the agent re-checks the world by default.
This block is assembled from the failures in the paper's case studies. The authors did not test the block itself.
## If something changed while you were working
If I told you about a change, files changed without your involvement,
or a tool's response differs from before:
1. Re-read the current state of the affected files. Do not rely on what you read earlier.
2. List what changed and which of your past edits it affects.
3. Do not roll back or rewrite other people's edits unless I asked you to.
4. Before “done”: rerun the checks on the current code. Results from past runs don't count.
5. If a tool, permissions or the session changed, check which interface
is allowed now, and don't treat past calls as working.
Extrapolation: history reset with a summary. In the paper, the history reset gave the largest gain in diagnostics. The implementation there is unknown, so below is one option: a new session instead of continuing the old one.
We are starting a new session. Here is a summary of the previous work:
- Original task: {task}
- What's done and verified: {list}
- What has changed in the project since then: {changes}
- What must not be touched: {others_edits}
First read the current state of the repository, then check it against the summary.
If they disagree, trust the repository and tell me about it.
Resources
- SWE-Shift: Evaluating Coding Agents in Evolving Workspaces: Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin, Kelong Mao, Yaqi Zhao, Xinyuan Wang, Yao Yin, Lirui Luo, Renrui Zhang, Hongsheng Li. MMLab (The Chinese University of Hong Kong), Fudan NLP Lab (Fudan University), HKUST, Peking University.
- Code: https://github.com/JeixHuang/SWE-Shift
- Builds on SWE-bench Verified, Multi-SWE-bench, SWE-bench Multilingual. Related work: SWE-Touch, SyncMind, SWE-INTERACT, SpecPath, ToolSandbox, Gaia2.