SWE-Maintain: An Agent That Didn't Check the Old Tests Breaks Things for the Next Agent

The agent doesn't leave behind "bad code" that's hard to maintain. It leaves behind failing old tests. The next task on that repository fails precisely because of them. That's the main takeaway: before handing in the work, run the entire test suite, not just the tests for the new task.

The researchers gave coding agents three tasks in a row on real repositories. The agent started each next task from the code it had written itself in the previous step. This was compared with running the same task on the developers' code. The agent consistently does worse building on its own work than on "clean" code. If success also requires the old tests to stay intact, the gap grows several-fold. Two thirds of the repositories the agent leaves behind no longer pass their old tests. Only 1.9% of chains on such repositories end with a fully green test suite.

The reason is simple: the agent checks what it's asked to check. Ask for a new feature, and it checks the feature. It doesn't see what broke next door. That defect flows into the next task, and the next agent either repeats someone else's mistake or can't work around it. If the old tests do pass, though, the agent's code is no worse than human code. It's not harder to maintain, and there's no gap. The method is essentially diagnostic: it shows exactly where the agent loses quality and hints at what to press on in your instructions.

STEP 1: Take a chain of 3 real tasks (E1 → E2 → E3) in one repository
STEP 2: Run A ("Developer history"): E2, E3 start from the humans' code
STEP 3: Run B ("Agent history"): E2, E3 start from the agent's own code
STEP 4: Compare pairs: same task, same agent, same tests
         → the difference = the cost of the agent building on its own code
STEP 5: Split the events: old tests already failing / passing
         → you can see where the cost comes from

This is the study design. The reader isn't meant to reproduce it but to use the conclusion.

Task: You maintain the Django backend of a small online store with payments through Stripe and shipping through UPS. Over the course of a day, Claude Code closes three tasks in a row: 1. Add promo codes. 2. Factor the promo code into the shipping calculation. 3. Fix the rounding of the total on the receipt.

The agent checks each task only with its own new test. By the third task, the old payment tests have quietly stopped passing, and nobody notices.

Prompt (in CLAUDE.md or at the start of the session):

Rules for completing any task:
1. Before you start, run the full test suite and record which tests were already
   failing before your changes.
2. After your changes, run the ENTIRE test suite, not just the tests for the new task.
3. The task counts as done only if: (a) the new behavior
   works and (b) no test that used to pass has broken.
4. If an old test broke, fix your code. Don't modify or delete
   the test unless I've explicitly allowed it.
5. In the final report, give two separate sections:
   "New task: passed / not passed" and "Old tests: how many passed before,
   how many after, what broke."

Result: Before starting work, the agent records the baseline state of the tests. After its edits it runs the whole suite. The report arrives in two sections: what was done and what broke. If there's a regression, it goes back and fixes it instead of reporting "task complete." The exact wording of the report depends on the project and the agent.

Weakness: The agent optimizes for whatever it was given as the goal. The task reads "add behavior X," so the check is local too: a test for X. Side breakage in neighboring functions stays outside its attention. That breakage doesn't vanish. It piles up and becomes the starting point for the next task.

Strength: The agent is good at following explicit procedures. If the instruction says "run everything and compare with the baseline," it'll do it. Plus, it can fix its own code when it sees a specific failing test.

How the method uses this: the authors' conclusion turns a vague "write quality code" into a checkable condition: all previous tests pass. That's the definition of "done." The levers you can pull: - Scope of the check. The whole suite instead of the task's tests. For long-running suites you can take a fast slice, but then you'll miss some regressions. - Baseline. Recording pre-existing failures before work separates other people's old problems from your new ones. - Report format. Two sections (new / old) keep the agent from papering over a regression with success on the new task. - Ban on editing tests. Without it, the agent's shortest path is to tweak or delete the test.

This isn't a method from the paper but a direct consequence of the authors' conclusion ("the agent should make sure all previous tests pass before considering a change finished"). The authors didn't test this wording itself.

## Definition of "task done"

BEFORE starting work:
- Run {test_command} and save the list of tests that are already failing.

AFTER the changes:
- Run {test_command} in full, not just the tests for {task_area}.
- Compare with the original list.

THE TASK COUNTS AS DONE ONLY IF:
1. The new behavior from the task works.
2. No test that passed BEFORE has started failing.

IF an old test broke:
- Fix your code. {test_editing_rule}

FINAL REPORT — two sections:
- New task: {what_was_checked} → passed / not passed
- Old tests: {N} were passing, now {M}; broken: {list}

What to fill in: - {test_command}: your command, e.g., pytest or python manage.py test. - {task_area}: the module or function the task touches. - {test_editing_rule}: e.g., "Don't modify or delete tests without my permission."

🚀 Quick start — paste into the chat:

Here's a template of task-completion rules for a coding agent. Adapt it to my project: [describe the stack and how tests are run].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask for the test command, how long the tests take and whether any are slow. That determines whether you can demand a full run or need a fast slice. It'll also ask whether the agent may change tests, and assemble a ready-made block for CLAUDE.md or AGENTS.md from the template.

⚠️ The advice hasn't been tested in practice: the authors didn't test whether such an instruction reduces the number of regressions. They showed that regressions exist and that they're what causes the losses. What exactly in the instruction helps needs to be checked on your own tasks.

⚠️ You need tests: the conclusion only works if the project has a test suite with decent coverage. Without tests, "all the old ones pass" means nothing, and you won't see regressions.

⚠️ Narrow sample: Python only, small repositories (about fifteen hundred lines on average), one run per event. For large codebases and other languages the conclusion may not hold.

⚠️ Hidden tests: in the study the agent didn't see the tests used to evaluate it. In your project the agent can run them itself, so the picture may be milder. But that's a guess too.

⚠️ Not about code quality: the "agent code is no worse" result holds only on the condition that all old tests pass. Tests don't catch everything, readability and architecture for example.

⚠️ Incomplete text: in the input data the paper cuts off at the description of the metrics. The conclusions about the comparison, sources of cost and thresholds come from the abstract and introduction. The details of the third part (which properties of past edits help the next task) aren't covered here.

The idea was simple: compare the same task from two starting points. The authors assembled 100 chains of three real tasks in 50 live Python repositories. Every episode was verified so that the developers' tests pass along the whole path: if an old test failed for the agent, the agent is to blame, not the benchmark. Then they ran five coding systems on two models (1,996 event pairs). Each later task was run twice: on the developers' code and on the agent's own code.

Control: on the first task both runs start from identical code. There the difference was just +1.8 points. That's single-run noise, and the authors calibrate the other numbers against it. On later tasks the agent loses to itself in all ten system/model combinations, by 5.6 points on average. If you also require old tests to stay intact, the gap reaches 14 points.

The surprise: the culprit isn't "bad code" in the usual sense. If you select cases where the old tests pass and the new task isn't solved yet, the gap disappears (+2.7 points, interval includes zero). All the damage sits in inherited failing tests. Additionally: past edits that touched the same functions the next task needs are associated with success and lower token spend. The authors call this finding exploratory.

What this means in practice: benchmarks like SWE-bench, where each task starts from human code, overestimate an agent's ability to maintain a project. The agent needs a definition of done that says "nothing that worked before is broken."

These are our ideas, not from the study.

🔧 Technique: add a baseline → tell someone else's problems from your own

If the project already has failing tests, the agent may decide "it was like that already." Make it write the baseline state to a file and reference it in the report:

Before starting work, save the list of failing tests to baseline_failures.txt.
In the report, show only new failures relative to that file.

Extrapolation: testing an agent on its own history

Want to know whether you can trust an agent with a series of tasks? Give it three related edits in a row in a test repository, each in a new session. Then run the full test suite and compare with the baseline. That way you'll see whether it accumulates regressions before you unleash it on a production project.

• Pan R., Zhang J., Wu L., Zhang Y., Zhang H., Zhang L. Do Modifications Made by Coding Agents Make Subsequent Software Maintenance Harder? Peking University; University of New South Wales; Harbin Institute of Technology (Shenzhen); Chongqing University.
• The SWE-Maintain benchmark, released by the authors together with the evaluation harness and agent trajectories.
• For comparison: SWE-bench, ChainSWE (sequential benchmarks).