FeatureLoop is a benchmark that doesn't ask "did the agent solve it on the first try" but how many follow-up instructions it needs to get the program into working shape. The task grows in stages: first a request, then new requirements, then "it's broken" complaints. At every stage, both the new stuff and everything built earlier gets tested.
The headline finding for practitioners: the agent is bad at fixing what broke, and it forgets old requirements. You write "the connector stays put when I drag the shape," and the agent fixes exactly that. The symptom goes away, but something right next to it breaks, and a part that used to work drives into a ditch. A single fix rarely brings everything back to working order. One more thing: the agent often "verifies" not what the user actually sees, but some convenient internal piece.
The researchers ran 12 strong models through three ways of handing over a task: the whole spec at once, requirements in chunks, and requirements in chunks + bug reports. The third way came out on top, while dumping the whole spec at once was the worst. And even then, after a failure complaint only a small share of cases get fully cured.
Method Blueprint
STEP 1: Take a real repository and a development goal → split it into stages (2 to 8)
STEP 2: The agent gets the first request → implements it in the same working directory
STEP 3: Tests run → if something doesn't work, the agent receives a complaint (completion feedback)
STEP 4: If everything works, the agent receives the next requirement (evolution feedback)
STEP 5: After each stage, check the new behavior + all previous requirements + the project's original behavior
Three scenarios for handing over the same task: Full (everything at once), Evo (in chunks), Evo+Comp (in chunks plus complaints). The agent wasn't trained or fine-tuned; this is a way to measure.
Example in Practice
First, the boundaries. The benchmark doesn't hand you a ready-made prompt and didn't test any tool. It shows how agents behave during long work on a single program. The takeaways apply to long, multi-step tasks; for a one-off 20-line script they're overkill.
Task: You're building out a booking chatbot for a hair salon in Python (Claude Code). You ask: "Add stylist selection." Then: "Now respect each stylist's schedule." Then: "Canceling a booking frees up the slot." The bot crashes on cancel. You write: "cancel doesn't work" — the agent fixes cancel, and stylist selection suddenly breaks.
Prompt (a bug report written with the paper's findings in mind):
Bug report. Scenario: a client books stylist Emma for 15:00 and then taps "Cancel booking".
Expected: the booking is deleted, Emma's 15:00 slot is free again.
Actual: the bot replies "Error", the slot stays taken.
Fix it, then verify ALL the accumulated functionality through the bot's real entry point (the message handler), not individual functions:
1. Stylist selection.
2. Stylist schedule.
3. Booking cancellation.
Run each scenario and show the output. If any of the earlier items broke because of the fix, fix it, re-run the check, and don't stop until all three pass.
Result: The agent fixes cancellation, then runs all three scenarios one by one and shows the output of each. If the fix broke earlier behavior, it should catch that in its own check instead of leaving it for the user to discover.
Why It Works
The model's weakness. The agent fixes whatever it was pointed at and considers the job done once the symptom disappears. It rarely checks the rest on its own. Often its check turns out to be "parallel": a helper function works, while the real entry point still behaves the old way.
The strength. When the agent is clearly given a scenario, the expected outcome, and the actual result, it finds and fixes the problem. It also handles requirements fed in chunks pretty well: new behavior gets implemented on the first try in most cases.
How to use it. The paper's findings show where to put the scaffolding: feed requirements in chunks, phrase complaints as a testable scenario, and define done as "all active requirements pass", not "the last symptom went away." The paper didn't directly test this combo as a technique, so treat it as a working hypothesis.
Control levers: - Chunk size → smaller for tangled projects, so the agent doesn't lose the thread. - Stop condition → "all previous scenarios pass" instead of "fixed the thing I named." - Check point → require runs through the public entry point (bot, endpoint, UI), not an internal function. - Number of attempts → after two failed fixes, stop and restart the diagnosis from scratch instead of filing yet another complaint.
Prompt Template
The paper has no ready-made prompt. Below is a template assembled from its findings (check everything accumulated, check through the public entry point), not the authors' method.
We're building out {project} step by step. Accumulated requirements that must always work:
{list_of_previous_requirements}
New task or complaint:
- Scenario: {what_i_do}
- Expected: {expected_result}
- Actual: {what_happens}
Working rules:
1. First reproduce the problem and show that it actually reproduces.
2. Fix the cause, not the symptom.
3. Don't verify internal functions; verify the real entry point: {entry_point}.
4. Run ALL accumulated requirements and show the output for each one.
5. If something broke because of the fix, fix it and repeat step 4. Finish only when everything passes.
6. At the end, list: what was fixed, what was checked, what couldn't be checked.
Fill in {project}, {list_of_previous_requirements} (one line per feature), and {entry_point} (bot, URL, command).
🚀 Quick start — paste this into the chat:
Here's a template for iterating on code with an agent. Adapt it to my project: [your project].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask which features must already work and how to run the real entry point — because "the full accumulated list" and "a public check point" are exactly what protect you from breaking old stuff and from verifying the wrong thing. It will keep the structure of the rules and tailor it to your project.
Limitations
⚠️ This is a measurement, not a cure: the paper didn't test any prompt or instruction. All conclusions about how to work are our hypotheses based on its data.
⚠️ One harness: all models ran through Claude Code, and the complaints came from a fixed generator, not a live human. Real complaints tend to be messier.
⚠️ Small task sample: 61 tasks from 34 repositories. For any single model, a difference of a couple of tasks may just be noise.
⚠️ No more than two corrections per stage: what happens with three or four, we don't know.
⚠️ Tests measure predefined behavior: if you ask the agent for something those tests don't cover, results may differ.
How They Studied It
Researchers from Tencent and CUHK assembled 61 tasks from 34 open-source repositories in five languages (Go, Rust, JavaScript, TypeScript, Python). Each task was drawn from real user requests and broken into 2–8 stages. For each stage, checks were written separately from the reference solution so the tests wouldn't simply mirror the ready-made patch. Then human reviewers ran the tasks with four models and fixed any flaws.
Next, 12 models were run in three scenarios, 2196 runs in total. Result: the full success rate was 40.3% when the whole spec was given at once, 51.6% when it was given in chunks, and 64.1% when given in chunks with complaints. The best model carried 72.1% of tasks across the finish line.
The most telling part: after the first failure, only 15.4% of broken states return to working order within two fixes. Even when the specific complaint gets resolved, in 20.9% of cases some other requirement stays broken. And 11.4% of what already worked breaks later on.
Digging into the failed runs revealed why: in 75.3% of cases the agent skipped verification along the user's path or checked a helper piece instead of the real entry point. On tasks no model solved, 32.4% of errors were inconsistent shared state (registry, storage, lifecycle handlers out of sync). On tasks at least one model solved, that figure was only 2.6%.
The surprise: the final score hides different working styles. Two models with the same result differ in that one more often fixes what's broken, while the other is better at preserving what works.
Adaptations and Extrapolations
🔧 Technique: add a completion rule to the agent's instructions file → the agent doesn't walk away after the first pass
"The work is done only when every item on the requirements list passes, not just the last one. If a fix breaks an earlier item, fix it."
🔧 Technique: a three-line bug report → fewer misunderstandings
Always phrase a complaint as "Scenario / Expected / Actual." The paper shows that a local fix doesn't guarantee a complete one, but a clear scenario gives the agent something concrete it can verify.
Extrapolation: before handing over the next chunk of requirements, ask the agent to list all previous requirements and confirm they still pass. It's cheap insurance against silent breakage.
Resources
FeatureLoop: Benchmarking Coding Agents under Evolving Requirements. Authors: Jia Feng, Alyssa Wu, Cuiyun Gao, Yunlong Wang, Songyuanyi Lu, Zhichao Hu (Hunyuan Team, Tencent; Chinese University of Hong Kong). Environment used: Claude Code. Related benchmarks: SWE-bench and its extensions.