TL;DR
The FeatureArc study tests what happens when a coding agent grows a single feature stage by stage: "build lesson booking," then "add payments," then "add rescheduling." Every new spec lands on code the agent wrote itself. Between stages, nobody tells it whether the previous stage passed. The paper has exactly one method: hidden checks after every stage plus a final check of the whole feature. The agent knows nothing about these checks.
The headline finding: an early mistake stays in the code and drags the following stages down with it. About 63% of agents pass the first stage, but the whole feature works for only 28%. After a failed stage, the next one passes just 23% of the time; after a successful one, 70%. Meanwhile, old code rarely breaks (3% of cases). The real problem is elsewhere: in 95% of cases the agent never notices that the previous stage was done wrong. Its self-checks look at the current stage and almost never look back.
The rule that follows: don't hand an agent a chain of extensions blindfolded. You need a checkpoint between stages, where a human, or the agent itself on an explicit command, re-verifies the earlier requirements. The authors didn't test this measure separately; it's a conclusion drawn from their data.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
Here's how the authors set up the test:
STEP 1: Agent gets the code and the stage 1 spec → writes the implementation
CHECK: hidden tests verify (a) the new behavior, (b) that old behavior isn't broken
→ the agent is NOT told the result, NO rollback is done
STEP 2: Agent gets the stage 2 spec → builds on top of ITS OWN code (same session)
CHECK: the same checks
...
STEP N: final stage
OUTCOME: the whole feature is checked end to end on the final code
The delegation levels the authors introduce:
L1: the user tells the agent HOW to fix the failed stage
L2: the user points out WHICH behavior doesn't work
L3: the user only says "passed / failed"
L4: the user says nothing about past stages and just gives the next spec ← this is what was tested
Example in Practice
The authors' method is a benchmark; the paper contains no ready-made prompts. Below is a practical application of the finding, built on top of the results, not a technique from the authors.
Task: You're using Claude Code to build a Discord bot for an online Spanish school. Stage 1 is lesson booking. Stage 2 is payments through Stripe. Stage 3 is reminders and rescheduling. In the past you'd fire off three specs in a row and discover at the end that booking no longer respects payment and rescheduling creates duplicates.
Prompt (before every new stage, starting with the second):
Before stage 2 ("Payments through Stripe"), stop and verify stage 1.
1. List every acceptance criterion for stage 1 from my spec.
2. For EACH criterion, run a check (a test or a manual run) and show the command output.
Don't write "should work." Only the actual result.
3. If even one criterion fails, fix it, rerun the check,
and show that it now passes. Do not move on to stage 2.
4. If everything passes, write "Stage 1: all criteria confirmed"
and start stage 2.
After stage 2, run the checks for stages 1 and 2 together. This is
to make sure payments didn't break lesson booking.
Result: The model first lists the criteria from the previous stage. Then it shows the check output for each criterion. If something doesn't add up, it fixes it before the new stage begins. Only then does it move on to payments. At the end of the stage, you get a combined run of old and new checks.
Why It Works
The LLM's weakness. The agent judges its work by what it checks itself. And it mostly checks the current task. Once a stage fails, it almost never goes back to it, because in its picture of the world everything is "green." The new stage gets built on a crooked foundation and the error snowballs: in most failed later stages (73%), the culprit is a defect inherited from earlier.
The strength. The agent is good at following a direct, specific command: "list the criteria, run the check, show the output." It can read its own code and run tests. What it lacks isn't skill, it's the habit of looking back without being asked.
How the checkpoint exploits this. You turn "the agent will figure out on its own to check past work" into a mandatory step in the instructions. Your control levers: - Check depth. "Only the previous stage" is cheaper; "every stage from the first" is safer for long chains. - Demand proof. "Show the command output" kills the phrase "looks like everything works." - Criteria from the spec. Have the agent copy them verbatim. The authors saw that green self-tests often check something other than what was required. - Who checks. It can be the agent itself, a second agent in a fresh session (fresh eyes), or you by hand.
Prompt Template
This template is based on the paper's findings, not taken from the authors:
We are growing the feature "{feature}" in stages. We are now moving to stage {number}: "{stage_name}".
<BEFORE_START>
1. List the acceptance criteria for all completed stages ({stage_list}).
2. For each criterion, run a check and show the actual output.
3. If a criterion fails: fix it, rerun, show the result.
Do not move to the new stage until everything is confirmed.
</BEFORE_START>
<NEW_STAGE>
{stage_spec}
Interfaces, names and error messages: {contract}
</NEW_STAGE>
<AFTER_STAGE>
Run the checks for the new stage AND all previous ones together.
Report as a table: stage — criterion — result.
If anything from earlier broke, fix it before handing off.
</AFTER_STAGE>
What to fill in: {feature} is what you're growing, {stage_list} is the stages already done, {stage_spec} is your description of the behavior and criteria, {contract} is function names, fields and error texts, so the checks don't have to guess.
🚀 Quick start: paste this into your chat with the agent:
Here is a template for a checkpoint between stages. Adapt it to my project: {describe the project and stages}.
Ask me questions to fill in the fields.
[paste the template above]
The model will ask how tests are run in your project, which stages are already done, and what their acceptance criteria are. It needs this so the check follows your requirements, not whatever the agent decided to check on its own.
Limitations
⚠️ This is a benchmark, not a proven technique: the authors measured how agents fail without feedback. They did not test intermediate checks or prompts like ours. The measure is inferred from the data, not proven by experiment.
⚠️ The agent checks the wrong thing: the authors separately analyzed cases where the agent's self-tests were green while the hidden tests failed. The agent may have picked the wrong expectations. So the command "run the checks" doesn't guarantee correctness. The criteria must be written out explicitly in your spec.
⚠️ Results depend on the model + harness combo: complete off-the-shelf agents were compared; the contributions of the model and the tool weren't separated.
⚠️ The test is harsh: no feedback, no rollback, one session. In real work you usually check something, so 28% is a lower bound, not a forecast.
⚠️ Code only: the finding is about multi-stage development. You can carry it over to writing or data analysis by analogy, but the paper doesn't cover that.
⚠️ The paper text was truncated: detailed per-model results and the root-cause analysis are missing from the provided excerpt. The conclusions rely on the abstract and introduction.
How They Studied It
The idea was simple: a real feature grows piece by piece, so the test should too. The Tencent team took the histories of real repositories (Python, TypeScript, Rust, Go) and pulled out chains of commits that successively extend the same feature. For each stage, LLM agents wrote a spec and an interface contract, and the existing tests were split into two kinds. The first check the new behavior; the second check that old behavior isn't broken. After filtering and manual review, 73 tasks, 263 stages, 53 repositories remained, with 2 to 10 stages per chain.
Ten coding agents worked in a single session. Hidden tests ran after every stage, but the agent never learned the results. There were no rollbacks, and the budget was four hours per stage.
The surprise was that new code barely breaks old code (3.3%). The main trouble isn't regressions, it's undetected defects: after a failure, the agent misses the error 95% of the time. Even the best agent, which passes the first stages 84% of the time, delivers a complete working feature in only 49%. The practical takeaway: success at one stage tells you nothing about the state of the whole feature.
The agents' trajectories were annotated by three LLM judges using a shared codebook. Decisions were made by majority vote, and disputed cases were resolved by the first author. He checked his own annotation blind on a sample, and agreement with the judges was 89–100%.
Adaptations and Extrapolations
🔧 Technique: independent reviewer → fresh eyes on old stages
Instead of self-checking, run a review in a new session:
You are signing off on another developer's work. Below are the specs for stages 1–2 and the code.
Don't trust their assurances. For each acceptance criterion, find confirmation in the code
or run a check. Return a list: criterion — confirmed / not confirmed — how.
Effect: the reviewer has no "it works anyway" habit. The paper did not test this hypothesis.
Extrapolation: a rule in CLAUDE.md / AGENTS.md
## Rules for staged work
- Before starting a new stage, list the acceptance criteria of all previous ones
and confirm each by running a check, with the output in your reply.
- Don't write "checked" if you didn't run the command.
- If a previous stage fails, fix it before starting a new one.
- A stage is done = new and old tests run together.
This way the check is baked into your standing instructions and doesn't depend on whether you remember to ask for it.
Resources
- FeatureArc: Benchmarking Coding Agents on Evolving Features — Yunlong Wang, Alyssa Wu, Cuiyun Gao, Jia Feng, Songyuanyi Lu, Zhichao Hu
- Hunyuan Team, Tencent; The Chinese University of Hong Kong
- Tools mentioned: Claude Code (code.claude.com/docs/en/overview), Codex CLI (github.com/openai/codex)