TL;DR
SpecForge is a method in which a coding agent writes a behavioral specification for every function and class. It's natural-language text: what the function takes, what it returns and what hidden rules it follows. The agent then writes code from these specs alone, runs the tests and edits the spec based on what failed. The loop repeats, and the model weights never change.
The core problem: fixing one function breaks others. You fix the description of calc_tariff, and suddenly round_weight starts failing. It's whack-a-mole; the authors call it output coupling. There are two causes. A caller depends on its callees, so its tests are misleading until the lower layer is described correctly. On top of that, the model reads all specs at once, so editing one changes how the others get generated. If you rewrite the whole spec bundle in one go or follow a single line of edits, progress hits a ceiling.
SpecForge gets around this with three tricks. Edit one entity at a time, and only one whose callees are all already solved (bottom-up). Use different subsets of failing tests for each edit, because some tests are noise. Keep several competing versions of the spec bundle, not one "best" one. A side finding: documentation written "for humans" is nearly useless as a spec for an agent.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
START: code → a spec for every entity (function/class/method)
LOOP (repeat B times):
STEP 1 (where to search): pick one of the saved versions of the spec bundle
→ the archive keeps versions with DIFFERENT strengths
STEP 2 (what to edit): take one unsolved entity
whose callees are ALL already solved → bottom-up order
STEP 3 (how to edit): produce W variants of the new spec,
each from a DIFFERENT subset of failing tests
STEP 4: for each variant, regenerate the module code from the specs
→ run ALL tests
STEP 5: variants that didn't make anything worse elsewhere go into the archive;
the best result hints at which entities to edit next
FINISH: return the bundle with the most solved entities
This is a multi-request process with tests. In the original a program drives it: entity selection is scored with a bandit (an algorithm for allocating attempts), and versions live in a Pareto archive. In a chat or an agent you reproduce the logic, not the automation.
Example Application
Task: An online store's Python code calculates shipping costs: round_weight → apply_zone_coeff → calc_tariff. The team decided to rewrite the service in Go to take load off the backend before Black Friday. The old code was written by three different developers, there are no comments, and it's understood only through its behavior. A straight line-by-line port is risky: the agent guesses at rounding and edge cases. So you first need specs that have been verified against tests.
Prompt (for Claude Code or Cursor in a repo with pytest tests):
You are working in behavioral-spec improvement mode.
Module: delivery/tariff.py
Entities and call order: calc_tariff → apply_zone_coeff → round_weight
Tests: tests/test_tariff.py (pytest)
Goal: for each function, produce a text specification (inputs, outputs,
edge cases, hidden rules: rounding, validation, order of operations)
from which a correct implementation can be recreated WITHOUT looking at the source.
Process:
1. Write draft specs for the three functions in specs/tariff.md.
2. Generate an implementation of each function seeing ONLY specs/tariff.md,
no source code. Put it in /tmp/regen/tariff.py.
3. Run tests/test_tariff.py against /tmp/regen/tariff.py.
4. Improve the specs strictly bottom-up: first round_weight, then
apply_zone_coeff, then calc_tariff. Move to the next function only
when all of its tests pass.
5. In one round, edit the spec of ONLY ONE function.
6. For that function, produce 3 edit variants. Build each variant from a
DIFFERENT subset of failing tests (e.g. one, two,
all). Don't mix tests that are clearly about different things in one variant.
7. For each variant, regenerate the code and run ALL tests.
Accept a variant only if no other function got worse.
If variants are good in different ways, keep the two best and
keep developing both.
8. Stop when all tests pass or after 8 rounds.
Show a table: round, which function, what was fixed, how many tests pass.
Result: The agent creates a spec file and works through the rounds bottom-up. In each round you see three edit variants for one function and the result of the full test run. Then comes a progress table marking which variants were accepted and which were rejected for regressions. At the end you get specs/tariff.md: you can ask the agent to write the Go version from it, and when reviewing you can see which rules have already been confirmed by tests.
Why It Works
The weakness. Documentation for humans is vague: "calculates the tariff taking the zone into account." A model writing code from such a description takes fuzzy phrases literally and resolves ambiguities unpredictably. That's why RepoAgent- and DocAgent-style descriptions solved only 14–35% of functions in the paper. The second weakness is coupling: the shared spec bundle affects how every function gets generated, so a local edit shifts other functions' results.
The strength. The model is good at turning a failed test into a clarification: "aha, round up to 0.5 kg, not to the nearest." It's also good at writing code from a precise description. Tests provide an objective signal that doesn't depend on the model itself.
How the method uses this. The "spec → code → test → spec edit" loop turns a failure into concrete knowledge about behavior. Bottom-up order removes false signals: until the lower function is described, the upper function's tests lie. Editing one entity makes it clear what broke or fixed the result. Different subsets of failures hedge against noise: some tests only confuse the model, and among several edit variants there's usually a cleaner one.
Control levers: - Number of edit variants (W) → more variants mean better odds of a clean edit, but more requests; 2 is enough for simple modules. - Number of rounds → cap it so the agent doesn't go in circles. - Acceptance rule ("nothing else got worse") → a strict rule protects against regressions; a loose one speeds things up but brings back whack-a-mole. - Traversal order → if the code has mutual recursion, describe that group of functions together as one entity. - How many tests to show → the full set of failures isn't always better. The authors show that extra noise makes edits worse.
Prompt Template
The paper's original prompts are in the appendix, which isn't in the available text. This is a manual version of the method: it keeps the bottom-up order, single-entity edits and different subsets of failures. The bandit search tree is replaced with a simple rule: keep the two best variants.
You are improving behavioral code specifications based on test results.
Module: {module_path}
Entities and who calls whom: {call_graph, from callers to callees}
Tests: {test_path_and_run_command}
Number of rounds: {number_of_rounds}
Edit variants per round: {number_of_variants}
GOAL: for each entity, write a text specification (inputs, outputs,
edge cases, hidden rules) from which the code can be recreated
WITHOUT looking at the source.
RULES:
1. Generate code only from the specs; don't open the source.
2. Edit order: first entities that call nothing, then those
whose callees are already solved (all tests pass). Don't touch a caller
until its callees are solved.
3. Describe recursive or mutually calling entities as one group.
4. In each round, edit the spec of exactly ONE entity.
5. For it, produce {number_of_variants} edit variants. Build each from a
different subset of failing tests. Don't mix noisy and repetitive failures
with informative ones.
6. For each variant, regenerate the module code and run ALL tests.
7. A variant is accepted if no other entity's result got worse.
If several variants are good in different ways, keep the two best.
8. Stop when all entities are solved or you run out of rounds.
REPORT FORMAT after each round:
Round N | entity | variants (which tests were used) | full-run result | accepted/rejected and why
FINISH: spec file {spec_path} and a list of unsolved entities.
Fill in the module path, call graph, test command, number of rounds (5–10) and variants (2–3). If you don't have a call graph, ask the agent to build one as the first step.
🚀 Quick start — paste into a chat or give it to your agent:
Here's a SpecForge template for improving code specs. Adapt it to my
task: {what my project is and what I want to do with the specs:
port to another language, add a feature, update docs for the agent}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask about the call graph, the tests and the run command, and the number of rounds and variants. The graph is needed for bottom-up order, the tests serve as the objective signal, and the limits keep the loop from spinning out of control.
Limitations
⚠️ You need tests: The method rests on execution checks. In the paper, tests were generated from a reference implementation and verified by running them. Without tests that pass on correct code, the loop turns into overconfident back-and-forth.
⚠️ Python only, one benchmark: Evaluation covered 20 Python modules. How the method behaves on huge repositories with dynamic calls, decorators and external services is unknown.
⚠️ The full version requires code: The Pareto archive, the bandit for entity selection and branch scoring are a program. In a chat you get a simplified version without automatic selection of the best branches.
⚠️ Expensive in requests: Every round means several variants, regenerating the whole module and a full test run. For small modules that's overkill.
⚠️ Not about readability: The output doesn't replace documentation for humans. The specs are optimized for the agent to recreate behavior, not for easy reading.
⚠️ Effect on new tasks: Transfer to cross-language translation and new features is shown, but those are secondary experiments. The main evidence is code reconstruction.
How It Was Studied
The authors built the DevEval+ benchmark: 20 Python modules, 247 entities and 760 tests. The modules come from DevEval, supplemented with repositories from finance, bioanalysis and signal processing. Average dependency depth is 4.2 levels, so the entities really do hook into each other. Spec quality was measured by what share of entities fully pass their tests after recreating the code without seeing the original. During search the method saw only generated tests; the final check used hidden reference tests.
It was compared against five groups: documentation for humans (RepoAgent, DocAgent), prompt optimizers (MIPROv2, GEPA, ACE), commercial agents (Cursor and Claude Code, including with a call-order hint and self-generated tests), simple sequential refinement, and the closest competitor, DocSearch. Five models were tested, including Claude-4.5-Sonnet and Gemini-3.1-Pro.
SpecForge won on every model: from 88.9% solved entities on Qwen3-235B to 94.9% on Claude-4.5-Sonnet. The competitor DocSearch trailed by 5.7 pp on average. It also goes bottom-up and diversifies errors, but it follows a single line and accepts only regression-free edits.
It's striking how badly the usual approaches do. Documentation for humans gets only 14–35%. Cursor and Claude Code agents, even with an order hint and tests, stay at 52–62%. Prompt optimizers that rewrite the whole bundle at once can't tell which entity caused a failure and amplify coupling. Simple sequential refinement hits a ceiling because of whack-a-mole.
The practical takeaway: order, edit locality and version diversity matter more than "smart" rewriting. Specs produced by the loop improved cross-language translation and implementation of new features, where they beat the requirements documents from the benchmark itself. The available text of the paper cuts off at the main results, so ablation and appendix details aren't covered here.
Adaptations and Extrapolations
💡 Adaptation for AGENTS.md / CLAUDE.md: The "a spec is verified by recreation" principle can be applied to agent instruction files. This is a transfer of the idea, not a result from the paper.
Read AGENTS.md and, based on it alone, describe how to implement {function_or_module}.
Don't look at the code. Compare the result with the real code and tests.
For each discrepancy, propose ONE line of edit to AGENTS.md —
specific, with an edge case. Fix one discrepancy at a time
and repeat the comparison.
🔧 Technique: strip noise from feedback → sharper edits. When you feed the agent the full pytest output with 40 failures, it mixes up cause and effect. Give it only the failures of the specific function and only the tests at the lower call level. Same logic as "different subsets of failures."
Resources
- SpecForge: Recursive Self-Improvement for Coding Agents via Behavioral Specifications
- Authors: Yutong Cheng, Peng Gao (Virginia Tech), Haifeng Chen, Wei Cheng (NEC Laboratories America); NeurIPS 2026
- Code and data: https://github.com/ccsnow127/SpecForge
- Mentioned: DevEval, DocSearch, RepoAgent, DocAgent, MIPROv2, GEPA, ACE, Cursor, Claude Code