TL;DR
A coding agent writes lots of valid tests, but weak ones. Almost all of them pass on the reference implementation. But introduce a bug against a specific requirement, and most of the tests won't notice. The researchers measured this with mutations: take a working program, deliberately break one requirement in it, and see whether even one test fails. The best agent caught only about 55% of these breakages; most of the others, under 40%.
The headline finding: the agent understands the specification but is bad at designing checks. It writes a test for the main scenario and skips input combinations and edge cases. For example, it tests a command with one file even though the requirements say "accepts multiple files." It also produces noticeably fewer tests than the human reference: for one of the models, about five times fewer. The problem isn't that the requirements are unclear. When the agent was also given a working program to check against, test strength barely changed.
The practical takeaway: a message saying "all 120 tests passed" tells you nothing about the quality of the checks. Demand tests for every requirement and for combinations of inputs, and check their strength by breaking the code. Two more levers confirmed in the paper: a detailed specification and a higher reasoning level for the model.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
The paper is a benchmark, not a technique. But you can reproduce its way of measuring test strength by hand:
STEP 1: Give the agent a detailed specification (organized by requirement) → a test suite
STEP 2: Run the tests on the working version → share of valid tests (V)
STEP 3: Break the code in exactly one requirement (a mutant) → one mutant per requirement
STEP 4: Run the tests on the mutant → did at least one fail? Yes = mutant killed
STEP 5: Share of killed mutants (M) = the real strength of the tests
Steps 3–4 can be handed to the agent itself as a separate request. Then it becomes a self-check, which nobody in the paper measured (see "Adaptations").
Example
Task: You lead a small team building a shipping-cost calculator for an online store. The rate rules: weight, zone, cash on delivery, free shipping on orders of $75+, pickup points and courier. A developer using Claude Code wrote the implementation and the tests, "all green." You want to make sure the tests would catch a bug, for example a forgotten discount on the "cash on delivery + pickup point" combination.
The paper has no ready-made prompt. Below is one assembled from its conclusions: tests per requirement, input combinations, and checking by breakage.
Prompt:
Below is the specification for a shipping calculator. Each requirement is numbered.
[R1] Free shipping on orders of $75 or more — courier only.
[R2] For pickup points, a 10% discount on the rate for orders of $50 or more.
[R3] Cash on delivery adds 3% of the order total, but no less than $2.
[R4] Weight over 40 lb — a surcharge of $0.50 for each additional pound.
[R5] Zone 3 (Alaska and Hawaii) — a 1.8 multiplier on the base rate.
[R6] If the pickup-point discount and cash on delivery combine, apply the discount first, then the fee.
Write the tests in tests/test_delivery.py.
Rules:
1. Separate tests for each requirement. Start each test with a comment giving the requirement number.
2. For each requirement, test the boundaries: exactly at the threshold, $1/1 lb below and above.
3. Separately, write tests for combinations of requirements: every pair from R1–R6 that can overlap, plus the order of application from R6.
4. If a requirement contains a list ("several," "any of"), feed the test at least two or three items, not one.
5. At the end, output a table "requirement → number of tests → which combinations are covered." If a requirement has fewer than three tests, explain why.
Don't tell me the tests are good. Show the coverage in a table.
Result: The agent returns a test file where each test references a requirement. At the end there's a coverage table: for each of R1–R6 you see the number of tests and the combinations. There will be noticeably more tests than with a plain "write tests" request. In a follow-up you can ask the agent to break one requirement at a time and show which tests failed.
Why It Works
The LLM's weakness. The agent considers "write tests" done once it has a set of sensible, passing tests. It covers the main scenario and calls it a day. The weak spots sit at the seams: two files instead of one, thresholds, the order in which rules apply. By default it doesn't go looking for them.
The strength. The agent reads specifications well. Implementations built from the same specifications passed up to 93% of the reference tests. If you spell out what to cover and how, it can list it and write it.
How the method uses this. It replaces the question "are there enough tests?" with "what would catch a breakage of this requirement?". That turns a fuzzy goal into a checkable one: every requirement needs tests for boundaries and combinations.
Control levers: - Specification detail. The strongest lever in the paper. With a detailed specification, Sol's test strength was 25.5%; with ordinary repository documentation, 4.0%. The fuzzier the requirements, the weaker the tests. - Specification size. The more requirements, the worse the agent covers them. Cut a large specification into parts and ask for tests section by section. - Reasoning level. For Sol, going from low to xhigh raised test strength from 31.3% to 46.5%, and the cost from $0.45 to $2.4 per task. Diminishing returns, not a miracle. - Number of tests. The models with the weakest results wrote noticeably fewer tests. Requiring "at least N tests per requirement" pushes in the same direction, but the paper didn't test that directly.
Prompt Template
Here is the specification. Each requirement is numbered.
{specification_with_numbered_requirements}
Write the tests in {test_file_or_folder}.
Rules:
1. Separate tests for each requirement. Start each one with a comment giving the requirement number.
2. For each requirement, test the boundaries: at the threshold, just below, just above.
3. Separately, write tests for combinations of requirements that can overlap: {list_of_overlaps_or_"find them yourself"}.
4. If a requirement mentions multiple values ({example: "multiple files," "any of the statuses"}), feed the test at least two or three, not one.
5. At the end, output a table "requirement → number of tests → combinations covered".
If a requirement has fewer than {min_tests} tests, explain why.
Don't assess the quality of the tests in words. Show it in a table.
Put your numbered requirements into {specification_with_numbered_requirements}. If your document doesn't have them, first ask the agent to break the text into numbered requirements.
🚀 Quick start — paste into a chat:
Here's a template for requesting test generation. Adapt it to my task: [your task].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask where the requirements live, which of them can overlap, and what thresholds and value lists they contain. That's what makes the boundary and combination rules work on your data instead of staying generic.
Limitations
⚠️ No tested prompt: The authors did not test whether asking to "cover combinations" improves test strength. They measured the problem and a few factors (specification, reasoning, an executable reference). The template above is our assembly from their conclusions; test it on your own tasks.
⚠️ Narrow project type: 165 command-line utilities in Rust, Go, and C/C++ with "input → output" tests. Web UIs, databases, and integrations with external services weren't tested.
⚠️ Correlation, not causation: The model that created mutants and ran programs on its own wrote the best tests. But Sol beat DeepSeek without doing that. The advice "let the agent break the code itself" is a hypothesis, not a proven technique.
⚠️ Expensive and slow: The best result cost about $10 and 37 minutes per task. Higher spend brings ever smaller gains.
⚠️ The authors' specification isn't yours: Their specifications grew out of reference tests and went through several rounds of refinement. A typical requirements doc is almost always worse, and your result will be closer to 4% than to 25%.
⚠️ Correct specifications don't save you from weak tests: Even with a working program to check against, test strength barely changed. Detailed requirements are necessary but not sufficient.
How They Studied It
Researchers from ETH Zurich took 165 open-source programs with reference code and reference tests. From the tests, an agent assembled a detailed specification, about 5,000 words on average. Then they created mutants for each requirement: copies of the program broken in exactly one place. The mutants were refined so that a single test killed them, not half the suite. The share of such mutants rose from 32% to 73%.
The specifications were cleaned up in cycles: agents wrote tests and code from a draft, and the errors were analyzed and classified. If the specification was at fault, it was fixed, until errors caused by inaccuracies fell below 5%. Then six current agents got only the specification, with no internet access, and had to write a full test suite.
The surprise was the gap between the metrics. Tests were 88–96% valid (except for two weak models), implementations from the same specifications passed up to 93% of the reference tests, yet only 38–56% of mutants were caught. So the specification is clear and the agent understands it; what's missing is the skill of designing checks. Two control experiments confirmed this. A working program to check against raised test validity from 94.4% to 98.4% but barely moved strength (37.7% → 38.1%). Replacing the specification with repository documentation crashed test strength from 25.5% to 4.0%: the agent wrote 40% fewer tests, and they were more often wrong. The error analysis showed a typical failure: core functionality is tested, while input combinations and edge cases are skipped.
Adaptations and Extrapolations
🔧 Technique: add self-mutation → see the holes in your tests right away
The paper showed that the best models created mutants on their own, but rarely: Opus did it in only 33 of 165 tasks. The idea is to turn that step into an explicit instruction. This is our hypothesis, not a proven technique:
Now take the requirements R1–R6 one at a time. For each one, break the code in exactly one place,
without touching anything else. Run the tests. Show a table: requirement → what was broken → did a test fail
(which one). If a mutant survived, write a test that kills it.
🔧 Technique: split the specification into parts → less coverage loss
In the paper, test strength dropped as the number of sections grew. Instead of one request covering 40 requirements, give the agent 8–10 requirements at a time and check the coverage table after each block. This is also our hypothesis, based on their correlation.
Resources
- TestSuiteBench: CodingAgentsStruggle to Generate Strong Test Suites from Specifications (Preprint)
- Authors: Kazuki Egashira, Niels Mündler-Sasahara, Martin Vechev (ETH Zurich), Mark Niklas Müller (LogicStar.ai)
- Website: testsuitebench.com
- Code: github.com/eth-sri/testsuitebench; data: huggingface.co/collections/eth-sri/testsuitebench
- Related work: ProgramBench (Yang et al., 2026), SWT-Bench (Mündler et al., 2024), SWE-Mutation (Sun et al., 2026), NL2Repo-Bench (Ding et al., 2025)