TL;DR
If you ask an LLM to write tests for fresh code and accept everything that passes, the bug turns into "expected behavior". The model looks at what the code does right now, not at what it's supposed to do. The researchers call these tests fault-enforcing: they pass on the broken code and fail on the fixed code.
The painful part is that everything looks great. The tests are green, coverage went up, and there's no signal of a problem. Fault-enforcing tests turned up several times more often than tests that actually catch the bug. The reason is simple. The model's only "source of truth" is the code, and the "test passes" check rewards fitting the test to the code. When a test fails, the model fixes the test or throws it away instead of saying "looks like there's a bug here."
The practical takeaway: tests generated from code without a separate statement of intent can't be treated as verification. You need manual validation of the expected values. The defense against this is a dedicated prompt (see below). The authors didn't test it; it's our own derivation.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Mechanism Blueprint
STEP 1: A developer opens a PR with a bug (the project's tests don't see it)
STEP 2: The LLM gets the code BEFORE, the code AFTER, and the diff → writes a test
STEP 3: Test failed on the code? → the model edits the test to fit the code (up to 6 attempts)
STEP 4: Test passes → accepted into the suite
STEP 5: The bug is now "confirmed" by a test
STEP 6: Later the bug gets fixed → the test fails → "regression or bad test?"
Test outcomes relative to the bug:
Fails on the bug, passes on correct code → catches the bug (rare)
Passes on the bug, fails on correct code → locks the bug in (several times more common)
Passes on both → useless (the majority)
Fails on both → broken
Example in Practice
Task: You're writing the backend for a delivery service in Cursor. The spec says: "up to and including 5 kg: $5.00; over 5 kg: $5.00 + $0.75 for each full kg above 5." In the PR you accidentally wrote weight >= 5 instead of weight > 5. You ask: "write tests for my PR." The model takes the values from the code, and the 5 kg edge case gets a "correct" answer that matches the bug.
Prompt (defensive version, our derivation, not from the paper):
Write pytest tests for the calc_delivery_price function.
<spec>
Up to and including 5 kg: $5.00.
Over 5 kg: $5.00 + $0.75 for each full kg above 5.
Weight is a positive number, in steps of 0.1 kg.
</spec>
<diff>
{diff_PR}
</diff>
<rules>
1. Derive expected values ONLY from <spec>. Not from the code and not from the diff.
2. First calculate the expected values by hand for the edges: 0.1, 4.9, 5.0, 5.1, 6.0, 7.5.
3. If a test fails on the current code, do NOT edit the test to fit the code.
Stop and write: "code ↔ spec mismatch."
4. If the spec doesn't cover a case, mark it "NOT IN SPEC."
Don't invent an expected value.
5. Output a table: test | input | expected | where in the spec it comes from.
</rules>
Result: The model will produce a table of tests tied to specific lines of the spec. The edge values will be calculated separately. At 5.0 kg the test will most likely fail, and the model will report a mismatch between the code and the spec instead of rewriting the expectation. Cases outside the spec, such as negative weight, it will mark as undefined.
Why It Works
The weakness. When the prompt contains only code, the only reference point for the "right answer" is the code itself. The model derives assertions from what it sees. The "test passed, coverage is there" criterion rewards fitting: rewriting a failed test is easier than suspecting the code. This is the well-known oracle problem (where do you get the right answer?), and here it's demonstrated on real project histories.
The strength. The model is good at computing from explicit rules and cross-checking two texts against each other. Give it a specification and it can compare the code to the spec. Forbid it from fixing the test to fit the code and it will report the mismatch.
Control knobs: - Source of expected values: spec, ticket, requirements doc, docstring. The more precise the source, the less invention. - The "failed → stop and report" rule: this is the main knob. Without it, the model slides back into fitting. - The "NOT IN SPEC" label: it forces the model not to fill gaps with guesses. - The "where it comes from" column: reviewing tests gets faster because you look at the reference, not the code.
Prompt Template
Write tests for {function_or_module}.
<spec>
{requirements_description: spec, ticket, rules}
</spec>
<diff>
{code_changes}
</diff>
<rules>
1. Derive expected values ONLY from <spec>, not from the code.
2. First calculate the expected values by hand for {edge_cases}.
3. If a test fails on the current code, do NOT edit the test. Stop and describe the "code ↔ spec" mismatch.
4. If the spec doesn't cover a case, mark it "NOT IN SPEC"; don't make things up.
5. Output a table: test | input | expected | spec item.
</rules>
Fill in:
- {function_or_module}: what you're testing.
- {requirements_description}: a human description of the behavior, not code.
- {code_changes}: the diff or the function itself.
- {edge_cases}: thresholds, zeros, empty values.
For agents (Claude Code, Cursor): you can move the same rule into an instructions file (CLAUDE.md, AGENTS.md): "when a test fails, first check whether the bug is in the code; don't change an expected value without a reference to a requirement." This is our assumption; the paper didn't test it.
Limitations
⚠️ The defense is untested: the authors measured the risk, not the cure. They didn't show that a "spec-based" prompt reduces the number of fault-enforcing tests. It's a reasonable hypothesis, but not a result of the paper.
⚠️ One model: only Claude Haiku 4.5, a small fast model, generated the tests. With stronger models the share may differ. Which way is unknown.
⚠️ The bugs are synthetic: they were injected through automatic mutations (replacing
<with<=, changing constants, and so on), not taken from real developer mistakes. Real bugs may behave differently.
⚠️ Narrow sample: three large Python libraries, with tests generated for uncovered changed lines. For your web service the numbers will differ, but the mechanism is the same.
⚠️ You need a specification: without a spec, the model has nothing to go on. You'll have to write the requirements out in words first, and that's manual work.
⚠️ Manual review stays: the defensive prompt doesn't replace human eyes. It only makes review easier: you can see where every expectation comes from.
How They Tested It
The researchers took 145 real pull requests from SciPy, Qiskit, and pandas. Bugs were automatically injected into each PR (about 60 thousand mutants in total), and only those were kept that the project's existing tests miss and that can be caught by some test. This simulated the situation "the bug got through review and CI." Then the model got the code before, the code after, and the diff, but not the correct version. It wrote a test for the changed lines and revised it up to six times based on the run results. Every test was run on both the broken and the correct version of the code. That produced four categories: catches the bug, locks the bug in, useless, broken.
Fault-enforcing tests made up 8–17%, bug-catching ones only 2.4–4.8%. The rest either passed everywhere or failed everywhere. Next, the same bugs and tests were carried through subsequent PRs. Several commits later, 83–91% of the fault-enforcing tests still passed only on the broken code. That means fixing the bug would break them. When bugs from different PRs were gathered into a single codebase, 83–92% of them stayed locked in by the end of the history. The project's own tests found only 14–30% of these bugs.
What's striking here isn't that the model makes mistakes, but that the mistake is invisible. All tests are green, coverage keeps rising, and the risk piles up like technical debt. The practical rule that follows is simple: "the test passed" says nothing about correctness if the expectation was taken from the code itself.
Adaptations and Extrapolations
🔧 Technique: add an "expectation source" column → fast manual review
For each test, state the source of the expected value:
[spec, item N] / [docstring] / [derived from code: MARK AS QUESTIONABLE].
Check the tests marked "derived from code" first. Those are your prime fault-enforcing suspects.
Extrapolation: a two-pass review. The first pass writes tests from the spec only, with no access to the code. The second compares the failures against the code. This is our idea; it's not in the paper.
Step 1 (chat without code): from <spec>, write an "input → expected output" table.
Step 2 (separate chat): here's the code and the table. Run the table against the code
and list every mismatch. Don't change the table.
Resources
- "On the Risks of using LLM-Generated Tests for Regression Testing," Mohammadali Charoosaei, Cedric Richter, Mike Papadakis, SnT, University of Luxembourg.
- Test-generation approaches mentioned: ChaCo, SymPrompt, CoverUp, TestForge, LLMLOOP.
- Projects studied: SciPy, Qiskit, pandas. The full prompts and parameters are in the paper's replication package (section 9, not reproduced in the text).