Checks "by the Spec, Not by the Code": Why Agents Write Tests to Fit the Existing Implementation and How to Catch It

Task: A Python online store, like a Shopify shop or an Etsy-style marketplace. There's a function that calculates the cart total with a promo code. Spec: "A promo-code discount cannot exceed 30% of the cart total. The discount does not apply to items marked "clearance". The total is rounded down to the whole dollar." The code is already written. You want the agent to write checks that will catch future bugs during refactoring but won't break on harmless changes like a different order of line items on the receipt.

Prompt:

You are writing checks for the apply_promo function in cart/pricing.py.

The standard of correctness is the spec below, not the current code. The code may not match the spec.

Spec:
1. The promo-code discount does not exceed 30% of the cart total.
2. The discount does not apply to items marked "clearance".
3. The final total is rounded down to the whole dollar.

For EACH spec item separately:
A) Scenario generator: a list of different calls (empty cart, single item,
   clearance only, a 50% promo code, cents in the total, a mix of regular and clearance items).
   Give concrete arguments for each.
B) Checks: assertions about the result that follow ONLY from the spec text.
   A check is not tied to a specific scenario and works for any call.

Rules for checks:
- Check no more and no less than the spec requires.
- Don't check what the spec is silent about: the order of items in the output, message text,
  internal field names, the number of helper-function calls.
- If you're unsure whether an assertion follows from the spec, drop it and add it to the list
  "Not checking, because the spec is silent".

At the end, for each check answer: "What correct code could it
wrongly reject?" If such code is easy to imagine, loosen the check.

Result: For each spec item, the agent produces a set of scenarios and checks that aren't tied to a specific call. In a separate list it names what it deliberately isn't checking. At the end comes a self-critique of each check: what correct variant of the code it might reject. Most likely the checks won't be hard-wired to line-item order or error text. But this is no full guarantee: even the best agents in the study missed on a third of tasks. So you still need to run the checks against a known-good rewrite of the code and a known-broken one.

The weakness. If finished code sits in the context, the model treats it as the template. A test built from such checks turns into a "retelling of what the code does today." It sees any difference from the code as an error, even one the spec allows. If the code has a bug, the check locks in the bug too. The experiment showed this: when given an implementation with a violation, every agent lost 19–31 points of correctness.

The strength. The model can read written requirements and derive assertions about the result from them. It's also good at inventing varied inputs. It does manage to find violations when there's something to check: agents' detection rate is noticeably higher than their correctness. The weak spot is in the checks, not in the number of scenarios.

What the approach does. It separates the sources: the spec is the standard, the code is reference only. It forces a separate answer to "what does this spec actually require?" and striking out anything extra. An explicit "not checking, because the spec is silent" list acts as a barrier against the habit of listing everything visible in the code.

Control levers: - Remove the implementation from context entirely → the strongest lever against copying the code (but then the agent can't run scenarios). - Strictness of striking out ("when in doubt, drop it") → loosen it if bugs slip through; tighten it if the agent rejects correct code. - Splitting the spec into atomic items → one requirement per check, so it's easier to see exactly what broke. - The final "what correct code are you rejecting" question → can be asked in a separate pass by a different model.

You are writing checks for {function_or_module} in {file}.

The standard of correctness is the spec below, not the current code.
The code may not match the spec; don't copy its behavior.

Spec:
{numbered_list_of_requirements}

For EACH requirement separately:
A) Scenario generator: a set of concrete calls (arguments), including
   edge cases: {which_edge_cases}.
B) Checks: assertions about the result that follow ONLY from the spec text.
   A check is not tied to a specific scenario and works for any call.

Rules:
- Check no more and no less than the spec requires.
- Don't check what the spec is silent about: {what_the_spec_does_not_fix}.
- In doubt, drop the assertion and add it to "Not checking, because the spec is silent".

At the end, for each check answer: what correct code could it wrongly
reject? If that's easy to imagine, loosen the check.

What to fill in: - {numbered_list_of_requirements} — the spec split into separate items. Each requirement should be breakable independently of the others. - {which_edge_cases} — empty data, limits, cents, non-standard types. - {what_the_spec_does_not_fix} — element order, error messages, internal fields, side effects. You know these items better than the agent does.

🚀 Quick start — paste into a chat:

Here is a prompt template for checks "by the spec, not by the code". Adapt it to my task: {describe_the_function_and_spec}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what requirements are in the spec, which edge cases matter, and what the spec deliberately leaves open. That's what makes the method work: without a clear list of what the spec is silent about, the agent can't cut the extra checks.

⚠️ The paper is a benchmark, not a ready-made method. The authors measured where agents go wrong but didn't test how much a prompt like the template above helps. The template is built from the paper's task framing and main findings. Its gain wasn't measured in the paper.

⚠️ The best result is still weak. No agent built a correct harness on more than 63% of tasks. Run the checks against correct and broken versions of the code yourself.

⚠️ You need a clear specification. The benchmark tasks are "atomic" requirements pulled from a good spec. If the spec is vague ("runs fast," "user-friendly"), the method won't work: there's nothing to strike out and nothing to check against.

⚠️ Python and synthetic violations only. The bugs were injected by another model, not a real dev team. Tested on open-source Python repositories.

⚠️ Keeping code out of context has a price. If you remove the implementation, the agent can't run scenarios. The balance: keep the code, but firmly order the agent to treat the spec as the standard.

The researchers took 30 real functions from open-source Python projects (13 repositories) and split each spec into small requirements. That gave 291 tasks. For each, the requirement was violated: Claude Opus 5 injected a bug into the code that the original tests didn't catch. A second model from a different family, GPT-5.6-sol, confirmed by a three-answer vote that it was a real violation, not a harmless difference. Failed variants were discarded. Bugs from neighboring requirements became the "valid alternatives": correct for the current spec item, but not for their own. Two authors manually checked 50 tasks and agreed with each other 94.7% of the time; they matched the pipeline's decision 98% of the time.

Then four agents (Codex with GPT-5.6-sol, Claude Code with Opus 5, OpenCode with DeepSeek-V4-Pro and GLM-5.3) wrote harnesses with 80 model calls per task. Checks counted as correct if the bug was rejected and everything correct was accepted. Correctness came out at 45–63%, detection at 65–82%, false alarms at 15–27%. The rankings were the surprise: GPT-5.6-sol detected violations 8 points more often than GLM-5.3, but it also raised 8 points more false alarms, and their correctness ended up nearly equal. False alarms made up 42–60% of all failures.

To figure out whether the scenarios or the checks were to blame, the researchers gave the agents "perfect" scenarios guaranteed to expose the bug. For three of the four agents, most missed violations remained: the checks still accepted the bug. So the bottleneck is in the checks themselves. They also showed agents the buggy implementation instead of the reference one. All four lost 19–31 points of correctness and started rejecting the reference implementation they'd never seen on 12–41% of tasks: the agent tailored its checks to what it saw.

🔧 Technique: add an "alternative implementation" → catch false alarms

After the agent writes the checks, ask it (or another agent) to rewrite the function differently: different order, different internal structures, different error messages, but the same behavior per the spec. Run the checks on it. If something fails, that's a false alarm: the check demands more than the spec says. The idea comes from the benchmark's design, but the paper uses this procedure for evaluation, not as a practitioner's technique.

🔧 Technique: break the code on purpose → catch missed bugs

Ask another agent: "Introduce exactly one violation of spec item {N} into the code, one that isn't visible from reading the code but shows up when it runs." If the checks let the violation through, they're weak.

Extrapolation: two passes with split roles.

Pass 1 (agent A, WITHOUT access to the code):
Here is the spec: {spec}. Write checks from the spec text only. You are not shown the code.

Pass 2 (agent B, WITH access to the code):
Here is the spec, here is the code, and here are agent A's checks. Run the checks on the code.
If a check fails on correct code, use the spec to decide: is the bug in the code,
or does the check demand more than the spec? Don't change a check if the spec supports it.

This is my combination, not from the paper: it applies the "agent copies the code" finding through context separation.

• SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications? (Preprint)
• Code and data: github.com/coxhin/SpecGuardBench, huggingface.co/datasets/coxhin/SpecGuardBench
• Authors: Zhiqing Zhong, Mingyu Chen, Aoyang Fang, Yuzhong Zhang, Haoye Tian, Lionel Briand, Boxi Yu, Pinjia He
• Affiliations: The Chinese University of Hong Kong, Shenzhen; Aalto University; Lero (Research Ireland Centre for Software); University of Limerick; University of Ottawa
• Dataset basis: FeatureBench (Zhou et al., 2026)