TL;DR
An independent verifier is a setup where one agent writes the code and another checks it. The checker starts in a clean session and sees only the spec and the program's public interface. It writes its tests before opening the code, and sends the builder only counterexamples: the input, the response and the rule that was broken. No test code and no ready-made fixes.
A typical agent writes the code, writes its own tests, fixes things until everything goes green, and reports "done." But green tests only check what the agent itself came up with. That's three problems. Tests become the reward: the agent bends the code to fit them. Code and tests can grow out of the same misreading of the spec and vouch for each other. And even with a correct reading, the agent only tests what it thought of. In the experiment, the builder's code was correct, but its tests missed subtle interactions between functions.
The method splits the roles. The builder writes code. The verifier checks it against the spec: first blind (phase A), then with access to the code (phase B). A human gets a report on every requirement (verified, violated, not verified) and decides whether to ship. The verifier in a fresh context found more seeded bugs than the builder's own tests: 95.4% vs 85.1% on average across three small services (no difference on one of them). It also surfaced ambiguities in the spec.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
ROLES: builder (writes code) | verifier (only proves defects) | human (decides)
STEP 0: The human locks in a "verification contract": what must be proven, and how,
before release → list of guarantees + required evidence + open questions
STEP 1: The builder writes code and its own tests → program P0
STEP 2 (phase A, new session): the verifier gets the spec + interface (no code bodies)
→ writes properties, forbidden scenarios, acceptance tests → FREEZES them
STEP 3 (phase B, read-only code access): looks for test weak spots, edge cases,
implicit paths → adds tests without changing the frozen ones
STEP 4: Counterexamples to the builder (input → actual → violated guarantee), no test code
→ builder fixes → verifier re-checks everything. 3 rounds max
STEP 5: Report on every requirement: status + what confirms it + what was NOT verified
→ human decision
Steps 2–5 are separate requests or separate agent sessions.
Example in Practice
Task: You're building the promo-code and refunds backend for an online store like Sephora or Instacart. Claude Code wrote the service and wrote its own tests — all green. But when money is involved, a bug costs real dollars, and you don't want to take the builder's green word for it. This is the method's sweet spot: a stateful service (orders, payments, retried requests) and a spec with clear rules.
Prompt (new session or subagent, the builder's code is not shown):
You are an independent verifier. You are NOT the builder. Your job is to prove
defects, not to fix them.
<Specification>
REQ-1: A promo code applies to an order once per user.
REQ-2: The promo discount doesn't exceed 30% of the order total and is capped at $50.
REQ-3: An expired promo code is rejected with code 410.
REQ-4: A repeated payment request with the same idempotency key does not create
a second payment. The key is scoped to the user, not the session.
REQ-5: A refund is only possible for a paid order and only up to the amount paid.
</Specification>
<PublicInterface>
class PromoService: apply_promo(user_id, order_id, code), pay(user_id,
order_id, idem_key), refund(user_id, order_id, amount)
</PublicInterface>
You do not see the method bodies and do not ask to see them.
Phase A: for each requirement, write guarantees (testable statements),
tests on sequences of operations, and forbidden scenarios (for example:
paying twice, refunding an unpaid order, the same code from the same
user on two orders). If the spec is ambiguous — do NOT silently pick an
interpretation; write it under "Open questions."
Output: guarantees → tests → open questions.
Expected result (our illustration, not from the paper): the model should give you a list of guarantees for each of the five requirements and a set of tests, including multi-step scenarios (pay → timeout → retry → refund). A separate block will hold "Open questions": for example, what counts as "once" if an order was canceled, or how to round 30% at the cent level. After running against the builder's code, you send it only the counterexamples and then get a report with requirement statuses.
Why It Works
The weakness. The agent checks itself with the same head it wrote with. If it misread the spec, the tests will confirm the flawed code. If it read it right, it still only tests what it thought of. And once tests sit inside a "fix until green" loop, they turn from a check into a target.
The strength. Models write good tests and scenarios from a spec when their view isn't polluted by someone else's code. In a clean context, the model reads the spec fresh and spots the ambiguous parts on its own.
How the method uses this. Blind-written tests encode the verifier's reading of the spec, not the builder's. That removes the shared source of error. Freezing the tests and giving the builder only counterexamples keeps it from tailoring the code to the check.
Your levers: - What the verifier sees. Only the spec and interface: pure independence. Add the code: you can aim at weak spots, but the view is already skewed. That's why phase A comes first, then B. - Number of fix rounds (3 in the study). Lower it to save tokens. Raise it for critical areas, but any remaining violations must make it into the report. - Open questions. Require ambiguities to be written down, not silently resolved. That's how you see the holes in the spec. - No fixing allowed. A verifier that fixes things becomes a second builder with the same incentives.
Prompt Template
Verifier, phase A:
<Role>
You are an independent verifier of a software module. You are not the author of the code.
Your goal is to find provable defects against the specification, not to get
green tests.
</Role>
<Principles>
P1. The builder's tests and its "done" assurances do not count as evidence.
P2. You do not receive the builder's code in phase A. The builder does not receive your tests —
only counterexamples.
P3. Spec-based tests first, then (phase B) work with the code.
P4. You prove defects but do NOT fix them and do NOT suggest fixes.
P5. For each requirement, state honestly: verified / partial / violated /
not verified — and what remained out of scope.
</Principles>
<Input>
Specification: {specification}
Public interface: {signatures or API schema}
Verification contract (if any): {guarantees and required evidence}
</Input>
<PhaseA>
For each requirement:
1. State guarantees — testable statements (what must always hold,
what must never happen).
2. Write tests: property checks on sequences of operations,
forbidden scenarios, acceptance examples.
3. If the specification is ambiguous or contradictory — don't decide yourself,
write it under "Open questions."
Output: guarantees → tests → open questions. The tests are frozen after this.
</PhaseA>
<PhaseB>
(Run in the next request, once you receive the code {code} as read-only.)
1. Find which defects your frozen tests would NOT catch: come up with
up to {mutation_count} small code edits and check whether the tests would fail.
2. Add tests for boundary values and errors that the specification
leaves implicit. Don't change the frozen tests.
</PhaseB>
<Report>
For each requirement: status, what confirms it, what was NOT verified.
For a violation — a minimal counterexample: input/request sequence →
observed behavior → violated guarantee. No test code and no fix.
Do not output an overall "quality" score as a single number.
Finish with a recommendation (ship / block); the human decides.
</Report>
What to plug in. {specification} — a spec with numbered requirements. {signatures} — public methods without bodies. {mutation_count} — 5–6. {code} — phase B only.
What to send the builder (counterexample format):
Guarantee {number} violated: {statement}.
Input/sequence: {steps}.
Observed behavior: {actual}.
Fix the code. If you believe the counterexample contradicts the spec, push back and
quote the spec clause.
🚀 Quick start — paste into a chat:
Here's an independent verifier template. Adapt it to my task: [your task:
what service or module it is, where the spec lives]. Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask where the requirements live and what public methods the program has. It'll also clarify which bugs would be most costly (money, access rights, retried requests) and how many fix rounds are acceptable. Those questions make sure the tests are built from the spec's rules rather than guesses, and that the verifier doesn't see more than it should.
Important about phase B. In the study, code mutations were generated and tests were run automatically in a harness. In a regular chat, the model only guesses whether the tests would fail. For an honest check you need an agent that runs commands (Claude Code, Cursor).
Limitations
⚠️ False oracles: the verifier sometimes writes a test with the wrong expected result because it misread the spec itself. Good code will "fail" such a test. That's the price of finding more bugs, though in the study genuine false oracles were rare (0.7% of tests in the two-phase run). Check every failing test against the spec text. That's why the builder is allowed to push back on a counterexample.
⚠️ A different verifier model didn't help: in this experiment, a GPT-5.5 verifier found fewer bugs than a fresh Claude Sonnet 5.5 session (85.1% vs 95.4%). Only one model pair was tested, so this says little about other models. Don't spend on switching models first. Start with a clean context.
⚠️ Only spec compliance is checked: if the spec itself is wrong or ignores the real environment (load, multiple replicas, external services), the verifier won't fix that. It will just honestly record those spots as unverified or open questions.
⚠️ The paper's contract and report weren't tested: in the study, verifiers were given the spec directly. The verification contract and report are shown as an illustration of the approach; their effect wasn't measured. What was mainly measured is context independence and the role of the phases.
⚠️ A narrow test: three small Python services, artificially seeded bugs, one builder. On huge codebases and fuzzy specs the result may be weaker. Without a spec the method is nearly useless: the verifier has nothing to work from.
⚠️ The human still decides: the method doesn't replace you; it shrinks what you have to review from the code down to a report.
How They Studied It
The idea was simple: let an agent write a program with its own tests, then compare how many bugs its own tests catch versus how many an independent verifier finds. First the authors took 20 tasks from HumanEval+ and MBPP+. The builder solved 19, almost no bugs remained, and there was nothing to compare. So the main experiment ran on three stateful services (documents and payments, room booking, equipment checkout). Each service had a 10-requirement spec and 46 seeded bugs. Every bug passed the visible tests and failed the hidden ones. Two of the bugged services were written by third-party models (Mistral, Qwen), so the authors couldn't tailor the task to their own method.
They compared five conditions. First: the builder's own tests (Claude Sonnet 5.5). The rest: the verifier in phase A only, in phase B only, in both phases with the same model, and in both phases with a model from another provider (GPT-5.5). Each service was run 5 times, and the protocol and hypotheses were preregistered. There are several metrics, not collapsed into one score. Among them: the share of bugs found and the share of tests that fail on the reference implementation (false oracles).
Result: the builder's programs were correct, but its tests missed subtle interactions. For example, two seeded bugs tied an idempotency key or a pagination cursor to the session instead of the user: the builder's tests caught them in 0 of 10 runs, the verifier in 6 of 10. Overall, the verifier in a fresh context (both phases) caught 95.4% of seeded bugs vs 85.1% for the builder's own tests (p = 0.006 with correction for multiple comparisons; no difference on the equipment service) and surfaced spec ambiguities. The cost: occasional false oracles (genuine ones were rare, under about 2% of tests) and extra tokens. The GPT-5.5 verifier caught fewer bugs (85.1%), but only one model pair was compared, so provider and model strength can't be separated. The practical upshot in this study: a clean session was enough; switching models didn't help.
Two more findings. Phase A did most of the work: within the two-phase run it alone caught 27 bug-runs, phase B alone 7, and both 186. On "ship/block": the builder produced no faulty programs, so the paper couldn't measure whether the gate stops bad releases. On correct programs the verifier's gate blocked too often, mostly because of a harness bug; counting valid tests only, 2 blocks remained, both tied to a question the spec leaves open.
Adaptations and Extrapolations
🔧 Technique: open questions instead of a silent choice → spec holes become visible
Keep this line in the verifier prompt: "If the spec is ambiguous — don't pick an interpretation, write it under 'Open questions.'" The output will be a list of places where the builder and the verifier could read the spec differently. It's a cheap way to find holes in the spec before release.
Below is my extrapolation; it's not in the paper. A rule for the agent's instructions file (CLAUDE.md or AGENTS.md):
## Work handoff rule
You may not say "done" based only on your own tests.
Before handing off:
1. Launch a verifier subagent in a clean context. Give it the spec and
the public interface. Do not pass it the code or your tests.
2. Fix only based on the verifier's counterexamples. If a counterexample
contradicts the spec — push back and quote the clause.
3. In the final message, state the status of each requirement and what wasn't verified.
Resources
- Paper: Who Verifies the Builder? Toward Independent Assurance for Agent-Produced Software (Sion, April).
- Authors: Sion Israel Sion, Alain April, École de technologie supérieure, Montreal, Canada.
- Open-source harness and prompts: https://github.com/besionisrael/assure-harness
- Related work: SpecBench and ImpossibleBench (agents tailor code to tests), EvalPlus (extended tests lower measured correctness), Knight & Leveson (independently written versions fail in correlated ways), the IEEE 1012 and DO-178C standards (verification independence).