TL;DR
When an agent writes checking code (a verifier: a script that answers “pass / fail”), it rejects correct solutions more often than it lets wrong ones through. All 19 tested models err in the same direction. They confidently reject bad solutions but are noticeably worse at accepting good ones. The core problem: almost all false rejections come from verifiers that accept no solution at all for their task, not even a correct one.
Here's the painful part: the agent writes the check, runs it on its own “correct” example and calls it done. Meanwhile the check contains an extra condition the task never asked for. Say, a list of allowed tools that's missing one legal tool. Or a requirement to modify files in only one folder. Or a value format that's only partially allowed. The agent's tests miss this for two reasons: it only tests solutions it already considers correct, which share its mistaken assumption, or it ships the check without ever seeing a single full “pass”. Giving it more “thinking time” barely helps, and swapping the agent's harness moves results in both directions.
The authors' practical takeaway: don't check whether “the verifier accepts my solution” — check whether “it accepts other valid solutions to the same task”. Every condition in the check must be justified by the task text. The agent shouldn't ship the check until at least one complete correct solution gets a “pass”.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
This is a finding, not a method. Below is a blueprint of the trap and the defense against it that the authors suggest in their conclusions.
THE TRAP (how the check breaks):
Agent writes a check → adds an extra condition “just in case”
→ tests only on its own solution, where the condition holds
→ the condition never shows up → the check rejects every other valid solution
THE DEFENSE (in one prompt or in the agent's instructions):
STEP 1: Split task requirements into MANDATORY and FLEXIBLE → list
STEP 2: For every check condition, cite the task text → “condition — source” table
STEP 3: Write at least 2-3 valid variants (different path, format, order) → test solutions
STEP 4: Run: reference and valid variants = “pass”, wrong ones = “fail”
STEP 5: Don't ship until at least one complete correct solution gets a “pass”
Example in Practice
Task: You're building a support agent for an online store selling on Amazon. The agent processes a return: it looks up the order, checks its status, creates a return request and messages the customer. You need an automatic checker that verifies whether the agent handled the ticket correctly. You ask Claude Code to write the checker. The classic trap: the checker only knows one order of calls. It doesn't know the agent is allowed to also look at the customer's ticket history, so it flags that as an “extra action”.
Prompt:
Write a check script (check_refund.py) for the “product return” scenario.
Input: the agent's action log (tool calls with arguments).
Output: PASS or FAIL and the reason.
Working rules (mandatory):
1. First, extract from the scenario description and the tool list:
- MANDATORY: what any correct agent must do
(create a return request with the right order_id and reason);
- FLEXIBLE: what may differ between correct solutions
(order of reading data, any read-only tools,
wording of the reason, case and format of values).
2. Give every condition in the check a comment citing a phrase
from the scenario description. A condition with no citation — delete it.
3. Write test solutions:
- a reference one;
- at least 3 valid variants (different call order,
an extra read of the ticket history, a different amount format);
- at least 3 wrong ones (wrong order, no request, refund for someone else's amount).
4. Run the check on all tests. The reference and valid variants
must get PASS, the wrong ones — FAIL.
5. Don't ship the check until at least one complete correct solution
has gotten PASS. If you couldn't run it — say so plainly.
6. At the end, answer: “Which valid solution will my check
reject?” Name at least two candidates and test them.
Result: The agent first outputs a “mandatory / flexible” list, then a script with a justification comment on every condition. Next comes a set of test solutions and a run report. At the end it honestly says which valid variants might have been rejected and shows how it checked. If any valid variant got FAIL, the agent has to fix the condition rather than ship the check.
Why It Works
The weakness. The agent writes the check and its tests from the same understanding of the task. If there's a mistaken assumption in its head (“only tools from this list”, “edits only in this folder”), it lands in both the check and the tests. The test becomes a mirror of the error. Negative tests don't help: a bad solution can violate a real requirement and trip the extra condition at the same time. The fact that it got rejected doesn't prove the extra condition is justified.
The strength. The model is good at answering narrow, text-grounded questions: “where in the task does this requirement come from?”, “come up with another valid solution”. It can generate alternative solutions. What it's bad at is doing this on its own initiative.
How the method uses the strength. The prompt forces the agent to do what the agents in the paper didn't do by themselves: justify every condition with a citation from the task, come up with valid alternatives, and require at least one real “pass” before shipping.
Control knobs: - Number of valid variants (3 → 5): more variants — higher chance of catching an extra condition, but more tokens. - The “no citation — delete it” rule: make it stricter (“ask me first”) if mistakes are expensive. - Shipping condition: replace “at least one PASS” with “all valid variants PASS” for critical checks. - The final “which valid solution will it reject” question: move it into a separate request (see adaptations) to look at the check with fresh eyes.
Prompt Template
Write {what_we_check}: a script/rule/checklist that, given {input_data},
answers PASS or FAIL.
Working rules (mandatory):
1. First, extract from the assignment:
- MANDATORY: what any correct result must satisfy;
- FLEXIBLE: where correct results may differ
(order, format, approach, allowed extra actions).
2. Accompany every check condition with a citation of a phrase from the assignment.
A condition with no citation — delete it.
3. Build test results:
- a reference;
- at least {num_variants} valid variants (different path, format, order);
- at least {num_errors} wrong ones (typical mistakes: {typical_mistakes}).
4. Run the check on all of them. The reference and valid ones must get PASS,
the wrong ones — FAIL.
5. Don't ship the check until at least one complete correct solution
has gotten PASS. If you couldn't run it — say so plainly.
6. At the end, answer: “Which valid solution will my check reject?”
Name at least two candidates and test them.
What to fill in:
- {what_we_check}: what exactly the check verifies (“processing a return”, “correctness of an export”, “a patch for a bug”).
- {input_data}: what the check receives (call log, final table, diff).
- {num_variants}: usually 3.
- {num_errors}: usually 3.
- {typical_mistakes}: 2-3 of your own examples of how people get it wrong.
🚀 Quick start — paste into the chat:
Here's a template for a reliable check that doesn't reject valid solutions.
Adapt it to my task: [describe what the agent should do and what you want to check].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what counts as mandatory and what's allowed to vary, what input the check will receive, and which mistakes are typical. It needs this to split requirements into “mandatory” and “flexible” and to build realistic valid variants. Without them, the check is once again built around a single example.
Limitations
⚠️ The recommendation wasn't tested experimentally: The authors showed on real agent runs how the error arises and suggested testing on valid variations. But the paper has no “better with this kind of testing than without” comparison. The template above is our assembly from their conclusions, not a ready-made, proven technique.
⚠️ Root causes were confirmed selectively: The breakdown of specific checks was done on hand-picked cases to explain the mechanisms. The authors don't estimate how often each error occurs.
⚠️ More “thinking” won't save you: Raising reasoning effort barely changes the outcome. It improves some task types and hurts others. Don't count on a “think longer” mode figuring out on its own that the check is too strict.
⚠️ The agent harness matters, but unpredictably: Switching tools (Claude Code, Codex, Gemini CLI and others) moves check quality in both directions — up for some models, down for others. There's no universally best option; test on your own setup.
⚠️ The problem is worst on hard tasks: On tasks about the final system state and code edits, agents accept valid solutions noticeably worse than on function-call checks. For simple format checks the effect is weaker.
How They Studied It
The team collected 222 tasks from six well-known agent benchmarks: function calls, final system state and code fixes. For each task they built a set of hidden tests, about five thousand in total. It includes the reference solution, valid variants (different order, format, path), ordinary mistakes and attacks on the check (for example, faking a success signal). The agent got the same task description as a regular solver and had to write a verifier. It never saw the hidden tests, so its only source of confidence was its own tests.
To make the tests trustworthy, the authors validated the tests and the reference verifiers themselves with six checks: re-runs, reproducibility, mutations and others. 45 tasks were dropped because of bugs. In one of the benchmarks, for example, the checking parser counted skipped tests as passed. Then they ran 19 models in one shared harness plus 17 more configurations in the vendors' “native” tools.
The main result: every model rejects valid solutions more often than it lets bad ones through. The surprise is that this isn't spread-out imprecision. The overwhelming share of false rejections comes from checks that reject absolutely everything in their task, i.e. it's a systematic error in a single condition, not random noise. Trace analysis revealed two mechanisms: the agent tests only on solutions the extra condition lets through anyway, or it never gets a single full “pass” and ships the work regardless. Hence the conclusion: agent self-checking without a “show me valid alternatives” requirement creates a false sense of reliability.
Adaptations and Extrapolations
💡 Adaptation for LLM judges and rubrics: the same principle applies to model-based answer grading. A rubric saying “the answer contains a 10% discount” will reject “a discount of ten percent” and “10% off”. Before running it, ask the model to come up with three validly different answers and run the rubric on them.
Here's the answer-grading rubric: {rubric}.
Come up with 4 answers that are essentially correct but worded and structured differently
(different order, different words, number format, an extra allowed detail).
Apply the rubric to each. If even one correct answer got “fail” —
rephrase the condition, pointing out which exact phrase in the assignment requires it.
🔧 Technique: move the search for valid alternatives into a separate request → fresh eyes. The authors showed that the agent that wrote the check misses its own assumptions. Open a second chat, give it only the task description (no check) and ask: “Come up with 5 correct solutions that differ noticeably from one another.” Then run the check on them. The second agent doesn't share the first one's assumptions.
This is our extrapolation from the paper's logic; the paper itself didn't test it.
🔧 Technique: the “check rejects everything” trap. If the check gave FAIL on all valid variants, and FAIL on every test overall, that's not a “strict check” — it's a broken one. Add to the agent's instructions: “If your check returned FAIL on every test, including the reference, treat it as a bug in the check and start by hunting for an extra condition.”
Resources
- VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
- Authors: Jianing Yin (University of Pennsylvania), Ziyi Chen (University of Illinois Urbana-Champaign), Hao Wang, Yiyou Sun, Dawn Song (UC Berkeley), Zijian Jin (New York University), Daoan Zhang (University of Rochester)
- Project page: https://verifierbench.github.io/
- Dataset: https://huggingface.co/datasets/Catherine1213/VerifierBench
- Task sources: BFCL v4, ACEBench, τ²-bench, AppWorld, AgentDojo, SWE-bench Verified