TL;DR
Given a spec (a PRD, a product requirements document) for a large, living codebase, an agent usually writes separate pieces but never quite finishes the job. Some features are half-built, others are written but never wired in to the rest of the system. To check that kind of work reliably, you don't ask "is everything done?" — you need a checklist of small criteria. The criteria are locked in before the review. For each one, the judge finds evidence in the code and quotes it. The final score is computed by a strict rule, not by eyeballing.
The pain is familiar. The agent says "done", but the handler isn't registered anywhere. Or the method exists, but its body is empty. Or it missed a condition that the spec never spelled out but that is obvious from context. The best agents fully completed the spec in only a small fraction of tasks. Two failure causes account for about three quarters of all losses: incomplete implementation and missing integration. What makes a task hard isn't the length of the spec or the number of files changed, but the share of implicit requirements that have to be inferred from context.
The review method works in four steps: 1) break the spec into atomic criteria; 2) find evidence in the code; 3) give a verdict for each criterion with a quote; 4) compute the total with a fixed rule. Without the criteria "contract" and without rule-based aggregation, agreement with experts drops noticeably.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 0 (separate request): spec → list of atomic criteria
(behavior, acceptable alternatives, weight) → FREEZE
STEP 1: Hard violations of explicit constraints → Missing
(if no violation is proven — don't decide, pass it on)
STEP 2: For each criterion, find the code + the links
that make the behavior reachable → evidence package
STEP 3: For each criterion, a verdict with a quote:
Covered / Partial / Missing (do not add new requirements)
STEP 4: Total by rule: Covered=1, Partial=0.5, Missing=0,
weighted average. "Done" = score ≥ 95 AND zero Missing
Steps 1–4 fit into a single prompt. Step 0 is better done as a separate request, with a human looking over the criteria.
Example in Action
Task: You run engineering for an online marketplace and ask Claude Code to add a loyalty program along the lines of Amazon or Starbucks Rewards: redeeming points at checkout. The agent reported "done". You don't have time to review a 1,200-line diff by hand, and simply asking "is everything done?" is useless.
Prompt:
You are a strict acceptance reviewer. Check the implementation against the spec.
<Spec>
"Points" loyalty program:
1. At checkout, the user can redeem points, 1 point = $0.01.
2. No more than 30% of the order total can be paid with points.
3. Points are deducted only after successful payment.
4. If the order is canceled, the redeemed points are refunded.
5. Order history shows how many points were redeemed.
</Spec>
<Criteria>
C1 (weight 3): discount calculated at 1 point = $0.01 at checkout.
C2 (weight 3): 30% cap on the order total; anything above is rejected or trimmed.
C3 (weight 3, implicit): deduction happens after payment is confirmed, not when the order is created.
C4 (weight 2): points refunded when the order is canceled.
C5 (weight 1): a "points redeemed" field is visible in order history.
C6 (weight 2, integration): the new handler/service is registered and called from the checkout flow.
Acceptable alternatives: any method names and class structure, as long as the behavior is the same.
</Criteria>
<Diff>
{paste the diff or the list of changed files with code}
</Diff>
Rules:
1. You may not change or add criteria.
2. For each criterion, find in the diff the code that implements the behavior and the code that makes it reachable (call, registration, route, configuration). Quote both.
3. A method name is not evidence. An empty body, a stub, or code that is never called is Partial or Missing.
4. Verdict: Covered (behavior implemented and reachable), Partial (implemented but incomplete or not wired in), Missing (absent or clearly violated).
5. If there is no evidence, write "not found" — don't fill in the gaps.
6. Total: Covered=1, Partial=0.5, Missing=0. Compute the weighted average ×100. The work is accepted only if the score is ≥ 95 and there is not a single Missing.
Response format: a table "criterion — verdict — quote — what's missing", then the score calculation and the conclusion "accepted / not accepted".
Result: The model will produce a table for the six criteria. Each row will have a verdict and a code quote. If the handler is written but never called from the checkout flow, criterion C6 gets Partial or Missing, not "ok". Then comes the weighted-score arithmetic and the "accepted / not accepted" conclusion. You'll see specific holes, not a vague "looks good".
Why It Works
The weakness. Ask a model to "assess whether the spec is met" and it looks at the diff as a whole and falls for surface matches: method names, similar-sounding class names. An empty stub with the right name gets partial credit. An eyeball verdict is unstable and depends on how the material is presented. Without pre-locked criteria, agreement with experts in the study dropped to 70%.
The strength. A model is good at checking one narrow condition when it's been shown where to look and required to produce a quote. It also handles simple rule-based arithmetic reliably.
How the method exploits this. Criteria narrow the question down to one condition. The evidence search forces the model to look not only for code but for links: the call, the registration. The aggregation rule removes the "gut-feel" score. Each of the three parts matters: remove any one and agreement dropped. A judge agent that wanders the repository on its own turned out to be no better than a regular judge.
Control knobs: - Criterion weights. Raise the weight for business-critical conditions (payments, access permissions), lower it for cosmetics. - The "done" threshold. Stricter: 100 and zero Partial. Looser: 90. - List of acceptable alternatives. The more detailed it is, the fewer false "not accepted" verdicts due to a different implementation. - Integration criterion. Add a separate item for every new entity: "registered and called". This is exactly where agents lose the most. - Implicit criteria. Flag them separately: they're the hardest for the agent and deserve the most attention.
Prompt Template
You are a strict acceptance reviewer. Check the implementation against the spec.
<Spec>
{spec_text}
</Spec>
<Criteria>
{id} (weight {1-3}, {explicit/implicit/integration}): {one checkable behavior}
...
Acceptable alternatives: {what counts as an equivalent implementation}
</Criteria>
<Work_result>
{diff_or_code_or_description_of_changes}
</Work_result>
Rules:
1. You may not change or add criteria.
2. For each criterion, find and quote: (a) the code that implements the behavior; (b) the code that makes it reachable (call, registration, route, setting).
3. A method or file name is not evidence. Empty body, stub, code that is never called = Partial or Missing.
4. Covered = implemented and reachable. Partial = implemented incompletely or not wired in. Missing = absent or clearly violated.
5. No evidence — write "not found".
6. Score = weighted average (Covered=1, Partial=0.5, Missing=0) ×100.
Accepted only if the score is ≥ {threshold} and there is not a single Missing.
Format: a table "criterion — verdict — quote — what's missing", then the calculation and the conclusion.
What to fill in: {spec_text} — the original spec, unedited. {id} — C1, C2… Write criteria one behavior per line. {threshold} — 95 by default. The work result is whatever the agent produced.
🚀 Quick start — paste into a chat:
Here is a template for accepting an agent's work against a spec. My case: {describe the task and what the agent did}.
First break my spec into atomic criteria (one behavior per criterion),
flag implicit requirements and integration items, and propose weights.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what the spec states explicitly and what is implied, and which parts of the system the new feature must connect to. That matters, because the quality of the review depends entirely on the criteria. The model will take the structure from the template and assemble criteria for your task.
Limitations
⚠️ A benchmark was studied, not a ready-made technique: the authors tested a judge with call-graph search and fixed rules. In a regular chat you'll get a simplified version. Its accuracy was not measured in this paper.
⚠️ Stubs still slip through partially: even the full pipeline gave a little partial credit to code with the right names and empty bodies. So write the empty-body rule explicitly and review Partial verdicts yourself.
⚠️ Criteria need a human check: in the study they were verified by domain experts. If the model breaks down the spec poorly on its own, the judge will faithfully check the wrong list.
⚠️ Conclusions about what to put in the agent's instructions were not tested directly: the paper shows where agents break: integration, incompleteness, no verification after the last edit. But the accessible part of the text contains no tested fix via instructions. It's a hypothesis — try it out.
⚠️ The "signal → failure" link is correlation, not proven causation: missing verification after an edit often goes hand in hand with unfinished work, but the authors themselves say this doesn't prove the cause.
⚠️ Narrow material: tasks from a single enterprise environment (Java and TypeScript, games, promotions, marketplace). For other stacks the patterns are likely but unproven.
⚠️ The paper text is truncated: the sections on static knowledge in prompts and on improving the agent harness aren't visible here. Conclusions about them rely only on the short summary.
How It Was Studied
A team from Alibaba (Taobao and Tmall) took 43 real tasks from production development. Each had a live repository, a dialogue with the user, and an accepted patch. From these they built a single spec and broke it into 750 atomic criteria, verified by domain experts. The median task: about 1,500 changed lines across 15 files.
They then ran 11 "model + agent harness" configurations. Eight models worked in one shared harness, and two pairs also ran in their own native harnesses. For evaluation they used a three-layer judge: hard checks, link search via the call graph, and a semantic verdict with a quote.
The judge was validated like this: three annotators manually re-reviewed all 750 verdicts. Agreement was 95.2%. A regular judge without structure scored about 77%. Without the criteria contract, agreement dropped to 70%; without rule-based aggregation, to 76%. Reference patches scored about 98 out of 100, while empty and unrelated patches scored zero. This showed the judge doesn't fall for surface matches.
The surprise: the best model fully completed only 6 of 43 tasks, even though its average score was around 78. A high average score hides unfinished features. Task difficulty depended on the share of implicit requirements: scores fell from 75 to 57 as that share grew. Spec length and file count had almost no effect.
One more finding: the agent itself produces noticeably different results across three runs on the same task. The judge's score on repeats is about 3.7 times more stable. One lucky run says little about reliability.
Adaptations and Extrapolations
🔧 Technique: add a "definition of done" to the agent's instructions file → close the most common failures
This isn't the paper's method but a hypothesis drawn from its diagnostics: the main losses come from unfinished work, components that aren't wired in, and no verification after the last edit. For CLAUDE.md or AGENTS.md:
## Definition of "done"
Before you write "done", for each item in the spec:
1. Show where the behavior is implemented (file, function).
2. Show where it is wired in: call, registration, route, config.
3. Make sure there are no empty bodies, stubs or TODOs.
4. Explicitly list the implicit requirements you inferred from the spec.
5. After the LAST edit, run a check (tests or build) and include the result.
If something isn't finished, write "not done", not "done".
🔧 Technique: surface implicit requirements before starting → reduce the main difficulty factor
Before the task, tell the agent: "List every requirement the spec doesn't state outright but that follows from its intent. Let me confirm the list." Task difficulty grew precisely with the share of implicit requirements, and a human can check such a list far more cheaply than rewriting code.
Combo: first the implicit-requirements request, then those criteria go into the judge from the template, then the "definition of done" goes into the agent's instructions. You get a loop from spec to acceptance.
Resources
- Paper: When Coding Agents Outpace Review: Evaluating PRD-Driven Development in Mature Enterprise Repositories (PACE-Bench)
- Code: https://github.com/TianqiLiu1/PACE-Bench
- Authors: Tianqi Liu, Yandong Guan, Songze Li, Wenkai Wang, Gang Yu, Chengfei Lv
- Organizations: Zhejiang University; Alibaba Taobao & Tmall Group
- Related work: OpenHands (agent harness), Agent-as-a-Judge (Zhuge et al., 2025), SWE-bench, DevEval, FeatureBench