Specification Grounding: Explicit Criteria Instead of "Check for Errors"

STEP 1: Build a specification -> an explicit list of requirements (N rules)
        Can be a separate request: "Give me a list of criteria for [task]"

STEP 2: Feed the model the spec + the material to review
        -> For each rule: find a violation or confirm compliance
        -> Output: specific violations with the rule they broke

STEP 3: Fix the material based on the violations found
        -> Targeted edits only, don't rewrite everything from scratch

Steps 1 and 2 can be merged into one request if the criteria are already known. If not, get the criteria list first, then run the check.

Task: You wrote a sales proposal for a major client worth $50,000. You want to review it before sending — you don't want to hear "looks good overall," you want to know exactly what's wrong.

Prompt:

You are an experienced sales manager who reviews sales proposals before they go out to a client.

Here's a list of requirements for a strong proposal:
1. The client's pain point is clearly stated — not our product, but their problem
2. A concrete, measurable outcome (numbers, timelines, units)
3. Explains WHY us specifically, not competitors
4. Transparent pricing structure — the client understands what they're paying for
5. A specific next step with a deadline
6. No corporate jargon ("leverage," "synergy," "utilize," "in scope of")
7. Length — no more than one page

Check my proposal against each of these 7 rules, one at a time. For each rule: status (✅ met / ⚠️ partial / ❌ violated) and a specific explanation of what exactly is wrong.

Here's the proposal:
[proposal text]

Result:
The model walks through each of the 7 rules separately. For each one — a status and specifics: not "the outcome section feels weak," but "rule 2 is missing numbers — it says 'we'll improve efficiency' but doesn't specify by what percentage or over what timeframe." At the end — a list of exactly the spots that need rewriting, each tied to a rule.

An LLM without explicit criteria is like an editor told to "just check the text." It'll check whatever it itself considers important. If the author forgot to include a price, the editor might skip that, because in their mental model a price is "usually there." The model doesn't know what the author actually left out — it's not a mind reader.

When you give it an explicit list of rules, something different happens: each rule becomes its own separate check request. The model isn't choosing what to check — it's checking everything. A non-obvious case ("what if the client submits an empty string?") becomes explicit ("rule 4: handling empty input"). The model can't skip it.

Prompt control levers: - Number of rules → 5–10 is the sweet spot. Fewer, and you'll miss something important. More than 15, and the model starts paying less attention to each one - Rule specificity → the more concrete the wording ("numbers and timelines" instead of "specifics"), the more precise the check - Output format → add "output only the violations" for a quick result, or "output everything with explanations" for a full audit - Source of rules → rules can come from a standard, a methodology, a client checklist — and they're already explicit criteria

Check {what we're reviewing} against the following list of requirements.

Requirements:
1. {requirement_1}
2. {requirement_2}
3. {requirement_3}
[... all requirements]

For each requirement, in turn:
- Status: ✅ met / ⚠️ partial / ❌ violated
- Specific explanation: what exactly is wrong (if not met)
- A quote from the material demonstrating the issue (if applicable)

At the end — a short list of edits: what exactly to rewrite and how.

Here's the material to review:
{text / plan / document}

Placeholders: - {what we're reviewing} — "sales proposal," "product spec," "article," "business plan" - {requirement_N} — specific criteria for your task (you can ask the LLM to generate the list first) - {text / plan / document} — the actual material to review

🚀 Quick Start — paste into the chat:

Here's the Specification Grounding template — checking material against an explicit list of criteria.
Adapt it to my task: [your task].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what material you're reviewing and whether you have criteria — because without a list of requirements, the method won't run. If you don't have criteria, it will first offer to generate some for your task, then apply the review template.

⚠️ Algorithmic logic: The method helps wherever there's a specification — clear requirements for behavior. For tasks where the main difficulty is the algorithm itself ("figure out how to optimize...") rather than rule compliance, the effect is minimal or absent.

⚠️ Spec quality: If the rules are incomplete or vague, the result is the same as having none. "The text should be good" is not a rule. "Each paragraph starts with a thesis explained in the following sentences" is a rule.

⚠️ False alarms without a spec: Without explicit rules, LLMs not only miss real problems but also flag nonexistent ones more often. In the study — 33% false-alarm rate for "check the edges" versus 0% for rule-based checking. Rules give the model an anchor: "this violates rule N" instead of a subjective judgment call.

⚠️ Subjective criteria: If the rules describe taste or style ("the text should feel alive"), the model will be less accurate than with objective criteria.

A team from TD Bank ran an elegant experiment: one variable, everything else fixed. They took 18 tasks involving writing small Python functions. Each function was generated by an LLM — and it almost always handled ordinary cases correctly but forgot the edge cases. They then ran two review approaches on the exact same code: FREE+ ("check for edge cases") and SPEC (same reviewer, same model, same test budget — but with explicit rules). Correctness was determined by an independent "gold" test suite the model never saw.

What was surprising: doubling the number of tests produced almost nothing (+4 pp). Combining 8 independent "check the edges" runs still fell short of a single pass through the rules. The researchers expected quantity to help — it didn't. Only the content of the specification helped.

They also checked whether it was about format rather than content. They gave the model the spec as a single connected paragraph (not a list) — it still found 27 out of 30 issues. They gave it only the task plan with no spec — 2 out of 30. That answers the "why": the model doesn't infer what the author left out. It needs to be told explicitly.

The effect held up across three model families (Claude, GPT, Gemini) and on tasks whose specs were drawn from public standards (RFC, CSS, ISO) — meaning the researchers couldn't "stack the deck" for the method by inventing convenient rules.

SPEC arm prompt structure (from paper §3.3):
- Receives: the prose ticket + the specification enumerated into K discrete rules
- Task: write one test per rule
- Expected value: computed from the ticket/rules, never from the candidate code

FREE+ arm prompt structure (fair baseline):
- Receives: only the ticket
- Additional instruction: "cover invalid inputs and edge cases"  
- Same K tests, same tester model, same repair loop

Key ablation results (§4.2):
- Plan only (task decomposition, no spec): 2/30 bugs caught
- Spec as plain paragraph: 27/30 bugs caught  
- Spec as enumerated checklist (one test per rule): 30/30 bugs caught

Context: The researchers lay out the exact prompt structures for two comparisons — SPEC and FREE+. The difference between them is exactly one line: the presence of a specification-as-checklist. The ablation shows that it's not the format (list vs. paragraph) but the content of the spec that carries 90% of the effect.

1. Adaptation: generating the spec before reviewing

If you don't have rules — ask the LLM to come up with them first, then apply the method.

💡 Two-step SPEC:

Step 1:

Copy

I want to review {what we're reviewing} before finalizing it.
Draft a list of 7-10 specific, checkable quality criteria for {type of material}.
Each criterion should be objective: answerable with YES or NO.

Step 2: use the resulting list in the main review template.

2. Technique: "SPEC for the other side"

🔧 Use the client's/audience's requirements as the spec → review from their point of view

Instead of your own criteria — the other party's explicit objections or requirements:

Copy

Here's a list of typical investor objections at a pitch session:
1. Market is too small — less than $100 million
2. No defense against a big player copying it (Google, Amazon)
3. Unit economics don't hold up at scale
4. Team has no experience in this industry
5. Dependent on a single customer-acquisition channel

Check my pitch against each objection. For each one: is it addressed in the pitch, and how convincingly?

3. Extrapolation: SPEC + iterative improvement

Standard SPEC gives you a review and a list of edits. You can add a repair loop:

[SPEC review prompt]

After reviewing: fix every item marked ❌ and ⚠️.
Output the corrected version and a list of changes.
Then go through the requirements list again — confirm every ❌ became a ✅.

This mirrors the repair loop from the study — just in a single conversation. The LLM first finds violations, then fixes them, then double-checks itself.

Paper: "Specification Grounding Drives Test Effectiveness for LLM Code"
Authors: Amin Haeri, Mahdi Ghelichi — Model Development Innovation, TD Bank, Toronto, Canada
Contact: [email protected], [email protected]

Related work cited in the paper: - AlphaCodium (Ridnik et al., 2024) — iterative code generation with tests - Self-Refine (Madaan et al., 2023) — LLM self-correction - Reflexion (Shinn et al., 2023) — feedback-based iteration - EvalPlus (Liu et al., 2023) — extended test suites for code evaluation - TiCoder (Lahiri et al., 2022) — formalizing intent through human-in-the-loop tests