TL;DR
VeriSpec is a method for auditing instruction text (a system prompt, a rulebook, a model behavior spec). First, every rule is rewritten in a structured form: source, priority level, applicability condition, requirement, context. Then the rules are grouped by topic and handed to an LLM for checking. The model doesn't just "look for contradictions"; it has to prove each one it finds: describe a concrete situation where both rules fire, and show that both can't be satisfied.
The key finding: if you simply ask a model to "find contradictions in this instruction," it produces a pile of false alarms. In the experiment, these naive variants found zero real contradictions across dozens of reports. The reason is that two rules that look conflicting are often already reconciled elsewhere in the document: by an exception, an example or a priority. The model sees "always A" and "never A" and happily reports a conflict. It doesn't notice that another section says "except in case B." The second problem: in a long text, the two relevant paragraphs get lost.
The method fixes this with three moves. Each rule gets its context (exceptions, examples) placed right next to it. Small groups of related rules are checked, not the whole document. A report is accepted only if it contains a concrete witness situation, an explanation of "why both rules apply here," and an argument for "why the context doesn't resolve the conflict." Precision rose from 0–12% for naive approaches to 38.5%, and the cost per confirmed contradiction fell by roughly 1.5–5x.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Break the instruction into rules → table:
id | priority level | condition (P) | requirement (Q) | context (C) | topic
(separate request; a long text goes in chunks)
STEP 2: Group the rules → by topic AND by priority level
(don't compare rules across levels: a conflict between levels
is already settled by the "chain of command"). Merge small groups with related ones.
Target group size: 5–26 rules.
STEP 3: Check each group → a report only with three pieces of proof:
(a) a witness situation, (b) why both conditions hold
and the requirements are incompatible, (c) why the context does NOT resolve the conflict
(one request per group, repeat ~5 times)
STEP 4: Collect the reports → a human manually reviews the pairs found
Example in Action
Task: You're building a support bot for an online marketplace. The system prompt has been growing for six months: a marketer wrote the tone section, a lawyer wrote the financial limits, a product manager wrote the format rules. The bot behaves unpredictably: sometimes it promises a refund, sometimes it dodges. You need to find the rules that fundamentally can't be followed at the same time.
Prompt (step 3, checking one group):
<Role>
You are a contradiction checker for a support bot's instructions.
You do NOT look for rules that "look like a conflict." You prove a conflict or stay silent.
</Role>
<Rules>
R1 | level: system | topic: tone
Condition: the customer is unhappy
Requirement: calm the customer down and assure them the problem will be solved
Context: "Always calm down an unhappy customer and assure them we'll sort everything out."
R2 | level: system | topic: money
Condition: the customer asks for a refund or compensation
Requirement: don't promise a refund until an agent has reviewed the request
Context: "Never promise a refund before an agent has reviewed it."
R3 | level: system | topic: format
Condition: any reply
Requirement: keep it to two sentences
Context: "Reply in no more than two sentences."
R4 | level: system | topic: refusals
Condition: the bot is turning the customer down
Requirement: explain the reason and describe the appeal process
Context: "When refusing, explain the reason and tell them how to appeal the decision."
R5 | level: system | topic: confidentiality
Condition: any reply
Requirement: don't mention internal policies or department names
Context: "Don't disclose internal policies or department names."
</Rules>
<Task>
Check every pair of rules. Two rules contradict each other only if
ALL of the following hold at once:
1. There is a realistic situation where the applicability conditions of both rules are true.
2. In that situation, no reply can satisfy both requirements.
3. The context of both rules (exceptions, examples, priorities) does NOT resolve the conflict.
If even one point isn't proven, leave the pair out.
</Task>
<OutputFormat>
For each conflict found:
- Rules: the ids of the two rules
- Witness situation: a concrete example of a customer message
- Why both rules apply: ...
- Why both can't be satisfied: ...
- Why the context doesn't resolve the conflict: ...
If there are no conflicts, write "No proven conflicts."
Don't suggest fixes.
</OutputFormat>
Result: The model goes through the pairs and returns a short list of reports in the specified format. Each comes with an invented but realistic customer message as the "witness situation." Likely candidates: the pair about calming the customer vs. the ban on promises, the pair about the two-sentence limit vs. the mandatory explanation with the appeal process, and the pair about the appeal process vs. the ban on naming departments. The model will weed out some pairs on its own, because those rules can be satisfied together. It's best to repeat the run 3–5 times and merge the lists. You make the call on whether "this is a real hole."
Why It Works
The weakness. Asked "are there contradictions here?", a model tends to answer "yes, here are three." Reporting a finding is easy; checking that the conflict is real takes extra work. On top of that, in a long wall of text the two relevant paragraphs drown among the rest.
The strength. The model is good at building concrete scenarios and checking against them: "here's a customer writing X, rule 1 says do this, rule 2 forbids it." It also works well with a short selection of related rules, when everything is in focus.
How the method uses this. The requirement to "show a situation + prove incompatibility + prove the exceptions don't save it" turns a fuzzy search into a verification. The model can't fall back on a "sense of conflict": either the scenario holds up or it doesn't. Grouping by topic removes noise: only rules about the same behavior get compared. And you don't have to check every pair in sequence (with 405 rules in the paper, that's 80,000 pairs).
Control knobs: - Group size (5–26 rules in the paper): smaller is more precise, but you might miss a pair from different topics. For a small prompt (up to 20 rules) you don't need grouping. - Number of runs (5 in the paper): improves recall. A single run often misses real conflicts. - Three mandatory pieces of proof: remove any one and false alarms go up. - Priority levels: if you have "hard" rules and "soft defaults," compare within a level. Between levels, the conflict is already settled by the priority order. - The "Context" field: the more completely you attach exceptions and examples to each rule, the fewer false positives.
Prompt Template
Step 1. Extracting the rules:
<Role>You are an instruction analyst. You turn text into verifiable rules.</Role>
<Task>
Here is the instruction text:
{instruction_text}
Find every statement that prescribes behavior in some situation.
Split a statement with several requirements into several rules.
For each rule, fill in:
- id: {R1, R2, ...}
- source: a quote or section
- level: {list_of_priority_levels}
- condition (P): when the rule applies
- requirement (Q): what must be done
- context (C): all exceptions, examples and clarifications from ANYWHERE in the text
that relate to this rule
- topic: a short behavior label (e.g., "reply format," "tone," "refusals")
</Task>
<OutputFormat>A table. Don't make anything up; take everything from the text.</OutputFormat>
Step 2. Grouping:
Here is the rules table: {rules_table}
Group the rules by topic and level. Don't put rules from different levels in the same group.
If a topic has only 1–2 rules, merge it with the closest topic of the same level.
If a rule belongs to several topics, include it in each.
Group size: from 5 to {max_per_group} rules.
Output: group name → list of ids.
Step 3. Checking uses the prompt from the example above. Put your group into <Rules> and leave the condition in <Task> unchanged.
What to fill in: {instruction_text} is your system prompt or CLAUDE.md. {list_of_priority_levels} is how your hierarchy works (e.g., "hard rule / default / preference"). {max_per_group} is 15–25.
🚀 Quick start — paste into a chat:
Here's the VeriSpec method template for finding contradictions in instructions.
Adapt it to my instruction: [paste your system prompt or describe the document].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what priority levels your rules have and whether there are exceptions and examples in other sections. That's because the method compares rules only within one level and relies on context that "resolves" imaginary conflicts. It will take the structure from the template and assemble a ready-to-use chain of three requests for your document.
Bonus from the paper (anonymization). If you're checking a document that names a provider or a model (e.g., OpenAI's policies), top models may refuse to analyze it because safety filters kick in. The authors replaced the names with fictional ones, and the refusals went away.
Limitations
⚠️ Low precision in absolute terms: even the best variant of the method yields about 4 real findings per 10 reports. The rest are imaginary conflicts. The output is a list of suspects for manual review, not a verdict.
⚠️ Confirmation is manual only: the authors checked the pairs themselves. Out of 405 rules, only five contradictions were confirmed. The method finds rare holes; it doesn't "fix" the document.
⚠️ One document, one verifier model: the experiment was run on a single specification. Precision may differ for your prompt.
⚠️ Large documents need engineering: the full pipeline (a link graph, merging small groups, repeated runs) on hundreds of rules requires code or an agent. For a short prompt, three requests in a chat are enough.
⚠️ Contradictions only: the method doesn't catch gaps (a situation the instruction says nothing about), ambiguities or outdated rules.
⚠️ Step 1 can go wrong: if the model records a condition incorrectly or drops an exception, the false positives come back. Spot-check the table against the original.
Resources
- Paper: Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning (VeriSpec), preprint.
- Code: https://github.com/HIPREL-Group/VeriSpec
- Authors: Zichen Xie, Mrigank Pawagi, Lize Shao, Yang Hu, Wenxi Wang. University of Virginia, University of Pennsylvania, The University of Texas at Austin.
- Subject of the audit: the OpenAI Model Spec (a public specification of model behavior). Claude's Constitution is also mentioned.