TL;DR
MARQ turns an LLM into a reviewer with an expert's eye. The prompt doesn't just define each criterion ("unambiguous," "complete"); it adds a checklist of concrete rules under every criterion. The logic is simple: "break any sub-rule and the criterion fails." The model gets a single prompt with context, criteria, rules and an output format, and in one call returns a "statement × criterion" table.
Without checklists, the model only knows the dictionary definition of a criterion. It misses up to half of the real defects and invents some "defects" of its own. On top of that, the results bounce around from run to run. A definition like "a requirement must be correct" says nothing about how an expert actually checks it. So the model fills in the blanks itself, differently every time.
With checklists, quality (F1, a composite score where 1.0 is perfect) went up: Claude from 0.55 to 0.80, GPT from 0.52 to 0.62, and the small open model Gemma 3 12B from 0.18 to 0.53. The effect depends on the model: Claude both caught more defects and raised fewer false alarms. GPT caught more, but its false alarms went up too.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Capture HOW experts judge each criterion
→ notes: "what I look at, why I call it a defect"
STEP 2: Group by theme and rewrite as statements
→ "A good statement: ..." (3–8 rules per criterion)
STEP 3: A second person reviews the rules; disputes get discussed
→ an agreed-upon checklist
STEP 4: Build ONE prompt (one request covering all criteria):
context + criterion definitions + a checklist under each
+ logic "any rule broken = criterion failed"
+ output format (yes/no table)
STEP 5: Run it, compare against a manual review on a sample,
fix the rules where the errors are
Steps 1–3 are done once. Step 4 is the prompt itself, and you can assemble it in any regular chat.
Example in Action
Task: A product manager is putting together the spec for a new seller dashboard on a marketplace (think Amazon Seller Central or Etsy). The contractor promises "everything per spec," then argues about what counts as done. The spec has 60 items. You need to catch the bad wording fast, before the contract gets signed.
Prompt:
<Role>
You are a lead systems analyst reviewing a spec before the contract is signed.
</Role>
<Context>
Project: a seller dashboard on a marketplace. Sellers are small businesses from mid-sized US cities (Columbus, Boise, Tampa), 5–50 orders a day. The contractor is an outside agency. Timeline: 4 months. Budget: $80,000.
</Context>
<Requirements>
R1: The system shall quickly generate a sales report.
R2: The seller shall be able to export the report to Excel and receive a Slack notification.
R3: A report covering a period of up to 31 days shall be generated in no more than 10 seconds with 5,000 concurrent users.
R4: The interface shall be user-friendly and modern.
</Requirements>
<Criteria>
C1. Unambiguity. The requirement can be read only one way.
Rules (any one broken = criterion failed):
- No evaluative words without a number: "fast," "user-friendly," "modern," "reliable."
- No "etc.," "for example," "including," "as needed."
- No pronouns or references where it's unclear what they point to.
- Every term means the same thing to the client, the contractor and the tester.
C2. Singularity. One requirement = one capability.
Rules (any one broken = criterion failed):
- No "and," "as well as," "plus" joining two different actions or outcomes.
- The requirement cannot be partially accepted: it's either fully met or not.
C3. Completeness. There's enough information to build and accept the work.
Rules (any one broken = criterion failed):
- The actor is stated (who or what performs the action).
- Conditions and boundaries are stated (period, volume, load, platform).
- Measurable properties have units and an acceptable value.
C4. Template conformance. Format: "The system shall …".
Rules (any one broken = criterion failed):
- The subject is "The system" or a specific component.
- It uses "shall," not "should ideally," "it would be nice," "may."
- No implementation details ("using React") unless it's a constraint.
</Criteria>
<EvaluationLogic>
For each "requirement × criterion" pair, check every rule of the criterion one by one.
If at least one is broken, mark "violated" and give the rule number.
Otherwise, mark "ok."
</EvaluationLogic>
<OutputFormat>
Table: rows are R1–R4, columns are C1–C4. Each cell: "ok" or "violated: <rule>".
After the table, one sentence per violation: what exactly is wrong.
</OutputFormat>
Result: The model returns a 4×4 table showing exactly which rule is broken: "quickly" in R1, two actions in R2, no numeric boundaries in R4. Short explanations of each violation follow the table. R3 will most likely come through almost clean. Treat this as your final list for the conversation with the analyst, not gospel: double-check the borderline cells by eye.
Why It Works
The weakness. A criterion definition like "a requirement must be correct" is just a vague word. An expert checks it against a dozen unspoken tricks, and the model has no idea which ones you mean. So it guesses: it catches some defects, misses others, and behaves differently from run to run. Criteria that need context and systems judgment ("necessary," "correct," "complete") suffer the most.
The strength. LLMs are good at checking text against an explicit list of conditions. When a rule reads "no evaluative words without a number," the task shifts from "judge the quality" to "find a match for a feature." That's squarely in the model's wheelhouse.
How the method exploits it. You move the expert's judgment out of their head and onto the page. The "any rule broken = criterion failed" logic removes the question "how bad is it?" and leaves only "is there a violation or not?" One request for all criteria keeps every rule in a single context.
Control levers: - Number of rules per criterion → more rules: higher detection, but more false alarms. Fewer: cleaner, but more misses. The paper confirms this as a trade-off within individual criteria. - "Any rule broken" logic → switch to "two or more broken" if you get too many false alarms. - Rule number in the answer → forces the model to anchor on a specific rule, and you quickly see where a rule is poorly worded. - Project context → the more precisely you describe the system and the client, the better criteria like "necessary" and "complete" work. - Model choice → rules give a weak model the biggest boost, but run-to-run variance stays.
Prompt Template
<Role>
You are a {expert_role} who reviews {what_we_check} before {usage_context}.
</Role>
<Context>
{description of the project, audience, constraints — everything needed for judgments like "necessary" and "complete"}
</Context>
<Items>
{I1: text}
{I2: text}
...
</Items>
<Criteria>
C1. {criterion name}. {one-line definition}
Rules (any one broken = criterion failed):
- {rule 1}
- {rule 2}
- {rule 3}
C2. {criterion name}. {definition}
Rules (any one broken = criterion failed):
- {rule 1}
- {rule 2}
...
</Criteria>
<EvaluationLogic>
For each "item × criterion" pair, check every rule of the criterion one by one.
If at least one is broken: "violated" and the rule number. Otherwise: "ok."
</EvaluationLogic>
<OutputFormat>
Table: rows are items, columns are criteria. Each cell: "ok" or "violated: <rule>".
After the table, one sentence per violation.
</OutputFormat>
What to plug in:
- {what_we_check} — requirements, brief items, support replies, product listings, legal clauses. Anything that gets checked against criteria works.
- {rules} — how your best specialist actually makes the call. Pull them from their comments on past edits, not from a textbook.
- Pick criteria whose rules can be checked against the text. "Like/don't like" won't work.
🚀 Quick start — paste into the chat:
Here's the MARQ template: criteria-based evaluation where every criterion has a checklist of rules underneath it. Adapt it to my task: [what you're checking and against which criteria].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what you're checking, who the expert reviewer is, which criteria matter and how the expert tells good from bad. That matters because the method's power lies in rules drawn from the expert's real practice, not in generic definitions. From your answers, the model will assemble a ready-to-use prompt with checklists.
Limitations
⚠️ Rules and ground truth come from the same source: the checklist and the "right answers" were written by the same experts. So the study tested whether the model can reproduce a specific expert interpretation, not how "true" it is. If your rules are off, the model will faithfully reproduce off judgments.
⚠️ You need an expert and time to write rules: rules don't write themselves. You need someone who can articulate how they make decisions, and a second person to check them. Without that, you'll get a generic checklist, not an expert one.
⚠️ The effect depends on the model: on some models, the method cuts both misses and false alarms; on others, it catches more defects at the cost of more false alarms. The small open model reached the level of the paid ones but stayed very unstable from run to run.
⚠️ Trade-off within criteria: a rule that catches more defects usually adds false alarms. For some criteria the method barely helps. The gains are concentrated in "Correctness" and "Template conformance."
⚠️ Small sample: 13 statements from a single NASA space project. Only individual statements were tested, not sets, and the "feasibility" and "verifiability" criteria weren't tested. Transfer to other domains is plausible but unproven.
⚠️ Doesn't replace a human: the authors say outright that human review is required. The model produces defect candidates, not a verdict.
How They Studied It
The authors took 13 requirements from a real NASA task: a robotic arm that moves cargo inside the ISS. Three experts independently rated each requirement against 8 INCOSE quality criteria, then converged on a single set of labels. That gave 104 yes/no judgments, the ground truth for testing the model.
In parallel, one of the researchers recorded exactly how the experts reasoned, added recommendations from the INCOSE guide, and turned all of it into rule checklists for each criterion. A second researcher double-checked them. The rules didn't mention NASA, the robot or the specific requirements, so they're general-purpose.
Then they compared two modes: a plain prompt (requirements, context, criterion definitions, task) and the same prompt plus checklists. Nothing else changed: no examples, no model-specific wording tweaks, no document retrieval. Each "model × mode" combo ran 100 times, 600 runs total, on Claude Opus 5, GPT-5.6 Sol and Gemma 3 12B. All parameters were left at defaults.
Median F1 rose from 0.55 to 0.80 for Claude, from 0.52 to 0.62 for GPT, and from 0.18 to 0.53 for Gemma. The surprise: a weak small model with checklists caught up with the paid models without them. But the gains differ: Claude improved in both defect detection and precision, while GPT paid for its gain with more false alarms. The practical takeaway: rules work, but once you add them, tune them to the specific model and validate on a small sample.
Adaptations and Extrapolations
🔧 Technique: require a "quote the fragment" → fewer false alarms
The paper showed that false alarms go up for some models. To bring them down, change the output format (this isn't from the study, so test it on a sample):
In a "violated" cell, give the rule number and a verbatim quote from the item that breaks it. If there's no quote, mark "ok."
Extrapolation: have the LLM itself build the checklist from the expert's comments
The paper built the rules by hand. You can speed up this step:
Below are 15 of my edits and comments on other people's texts: what was wrong and why I decided so.
Identify the recurring principles and phrase them as checkable rules for the "{criterion}" criterion.
Each rule is one statement of the form "A good statement: ...". No more than 6 rules.
For each rule, show which of my comments it's based on.
{comments}
Then edit the rules by hand and drop them into the MARQ template.
Resources
- Paper: MARQ: A Modular, LLM-Agnostic Requirement Quality Assessor Infused with Systems Engineering Knowledge
- Authors: Jannatul Shefa, Taylan G. Topcu (Virginia Tech); Alejandro Salado, Paul Wach (University of Arizona)
- Criteria basis: INCOSE Guide to Writing Requirements / INCOSE Handbook (2023)
- Data: the Simple Deployment Mechanism task from the NASA Astrobee Challenge Series
- Authors' previous work: Shefa et al., 2026 — a benchmark of ten LLMs on the same task
- Prototype: Electronic Appendix A to the paper