TL;DR
When you ask a model to list everything — every synonym of a word, every correct answer variant, an entire checklist of criteria — it quietly drops a huge chunk of valid options. But if you hand it one specific candidate and ask "does this fit?" — it does much better. The reason is that enumerating a list and checking a single item are different tasks for the model, even though the underlying knowledge is identical: while building a list, the model writes items one by one and decides for itself when to stop, without ever seeing the options it hasn't "thought of" yet.
The real pain point is that the model errs by omission, not by adding junk. An extra wrong item on a list is visible and can be crossed out during review. A missing correct item is an invisible hole — you won't spot it unless you already know the full list. In the study, mid-size Qwen2.5 models (7B-14B) reviewing a synonym list caught "planted" extra items 6-7 times more often than planted omissions (the gap was smaller for the 72B model and for GPT-5.1). On the coding task this gets absurd: Qwen2.5 models that judge finished solutions with an F1 score of 0.74-0.90 will, when asked to write their own test cases, reject 58-81% of genuinely correct solutions — mostly because they invent requirements and wrong expected outputs.
The paper's fix: don't ask the model to list everything — ask it to write the rule as a small program and let the program check every candidate. On word-list tasks where the rule was given, writing it as a Python function instead of listing the matches raised F1 from 0.26-0.48 to 0.98-0.999 (Qwen2.5 3B-14B). The authors note this escape only exists when a compact rule exists. For model-written test suites they recommend checking them against known-correct behavior first: that cut false rejections to 5% or less, though only 5-39% of suites passed the check. The plain-language method below is our adaptation of these ideas for chat use.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Ask the model to state a RULE (criterion), not a list → a text description of the rule
STEP 2: Test the rule against a known-good example ("would it pass?") → yes/no; if no, rewrite the rule
STEP 3: Apply the rule to each candidate ONE AT A TIME, not all at once → a "fits/doesn't fit" verdict for each
All three steps can happen in a single conversation — just as consecutive messages. This chat recipe is our adaptation; the paper itself had the rule written as code and run by a program, and checked test suites against a reference solution.
Example in Action
Task (our illustration, not from the paper): An online school (think Coursera or Udemy style) wants to auto-grade students' open-ended answers to the question "Name a way to reduce customer churn." They need a checklist of "acceptable answers" for automated grading.
Prompt:
I'm building an auto-grader for students' open-ended answers to the question:
"Name a way to reduce customer churn."
Step 1. Don't try to list every possible correct answer.
Formulate a GENERAL RULE: what markers tell you a student's answer
should be counted as correct, and what markers tell you it shouldn't.
Step 2. Test your rule against this known-correct answer:
"Introduce a loyalty program with personalized discounts for repeat customers."
Does it pass the rule? If not, rewrite the rule so it does pass.
Step 3. Now run each of these answers through the finished rule, one at a time,
and give a verdict of "accepted / rejected" with a short explanation:
1. "Improve customer support quality"
2. "Cut prices by 50%"
3. "Introduce a rewards system for long-term subscribers"
4. "Run a satisfaction survey"
Result: The model first produces a verbal rule/criterion (e.g., "an answer counts if it proposes a concrete retention mechanism, not a vague general statement"). It then checks that rule against the control example and adjusts the wording if needed. In the end, you get a separate verdict with an explanation for each of the 4 answers — instead of one big "correct answers" list generated all at once.
Why It Works
The model struggles with "list every possible variant" because it writes the list item by item, without seeing the others, and decides on its own when to stop — it's a bit like being asked to name every word starting with "K" without a dictionary in hand. You'll stop long before you exhaust the real list, simply because you'll run out of things to recall.
On the other hand, the model is great at evaluating one specific candidate — here it doesn't need to "recall everything," just compare a single example against a criterion. That's its strong suit, and the method leverages it: instead of forcing the model to dream up the whole list at once, we split the task into "state the criterion" (which the model nails almost perfectly) and "check one candidate against the criterion" (which it also nails).
Levers you can pull: - Use a model with a reasoning mode switched on — in the study, low-effort reasoning on GPT-5.1 and Claude Opus 4.8 shrank the gap on word-list tasks (almost closing it for GPT-5.1), but did not reliably help with model-written test suites. - Test the rule against 2-3 control examples instead of just one (our suggestion) — the paper warns that a single hand-checked example is a much weaker test than a full known-correct reference. - If there is no crisp rule (for example, code described only in words) — the paper's evidence is weaker here. For code, its fix was checking the model's tests against a known-correct solution: false rejections fell to 5% or less, but most test suites failed the check and had to be thrown out or repaired.
Prompt Template
I need a {checklist / rubric / list of criteria / answer key} for grading {task}.
Step 1. Don't list every possible variant right away. Formulate a GENERAL RULE —
what markers tell you whether a candidate fits or not.
Step 2. Test this rule against a known-correct example: {known_correct_example}.
Does it pass? If not, rewrite the rule.
Step 3. Now apply this rule to each candidate one at a time and give a verdict
of "fits/doesn't fit" with an explanation:
{candidate_1}
{candidate_2}
{candidate_3}
Fill in {task} with whatever you're grading (an essay, a resume, code, quiz answers), {known_correct_example} with a known-good variant the rule absolutely must accept, and {candidate_N} with the specific items to check.
🚀 Quick start — paste into chat:
Here's a template for the "rule instead of a list" method for checking checklists and criteria.
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what your control example is and exactly which candidates need checking — because without that, Step 2 (testing the rule) and Step 3 (applying the rule) can't work.
Limitations
⚠️ Not a universal cure: the paper's near-perfect result needs a compact rule that a program can run. The authors note this escape is unavailable exactly where checking is hardest; on code, the model's own tests kept inventing extra requirements, and the known-correct check left only 5-39% of test suites usable.
⚠️ Checking candidates one by one is slower than a single request: if you have hundreds of candidates, checking them "one at a time" will take many messages or one long prompt with a full list — that's the price of accuracy.
⚠️ The control example needs to be truly representative: if your "known-correct" example is an edge case, the rule might pass the test but still be full of holes on real-world candidates.
How It Was Studied
The team deliberately built tasks where the correct answer could be checked mechanically, without another model in the loop — otherwise you'd end up with an unreliable judge checking another unreliable judge. For example: they gave a model a list of 15 words and the rule "pick the words with a double letter" — here the answer is fully computable, with zero ambiguity.
They tested open models (Qwen2.5 from 3B to 72B, Qwen3, Llama, Mistral, Gemma and others) plus GPT-5.1 and Claude Opus 4.8 across four constructions: word lists with a clear-cut rule, number lists with a clear-cut rule, program code with reference test suites (HumanEval+/MBPP+), and dictionary synonyms (WordNet). With one-shot answers and reasoning off, the models were more accurate at checking a single candidate than at generating the whole list — and the gap stayed even for the largest open models (about +0.29 F1 for Qwen2.5-72B on word lists). Turning on reasoning closed it for GPT-5.1 on word lists.
The most surprising finding: when researchers asked the model not to produce a list but to write the given rule as a Python function, which was then run over the word list, F1 jumped to near-perfect (0.98-0.999) — so the model understands the criterion, it just struggles to "materialize" it into a full list. The authors' advice: never ask a model to list a set it could instead describe as a rule, and let a program do the listing.
Resources
Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets — Wenhui Chen, Jianlin Chen, Ziyao Lin, Peiji Long, Chi Man Vong (University of Macau, South China University of Technology; Peiji Long is an independent researcher). Built on four controlled ground-truth constructions (word-list and number-list rules, executable code via HumanEval+/MBPP+, and lexical via WordNet).