TL;DR
When you ask a model to list everything — every synonym of a word, every correct answer variant, an entire checklist of criteria — it quietly drops a huge chunk of valid options. But if you hand it one specific candidate and ask "does this fit?" — it answers almost flawlessly. The reason is that enumerating a list and checking a single item are different tasks for the model, even though the underlying knowledge is identical: while building a list, the model writes items one by one and decides for itself when to stop, without ever seeing the options it hasn't "thought of" yet.
The real pain point is that the model errs by omission, not by adding junk. An extra wrong item on a list is visible and can be crossed out during review. A missing correct item is an invisible hole — you won't spot it unless you already know the full list. In the study, models catch "planted" extra items 6-7 times more often than they catch omissions. On the coding task this gets absurd: a model that judges a finished solution with 74-90% accuracy will, when asked to write its own test cases, reject 58-81% of genuinely correct solutions — because it invents requirements that were never part of the task.
The fix is simple: don't ask the model to list everything — ask it to formulate a rule that lets you check any given candidate. When the model states a rule instead of a list, accuracy jumps to nearly perfect (in the study, from ~0.48 to ~0.99). Bonus tip: before trusting a finished checklist or rubric, run one known-good example through it first — if the rule rejects it, the rule is broken and needs a rewrite.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Ask the model to state a RULE (criterion), not a list → a text description of the rule
STEP 2: Test the rule against a known-good example ("would it pass?") → yes/no; if no, rewrite the rule
STEP 3: Apply the rule to each candidate ONE AT A TIME, not all at once → a "fits/doesn't fit" verdict for each
All three steps can happen in a single conversation — just as consecutive messages.
Example in Action
Task: An online school (think Coursera or Udemy style) wants to auto-grade students' open-ended answers to the question "Name a way to reduce customer churn." They need a checklist of "acceptable answers" for automated grading.
Prompt:
I'm building an auto-grader for students' open-ended answers to the question:
"Name a way to reduce customer churn."
Step 1. Don't try to list every possible correct answer.
Formulate a GENERAL RULE: what markers tell you a student's answer
should be counted as correct, and what markers tell you it shouldn't.
Step 2. Test your rule against this known-correct answer:
"Introduce a loyalty program with personalized discounts for repeat customers."
Does it pass the rule? If not, rewrite the rule so it does pass.
Step 3. Now run each of these answers through the finished rule, one at a time,
and give a verdict of "accepted / rejected" with a short explanation:
1. "Improve customer support quality"
2. "Cut prices by 50%"
3. "Introduce a rewards system for long-term subscribers"
4. "Run a satisfaction survey"
Result: The model first produces a verbal rule/criterion (e.g., "an answer counts if it proposes a concrete retention mechanism, not a vague general statement"). It then checks that rule against the control example and adjusts the wording if needed. In the end, you get a separate verdict with an explanation for each of the 4 answers — instead of one big "correct answers" list generated all at once.
Why It Works
The model struggles with "list every possible variant" because it writes the list item by item, without seeing the others, and decides on its own when to stop — it's a bit like being asked to name every word starting with "K" without a dictionary in hand. You'll stop long before you exhaust the real list, simply because you'll run out of things to recall.
On the other hand, the model is great at evaluating one specific candidate — here it doesn't need to "recall everything," just compare a single example against a criterion. That's its strong suit, and the method leverages it: instead of forcing the model to dream up the whole list at once, we split the task into "state the criterion" (which the model nails almost perfectly) and "check one candidate against the criterion" (which it also nails).
Levers you can pull: - Add a reasoning step ("think step by step about possible options before stating the rule") — the study shows this noticeably narrows the gap when the task has a clear-cut rule. - Test the rule against 2-3 control examples instead of just one — more "known-good" test cases means a more reliable filter. - If the task is inherently "fuzzy" (no crisp rule, just a text description, like grading code against a docstring) — the method helps, but doesn't fully close the gap. In those cases, Step 2 (checking against a known example) matters even more.
Prompt Template
I need a {checklist / rubric / list of criteria / answer key} for grading {task}.
Step 1. Don't list every possible variant right away. Formulate a GENERAL RULE —
what markers tell you whether a candidate fits or not.
Step 2. Test this rule against a known-correct example: {known_correct_example}.
Does it pass? If not, rewrite the rule.
Step 3. Now apply this rule to each candidate one at a time and give a verdict
of "fits/doesn't fit" with an explanation:
{candidate_1}
{candidate_2}
{candidate_3}
Fill in {task} with whatever you're grading (an essay, a resume, code, quiz answers), {known_correct_example} with a known-good variant the rule absolutely must accept, and {candidate_N} with the specific items to check.
🚀 Quick start — paste into chat:
Here's a template for the "rule instead of a list" method for checking checklists and criteria.
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what your control example is and exactly which candidates need checking — because without that, Step 2 (testing the rule) and Step 3 (applying the rule) can't work.
Limitations
⚠️ Not a universal cure: if the correct answer can't be pinned down by a crisp criterion (for example, grading "code quality" against a general text description of the task rather than formal rules), the method reduces errors but doesn't close the gap entirely — the model can still invent extra requirements.
⚠️ Checking candidates one by one is slower than a single request: if you have hundreds of candidates, checking them "one at a time" will take many messages or one long prompt with a full list — that's the price of accuracy.
⚠️ The control example needs to be truly representative: if your "known-correct" example is an edge case, the rule might pass the test but still be full of holes on real-world candidates.
How It Was Studied
The team deliberately built tasks where the correct answer could be checked mechanically, without another model in the loop — otherwise you'd end up with an unreliable judge checking another unreliable judge. For example: they gave a model a list of 15 words and the rule "pick the words with a double letter" — here the answer is fully computable, with zero ambiguity.
They tested six model families (Qwen2.5 from 3B to 72B, Qwen3, Llama, Mistral, Gemma, Phi-4) across four task types: word lists with a clear-cut rule, program code with reference test suites, and dictionary synonyms. In every case, the model is far more accurate at checking a single candidate than at generating the whole list itself — and the gap doesn't close even for the largest models.
The most surprising finding: when researchers asked the model not to produce a list but simply to state the rule, accuracy jumped to near-perfect — meaning the model knows the criterion perfectly well, it just struggles to "materialize" it into a full list. That's the key practical insight: don't make the model guess the whole list at once — let it name the rule first.
Resources
Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets — Wenhui Chen, Jianlin Chen, Ziyao Lin, Peiji Long, Chi Man Vong (University of Macau, South China University of Technology). Built on four controlled ground-truth constructions (algorithmic, executable code via HumanEval+/MBPP+, and lexical via WordNet).