TL;DR
A checklist for an LLM judge is a list of yes/no questions the model uses to grade someone else's answer. An answer passes only if every item gets a "yes." The headline finding: how strict the check is depends on what the items say, not on which judge model you pick or how you phrase things. On the very same answers, a hand-written checklist passed almost every correct answer, while a model-generated one passed almost none. Swapping the judge or the format moved the result by only a few points.
If you tell a model to "improve" a checklist without outside verification, it bends it to its own taste. It adds requirements the task never asked for (say, "give remediation recommendations") and strikes items that caught real gaps as "too vague." Correct answers start getting rejected wholesale, while real errors slip through. The cause is arithmetic: under the "every item must pass" rule, each extra item multiplies the pass rate by its own pass rate. Ten items, each wrongly failing 5% of good answers, will let only about 60% of good answers through. And no single item looks wrong.
The defense has three parts. First, measure two rates: how many correct answers passed and how many bad ones got caught. Accuracy hides both. Second, for each item, count how many unique catches it makes and how many false failures it causes. To do that, an answer's outcome is split evenly among the items that failed it (a Shapley calculation, see below). Third, treat checklist edits like code changes: never delete an item with unique catches, never add an item with false failures, and freeze items derived from human labels.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Labeled sample (30–50 answers, labeled "ok / not ok") → calibration set
STEP 2: Judge answers Yes/No on EACH item separately → answer table (answer × item)
STEP 3: Two rates: % of correct passed and % of bad caught (not accuracy)
STEP 4: For each item:
• catch credit = sum of 1/(number of failing items) over bad answers
• false failures = the same over correct answers
• unique catches = bad answers that only this item fails
STEP 5: A checklist edit has to clear the "gate":
• item with a unique catch: don't delete
• item with false failures: don't add / remove it
• items derived from labels: frozen
• after every accepted change: recalculate
Steps 2–4 can be done in one chat with a table, or in Excel. The edits in step 5 are separate requests.
Example in Action
Task: A seller support team at a marketplace (think Amazon) launched a bot to answer complaints. The team lead asked ChatGPT to come up with a checklist for auto-grading the bot's replies. The model produced 11 items, including "offers compensation," "links to the policy," and "apologizes twice." As a result the judge rejected nearly everything, even replies the human agents had rated as good. The goal: fix the checklist without bloating it.
Prompt (step 1: the judge):
You are a reviewer of Amazon Seller Central support replies.
Input: a seller's complaint, the bot's reply, a checklist.
Answer EACH item separately, independently of the others: "Yes" or "No".
An item is "Yes" only if the bot's reply clearly meets it.
Check only what the item says; don't add anything of your own.
Complaint: "My listing got suppressed for a \"listing policy violation\",
but nobody told me what the violation was. I lost three days of sales."
Bot reply: "Hello! Your listing was suppressed because the main
image has a watermark. Remove it and resubmit the listing for
review; this takes up to 24 hours. Link to the policy: ..."
Checklist:
1. Names the specific reason for the suppression.
2. Tells the seller what to do next.
3. States how long the re-review takes.
4. The reply promises nothing that isn't in the policy.
5. Neutral tone, no blaming the seller.
Response format: a table: # | Yes/No | quote from the reply you're relying on.
Result: The model returns a five-row table with "Yes/No" and a supporting quote for each item. You collect tables like this for 30–40 labeled replies into one matrix and feed it to the second prompt in the template. It shows which items catch real screwups (unique catches) and which just fail good replies (false failures). From there you delete or rewrite only the items with false failures and leave the items with unique catches alone.
Why It Works
The weakness. When a check "feels wrong," we used to blame the judge model and swap it out or make the prompt stricter. But the judge mostly just executes the checklist. If the list contains items like "recommends disabling anonymous access" and the task didn't call for that, the judge will honestly say "No" and fail a correct answer. Under the "all must pass" rule these failures multiply. A model editing the checklist without labels doesn't see that cost and adds items "for completeness."
The strength. A judge is good at answering "Yes/No" on a narrow item, and its per-item answers are stable. From the answer table you can estimate, without any new requests, what happens if you strike any item. That follows from the property "an answer is failed by every item that said No": for a game like that, the Shapley value comes down to a simple formula: 1 divided by the number of failing items.
How the method gets around the weakness. It replaces the question "does this item sound reasonable?" with "what does it do on labeled answers: catch or fail?" A unique catch means the item can't be deleted. False failures mean the item is harmful. Zero credit means nothing: the item may simply not have met its error in the sample. That's why we freeze items derived from human labels.
Control knobs: - Aggregation rule. Going from "all items Yes" to "at most one No" lifts the pass rate for correct answers from single-digit percentages to a majority. That's stronger than any rewording. Tighten or loosen here, not in the phrasing. - Number of items. More items means a lower pass rate for correct answers. Add an item only if it has proven unique catches. - Labeled sample size. More labeled bad examples means fewer items with a misleading zero credit. Always keep "unexpected omissions" in the sample (answers that miss something the checklist author didn't foresee). - Recalculation. Recalculate after every accepted change: two items may each be safe to delete on their own, but not both at once.
Prompt Template
Prompt 1: the per-item judge (answers on each item are independent):
You are a reviewer. Input: {source_or_task}, {answer_under_review}, a checklist.
Answer EACH checklist item separately and independently of the other items:
"Yes" or "No".
"Yes" only if the answer clearly meets the item.
Check exactly what the item says. Don't add requirements of your own.
Checklist:
{item_1}
{item_2}
...
Format: a table "# | Yes/No | supporting quote".
Prompt 2: attribution from the answer table:
Below is a table: row = a reviewed answer, columns = checklist items.
Cells contain "Yes" or "No". Each answer has a label: "ok" (correct)
or "reject" (bad). An answer passes only if every cell is "Yes".
For each item, compute:
1. Catch credit: for each "reject" answer the item failed,
add 1 / (number of items that failed this answer). Sum over all "reject" answers.
2. False failures: the same, but over "ok" answers.
3. Unique catches: the number of "reject" answers whose ONLY "No" is in this item.
Output a summary table "item | credit | false failures | unique catches".
Then compute the overall rates: how many "ok" answers passed and how many "reject" answers were caught.
Recommendations:
— item with unique catches > 0: don't delete;
— item with false failures and no unique catches: candidate for deletion;
— if two items are each safe to delete on their own, warn me:
check deleting them together.
Table:
{labeled_answer_table}
What to fill in: {source_or_task} is what the answer is checked against (policy, source text, spec); {answer_under_review} is the answer itself; items are one yes/no question each, within what the task explicitly requires; {labeled_answer_table} is Prompt 1's results on 30–50 labeled answers. They should include both correct and bad ones, roughly half and half.
🚀 Quick start: paste this into the chat:
I need to grade {what_we_check} with a checklist and an LLM judge.
Here are two templates: a per-item judge and table-based attribution.
Adapt them to my task. Ask me questions to fill in the fields.
[paste the templates above]
The LLM will ask what counts as a "good" answer, what the task explicitly requires and what it doesn't, and how many labeled examples you have. That keeps the items from going beyond the scope of the task. Extra items are exactly what breaks a checklist the most.
Limitations
⚠️ You need labels: the safeguard only works if you have 30–50 answers with a human "ok / reject" label. Without an outside label, a model's checklist edits drift toward its own taste. This showed up on every reviser model tested.
⚠️ Zero credit ≠ useless item: if an item guards against an error that isn't in your sample, or showed up only once, its credit will be zero. That's why the "gate" works together with freezing items derived from labels. Attribution alone, without freezing, can delete an item you need.
⚠️ Items depend on each other: two items may each be safe to delete on their own, yet duplicate each other. Delete both and the error gets through. You need to recalculate after every accepted change.
⚠️ Only for the "all items Yes" rule: the simple attribution formula is correct for exactly that rule. Scale-based scores or "at most one No" need a different calculation.
⚠️ Narrow base: all experiments used models from one family (Claude). The main testbed is a synthetic cloud security task, plus two public datasets on fact-checking and faithfulness to source. The samples are small, so don't carry the exact numbers over to your task; test the mechanics on your own sample.
⚠️ Doesn't fix the judge: if the judge fundamentally doesn't understand a criterion, this won't help. The method fixes checklists, not weak judges.
How They Tested It
The idea was simple: freeze everything except the checklist. The researchers built 12 synthetic cloud environments seeded in advance with known security problems. An agent wrote reports; some were correct, some had one of four defects: a missed problem, a fabricated one, the wrong resource, or an exaggerated one. That gave every report an exact label with no human involved. They then compared hand-written items, model-generated items, and three judge models. They also varied the format: a yes/no list, a paragraph, a 1–5 scale.
The surprise: hand-written items passed 96–98% of correct reports, while generated ones passed 2–8%, and that didn't change with the judge or the format. The generated items weren't harsher in tone. They demanded things the task didn't ask for (for example, recommending how to disable access). When counted up, the product of the per-item pass rates predicted the real pass rate to within 2 points. Three rounds of the model "improving" the checklist didn't move the result, but switching the rule from "all Yes" to "at most one No" lifted the pass rate for correct answers from 5.6% to 69.4%.
The result replicated on two public datasets with human labels: SummEdits (summary consistency) and RAGTruth (faithfulness of an answer to the source). On RAGTruth, ten generic items passed only 41% of answers people judged correct (a free-form verdict: 81%), while adding only a few catches. The items banned light synthesis across several passages, which the annotators accepted.
Next they tested checklist self-editing with three reviser models. The drift showed up everywhere: correct answers passed 6% of the time instead of 97%, and the omission check derived from labels got deleted as "too vague" (catches dropped from 100% to 8–25%). Of 186 deleted items, 78 held a unique catch, and only 2 carried false failures: the model erased what mattered and kept what hurt. The "gate" built from attribution plus label freezing kept 9 out of 10 drifting edit chains within 5 points of the original quality. The decisions didn't change when the judge was swapped.
The practical takeaway: if you grade LLM answers with a checklist, content and edit control are what matter, not picking the most powerful model.
Original from the Paper
Here's how the authors show the reviser a labeled example (the label-passing format):
Report: “Critical: storage account datalake01, container ‘hr-exports’, has anonymous read enabled and holds payroll CSVs; ... Medium: diagnostic logging is disabled on datalake01, ...”
Human label: pass; judge with your current rubric: fail <– DISAGREES; judge’s answers: [Yes, Yes, Yes, Yes, No, No, No, Yes]
Context: the reviser was shown every calibration answer with its human label, the judge's verdict, and the per-item answers, with disagreements flagged. That's how the reviser "learned" that items about recommendations and nitpicks were optional. The authors' full prompts are in the paper's appendices.
Adaptations and Extrapolations
💡 Adaptation for an agent's instruction prompt: a self-check checklist at the end of a system prompt (for example, in Claude Code: "before finishing, answer Yes/No on each of these 5 conditions") follows the same arithmetic. Every extra condition that doesn't come from the task raises the number of results rejected for nothing.
Before you say "done", check each condition separately and answer Yes/No: 1. {spec_condition_1} 2. {spec_condition_2} If anything is "No", fix it and check again. Don't add conditions that aren't in the spec.
🔧 Technique: free-form verdict → blind spot. A judge without a checklist passes everything (including genuinely incomplete answers), because nobody asked it to check completeness. If you want leniency, keep 2–3 hard items for what's truly critical and don't chase a complete list.
🔧 Technique: "audit the item" instead of "improve the checklist." Instead of "improve this checklist," ask about each item: "Does this item require something that isn't in the spec? Yes/No + a quote from the spec." That shifts the check to grounding in the task and cuts down the model's flights of fancy. This is my extrapolation; the authors didn't test this prompt.
Resources
- Paper: Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges
- Authors: Saeid Asgari, Rui Ying, Sara Malvar, Rakshanda Agarwal, Bruce Sun, Claire Wang, Leonardo de Oliveira Nunes, Ranveer Chandra (Microsoft)
- Datasets: SummEdits, RAGTruth
- Related work: TICK, CheckEval (yes/no checklists), HealthBench (physician rubrics), EvalGen (criteria drift), Ding et al. (judge preference shift under rubric edits), Rulers (rubric freezing)