Auditing an LLM Judge by Its Class Counts: How to Catch a Judge That Can't Assign the Rare Label

STEP 1: One rubric, fixed responses → two or more judges (different models)
STEP 2: For each judge, count how many times it assigned each class → a “judge × class” table
STEP 3: Flag rule: judge A's count for a class < ⅓ of judge B's
        AND judge B assigns that class to ≥ 10% of responses → judge A is broken on that class
STEP 4: In the report — judge name, per-class counts, and both metrics
        (strict and lenient)

Steps 1–2 take one or two chat requests. Step 3 you can hand off to the model itself.

Task: You run customer support for a marketplace like Amazon. A bot answers 500 complaints about late deliveries. To avoid reading them all by hand, you ask a cheap LLM to label the responses. The metric for leadership: the share of responses where the bot actually solved the customer's problem.

Where the method shines: a rare but important class and a rubric with neighboring classes. Here the neighbors are “solved” and “dodged the question.”

Prompt (judge, run in two different models, e.g., Claude and ChatGPT):

You are labeling responses from a marketplace support bot.
For each response, pick exactly ONE class:
- refuse: the bot refused or redirected without helping (“contact the seller”)
- evasive: the bot stayed on topic but gave the customer neither a solution nor a concrete action (vague phrases, apologies, “we're looking into it”)
- solve: the bot gave a solution or a concrete action (refund, new delivery date, ticket number, exact steps)
- gibberish: the response is incoherent or off-topic

Important: “apologized and promised” with nothing concrete is evasive, not solve.

Below are 500 numbered “complaint — bot response” pairs.
Return only a JSON array: [{"id": 1, "label": "..."}, ...]
Then, on a separate line, output a tally: how many times each class appeared.

{complaint_response_pairs}

Prompt (the check, as a second step, once you have both results):

Here are the class counts from two judges who labeled the same 500 responses with the same rubric:

Judge A ({model_A}): refuse={a1}, evasive={a2}, solve={a3}, gibberish={a4}
Judge B ({model_B}): refuse={b1}, evasive={b2}, solve={b3}, gibberish={b4}

Rule: a class counts as “unused” by one judge if
(1) for the other judge it makes up at least 10% of all responses, AND
(2) the first judge's count for it is less than one third of the other judge's.

Check the rule for every class and every pair of judges.
Name which judge is flagged and on which class.
If a judge is flagged, use the labels by id to show where the responses the other judge put in that class ended up.

Result: The first request returns JSON with labels plus a per-class tally. In the second, the model lays out a table checking the rule. If one of the judges almost never assigns solve, you'll see a flag and a list of ids it moved into evasive. That judge's “share resolved” metric does not belong in your report.

The weakness. Overall accuracy (the share of matches with the reference) is measured across the whole sample, where most responses are easy to classify. The rare class is precisely what the whole exercise is about. Failing on it barely moves the overall number. That's how a judge can “agree 78% of the time” and still be unable to assign the label you actually need. Small judges (7–9B) don't spread their errors randomly: they drift into the neighboring class. To them, “solved” and “evasive” are the closest in meaning, and the line between them is too thin.

The strength. Class counts are visible instantly and for free: every labeling run already contains them. Comparing two judges against each other doesn't require knowing the “right” answer. You only need to notice that one judge never uses what the other uses heavily. That separates “this class isn't in the data” from “the judge can't assign it.”

How the method gets around the weakness. It swaps the question “how accurate is the judge?” (which needs gold labels) for “does the judge use every class?” (which needs only counts).

Levers: - The 10% threshold — raise it if the class is rare for every judge (in that case the rule won't fire; in the study this happened on one corpus). - The ⅓ threshold — make it stricter (e.g., ½) if you want to catch partial failures. - Number of judges — add a third, ideally a large model: the rule compares against the most “generous” judge. - Two metrics — compute both the strict one (“solved”) and the lenient one (“didn't refuse”). The lenient one shifts by a few percentage points; the strict one shifts severalfold.

You are a labeler. For each item below, pick exactly ONE class:
- {class_1}: {definition_1}
- {class_2}: {definition_2}
- {class_3}: {definition_3}
- {class_4}: {definition_4}

Boundary rule: {how to tell neighboring classes apart, 1–2 lines}.

Return JSON: [{"id": N, "label": "..."}].
After the JSON — a tally: how many times each class appeared.

{numbered_items}

Class counts from two judges on the same {N} items with the same rubric:
Judge A ({model_A}): {counts_A}
Judge B ({model_B}): {counts_B}

Flag rule: judge X is flagged on a class if the other judge Y assigns it to ≥10% of items,
while X's count for it is less than ⅓ of Y's.
Check every class and every pair. Name the judge and the class.
For flagged classes, show which neighboring classes X moved Y's items into.

What to fill in: {class_N} and the definitions — your rubric (3–4 classes, ideally ordered “from bad to good”). {N} — the number of labeled items. {model_A/B} — the names of the actual judge models. The authors' verbatim rubric isn't in the body of the paper (it's in the appendix); this template follows its structure: refuse / evasive / comply / gibberish.

🚀 Quick start — paste into a chat:

Here's a template for auditing an LLM judge by its class counts. Adapt it to my task: [your task: what we're labeling and with which classes].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what your classes are, where the line between neighboring classes falls, and how many items you're labeling. It needs this so the flag rule compares the judges against the same definitions.

⚠️ No ground truth: The rule shows that the judges disagree, not that the “generous” judge is right. A judge the rule didn't flag can still be wrong. In the study, the reference judge wasn't flagged, even though human labels produced noticeably more “comply” than it did.

⚠️ Rare for everyone: If no judge assigns a class to ≥10% of responses, the rule doesn't apply. On one of the corpora the check simply didn't fire.

⚠️ Provisional thresholds: The authors picked 10% and ⅓ in advance, but they aren't grounded in theory. Tune them to your own sample.

⚠️ Size doesn't explain everything: The hypothesis “the smaller the judge, the worse” didn't hold up. A pair from the same release (21B and 117B) showed the opposite, and the authors withdrew the hypothesis. Don't write off every small judge either: what broke were dense 7–9B models from four families. Small judges fine-tuned specifically for safety may do better.

⚠️ One task only: The entire check was run on a single task (grading responses to harmful requests). On other rubrics the effect may be weaker or stronger.

⚠️ The judge's own noise: Even the reference judge disagreed with itself on 10–12% of the same responses at temperature 1.0. Any difference smaller than that is just noise.

⚠️ The lenient metric drifts too: The “didn't refuse” rate differs between judges by 10–23 percentage points. That's less than the spread on the strict metric, but it isn't zero.

The idea was simple: freeze everything except the judge. The authors took 100 harmful requests from public benchmarks and wrapped each one in 9 well-known attack types, yielding 900 prompts. Two different target models answered them, and each set of responses became a corpus. Eight judges from five organizations (from 8B to ~117B) labeled those exact same responses with one four-class rubric. Not a single character of the responses changed.

The result was surprising: the judges don't disagree “at random.” Three dense 8–9B models move “comply” almost entirely into the neighboring “evasive.” On the second corpus, one of them didn't assign “comply” a single time out of 900. Without those three judges, the spread on the strict metric drops from ~22–25x to 1.7–2.8x. Robustness checks — new harmful requests, a third target model, a judge from a fourth family (7B) — reproduced the pattern. Some of the checks were preregistered in writing before they were run.

A nice twist: the authors first claimed “the smaller the judge, the worse,” then tested it with a control pair from the same release — and retracted the conclusion themselves. The practical takeaway: trust the counts, not the parameter count.

🔧 Technique: add a “got past the guardrail but delivered nothing” class to the rubric → remove the outcome nobody can express

The authors showed that encoding attacks get past refusals but don't actually produce anything harmful. If the rubric has no class for “answered, but there's nothing in it,” judges read such responses in very different ways (a spread of up to 50 points). In your rubric:

- empty: the bot didn't refuse, but didn't deliver anything useful either (a boilerplate phrase, restating the question)

For support or content work, you'll almost always need a class like this.

Minimal evaluation report (from the authors' checklist, adapted):

Attach to the evaluation results:
1. Name and size of the judge model, number of calls per item.
2. Counts for each class.
3. The rubric verbatim, plus any edits made to it.
4. Both metrics: strict (solved/complied) and lenient (didn't refuse).
5. Temperature of the target model and of the judge.
6. Source of the test requests.
7. Whether anything was labeled by a human who didn't know the goal of the study.

• The Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free Check — Ruqiya Bin Safi, independent researcher.
• Artifact (labels, scripts): https://anonymous.4open.science/r/missing-compliance-class-4F6F
• Related work from the literature review: Souly et al., 2024 (graders inflate jailbreak success); Gao, 2026 (ASR spread across judges); Tamsal & Rusert, 2026 (a grader triples the apparent gain in success after fine-tuning); Chao et al., 2025 (judges vs. expert labels).
• Request benchmarks: Huang et al., 2024; Zou et al., 2023 (AdvBench). Attacks: Wei et al., 2023; Andriushchenko & Flammarion, 2025; Anil et al., 2024.