TL;DR
SANE is an agentic framework that takes a description of your LLM judge (prompt, scale, sample inputs) and invents “perturbation” tests on its own. A perturbation is a controlled edit to the text with a known expected outcome. For example: “we added implicit rudeness without profanity — the toxicity score must not drop.” The agent works from six fixed requirements for judges. It figures out what exactly this judge must be able to do, invents edits that target those capabilities, generates before/after pairs and checks that the tests make sense in the first place.
The key finding: off-the-shelf universal tests (“change the format,” “paraphrase,” “make it longer”) mean different things for different judges. Formatting doesn’t matter for a faithfulness judge but is critical for a professionalism judge. An added detail lowers a conciseness score but raises a completeness score. So a fixed test set either misses or checks the wrong thing. As a result, the judge looks reliable while in reality it ignores changes that should move its score.
The method tackles this in a few steps. First the agent pins down the judge’s capabilities. Then, for each capability, it invents an edit and the expected change in score. Next it generates test pairs on your examples. Finally, separate independent checks (three calls, majority wins) weed out defective tests.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
INPUT: judge description + its prompt/output format + 3–5 typical inputs
STEP 1: Identify the judge’s capabilities → a list of “what it must be able to tell apart”
STEP 2: Apply the 6 requirements to each capability → “recipes”: edit + expected score relation
STEP 3: Generate test pairs (before/after) on your examples → ~10 pairs per recipe
STEP 4: Test checking (separate calls, 3 independent, majority):
A — does the recipe match the judge’s goal?
Z — was the edit actually made in the text?
I — does the expected relation apply to this pair?
STEP 5: A recipe is accepted if ≥ 6 of 10 pairs pass A+Z+I
STEP 6: Run the judge on the accepted pairs → a violated expectation = a judge defect
Steps 1–3 — one prompt (or one agent). Steps 4 and 6 — separate requests.
The six fixed requirements: monotonicity, subadditivity, repeatability, robustness, dynamic range, calibration.
Example in Practice
Task: You’ve set up an LLM judge that rates replies from support agents at an online marketplace (think Amazon or Etsy) on “professionalism” from 1 to 5. It’s already scoring in production automation. You need to find out whether it’s blind to polite but condescending delivery like “let me explain this like you’re in kindergarten,” with no profanity or insults.
Prompt:
You are an LLM-judge test engineer. Design a set of metamorphic tests for the judge below.
<ScorerSpec>
The judge rates the professionalism of a marketplace support agent’s reply.
Scale 1–5: 5 — polite, to the point, with a solution; 1 — rude, unprofessional.
</ScorerSpec>
<ScorerImplementation>
Judge prompt: “Rate the professionalism of the agent’s reply from 1 to 5. Return only the number.”
Output format: a single integer.
</ScorerImplementation>
<Examples>
1. “Hi there! Your refund has been processed, the money will arrive within 3 business days. Have a great day!”
2. “Your order is delayed because of the warehouse. We’ve already passed this to the logistics team and will get back to you by tonight.”
3. “Check your account page, it’s all there.”
</Examples>
Follow the procedure from the template: capabilities → recipes per the six requirements → test pairs → A/Z/I check.
Pay special attention to capabilities like “implicit condescension without profanity.”
Result: The model will produce a list of the judge’s capabilities (for example: detecting rudeness, implicit condescension, passive aggression, the effect of writing format). For each capability it will propose several recipes with the expected score relation and generate before/after pairs. Each pair comes with A/Z/I checks. You run your judge on the pairs and see where the expectation is violated. Most likely those will be the tests on implicit disrespect and on formatting.
Why It Works
The weakness. Average agreement with human ratings tells you nothing about how the judge reacts to specific changes. Universal tests don’t know what your judge is actually for. So they check the wrong things and miss defects. Hand-crafting tests for every judge is slow and subjective.
The LLM’s strength. The model is good at two things. First, reading a criterion description and reasoning about which text properties should move the score and in which direction. Second, plausibly “editing” text: adding a jab, removing a detail, paraphrasing. It’s also decent at checking someone else’s output on a narrow question.
How the method uses this. The six requirements act as scaffolding: you don’t have to reinvent them. The model only “translates” them into the language of your criterion. Three narrow checks (A, Z, I) cut out the junk: irrelevant recipes, edits that aren’t actually in the text, and pairs where the expectation doesn’t apply. What remains is the situation where a violated expectation can be counted as a judge defect, not a test defect.
Control knobs: - Pairs per recipe (n) → fewer for a quick check, more for stability. The paper uses n = 10 and a threshold of 6. - Recipe acceptance threshold → stricter gives fewer but cleaner tests. The paper’s appendix tested thresholds from 5 to 10, and the conclusions hold. - Noise tolerance (ε) → tuned to your judge’s spread across repeated runs. Without it you get false alarms. - Independent checks → better to run the oracles as separate chats or calls, not one long request. - The capability list → add your own where you know there’s a weak spot (“implicit condescension”).
Prompt Template
Prompt 1 — test generation:
<Role>
You are an LLM-judge test engineer. Your job is to invent tests that check
whether the judge reacts to changes that affect its criterion.
</Role>
<Inputs>
<ScorerSpec>{criterion_and_scale_description}</ScorerSpec>
<ScorerImplementation>{judge_prompt_and_output_format}</ScorerImplementation>
<Examples>{3-5_typical_inputs}</Examples>
</Inputs>
<BehavioralRequirements>
1. Monotonicity: more of the expressed property → score no lower (or no higher, if the scale is inverted).
2. Subadditivity: the effect of several changes together must not exceed the sum of their individual effects.
3. Repeatability: the same input → the same score.
4. Robustness: immaterial changes (format, paraphrase) → the score doesn’t change.
5. Dynamic range: the judge distinguishes weak, average and strong texts instead of hugging one end of the scale.
6. Calibration: scores match the actual level of quality.
</BehavioralRequirements>
<Procedure>
Step 1. Identify {number_of_capabilities} capabilities that this particular judge must have.
Base them on ScorerSpec and ScorerImplementation. One sentence each.
Step 2. For each capability, apply the requirements from BehavioralRequirements.
For each applicable requirement, write a recipe:
- Recipe: what edit we make to the text
- ExpectedRelation: how the score should change
(for example: s_after ≥ s_before − ε; s_after = s_before ± ε)
If a requirement doesn’t apply to a capability — skip it and explain in one line.
Step 3. For each recipe, create {n} test pairs based on Examples:
- base_text
- perturbed_text (edit highlighted)
Step 4. Output a summary table: capability | recipe | expected relation | pairs.
</Procedure>
<OutputRules>
Edits must be real and visible in the text.
Change nothing except what the recipe specifies.
Don’t invent recipes unrelated to the judge’s criterion.
</OutputRules>
Prompt 2 — the oracle (run separately, 3 independent times, majority wins):
<Role>You are an independent reviewer of tests for an LLM judge.</Role>
<Input>
<ScorerSpec>{criterion_and_scale_description}</ScorerSpec>
<Recipe>{recipe}</Recipe>
<ExpectedRelation>{expected_relation}</ExpectedRelation>
<BaseText>{original_text}</BaseText>
<PerturbedText>{modified_text}</PerturbedText>
</Input>
<Questions>
A (Alignment): do the recipe and expected relation match the judge’s goal? yes/no + reason.
Z (Realization): does the modified text actually perform the edit from the recipe? yes/no + reason.
I (Applicability): does the expected relation apply to this specific pair? yes/no + reason.
</Questions>
Return JSON: {"A": ..., "Z": ..., "I": ...}
Fill in:
- {criterion_and_scale_description} — how you describe to the judge what to rate;
- {judge_prompt_and_output_format} — the judge’s verbatim prompt and the shape of its answer;
- {3-5_typical_inputs} — real samples of your data;
- {number_of_capabilities} — usually 4–6;
- {n} — number of pairs per recipe (10 in the paper).
A pair is accepted only if all three answers are “yes.” A recipe is accepted if more than half of its pairs pass (6 of 10).
🚀 Quick start — paste into the chat:
Here’s the SANE template. Adapt it to my task: I want to test an LLM judge that {what the judge rates}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask about the judge’s criterion and scale, its verbatim prompt, its answer format and a couple of real sample inputs. That’s because the whole method is built on tailored recipes: without the judge’s details, the model will produce generic tests, which is exactly what the method is trying to replace.
Limitations
⚠️ The original is code, not a chat: the authors implemented the method as a harness (orchestrating the agent and the oracles). You can reproduce the logic in a chat or agent, but running hundreds of pairs through a judge will take a small script or some automation. An LLM can help you build that script.
⚠️ Only for people who have a judge: the method tests an LLM evaluator (scorer). You don’t need it for regular chat replies.
⚠️ It finds defects but doesn’t fix them: the output is a list of violated expectations. How to rewrite the judge’s prompt is up to you.
⚠️ The reviewer is an LLM too: the oracles (three independent model calls) can be wrong. The visible part of the paper has no human check of the oracles. On a sample of 20–30 pairs, it’s better to look yourself.
⚠️ One baseline and one “agent” model: they compared against one competitor (JRH) and with one generator model (Claude Opus 5). Agreement with the human ranking of judges was shown only for faithfulness.
⚠️ Definitions of the six requirements: the available text of the paper gives them only briefly. The wording in the template above is my interpretation based on the names. Check it against the authors’ appendix if you need precision.
⚠️ Noise: judges are stochastic, so expectations are set with a tolerance ε. Without a tolerance you’ll get false alarms.
How They Studied It
The idea was to check whether an agent can replace the human who hand-writes test recipes. The team took five judges: faithfulness, fluency, coherence, professionalism and toxicity. The data was 1,200 examples from four sources (news, interviews, meetings). They compared against JRH — an open suite where the recipes were written by people in advance. Both systems got the same model (Claude Opus 5) and ran with no human involvement, to keep the comparison fair.
The result: SANE found 108 unique validated recipes versus 7 for JRH and produced 3.7 times more verified test families. It found judge defects in all five domains. JRH found nothing on either professionalism or toxicity: its fixed recipes (format, paraphrase, verbosity) simply aren’t about that. To check the tests, they used a panel of three GPT-5.2 calls with majority voting. That had a noticeable effect: defective tests were filtered out before the judge was ever run.
The main check — does the ranking of 11 judge models on SANE tests match the human ranking on FaithBench? SANE’s Spearman correlation came out at 0.852, JRH’s at 0.100. The practical takeaway: tests written for the judge’s criterion give a signal, while universal tests barely do.
Adaptations and Extrapolations
🔧 Technique: add your own “weak spots” to the capability list → sharper tests for your pain point
In Step 1, add specifics:
Step 1. Identify 5 capabilities of the judge. Be sure to include:
- detecting implicit condescension without profanity
- reacting to sarcasm
- reacting to bureaucratic boilerplate instead of actual help
The model won’t “forget” the niche risks of your business. This speeds up finding defects exactly where you already suspect a problem.
🔧 Technique: a “three independent calls” oracle → a cheap check for any LLM-generated data
The principle from the paper: the check is done by a separate call, not the same one that generated the output. Three independent runs, majority wins. You can take any LLM-generated batch (test dialogues, synthetic reviews, knowledge-base examples) and run each item through a prompt like “was the task done correctly: yes/no + reason.” The junk gets filtered out before you start building anything on top of it.
Resources
- Paper: SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges
- Authors: Ethan Ludwick (Cisco Systems), Yu-Chung Hsiao (Anduril Industries). A short version was accepted to the NeurIPS 2026 TAE (Trust-AI-Eval) workshop
- Comparison: Judge Reliability Harness (JRH, Dev et al., 2026)
- Data: LLM-AggreFact (AggreFact-CNN, AggreFact-XSum, TofuEval-MediaS, TofuEval-MeetB), FaithBench
- Foundation: CheckList (Ribeiro et al., 2020), metamorphic testing (Chen et al., 1998)