TL;DR
The study shows how LLMs go wrong when picking the better of two experiment plans. The authors took real experiments from machine learning papers and made a second version of each with one planted methodological flaw, changing about 2% of the text. The model has to pick the more reliable plan. Every pair was shown in both orders: original first and original second.
The headline finding: the answer depends on presentation order. A model picks the right plan in one order and the wrong one in the other, even though the text is identical. Some models hit about 66% accuracy “in each order on its own,” but only 43% “correct in both orders.” Even the best models land around 78% when they have to be right both ways. They don't fail out of ignorance: they often name the exact flaw and then explain it away. They mistake detail for rigor (“the author laid it all out honestly”), popularity for correctness (“everyone does it this way”), and parrot the weak plan's persuasive self-justification.
The way out is hinted at by the “hints” test: tell the model outright that one of the plans has a flaw, name its category, or give it a separate review of each plan by severity level, and accuracy rises noticeably. That adds up to a working procedure for an LLM judge: run the comparison in both orders, review each option separately, and check it against a checklist of common flaws.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
This isn't a new technique, it's a reliability test for an LLM judge. Here's how it's set up:
STEP 1: Take the original experiment plan → O
STEP 2: A copy with one planted flaw (edit ≤10% of the text) → T
STEP 3: The model picks the “more methodologically sound” one → two requests:
• O first, T second
• T first, O second
STEP 4: Scored as correct only if O is chosen in BOTH orders
The “hints” test (in the original, run on the hard subset of the sample):
Base question: “Which plan is more methodologically sound?”
+ Flaw disclosure: “One of the plans has a flaw, find it”
+ Category disclosure: + the name of the flaw type
+ Tiered review: a separate review of each plan by severity tier
+ Full set: all of the above together
How They Studied It
- Data: 1,858 pairs from accepted papers at major ML conferences, 2024–2026.
- Pair validation: multi-stage filters. A human check by PhD students put the share of valid pairs at 86%.
- Models: about two dozen current models (Claude, GPT, Gemini, Grok, DeepSeek, Qwen, and others).
- Reasoning analysis: the authors read the models' explanations in cases where they picked the flawed plan.
- Luck check: another LLM checked whether the error was named correctly in cases where the model got it right in only one order. For strong models the error was named correctly 86–92% of the time, so this isn't just blindly picking the first option.
Which Flaws They Planted
This doubles as a ready-made checklist for reviewing any experiment plan, test, or study:
- Confounded comparison — something else changes along with the factor being tested.
- Blocked mechanism — the setup removes the very situation where the tested idea could work at all.
- Wrong metric — it measures something other than what the conclusion claims.
- Unequal conditions — one option gets more tuning, more information, or friendlier scoring.
- Degenerate comparison — the options are identical, all of them fail, or all of them hit the ceiling.
- Unmet theoretical assumptions — it leans on a rule that doesn't hold here.
- Circular evaluation — the evaluation depends on the very method being tested, or on settings chosen by looking at the final result.
- Data leakage — it uses information that shouldn't be available.
7 Typical Reasoning Errors of an LLM Judge
| Error | What happens |
|---|---|
| Detail as rigor | The more detailed and “transparent” the weak plan's description, the higher it scores |
| Standard as correctness | “That's how the field does it” is taken as proof of reliability |
| Self-justification echo | The model restates the plan's own argument in its defense |
| Failure to weigh | Criticizes both plans but never decides which flaw is decisive |
| Self-contradiction | Slams a technique in one order, praises it in the other |
| Misapplied principle | Cites the right rule but applies it backwards |
| Hallucinating the paper | “Remembers” the source paper and bends the answer to fit it |
Example Use Case
The paper tested only ML experiments. The transfer to business tests below is an adaptation the authors themselves did not test. The method's sweet spot: comparing two similar test plans that differ in a single methodological detail.
Task: The product team at FreshMart, a grocery chain, wants to check whether an email newsletter boosts repeat purchases. An analyst wrote two plans. Plan A is short: randomly split the customer base, send the newsletter to one group and not the other. Plan B is long, with metrics, timelines, and “what the big retailers do”: compare people who subscribed to the newsletter with everyone else, and launch a new promo code while we're at it. You need to figure out which one to trust. The model will happily pick B for its detail and “standard practice.”
Prompt:
You are a reviewer of experiment plans. Below are two plans for testing the same hypothesis.
<Context>
Hypothesis: an email newsletter increases the share of repeat purchases
within 30 days among FreshMart customers in Chicago.
</Context>
<Plan_1>
[text of Plan B]
</Plan_1>
<Plan_2>
[text of Plan A]
</Plan_2>
Steps:
1. For each plan separately, list its methodological
problems by level: CRITICAL (makes the conclusion impossible),
SERIOUS, MINOR. Check these categories: confounded comparison,
wrong metric, unequal conditions, circular evaluation, information
leakage, degenerate comparison, blocked mechanism.
2. Give no credit for a detailed description or for references to
“accepted practice.” Judge only what the plan can actually
prove.
3. Determine which of the critical problems is decisive.
4. Pick the more reliable plan and explain why in 3 sentences.
Then, in a second request, repeat the same thing with the plans swapped (A as Plan_1, B as Plan_2). If the choice changes, don't trust the answer and work through the discrepancy by hand.
Result: The model will produce two lists of problems by severity, name one decisive problem, and pick a plan. Expect it to notice that Plan B mixes two factors (the newsletter and the promo code) plus subscriber self-selection. Then look at the two requests: did the choice match in both orders?
Why It Works
The weakness. Faced with two coherent texts, the model anchors on whatever it read first. A detailed, confident description looks more reliable than a short one. “That's standard practice” sounds like an argument. If the model isn't forced to weigh the decisive flaw, it criticizes everything a little and ends up deciding nothing. That's why presentation order flips the answer.
The strength. LLMs are good at naming problems and know their methodological principles. The authors explicitly note that the model often finds the right flaw on its own. What trips it up isn't knowledge, it's how it weighs the arguments.
How the method works around the weakness. Running both orders catches instability: any discrepancy jumps out immediately. A separate review of each plan by severity tier forces the model to single out the decisive flaw before choosing. A category checklist points attention at the mechanics of the error, not the polish of the write-up. Saying up front that a flaw exists flips the mode from “approve” to “find the problem.”
Levers: - Plan order. Always two requests; count the result only if the choice matches. - Severity tiers. Add “what decides the outcome” — it forces weighing instead of listing. - Category checklist. Tailor it to your field: for marketing, add “survivorship” and “seasonality.” - Anti-heuristics. The phrases “detail is not proof” and “standard practice is not proof” directly shut down two common errors. - Expecting a flaw. “One of the plans may contain a critical error” — nudges the model into search mode, but can provoke nitpicking at a flawless plan.
Prompt Template
The paper has no ready-made prompt, only a description of the conditions. The authors' base question: “which design is more methodologically sound.” The template below is assembled from elements described in the paper: tiered review, flaw categories, two orders. It's an adaptation, not the authors' text.
You are a reviewer of a {what_we_test} plan.
<Context>
{hypothesis_or_question}
</Context>
<Plan_1>
{first_plan_text}
</Plan_1>
<Plan_2>
{second_plan_text}
</Plan_2>
Steps:
1. For each plan separately, list its methodological
problems by level: CRITICAL / SERIOUS / MINOR.
Check these categories: {flaw_category_list}.
2. Do not treat a detailed description or references to “standard practice”
as an argument. Judge what the plan can actually prove.
3. Identify the single most decisive critical problem.
4. Pick the more reliable plan. Answer: “Plan_1” or “Plan_2”
plus 3 sentences of justification.
Fill in: {what_we_test} — an A/B test, a study, a pilot, a migration plan. {flaw_category_list} — paste the 8 categories above or your own. Run it a second time with Plan_1 and Plan_2 swapped.
🚀 Quick start — paste this into the chat:
Here's a template for comparing two plans in both orders. Adapt it
to my task: [your task]. Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what you're testing, which hypothesis, and what typical mistakes crop up in your field. That's so it can swap the generic category list for yours and pin down the context for the comparison.
Limitations
⚠️ A different domain: The test was built on ML experiments from research papers. Carrying it over to business tests, marketing, or analytics is a reasonable extrapolation, but the authors didn't test it.
⚠️ Even the best models aren't reliable: The top systems got about 78% of pairs right under the strict “both orders” criterion. Don't trust them as the sole arbiter for high-stakes decisions.
⚠️ Some hints are unrealistic: Disclosing the flaw and its category is a diagnostic. In real work you don't know where the error is. What you can actually use is the tiered review and a general checklist, not a precise hint.
⚠️ Order bias isn't fixed by a prompt: The position effect is shown to be robust. The authors found no explanation and don't offer a way to remove it with a single phrasing. Running both orders is what's left.
⚠️ The planted flaws are artificial: Real plans break less “cleanly.” The authors estimate that about 14% of the pairs in the benchmark are debatable.
⚠️ The paper text was truncated: The numerical gain from each hint isn't visible here. All we know is that it “noticeably improves” the result on the hard subset.
Resources
- Tampered ML: Evaluating LLM Judgment of Machine Learning Research — Soroush Ebadian, Parand A. Alamdari (Anthropic Fellows Program, University of Toronto), Joe Benton, Hailey Joren (Anthropic).
- Code and data: https://github.com/tamperedml/tamperedml
- Related work: FLAWS (Xi et al., 2025), PaperAudit-Bench (Tu et al., 2026), SPOT (Son et al., 2025).