arXiv:2607.10139 ★ 71 Jul 11, 2026

LLM Juries: Multiple Different Models Beat One Model Asked Repeatedly

Key result

When you hit "regenerate," you're not checking the answer — you're re-voting for the same mistake. Cross-model consensus lets you get a reliable answer on math, code, and fact-based tasks — no fine-tuning, no special tools, right in the chat window. The trick: errors from different model families don't overlap — Claude slips up in one place, GPT in another, Gemini in a third. Wrong answers scatter in different directions, while the correct one accumulates votes: cross-model consensus on math benchmarks has beaten purpose-trained verifiers (Process Reward Models) — without a single line of fine-tuning.

TL;DR

Cross-model consensus is an answer-selection technique: ask the same question to several independent LLMs from different families, and go with whatever answer the majority agrees on. Each model solves the task on its own, without seeing the others' answers — like jurors who deliberate in isolation before announcing a verdict.

The core finding: different models fail differently. When several independent LLMs tackle the same task, their wrong answers scatter in different directions, while the correct one accumulates agreement. A single model asked repeatedly, on the other hand, tends to repeat the same mistake over and over — and majority vote "confirms" the wrong answer. It's like measuring something ten times with the same faulty instrument: you'll get consistent — and consistently wrong — numbers.

The method works in two moves: give the task to several different models separately, then take the majority answer. Unanimity is a signal of high confidence; disagreement is a cue to dig deeper.

Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.

Method Blueprint

Steps are run in separate chats / separate services:

STEP 1: Ask model A (Claude) the question → record the answer
STEP 2: Ask model B (ChatGPT) the same question → record the answer
        ⚠️ Do NOT show it model A's answer
STEP 3: Ask model C (Gemini / Grok) the same question → record the answer
        ⚠️ Do NOT show it the previous answers

OUTPUT: The answer 2-3 models agree on → accept it
        All three gave different answers → question is in doubt, needs further checking
        All three matched → high confidence

Optional — "candidates + jury" mode:

STEP 1: Have one strong model solve the task 5-10 times (different attempts)
        → get a pool of candidates
STEP 2: Give the same task to 2-3 other models independently
        → the jury solves the task on its own, without evaluating each other's answers
STEP 3: Pick the candidate from the pool that matches the jury's majority answer

Example

Scenario: Alex is the founder of a SaaS startup. He's planning to raise prices from $19.90 to $24.90 per month, with current monthly churn at 4%. He needs to know: at what churn level after the price hike will revenue still grow? Getting this wrong would be costly — the decision drives strategy for the whole quarter.

Prompt (paste into each chat separately):

SaaS unit economics problem:

Current parameters:
- Price: $19.90/month
- Monthly churn: 4%
- Customer base: 500

Planned parameters:
- New price: $24.90/month
- Expected churn increase after the hike: unknown

Question: what is the maximum monthly churn (in %) after the price increase
at which monthly revenue does not fall below the current level?

Show the calculation step by step and give the final number.

Run this same prompt in Claude, ChatGPT, and Gemini one after another, without showing any previous answers.

Result: Each model will produce a step-by-step calculation and a final number. If two or three models converge on the same churn value — that's your answer with high confidence. If the answers diverge, ask each model to spell out its formula explicitly and find where the logic splits.

Why This Works

Any single LLM has systematic blind spots — error patterns that stem from quirks of its training. When you re-query one model multiple times, it generates variations around the same (possibly wrong) understanding of the task. Majority vote then just locks in that flawed interpretation.

Models from different families (Anthropic, OpenAI, Google, DeepSeek) are trained on different data, by different teams, with different architectural choices. Their errors are decorrelated — they fail in different places. Wrong answers scatter: one model outputs X, another Y, a third Z. The correct answer accumulates agreement — and ends up as the majority.

Levers you can pull: - Number of models → 3 models give a solid signal; 4-5 boost confidence on hard problems - Unanimity as a threshold → if all three agree, accept without hesitation; if only 2 of 3 agree, that's still good, but the topic is tricky - Task type → math and code give the strongest effect; knowledge questions ("which law applies here") give less, because models can share identical misconceptions from the same textbooks - Isolation → it's critical that each model solves independently. Don't show the first model's answer to the second — that kills the decorrelation

Prompt Template

Use this prompt to have an LLM tally the jury's verdict for you:

I asked the same question to three different AI models independently.

Question: {your_question}

Model answers:
- Claude: {claude_answer}
- ChatGPT: {chatgpt_answer}  
- Gemini: {gemini_answer}

Task:
1. Find which answer 2-3 models agree on (or state that they all diverge)
2. If there's consensus, confirm it's correct by checking the logic
3. If they all diverge, explain the key point of disagreement
   and which approach you think is correct and why

Placeholders: - {your_question} — the actual question you asked all the models - {claude_answer} / {chatgpt_answer} / {gemini_answer} — paste in each model's final answer

🚀 Quick Start — paste into a chat:

Here's an LLM jury template for checking consensus across models.
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what question you asked and what answers you got — without that, it can't run the comparison and find the point of agreement.

Limitations

⚠️ Shared misconceptions: The method fails if all models share the same error due to similar training. On graduate-level science questions (GPQA), the effect is weaker — models learned from the same textbooks and can reproduce the same flawed heuristic.

⚠️ Not for subjective tasks: If there's no objectively correct answer (rate this text, which headline is better), majority consensus means nothing — each model is "right" in its own way.

⚠️ You need different families: Several versions of the same model (GPT-4o and GPT-4o mini) barely give you any decorrelation — their errors will look similar. What matters is a genuine cross-family lineup: Claude + ChatGPT + Gemini.

⚠️ Diminishing returns on easy tasks: On simple questions where any strong model gets it right (multiplication tables, a country's capital), the method adds nothing — everyone's already correct.

⚠️ Requires multiple subscriptions: A proper jury needs access to 3+ different platforms.

Resources

Paper: LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning (2025)

Author: Ning Liu, Independent Researcher (University of Michigan)

Contact: [email protected]

Related work: - Self-consistency (Wang et al., 2023) — the original "re-ask one model many times" method - Process Reward Models: Lightman et al., 2023; Wang et al., 2024 - Panel of LLM judges (Verga et al., 2024) - Benchmarks: AIME-2024/2025, MATH-500, GPQA Diamond, OlympiadBench, GSM8K, MMLU-Pro, HumanEval+

Research digest

Key result

Principle

Re-asking one model is like measuring with a bent ruler ten times. Consistency goes up, accuracy doesn't. The right move is to grab three different rulers from three different factories. Isolation matters just as much as diversity: each model has to solve the task on its own, without seeing the others' answers. Show the second model the first one's answer, and it'll catch the same mistake. Just like that, decorrelation is destroyed.

Why it works

Models from different families — Anthropic, OpenAI, Google, DeepSeek — were trained by different teams, on different data, with different architectural choices. Their blind spots don't line up. On a task with an objectively correct answer, that's the exact mechanism at play: wrong answers fan out in different directions, while the right one draws matches. Unanimous agreement among three independent models is a far stronger signal than ten "attempts" from one.

When to apply

Math, code, unit economics, legal calculations — any task with an objectively correct answer. Especially useful when the cost of a mistake is high: financial calculations, algorithms, facts that need verification. NOT suited for subjective tasks ("which headline is better," "rate this text") — there, majority consensus is just an averaged opinion, not the truth. It also works worse on questions from a single knowledge domain, where all the models learned from the same textbooks — a shared misconception baked into the source material won't go away.

Mini recipe

1. Open three windows: Claude, ChatGPT, and Gemini (or any three from different families — not three versions of the same GPT). 2. Paste the exact same prompt into each one — don't show the previous answers to the next model. Not at all. This is critical. 3. Record the answers and compare: if two or three models agree, go with that answer — confidence is high. If all three diverge, the question is hard — ask each one to show its calculation steps and find where the logic splits. 4. Optional — "candidates + jury" mode: ask one strong model to solve the task 5-10 times to get a pool of candidate answers. Then give the same task to two or three other models acting as a jury — they solve it independently, without judging each other's answers. Pick the candidate from the pool that matches the majority verdict from the jury.

Examples

[BAD]: `Are you sure? Recalculate. Try a different approach.` — the same model, the same systemic bias, re-voting for the same mistake. [GOOD]: The same prompt pasted into three different windows, one after another — `Problem: current price is $19.90/month, churn is 4%, 500 customers. New price is $24.90/month. What's the maximum monthly churn rate at which revenue won't drop below the current level? Show the calculation step by step and give the final number.` You record three answers. Two agree on 6.6% — that's your answer. The third gives 5% — you check where it went wrong in its steps.

In plain words

The core idea here is that modern models aren't calculators — they're more like brilliant experts who happen to be prone to flights of fancy. When you ask a single model to solve a hard problem, it might nail the answer perfectly, or it might confidently faceplant thanks to its own internal training quirks. The real problem is that if you ask that same model to double-check itself five times, it'll most likely just keep circling around its own mistake. This is called a systematic blind spot: the model genuinely believes in its own version of reality, even when that version is fundamentally wrong.

This works like a jury trial, where each juror is a completely different model from a completely different family. Imagine asking a question not to one person, but gathering a Harvard grad, an MIT engineer, and a seasoned dealmaker from Shanghai in the same room. They don't talk to each other, don't peek at each other's notes, and don't try to match their neighbor's opinion. If three totally different minds, trained on totally different data, land on the exact same number — that's probably the truth. The odds of all three screwing up in the exact same way are basically zero.

In practice, this looks like cross-model consensus. You take your tricky question — say, calculating unit economics for a startup — and feed it to ChatGPT, Claude, and Gemini at the same time. If ChatGPT says "raise your prices" while Claude and Gemini both scream "don't even think about it, you'll go broke," go with the majority. The research shows that this kind of "collective intelligence" is more accurate than even the fanciest self-verification systems running inside a single model. The opinion of a crowd of independent experts beats the intelligence of any lone genius, no matter how sharp it seems.

This approach is a lifesaver for tasks where the cost of being wrong is too high to leave to chance. The method was tested on math and logic problems, but the principle is universal: it applies to legal advice, code writing, or strategic planning — anywhere there's no clear-cut "correct answer" sitting in a database, and you actually need to reason your way to a conclusion. Model diversity is your insurance policy against hallucinations and dumb mistakes, which are simply unavoidable for any single LLM working alone.

Bottom line: stop taking one model's word for it when real money or your reputation is on the line. Instead of hounding a single model with "are you sure?" prompts, just assemble a panel from different model families. It costs more and takes longer, but what you get on the other end isn't just text — it's a verified result. Consensus is the new quality bar, and if the models agree, you can trust the answer. Everything else is a gamble.

AI lab assistant

Apply this study to your task

Describe what you're working on. AI takes the method and the measured results from this paper and adapts them to your case, or writes a ready prompt built on them.

Based on LLM Juries: Multiple Different Models Beat One Model Asked Repeatedly arXiv:2607.10139

Apply this study to your task

Related techniques

Get the week’s best papers by email