Selection Bottleneck: Why You Should Pick the Best Answer Instead of Blending Them All Into One

STEP 1: Generation (one request to each model)
        Diverse: Claude + GPT + Gemini → 3 different answers
        or: one Claude, 3 different prompts → 3 variants

STEP 2: Selection (a separate request to the judge LLM)
        [Paste all the variants] → the judge picks ONE best
        Does NOT synthesize, does NOT blend — only picks

Final: The best variant as is (or as a starting point)

Two steps — two separate requests. Or one request with three variants built in, if you're generating in a single session.

Task: Ethan is launching a B2B SaaS for automating HR processes. He needs a cold email to the head of HR at a large company. Three different models produced three very different drafts — now he needs to pick the best one.

Prompt (selection step):

You are an experienced B2B copywriter. Your task: pick the BEST 
of three cold email drafts. Don't merge them into one 
and don't rewrite them — just pick a single winner.

Evaluation criteria:
- How specific the client's pain point is (not vague generalities)
- How compelling the value proposition is
- Natural tone, no corporate jargon
- Likelihood the email gets opened and answered

---
VARIANT A:
[paste the Claude draft]

---
VARIANT B:
[paste the GPT draft]

---
VARIANT C:
[paste the Gemini draft]

---

For each variant: 2-3 sentences of assessment against the criteria above.
Final decision: VARIANT [X] is the winner. 
Main reason for the choice: one sentence.

Result: The model will break down each variant against the criteria, explain the weaknesses of the losers and give a clear decision — one winner with a rationale. You get the best of the three answers plus an understanding of why the others fall short.

An LLM synthesizing averages rather than improves. When you ask it to combine three answers into one, the model looks for common ground and smooths over contradictions. But it's exactly the sharp, unusual elements of the "best" answer that most often disappear — they don't match the others. You get an averaged-out Frankenstein instead of the best of three.

Different models make different mistakes. Claude may be too cautious, GPT too generic, Gemini too formal. The odds that all three slip up in the same place at the same time are low. So the pool almost always contains one strong variant. The judge's job is to find it. Synthesis destroys that strong variant; the judge preserves it.

Control levers: - Criteria for the judge → tailor them to the task: "specificity" and "persuasiveness" for sales, "accuracy" and "completeness" for analytics. The more concrete the criterion, the more precise the choice - Final decision format → add "give the top 2 and explain the gap" — if the gap is small, the variants are interchangeable - Diversity of variants → the more the generation prompts differ, the better the odds that one turns out much stronger

You are a judge. Your task: pick the best {content type} 
out of the {number} provided. Do NOT merge the variants into one.
Do NOT rewrite. Just pick the winner.

Evaluation criteria:
- {criterion 1}
- {criterion 2}
- {criterion 3}

---
VARIANT 1:
{variant 1}

---
VARIANT 2:
{variant 2}

---
VARIANT 3:
{variant 3}

---

Assessment of each variant: 1-2 sentences against the criteria.
Winner: VARIANT [N].
Reason for the choice: one sentence.

What to plug in: - {content type} — email, strategy, product description, argument - {number} — how many variants you're comparing (2-5; more and the judge loses focus) - {criterion 1-3} — concrete qualities that matter for your task - Variants — from different models, or one Claude with different prompts

🚀 Quick start — paste into the chat:

Here is a judge template for picking the best LLM answer.
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about the content type, the number of variants and the criteria — because without them it doesn't know what counts as "good" in your context. It will take the structure and adapt it to your task.

⚠️ A homogeneous team gives no boost: If your three variants are three requests to the same Claude with similar prompts, a judge won't help. All three will be nearly identical — there's nothing to choose from. You need real diversity: different models or fundamentally different instructions.

⚠️ Majority voting barely works: When each agent votes for the best variant itself, the result is close to random. You need an external judge who reads everything and makes an independent call.

⚠️ For simple factual tasks the effect is minimal: "Give me 5 synonyms for a word" — all the variants are roughly equal, and a judge won't add anything. The method works where answer quality really varies: strategic decisions, complex texts, unconventional tasks.

⚠️ The weak-model paradox is still just a research finding: Adding a weak model to the team improves the result and lowers the cost — an intriguing result, but not confirmed by other studies. Don't go out of your way to include "bad" answers — wait for replications.

A researcher from Uber set out to settle a debate: one camp says "diverse model teams are better", the other says "identical models are better". Are both right? Artem Maryanskyy suspected the contradiction arose because both sides used synthesis as the aggregator — and nobody tried swapping in a judge.

The design is simple and rigorous: 5 comparison cells, 42 tasks from 7 categories (code, ethics, math, reasoning, science, summary, creative). Teams: three Claude Opus, or Claude + GPT + Gemini, or Claude + Gemini + a weak Haiku. Aggregators: a judge panel of three models, majority vote of the agents, synthesis by a single model. 210 experimental runs in total.

The most surprising result: synthesis lost even to a single model on 82% of tasks (not "worse" — much worse). That contradicts the intuition that "more voices = better". The consistency of the findings was confirmed by an independent judge panel built from different model families: the rank order held completely (Spearman ρ = 0.90). The practical takeaway for choosing a pipeline: the quality of the selector matters more than the makeup of the generator team.

🔧 Technique: one Claude, three instructions → diverse variants

No access to three different models? You can create "synthetic diversity" through radically different roles:

Give three fundamentally different answers to the question: {question}

VARIANT A — from a maximalist: fast, bold, 
with concrete steps, no caveats.

VARIANT B — from a skeptic: what could go wrong, 
which risks are being overlooked, what needs checking first.

VARIANT C — from a systems thinker: structure, 
priorities, dependencies, long-term consequences.

After the three variants: which one is closest to optimal 
for a typical situation — and why that one?

These aren't identical variants — the roles create real perspectives with different emphases. At the end the judge chooses not among "the same thing three times" but among three genuinely different approaches.

🔧 Technique: a weighted judge instead of an abstract "best"

If the criteria aren't equally important, say so explicitly:

Evaluate on three criteria with different weights:
- Likelihood the client accepts (weight 50%)
- Speed of implementation (weight 30%)
- Cost (weight 20%)

Pick the winner by weighted sum.

This is especially useful for strategic decisions, where "best" depends on priorities.

Paper title: When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines

Author: Artem Maryanskyy (Uber), [email protected]

Key references from the study: - Mixture-of-Agents (MoA): Wang et al. — the original synthesis architecture - Self-MoA: Li et al. — homogeneous model teams - LLM-Blender (Jiang et al.) — a two-stage pipeline with ranking - Condorcet's jury theorem — the historical foundation of collective decision-making - Hong & Page: "Diverse problem solvers can outperform high-ability solvers" — the starting point of the debate - Chatbot Arena (57,477 battles) — data on correlation within model families