TL;DR
The study exposes a fundamental asymmetry in multi-agent LLM systems: diversity beats quantity. Two agents with different setups (different models, roles or tools) consistently outperform 16 agents with identical configurations. The difference isn't the number of attempts but the number of effective channels: independent sources of information the system can draw on.
When you multiply identical agents ("try solving this task 10 times"), the first 2-3 attempts raise accuracy, then the curve breaks and each new attempt adds less and less. The reason: correlated outputs. Agents with identical settings generate similar reasoning, essentially duplicating each other. If the first agent solved the task algebraically, the other 15 will go the same way: the system gets not 16 independent views but 16 variations on the same logic. The information value of each new attempt tends toward zero.
Diversity breaks the correlation. Different models "see" the task differently, different roles focus on different aspects, different tools open up different paths to a solution. Instead of 10 algebraic solutions with minimal variation, you get algebra + geometry + a numerical estimate: three truly independent information channels. The authors formalize this with information theory and introduce the K* metric, a count of effective channels that works without knowing the correct answer.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Why it works
LLMs given identical inputs and identical settings generate statistically correlated outputs. That's not a bug; it's the nature of a probabilistic model: temperature, top-p and other decoding parameters add randomness but don't change the basic reasoning patterns encoded in the model's weights. If a model "sees" a task a certain way, 100 samples with different random seeds will give 100 variations on the same view.
An LLM's strength is its ability to follow structured roles and contexts. Switching models (Qwen → Llama → Mistral) changes the weight architecture and training data. Switching roles ("mathematician" → "engineer" → "skeptic") changes the focus of attention and the evaluation criteria. Switching tools (calculator vs search vs code) changes the available solution paths. Each change shifts the probability distribution over the space of possible answers: the agents stop correlating and start exploring different regions of the solution space.
The method uses this as follows: instead of "more attempts," the system gets "more independent perspectives." Information theory shows that the gain in knowledge about the correct answer is limited not by the number of requests n but by the number of uncorrelated information channels K. The information-recovery formula: 1 - e^(-αK), where α is the probability that a new channel covers a gap in knowledge and K is the number of effective channels. When agents are identical, K grows slowly (it saturates quickly because of correlation). When agents are diverse, K grows linearly with the number of agents: each adds an independent view.
Diversity control knobs:
- Models (GPT-4 + Claude + Gemini vs 3×GPT-4): maximum independence, different architectures "see" differently
- Roles/personas ("optimist," "pessimist," "pragmatist"): medium independence, shifts the focus of attention
- Tools (text only vs with search vs with code): strong independence, opens different paths
- Prompting strategies (Chain-of-Thought vs Tree-of-Thought vs a direct answer): changes the structure of the reasoning
The more independent your sources of diversity, the higher K for a fixed number of agents. Two different models give more diversity than two different temperature settings on one model.
Example: voting and debate
Task: Evaluate a startup's business idea: a nanny-matching service powered by an AI that analyzes candidates' video interviews and matches them to the parents' parenting style (questionnaire + a 15-minute call).
Option 1: Voting with identical agents
Prompt (10 times to the same model):
Rate the potential of this business idea on a 1-10 scale and explain why:
[idea description]
Give a score and 2-3 key arguments.
Result (the typical pattern):
Ten times over, the model gives scores in the 6-8 range with similar arguments: "the nanny market is big," "AI is growing," "but regulatory risks," "you need a candidate pool." The spread of opinions is minimal, and after the 3rd attempt there's almost no new information.
Option 2: Voting with diverse agents
Prompt (to different models / roles):
Role: {role}
Evaluate this business idea from your point of view (1-10 scale + arguments):
[idea description]
{role} = "Venture investor (focus: unit economics and scalability)"
{role} = "HR expert in the domestic-staffing segment (knows the market's pain points)"
{role} = "ML engineer (assesses the technical feasibility of video analysis)"
{role} = "Skeptical lawyer (regulatory and ethical risks)"
Result:
Four qualitatively different assessments: the investor focuses on lifetime value vs CAC and ticket size, the HR expert on the fact that parents don't trust algorithms with this kind of decision, the ML engineer on how hard it is to detect "soft skills" from video, and the lawyer on the GDPR risks of storing children's biometric data. Each channel covers its own layer, and together they give a 3D picture in 4 requests instead of a shallow consensus in 10.
Option 3: Debate between diverse agents
Prompt:
Two roles discuss a business idea over 2 rounds:
IDEA: [description]
ROUND 1:
- Enthusiast (looks for opportunities): rate the potential, give 2-3 strong arguments FOR
- Skeptic (looks for risks): rate the same idea, give 2-3 strong arguments AGAINST
ROUND 2:
- Enthusiast: read the Skeptic's critique and respond (where you agree, where you can push back)
- Skeptic: read the Enthusiast's response and adjust your score if needed
FINAL: Synthesis: an overall score (1-10) that accounts for both views.
Result:
The model simulates a debate: the Enthusiast lays out the case (big market + the personalization trend), the Skeptic pushes back (parents won't trust an algorithm to pick a nanny + a cold start with no candidate pool), the Enthusiast adjusts (start with candidate verification rather than matching), and the Skeptic softens (if it's positioned as an assistant rather than a replacement). The final score reflects mutual cross-examination of the arguments, which goes deeper than isolated voting.
How to apply it
1. Create diversity, don't multiply attempts
Instead of:
[the same request 10 times, hoping to "get lucky"]
Do this:
Agent 1 (role A): [request]
Agent 2 (role B): [the same request, a different role]
Agent 3 (role C): [the same request, a third role]
2. Sources of diversity (from most to least independent)
- Different models (GPT vs Claude vs Gemini): different weights = maximum decorrelation
- Different tools (plain text vs search vs a code interpreter)
- Different roles with a clear focus ("mathematician," "engineer," "critic"), not abstract "expert 1/2/3"
- Different prompting strategies (CoT vs a direct answer vs "think like X")
Different temperature: a weak source, barely changes the underlying logic
3. The number of agents depends on difficulty
- Simple tasks (facts, translation, formatting): 1 agent is enough, no diversity needed
- Medium tasks (analysis, comparison): 2-3 different views are optimal
- Complex tasks (multi-criteria evaluation, strategy): 4-6 agents with narrow roles
4. Voting vs debate
Voting: agents work in parallel and don't see each other's answers. Faster, but with no cross-checking.
Debate: agents see each other's arguments and adjust their own. Slower (more tokens) but deeper, especially when the task is subjective and involves trade-offs.
5. Verification via K *(advanced)
If you want to check whether you have "enough diversity," ask the model:
Here are 4 answers from different agents to a task:
[paste the answers]
Assess: how INDEPENDENT are these answers from one another?
- If they're nearly identical (different wording, same logic) → 1 channel
- If they partly overlap → 2-3 channels
- If each looks from a unique angle → 4 channels
Give an estimate of the number of effective channels (1-4).
If the model says "1-2 channels" with 4 agents, the roles are too similar and you need more diversity.
Limitations
⚠️ Simple tasks: On questions with a single answer ("capital of France?") diversity is useless: every agent will say "Paris," K=1 no matter how many you run. Diversity helps where there are multiple possible solutions or trade-offs.
⚠️ Cost: Different models = different subscriptions. If you only have access to one model, diversity is limited to roles and prompting strategies, and the effect is weaker than when you switch architectures.
⚠️ Role quality: Abstract roles ("expert 1," "expert 2") give weak diversity: the model simulates similar behavior. What works is a concrete focus: "the optimist looks for opportunities," "the pessimist looks for risks," "the pragmatist runs the ROI."
⚠️ Conflicting conclusions: When agents diverge sharply, the final decision needs human judgment: automatic voting may pick the popular answer rather than the best one. Debate partly solves this through mutual critique.
Resources
Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity
Yingxuan Yang, Chengrui Qu, Muning Wen, Laixi Shi, Ying Wen, Weinan Zhang, Adam Wierman, Shangding Gu
Shanghai Jiao Tong University, Caltech, Johns Hopkins University, UC Berkeley
https://github.com/SafeRL-Lab/Agent-Scaling