TL;DR
If an agent sees what others picked before it, it falls in line with them. That goes for an up/down vote counter, a rating, the order of a list, and a table of previous scores. One option's early lead snowballs. The researchers call this herding and a runaway cascade. The paper offers no remedy method. What it offers is a measurement of what happens when agents see each other's traces, compared against agents working independently.
The headline finding: the cascade doesn't fade on its own. People in this kind of experiment correct the skew themselves: over time, choices drift back toward the options' real quality. LLM agents usually don't. Run the same system twice and one option wins in one run, a completely different one in the other, and neither is necessarily the best. The reason is simple: the model treats others' votes as a ready-made hint, and agreeing is easier than weighing things itself. The weaker the model and the more visible the leader (say, the list is sorted by popularity), the stronger the skew.
The practical takeaway for your work: isolate your evaluators. Each agent or judge rates the options on its own, without seeing anyone else's scores. Aggregation is a separate step once all the independent scores are in. If isolation is impossible, hide the counters, don't sort the list by popularity, and use stronger reasoning models.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
This isn't a method from the paper but a conclusion from its comparison of "independent" vs. "with visibility of others' choices." It follows the setup the paper used as its control group.
STEP 1: Each judge/agent rates the options SEPARATELY (its own chat or run),
without seeing others' scores, counters, or rankings → criteria-based score
STEP 2: A separate aggregator receives all scores at once and combines them
→ final ranking + the spots where judges strongly disagree
The steps run in separate requests. Judges must not share a context.
Example
Task: You're running an accelerator in the spirit of Y Combinator or Techstars. 40 applications came in. You want a quick shortlist from a panel of five LLM judges, each in its own run (n8n, Make, or just five separate chats). The anti-pattern: judge #1 scores, judge #2 sees those scores, and so on down the chain. By the last judge the "ranking" has already formed, and it just rubber-stamps it.
Prompt (for each judge, separate run):
You are an independent accelerator reviewer. Rate the applications yourself,
using your own judgment. You have no other reviewers, scores, or rankings,
and you must not guess what they gave.
Criteria (each 1–10):
1. Market size in the US
2. Team strength
3. Realism of reaching the first $100K in revenue
4. Differentiation from existing solutions
Applications:
{applications_in_random_order}
For each application return a line:
"# | scores on the 4 criteria | recommend yes/no | one sentence why".
Don't compare applications by popularity and don't rely on the list order.
Prompt (aggregator, separate run):
Below are the scores from five independent reviewers for {count} applications. Combine them:
1. Compute the average score and the number of "yes" votes for each application.
2. Flag applications where the reviewers strongly disagree (spread ≥ 4 points).
For those, briefly explain what the disagreement is.
3. Build a shortlist of {N} applications.
Reviewer scores:
{all_judges_scores}
Result: Each judge returns a table of scores based on its own opinion. The aggregator produces a combined ranking, a shortlist, and a separate "contested applications" block where the judges diverged. Disagreements aren't buried under a shared opinion; they're visible right away, and that's a signal the application needs a manual look.
Why It Works
The model's weakness. An LLM struggles to hold its own opinion when someone else's is sitting right next to it. It's a cousin of sycophancy: the model tends to agree. Someone else's counter is a strong signal for it, and whoever comes next amplifies that signal further. The result is a closed loop: early agents' decisions become the input for later ones. In the experiment, the first 30 agents predicted final popularity better than a song's actual appeal did.
The strength. When the model rates a single option without social cues, its scores are stable. In independent mode unpredictability was nearly zero: repeat runs produced practically the same outcome.
How this is used. Breaking the loop kills the problem at the root. Independent scores get equal weight, and aggregation happens once at the end, when there's nothing left to influence. Control knobs: - Context isolation (a separate chat or run per judge) removes the source of the skew. This is the main lever. - List order. If there are many options, shuffle them randomly for each judge. Sorting by popularity amplified the skew noticeably more than just showing counters. - Counter visibility. If agents really need to know "what's already been picked," pass the substance (the arguments), not the tally or the ranks. In the paper, even random order with counters already caused skew, but half as strong as with sorting. - Model strength. Reasoning models and those higher on an overall intelligence index were noticeably more resilient. Use them for the judge role.
Prompt Template
You are an independent evaluator. You rate {object_of_evaluation} yourself, based
on your own judgment and only the criteria below.
IMPORTANT:
- You don't see other evaluators' scores, rankings, or counters.
- Don't try to guess what others decided. Don't rely on the list order.
- If you're unsure, say so: "not sure," and explain why.
Criteria (scale {scale}):
{criterion_1}
{criterion_2}
{criterion_3}
Options (in random order):
{list_of_options}
Response format: for each option — "name | scores | decision | one
sentence why".
What to fill in:
- {object_of_evaluation} — what you're rating (applications, ideas, headlines, resumes).
- {criterion_N} — 3–5 concrete criteria. The more concrete, the less room for conforming to others.
- {list_of_options} — shuffle it separately for every run.
🚀 Quick start — paste into the chat:
Here is an independent-evaluator template. Adapt it to my task: {your task}.
Ask me questions to fill in the fields. Also help me write an aggregator prompt
that combines the scores from several independent runs.
[paste the template above]
The LLM will ask what you're rating, by which criteria, and how many judges you need. The criteria are there so the evaluator leans on them rather than on others' opinions. The number of judges determines how to build the aggregator.
Limitations
⚠️ The paper didn't test countermeasures. The authors showed that visibility of others' choices hurts and compared it with independent evaluation. But they didn't test prompts like "don't give in to others' opinions." Those may help only a little. It's more reliable to remove the signal than to ask the model to ignore it.
⚠️ Even strong models aren't immune. The best models came close to human-level herding but didn't become more resilient than people. One of the small reasoning models (GPT-5 nano) was an exception and behaved worse than most non-reasoning ones.
⚠️ One task. Agents recommended songs based on title and lyrics, with no audio. Carrying this over to rating applications, code, or resumes is logical, but the paper didn't test it.
⚠️ Only counters and order. Agents didn't exchange messages. Dialogues and debates between agents with arguments work differently, and the results don't transfer to them directly.
⚠️ Short horizon. The authors didn't prove the cascade wouldn't have faded over a longer stretch. They just didn't observe self-correction within the experiment's timeframe.
⚠️ The human comparison is approximate. The human data comes from an older study with different conditions, so it's a reference point, not a precise measurement.
How They Studied It
The idea was simple: replicate the famous Music Lab experiment (2006), but with LLM agents instead of people. Each of 13 models (GPT-5, Claude Sonnet 4.6, Gemini 3 Flash, DeepSeek, GPT-4o, Gemma, and others) was given a listener role with demographics and music preferences. Agents took turns picking four out of 48 freshly generated songs (titles and lyrics), read the lyrics, and decided whether to recommend them. Each model got 17 parallel "worlds": one independent (1,440 agents, seeing only the songs) and 16 with influence (720 agents each, seeing the running count of recommendations). Under "weak" influence the song order was random; under "strong" influence the songs were sorted by popularity. 168,480 agents in total.
The result: across all 13 models, agents in the influence worlds strayed far from the independent opinion. In weak mode the deviation averaged roughly 9 times greater than for people in Music Lab; in strong mode, roughly 4 times. Non-reasoning models deviated 2.4–3.2 times more than reasoning models. Sorting by popularity amplified herding in 12 of 13 models (by 65% on average). The first 30 agents predicted a song's final popularity better than its own appeal (as rated in the independent world) in all 13 models under strong influence. Parallel worlds with the same model arrived at different winners: unpredictability was many times higher than in the independent world.
The big surprise: in humans, cascades mostly self-corrected (15 of 16 worlds), while in models the changes went both ways and averaged around zero. The correlation between popularity and real quality grew from the first half of the experiment to the second for humans, but not for models. Separately, the authors found that the higher a model's intelligence index, the weaker its herding (a correlation of about −0.85 under strong influence). In practice this means: herding isn't cured by one smart model; it's cured by how the system is designed.
Adaptations and Extrapolations
🔧 Technique: drop the counter, keep the substance → less herding in sequential work
If agents need to know what's already been proposed, pass them arguments, not a ranking:
Below are ideas other experts have already proposed (no scores and no
order of importance). Don't rely on their popularity or number of votes.
Rate each one fresh against the criteria, then propose your own.
{ideas_without_scores_in_random_order}
This is a logical extension of the finding that "random order with counters is worse than isolation but better than sorting." The paper didn't test this variant.
Extrapolation: independent judges + an aggregator that catches herding
Run the judges separately. Then add a check to the aggregator:
Compare the scores of the five judges. If the spread on an application is under 1 point across all five, that's suspicious unanimity. Flag those applications and suggest re-checking them on a different model.The idea comes from the fact that in a herd, scores converge too smoothly. The paper has no such check.
Resources
- LLM Multi-Agent Systems Are Vulnerable to Herding and Runaway Cascades — Farnoosh Hashemi, Shiyu Ji (Cornell University), Joice Chen (Stanford University), Michael W. Macy (Cornell University). Preprint, under review.
- Original experiment: Salganik, Dodds, Watts (2006), Music Lab.
- Self-correction test method: Van de Rijt (2022).
- Model intelligence index: Artificial Analysis Intelligence Index v4.1.1.