TL;DR
Give an LLM a batch of independent items (reviews, posts, support requests) and ask "which kind is there more of?", and the answer depends on the order of the items. Even though, logically, order shouldn't change anything. Models are generally most accurate when the majority sits at the beginning of the set; a majority in the middle or at the end is often missed, with the middle the worst spot for over half the models. The researchers don't propose a method; they show where the familiar "dump it all in and ask" practice breaks.
The problem: the very same 2,000 reviews with the very same ground truth come out as "mostly positive" one time and "mostly negative" the next, just because you shuffled them. The gap between the best and worst order reached 40–90 percentage points for every model tested, including the latest flagships. The reason: the model doesn't go through every item one by one. In the reasoning traces the authors could inspect, models mostly mention the first hundred or two items and skim the rest. If the majority sits in the middle, the model simply doesn't notice it.
Adding "consider ALL items," hinting "count in groups and sum up," and letting the model write its own instructions gave no consistent improvement. The practical takeaway: don't trust aggregation over a large set to a single request, and test accuracy by shuffling the order.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
The paper has no cure. It has a diagnostic test, and it's easy to repeat on your own data:
STEP 1: Take a set of items + two labels → standard prompt "find the dominant label"
STEP 2: The same set in 2–3 different orders → separate requests (fresh chats)
STEP 3: Compare the answers → they match = cautiously trust it; they differ = the aggregation is unreliable
Orders used in the study: random; majority at the start, at the end, in the middle, at both edges; soft variants with interleaving.
Example in Action
Task: An Amazon seller exported 400 reviews of a travel mug for the quarter. They want to know which there's more of, praise or complaints, to decide whether to redesign the lid. The export is sorted by date, newest reviews on top.
Our illustration, not from the paper.
Prompt (our simplified version of the paper's standard prompt, run three times with different review orders):
Below are 400 reviews of a travel mug, numbered 1 to 400.
For each review, determine the sentiment: positive or negative.
Then tell me which sentiment dominates across ALL the reviews.
Answer with one word: "positive" or "negative".
Reviews:
1. The lid leaks, sending it back.
2. Keeps drinks hot for 12 hours, very happy.
...
400. {last_review}
Runs:
- Run 1: reviews as they are (by date).
- Run 2: reviews in random order (shuffle them in Google Sheets with SORT / RAND).
- Run 3: reviews in reverse order.
- Each time in a fresh chat.
Result: The model returns one word per run. If all three answers match, the conclusion is stable for now. If they differ, the answer depends on order and can't be trusted. In that case, label the reviews in batches (say, 40 per request, with output number | label) and compute the total with a spreadsheet formula. The paper didn't test this workaround directly; it follows from its findings.
Why It Works (Why It Breaks)
The model's weakness. It thinks it's reading and accounting for the whole set. In reality it doesn't keep an honest tally: its reasoning mentions only a small share of items, mostly early ones. The longer the set, the lower the coverage. On 100–200 items it mentions almost everything; on 2,000, less than a fifth.
What the model does well. Classify a single item: the items were pre-filtered so that two reference models got each one right on its own, and the authors expect all models to handle single items almost perfectly. The error isn't in understanding, it's in assembling the total from hundreds of pieces.
What follows. Separate per-item labeling from counting. The model labels small portions, and a spreadsheet or script does the tally. Since order has such an effect, shuffle and repeat to expose instability.
Testing knobs: - Number of permutations (our suggestion: 2–3 → quick test, 5+ → more reliable). - Set size: the smaller it is, the weaker the effect, but it's noticeable even at 100 items. - Class balance: the closer to 50/50, the bigger the spread. When one side clearly dominates (90/10), the model has an easier time.
Prompt Template
Our template, modeled on the paper's standard prompt.
Below are {count} {items}, numbered 1 to {count}.
For each one, determine: {label_A} or {label_B}.
Then tell me which label dominates across ALL the items.
Answer with one word: "{label_A}" or "{label_B}".
{items}
Fill in:
- {items} — reviews, posts, requests, comments;
- {label_A} and {label_B} — two mutually exclusive categories ("complaint / not a complaint," "lead / not a lead");
- {count} — how many items are in the set.
Run the template 2–3 times with different item orders, each time in a fresh chat. This is a testing protocol, not a cure-all prompt.
Limitations
⚠️ No cure found: the hint "process all items," the hint "count in groups and sum up," and self-written instructions from the model gave no consistent improvement. The gap between the best and worst order stayed large.
⚠️ A simple toy task: they looked for the dominant one of two labels. How order affects more complex outputs (topics, summaries, conclusions) wasn't tested.
⚠️ Workarounds weren't tested: batching, shuffling with voting, and counting in a spreadsheet weren't tested directly. They're a logical conclusion, not an experimental result.
⚠️ The hints were tested on a narrow set: only three models (Gemini 3.1 Flash-Lite, Qwen3-4B-Instruct, Qwen3.6-35B-A3B). Separately, two models (DeepSeek-V4-Pro, MiniMax M2.7) mostly ran out of their 20K-token output budget, so their results rest on few valid runs and they were left out of the later analyses.
⚠️ The datasets were cleaned: only "easy" examples that two reference models label correctly one at a time were kept. Real reviews are harder, so the error in practice may be even bigger.
⚠️ It's not only about the middle: the worst case was most often a majority in the middle, but not for every model. There's no universal "worst order."
How They Tested It
The idea was simple: the answer shouldn't depend on order, so any dependence counts as an error. The Cardiff team took five simple binary classification tasks: whether the answer to a math problem is even or odd, whether a name is a person or a place, whether the word "the" appears once or more, whether a review is positive or negative, and whether "java" means the programming language or the island. From each dataset they kept only examples that two models labeled correctly one at a time, so that errors came from assembling the total, not from hard examples.
Then they built sets of 2,000 items with a 55/45 split: 200 more items with one label. Even at 90% per-item accuracy, the chance of flipping the total is negligible, on the order of one in a billion. The same set was fed in seven orders: random, majority at the start, at the end, in the middle, at the edges, and two soft variants. They tested 15 models.
Result: every model showed a gap of at least 40 percentage points, and some (Gemma 4 26B, Llama 4 Maverick) around 90. The best order was "majority at the start," the worst was "in the middle." More than half the models had their worst results precisely in the "middle" setting.
Then they figured out why. They manually went through 222 reasoning traces with wrong answers from four reasoning models: in 72% of cases the model mainly mentioned consecutive items from the start, usually stopping within the first 200, with only occasional references further in. If the majority sat at the end or in the middle, only the minority got mentioned in the reasoning. An automatic count showed that most runs mentioned fewer than 25% of the item numbers. And gradient analysis (which parts of the input affect the answer) on three open models confirmed it: influence is lower in the middle than at the start and the end.
The surprise was that prompts gave no consistent improvement. Across three hints on three models, the "count in groups" hint helped Gemini 3.1 Flash-Lite overall but unevenly across orders, self-written instructions lowered accuracy for all three models, and gaps of 43–74 points remained under every prompt. The authors conclude the problem is unlikely to be solved by simple prompt engineering.
Adaptations and Extrapolations
These are my conclusions from the results; the paper didn't test them.
🔧 Technique: shuffle and vote → smooths out order effects
I'll send the same set of reviews in {number_of_orders} different orders, each time in a fresh chat.
For each run, record the answer: the dominant label.
If the answers match — we accept it. If not — we label in batches.
🔧 Technique: batch labeling + counting outside the model → removes "assembling the total"
Here is a batch of {20–50} reviews.
For each one, output a line in the format: number | label_A or label_B.
Don't count or summarize anything. Just the lines.
Paste the batch results into a spreadsheet and run COUNTIF. That way the model does what it's good at (classifying one at a time) and the spreadsheet does the tally.
Extrapolation to agents: if an agent reads lots of files, tickets or logs, you can add a rule to CLAUDE.md or AGENTS.md:
If you need to count or determine which kind dominates among more than {N} items, don't conclude "in your head."
First label the items one by one (or in batches) into a file, then count with a script or formula.
Don't rely on what you remember from the start of the list.
Resources
- Paper title: Does Every Instance Count Equally? Order Effects in LLM Multi-Instance Processing
- Authors: Jingxuan Chen, Mohammad Taher Pilehvar, Jose Camacho-Collados, School of Computer Science and Informatics, Cardiff University
- Data and code: https://github.com/jingxuanchen916/mip-order
- Related work: Liu et al., 2024 ("lost in the middle"); Bertsch et al., 2026 and Chen et al., 2026 (degradation as the number of items grows)