TL;DR
Ask a model to pick one correct answer out of 5 options, and it'll do it almost perfectly. Ask it to pick from 160, and accuracy drops by tens of percentage points, and not because the options turned into "too many letters" for the context. The model starts losing confidence in the correct answer as the list grows, not because the wrong options look more convincing. And there's a second problem: the model latches onto one of the first options it sees, and the longer the list, the harder it is to get it to change its mind in favor of an option further down.
The researchers tested the standard explanation, "the model gets lost in a long context" (the lost-in-the-middle effect), and deliberately ruled it out: they gave the model a pile of irrelevant text without increasing the number of options, and no degradation occurred. So the problem isn't the length of the text but specifically the process of comparing many competing candidates. Even step-by-step reasoning (Chain-of-Thought) and self-critique don't help; the degradation continues.
The only thing that works is restructuring the comparison process: split the big list into small groups and compare them in a tournament bracket (Hierarchical Partitioning), or shuffle the order of the options several times and take the one the model picks most often (Permutation-based Inference). On a list of 160 options, both methods deliver +20 percentage points of accuracy compared with the usual "dump everything into one prompt."
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
Method A: Hierarchical Partitioning (tournament selection):
STEP 1: Split the list of N candidates into small groups (e.g. 10-15 each) → a separate request per group
STEP 2: In each group, ask the model "which option is best" → get one winner per group
STEP 3: Gather all the winners into a new (now small) list → repeat step 2 until a final answer remains
Each round is a separate request to the model.
Method B: Permutation-based Inference (voting across shuffles):
STEP 1: Randomly shuffle the order of the options in the list
STEP 2: Ask the model "which option is best" → record the answer
STEP 3: Repeat steps 1-2 several times (3-5 times) with a different order
STEP 4: Take the option the model picked most often (majority vote)
Each shuffle is a separate request.
Example in Action
Task: You're a marketer, and an AI generated 80 subject-line options for a promotional email. You need to pick the single best one, but dumping all 80 into one prompt and asking "pick the best" is a bad idea: the model will almost certainly choose something from the first ten simply because it saw those first, not because they're actually the best.
Prompt (Method B: shuffles):
Request 1:
Here are 80 subject-line options for a promotional email (in random order).
Pick the ONE best subject line in terms of click appeal and clarity of the message.
Answer with the option number only.
1. [subject line]
2. [subject line]
...
80. [subject line]
Request 2 (the same list, but with the options shuffled):
[the same prompt, but items 1-80 in a different order]
Request 3 (shuffle again):
[the same prompt, a third random order]
Result: In each request the model returns the number of one option. If the pick matches across all three requests, that's a reliable signal the option really is strong. If the answers differ, take the option chosen most often (2 out of 3), and don't trust a single run with 80 options in one prompt.
For method A (tournament): split the 80 subject lines into 8 groups of 10, ask the model to pick the best in each group in a separate message, then compare the 8 finalists in a final request.
Why It Works
A model compares a small number of options well; there, its confidence in the right answer is high and barely erodes. But the more competing candidates there are, the more the model's "confidence" in the correct answer melts away, which shows up in the internal probability scores the researchers measured directly. On top of that, an anchoring effect kicks in: the model "settles" on a favorite early, among the first options it sees, and the later the truly best option appears in the list, the less chance it has of dislodging that anchor.
Both fixes lean on what the model does well: comparing small sets. The tournament turns one big, muddled comparison into a series of small, precise ones. Shuffling removes position bias through averaging: if the model's "favorite" depends on what comes first, changing the order and voting across several runs washes that effect out.
Control knobs: - Group size in the tournament: a smaller group (5-10 options) → more accurate at each step, but more rounds and messages. - Number of shuffles in the vote: more runs (5-7 instead of 3) → a more reliable result, but more requests. - Combining the methods: you can split into a tournament first and also shuffle the order inside each group for extra reliability.
Limitations
⚠️ Doesn't work in a single pass: both methods require several separate requests to the model. It's not a prompt patch; it's a mini-process spanning several messages.
⚠️ Chain-of-Thought doesn't help: even if you ask the model to reason step by step or critique itself, the degradation keeps going as the number of options grows. Making a single prompt more elaborate doesn't solve the problem.
⚠️ Pointless on short lists: if you have 10-15 options, the degradation is minimal, and a tournament or shuffles will only add extra messages with no gain in accuracy.
⚠️ Not a complete fix: even with these interventions, accuracy on huge lists (160+) is still lower than on small ones. The methods ease the problem rather than eliminating it.
How They Studied It
The team took five open models (the Gemma, Llama, Phi, and Qwen families) and ran them through six different tasks, from re-ranking search results to medical diagnosis (MIMIC) and classifying scientific texts, gradually increasing the number of choices from 5 to 160 (and for one strong closed model, all the way to 640). The same picture showed up everywhere: the more options, the lower the accuracy.
Next, the researchers tested the most obvious explanation, "the model drowns in long text," as in the classic lost-in-the-middle effect. To do this, they separately padded the prompt with irrelevant text without touching the number of real options, and the degradation didn't reproduce. So the problem isn't the volume of text but specifically the process of comparing the candidates against each other.
To understand the mechanics, they measured the internal probability scores (log-probability) for the correct answer and for the strongest "wrong" competitor, and saw that it's the correct answer's score that falls fastest, not that the wrong answers become more convincing. They also separately checked the models' attention patterns and found that attention is distributed differently across models, while the tendency to choose early options is the same for all of them. That demolished the hypothesis that "the model simply pays less attention to options further down": the issue isn't attention but how the model makes its final decision.