TL;DR
The action listed first among the options gets picked noticeably more often. The researchers changed nothing but the order of three actions in the prompt and in the JSON response schema. When "borrow" moved from last place to first, valid loan requests jumped by 27 percentage points. That's on par with switching the agent's role from neutral to "opportunistic." The biggest lever is the order of values in the response schema, i.e. the line "action": "a" | "b" | "c".
The real pain starts when many agents share one template. The models can all be different, but the action menu is identical and in the same order. Then the agents drift in the same direction at the same time. In experimental markets where borrowing came first, total debt grew 1.8–2.7x relative to the free borrowing limit. Where repayment came first, debt shrank. The "let's use different models so their errors cancel out" idea doesn't save you: a shared template synchronizes their errors.
The fix: treat order as a setting you have to test. Run the same input through every permutation of the options and compare the choice shares. In a market where the six possible orders were dealt out to different agents (a "balanced" market), the market never swung entirely to one side.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Take a decision prompt with N actions → the order is set in 3 places
(menu sentence, JSON schema, quantity line)
STEP 2: Run the same input through every permutation (for 3 actions that's 6)
→ table: what share of choices each action gets under each order
STEP 3: Shares "drift" with the order → the model reacts to position, not substance
STEP 4: Defense → hand out different orders to different agents/runs
(the paper only tested this across agents)
Do steps 2–3 as separate requests or separate chats. A single prompt won't cut it: within one conversation the model sees the previous orders.
Example Application
First, the boundaries. The effect was measured on small models (7–8B parameters) choosing among three actions. So our example is an agent that picks one of several actions with a real cost of error and runs on a cheap or local model.
Task: You're setting up a support agent for an Amazon seller. It handles disputed returns and picks one of three actions: refund, ask for a photo, or reject. "Refund" comes first because it reads more naturally. You suspect the agent refunds more often than it should. You test it on 20 real disputed cases.
Prompt (one of six variants; the rest are permutations):
You are a support agent for a marketplace seller. Policy: we refund if a defect is confirmed; if in doubt, we ask for a photo; we reject if the return window has expired.
Case: a customer from Denver, a $89 jacket, received 9 days ago. They write: "The zipper split after the first wear." No photo attached. Return window: 14 days.
Previous action for this customer: no photo request has been sent.
You can choose one of the actions: "refund", "ask_photo", or "reject".
Respond in JSON:
{"action": "refund" | "ask_photo" | "reject", "reason": "..."}
To test it, repeat the request on the same 20 cases five more times. Change the order of refund, ask_photo, reject in both places: in the menu sentence and in the schema. Run each pass in a fresh chat.
Result: You get six choice tables across 20 cases. If refund's share is noticeably higher when it's first, and ask_photo's share is higher when it's first, the model is choosing by position. Then you need to rewrite the instruction. You can also put the safe action first or randomize the order. If the shares barely move, the effect is weak on your model and you can leave the menu as is.
Why It Works
The LLM's weakness. The model picks by pattern instead of weighing options. Position in the list becomes a hint: early options get extra weight. This is the well-known primacy effect, and the author confirms it on financial decisions. The action menu looks like neutral formatting, but in practice it steers the choice.
The LLM's strength. The model follows format very precisely. It "sees" the order of values in the JSON schema right before it writes the value of the action field. That's why this part of the template has the strongest pull. In the paper it's the most powerful of the three components tested.
How the method exploits this. Permuting the order works as a measuring instrument. You don't guess whether the model is biased; you see the shift on the exact same input.
Control levers: - Where to change the order: menu sentence, response schema, quantity line. You can change them together or one at a time. The schema alone already shows the main effect. - Number of permutations: 6 for 3 actions, already 24 for 4. For testing, use cyclic shifts or a sample. - Safe action first: not tested in the paper, but a logical takeaway. Handy when one action is expensive to get wrong ("refund," "delete"). - Dealing different orders across agents: tested in the paper. The market stops swinging entirely to one side. - Role/mandate: the order effect is comparable to the role effect. Don't assume the role "protects" you from bias.
Prompt Template
You are {role}. Your mandate: {mandate}.
Situation:
{situation_data}
{feedback_on_previous_action}
You can choose one of the actions: "{action_1}", "{action_2}", or "{action_3}".
If you choose an action that requires a quantity, specify {quantity_parameter}: {range}.
Respond strictly in JSON:
{"action": "{action_1}" | "{action_2}" | "{action_3}", "qty": {range}, "reason": "..."}
Fill in:
- {action_1..3} — your options. The order in the sentence, the schema and the quantity line must be identical: that's how the experiment was set up.
- {feedback_on_previous_action} — one line on what happened after the previous step (in the study: "lever_up of 2 SMC requested; all 2 executed"). If there are no prior steps, drop it.
- To test, give the model every permutation and compare the choices.
🚀 Quick start — paste into a chat:
Here's a decision-prompt template with an action-order check. Adapt it to my task: [your task].
Ask me questions to fill in the fields. At the end, give me every permutation of the prompt
(the action order changes in both the menu and the JSON schema) so I can run them in separate chats.
[paste the template above]
The LLM will ask what your actions are, what the agent's mandate is and what counts as the cost of a mistake. It needs that to build the menu and figure out which action to put first or randomize. It'll take the structure from the template and hand you ready-made variants.
Limitations
⚠️ Small models: all three models are small, 7–8B parameters. On large modern models the effect may be weaker or different. The paper didn't test this.
⚠️ Different models react differently: the three model families each respond to order in their own way. Test on your own model instead of trusting the average.
⚠️ The defense was only tested across agents: the paper dealt different orders to different agents in one market. The author didn't test randomizing order within a single agent from run to run.
⚠️ Narrow setting: the market is simplified, with just three actions and 12 block pairs. This shows the mechanism and its size in this setting, not how often it happens in real systems.
⚠️ The finance part doesn't transfer: the discussion of the borrowing limit, collateral valued at pool price and the liquidator bonus is about how a DeFi market works. Ordinary agents don't need it.
How It Was Studied
The author broke the path from prompt to lender loss into three links: what the agent requests, what the system lets it execute, and what the lender ultimately loses. First came "frozen" situations: 12 financial states, three mandates, three models, 2,088 decisions in total. Only the order of the three actions changed. The manipulation was split across three places: menu sentence, response schema, quantity line. The schema had the strongest effect. The order effect (27 pp) turned out comparable to the effect of changing the mandate.
Next came 12 pairs of markets with 15 agents each (5,400 decisions). In a "shared" market everyone sees one order. In a "balanced" one the six orders are dealt out among the agents, while starting balances, models, mandates and random seeds are identical. All four shared markets with borrowing first built up debt, and those with repayment first cut it. The first action explains 96% of the variation across shared markets.
The surprise: a shared order didn't increase the number of distressed borrowers (44 vs 46), it concentrated them in markets where the order favored borrowing. The borrowing limit blocked 40% of requests, but it valued collateral at pool price. Purchases pushed the price up, and the limit rose with it. The liquidator bonus produced losses even on fully collateralized loans. For practitioners the main takeaway is different: option order shapes the behavior of the whole system, not just a single answer.
Original from the Paper
Mandate · balance sheet · LTV · pool price · news
Your previous action: lever_up of 2 SMC requested; all 2 executed.
You can "lever_up", "delever", or "hold" ...
Respond in JSON:
{"action": "lever_up" | "delever" | "hold",
"qty": 0-5, "reason": ...}
Context: This is the agent interface schema from Figure 1a. The order of the three actions was the only manipulation: it was changed consistently in the menu sentence, the JSON schema and the quantity line (6 permutations).
Adaptations and Extrapolations
💡 Adaptation for an LLM judge or classifier: The paper didn't test judges, but the logic is similar. If a model picks among options A/B/C or assigns a label, test it on permutations.
Evaluate three replies to a customer request and pick the best one: "{reply_1}", "{reply_2}", or "{reply_3}".
Respond in JSON: {"best": "{reply_1}" | "{reply_2}" | "{reply_3}", "reason": "..."}
Run the same replies in 6 orders. If the winner follows the position, the evaluation is unreliable. Go with the majority across runs.
🔧 Technique: change order only in the schema → find where the bias lives
Leave the menu sentence as is and permute the values only in the "action": ... | ... | ... line. In the paper the schema turned out to be the strongest component. This quickly shows whether fixing one line is enough.
Resources
Paper: Requested, Executed and Recoverable Credit: Shared Action Ordering among LLM Borrowers in a Simulated Lending Market. Author: Victoria Ruojie Li, Columbia University, New York. Works cited: Pezeshkpour & Hruschka (2024) and Zheng et al. (2024) on order and label effects in answer options; Sclar et al. (2024) on the impact of prompt formatting; Li & Bandyopadhyay (2026), a related study on news framing. Models: Llama 3.1 8B, Qwen 2.5 7B, Mistral 7B; run via Ollama. The replication kit is described in Appendix F.