Apply to my task
arXiv:2507.11538 ★ 92 Jul 15, 2025

The primacy effect: with long instruction lists, LLMs drop the later ones first

Key result

Under a long list of instructions, an LLM doesn't forget at random: it drops later items more often than early ones. In the study (up to 500 keyword instructions, 20 models) this bias was almost absent with few instructions, peaked around 150-200, and faded toward uniform failure at 300+. Controlling instruction order can help critical requirements survive when there are lots of instructions. The trick: put the most important things first: the role, the main goal, the key format requirement. Anything at the bottom the model may skip: as the count grows, errors are mostly outright omissions, not near-misses.

How Many Instructions CanLLMsFollow at Once?

Key Aspects of the Study:

The researchers tested how many simultaneous instructions modern LLMs can handle before their performance starts to drop. The test: write a business report that includes 10 to 500 specific keywords, one instruction per keyword. Every model degrades as the list grows, but in different ways: the best reasoning models (o3, gemini-2.5-pro) stay near-perfect up to 150 or more instructions, others decline steadily, and small models drop sharply after a few dozen. Even the best model reached only 68% at 500 instructions. Under heavy load, models also follow instructions near the start of the list better than those near the end (the "primacy effect"); this bias was almost absent with few instructions, peaked around 150-200, and faded again at 300+.

Key Result: When a prompt carries a long list of instructions, putting the most important ones near the top may help, because later items are dropped more often under load. With only a few instructions the study saw almost no order effect, and at extreme counts failures became uniform.


The Core of the Method, Explained:

The method that falls out of this study can be summed up as "managing the LLM's attention and cognitive load". Picture handing tasks to an intern. Give them 3 clear tasks and they will probably nail all of them. Give them a list of hundreds and even a strong intern starts dropping items, more often from the end of the list. LLMs behave in a similar way, though the breaking point depends heavily on the model.

Main takeaways and a playbook for users (the study tested only keyword-inclusion instructions; applying it to roles, goals and formats is our extrapolation):

  1. The "Important stuff first" rule (Primacy Effect): The instruction that matters most to you should sit at the very top of the prompt. It can be a role ("You are a seasoned marketer"), the core goal ("The main thing is to convince the reader to buy the product") or a format requirement ("Return the answer as a JSON object"). In long instruction lists, items that come later are more likely to be dropped.

  2. Know when to stop (Cognitive Load): Don't try to cram dozens of unrelated requirements into a single prompt. The more instructions you add, the higher the odds the model starts skipping them (under moderate load mostly the ones near the end; at extreme load more evenly). If your task is truly complex, consider splitting it into several sequential requests (our suggestion; the paper did not test splitting).

  3. Expect the right kind of mistake: As the instruction count grows, models mostly skip (omission) instructions rather than half-following them (in the test, using a variant of the required word). At low counts the two error types were more balanced, and reasoning models like o3 kept more near-misses. So when a long list is only partly done, check for missing items first.

  4. Watch the core task too: The study checked whether overload also hurts the main task (report coherence). For most models it did not, or only slightly; only o3 and o4-mini became markedly less coherent, partly because they wrote short reports packed with keywords. Trimming instructions you don't need is still sensible, but the paper does not show that fewer directions improve the text itself.


Practical Applicability Analysis:

  • Direct applicability: Extremely high. Anyone can improve their prompts right away just by reordering instructions. Instead of writing "Write a piece about the environment, be positive, use a bulleted list, and add a call to action at the end", the user writes: "Act as an environmental expert. Your main task is to write an upbeat piece with a call to action at the end. Use a bulleted list for the facts". With a short prompt like this the study found almost no order effect, so treat the reshuffle mainly as a habit that starts to matter once instruction lists get long.

  • Conceptual value: It gives users a simple, powerful mental model: "An LLM's attention is a finite resource that gets spent top to bottom through the prompt". That helps people stop treating the LLM as a magic box and start working with it as a system with clear rules and limits.

  • Adaptation potential: The paper tested only keyword-inclusion instructions in English business reports and notes the results may not generalize to other tasks or instruction types. Applying the idea to writing, code generation, data analysis or planning is our extrapolation. Adapting it is easy: before writing a complex prompt, ask yourself, "Which single instruction is absolutely critical to success? What comes second?" The answers define the structure of your prompt.


A Practical Example:

Our illustration, not from the paper.

Say a user wants an LLM to write a script for a short promo video (Reels/Shorts) about a new fitness app.

# TASK

You are an experienced screenwriter and creative director who specializes in short viral videos for social media (Reels, Shorts, TikTok).

Your main goal is to write a script for a 15-second video that makes viewers really want to download our new fitness app "FitGo".

---
# CONTEXT

*   **Product:** The "FitGo" mobile app.
*   **Key feature:** An AI coach that builds personalized workouts based on the user's goals and current fitness level.
*   **Target audience:** People aged 20-30 who want to start working out but don't know how, or find it boring.

---
# SCRIPT STRUCTURE AND REQUIREMENTS

1.  **Response format:** Present the script as a table with three columns: "Shot (visual description)", "Copy (voiceover or on-screen text)", "Timing (seconds)".
2.  **Tone:** Energetic and motivating, but not pushy. Avoid clichés like "change your life in one day".
3.  **Key message:** "FitGo is your personal trainer in your pocket that makes fitness simple and fun".
4.  **Must-have element:** The video must end with a clear call to action (CTA): "Download FitGo for free and start your first workout!".
5.  **Restriction:** Do not mention competitors.

Why It Works:

This prompt applies the ordering idea from the study (with this few instructions the study measured little order effect, so here the gain is mostly clarity):

  1. Primacy effect: The most important instructions sit at the top.
    • Role (You are an experienced screenwriter...): Sets the context and the quality bar. This is the first and most important instruction.
    • Main goal (...makes viewers really want to download...): Defines the end result the model should aim for. This is the second most important instruction.
  2. Managing cognitive load: Instead of one long list of requirements, they are grouped into logical blocks (CONTEXT, REQUIREMENTS). Key requirements are numbered, which also helps the model process them, and the most important ones (format, tone, CTA) lead the list.
  3. Clarity and prioritization: The prompt explicitly calls out the "main goal" and the "key message", helping the model see where to point its "attention" first. Less important details, like "do not mention competitors", sit at the end of the list, so even if the model ignores them, the core result won't suffer.

Another Practical Example

Our illustration, not from the paper.

Task: a user asks an LLM to help draft a polite but firm letter to a neighbor asking them to keep it down after 11 p.m.

# TASK

Act as an experienced conflict-resolution specialist and mediator.

Your main task is to help me write a letter to my neighbor. The main goal of the letter is to ask them, politely but very firmly and unambiguously, to stop making noise (loud music, shouting) after 11 p.m.

---
# KEY LETTER PARAMETERS

1.  **Tone:** Balance is everything. The letter must be **non-confrontational and respectful**, yet **confident and firm**. No passive aggression and no apologizing.
2.  **Structure:**
    *   Open with a friendly greeting.
    *   Describe the problem without blaming (e.g., "I've noticed that lately...").
    *   State the request clearly (stop the noise after 11 p.m.).
    *   Offer a solution or a conversation.
    *   End on a positive note.
3.  **What to avoid:** Do not use threats (calling the police, complaining to the landlord) in this first letter. Focus on a good-neighborly solution.

Why This Example Works, Mechanically.

This prompt works because it also applies the principles of managing LLM attention:

  1. Role and Goal come first: The prompt opens by assigning a role (conflict-resolution specialist) and stating the main goal (politely but firmly ask). That tunes the model in right away and sets the direction of the output.
  2. Focus on the hardest part: The trickiest part of the task is getting the tone balance right. That is why the tone instruction (non-confrontational, yet confident and firm) is first in the requirements list. Putting it first makes it less likely to be lost if the list grows long.
  3. Structure for clarity: Rather than just saying "write a polite letter...", the prompt breaks the task into clear structural pieces (greeting, problem description, request). That lowers the model's cognitive load and helps it work through every part of the assignment in order.
  4. Negative constraints at the end: The instruction about what not to do (Do not use threats) is placed last. It is a secondary constraint. If the model "forgets" it, that is far less damaging than forgetting the right tone. In other words, we deliberately gamble on a less important instruction to reliably land the more important one.

Usefulness Score: 92

02

Main Evaluation Criteria

  • A. Relevance to prompting techniques: Yes, the study directly analyzes how the number and position of instructions affect the result.
  • B. Better conversational answers: Yes, the findings help make answers more accurate and predictable when working with complex, multi-part requests.
  • C. Direct practical applicability: Yes, the key findings (especially the "primacy effect") can be used immediately in any chatbot without special tools or knowledge.
  • D. Conceptual value: Very high. The study gives users an intuitive grasp of an LLM's "cognitive load" and "limited attention", explaining why models "forget" or ignore some instructions.
  • E. New useful practice: The paper fits several clusters at once:
    • Cluster 2 (LLM behavioral patterns): The key finding is the "primacy effect", the model's tendency to follow instructions at the top of the prompt more reliably.
    • Cluster 3 (Prompt structure optimization): Shows directly that instruction order is critical.
    • Cluster 7 (Reliability and stability): Explains why reliability drops as the number of instructions grows and how to anticipate it.
  • Practicality checklist (+15 points): Yes, the study explains where to put important information in a prompt, how to structure complex requests, and reveals non-obvious quirks of LLM behavior.
03

Numeric Usefulness Score

The study earns a high score because its findings are fundamental and directly usable by anyone to improve their prompts. It doesn't hand you a mere "trick"; it gives you a conceptual understanding of LLM limits, which is far more valuable in the long run.

Arguments for a high score: 1. A direct call to action: The "primacy effect" finding is a golden rule of prompt engineering. The study puts science behind it and shows a systematic pattern across 20 models, strongest at around 150-200 instructions. Users can start putting their most important instructions first right away and get better results. 2. Explains "why it doesn't work": The paper explains why an LLM sometimes "ignores" part of your request. It's not the model being "dumb"; it's "cognitive overload". Understanding that helps users stop fighting the model and instead adapt their requests by splitting or simplifying them. 3. Broad relevance: The test used a specific task (including keywords), and the authors caution it may not generalize, but the ideas about attention and cognitive load are worth trying in other tasks, from essay writing to code generation.

Counterarguments (why the score could be lower): * An artificial test: The test task (fitting up to 500 keywords into a report) is pretty niche and rarely comes up in everyday use. People usually give instructions of different kinds (style, format, content limits), not just a list of words. So extrapolating the numbers directly (e.g., "the model breaks after 150 instructions") may not hold for other tasks. * No ready-made "templates": The study gives principles, not copy-paste prompt templates. Users still have to figure out for themselves which of their instructions matters most.


Research digest

Key result

Under a long list of instructions, an LLM doesn't forget at random: it drops later items more often than early ones. In the study (up to 500 keyword instructions, 20 models) this bias was almost absent with few instructions, peaked around 150-200, and faded toward uniform failure at 300+. Controlling instruction order can help critical requirements survive when there are lots of instructions. The trick: put the most important things first: the role, the main goal, the key format requirement. Anything at the bottom the model may skip: as the count grows, errors are mostly outright omissions, not near-misses.

Principle

Don't toss instructions in any old order; rank them by importance. The A→B→C approach: A: Role and main goal: whatever defines the whole result goes first. The model "sees" it before everything else. B: Key requirements: response format, tone, must-have elements. Second priority. C: Constraints and nice-to-haves: "don't mention competitors", "avoid clichés". If the model ignores these, it's not the end of the world. The rule is simple: the more critical the instruction, the higher it goes in the prompt. Knowingly risk the less important stuff to reliably land the important stuff.

Why it works

The study documented a "primacy effect": under heavy load, instructions in the last third of the list were violated more often than those in the first third. When overloaded, models mostly skip rather than half-follow. The tail bias is strongest at moderate-to-high load and fades when the model is completely overwhelmed. An LLM's attention is a finite resource that gets spent top to bottom. The main task mostly held up: report coherence stayed stable for most models, with o3 and o4-mini the exceptions. Fewer instructions means fewer chances for any one of them to be dropped.

When to apply

Prompts with long instruction lists (dozens or more); with only a few requirements the study saw almost no order effect. The study tested keyword instructions in business reports; writing, code generation, analysis or planning are our extrapolation. Especially critical for: — Complex tasks with format requirements (table + tone + structure + CTA) — Tasks where one missed element breaks the whole result (JSON format, legal wording) — System prompts with a long list of rules Ordering helps less at extreme densities (300+ instructions in the test); there, consider splitting the task into several sequential requests (our suggestion) or using a stronger reasoning model.

Mini recipe

1. Rank by criticality: before writing the prompt, ask yourself: "which single instruction matters most? what's second?". The answers define the structure. 2. Put role and goal first: not 'write a piece about X, and you're an expert in Y', but the other way round: first you're an expert in Y, then write a piece about X. 3. Group by importance: key format and tone requirements come right after the goal. Minor constraints go at the very end. 4. Trim the fat: if you have 10+ instructions, think about which ones actually affect the result. Our rule of thumb: drop the ones added only "just to be safe". 5. Check the tail: look at the last 2–3 instructions. If the model ignores them, is that a problem? If yes, move them up.

Examples

[BAD]: `Write a fitness post, be motivating, use a bulleted list, add a call to download the app, use plain language, no clichés, keep it under 150 words` [GOOD] (our example): `You're a copywriter for short motivational social media posts. Main goal: convince the reader to download the FitGo fitness app. Tone: energetic, no "change your life" clichés. Format: bulleted list of facts + a call to action at the end. Length: up to 150 words. Avoid jargon.` What changed: the role and goal moved to the front, so the model knows what it's aiming for before reading the rest. The "no clichés" constraint moved above the throwaway word-count line. With only seven instructions the study would predict little order effect, but the same habit protects the most important thing, convincing people to download, once the list grows long.

In plain words

Today's AI models act like know-it-alls, but in reality they're like a waiter at the dinner rush: order a coffee, a dessert and the check at once and he'll manage, but dump hundreds of conditions on him at once and he'll start forgetting orders. The study clearly shows that models have a cognitive load ceiling. In the test (a business report that must include up to 500 given keywords), the best reasoning models stayed near-perfect up to about 150 instructions before slipping, others declined steadily, and small models fell sharply after a few dozen. Even the best reached only 68% at 500.

It's like juggling chainsaws while singing the national anthem and riding a unicycle. Technically you're doing it all at once, but in practice either the song comes out mumbled or a chainsaw lands on your head. In the study it looked like this: as the list grew, models mostly skipped required words outright rather than half-using them, and under moderate load they skipped the later items in the list more often than the early ones. The report itself usually stayed coherent; it was the instructions that got dropped.

What the findings suggest: prioritization. Since later items get dropped first under load, put the instructions you can't afford to lose near the top; the authors note this helps less once the load is extreme. Model choice matters a lot too: reasoning models held up far longer, while gpt-4o and small models degraded early. Our own suggestion, not tested in the paper: if the list is huge, break the job into stages instead of stuffing every constraint into one prompt.

The test covered one task type only: simple "include this word" instructions in English business reports, and the authors warn it may not generalize. Still, the law of diminishing returns is a useful lens for code generation, scriptwriting or analytics too (our extrapolation). A handful of "don'ts" and "musts" is not where the trouble starts; long rule lists are, and each extra rule is another chance to be skipped.

Bottom line: stop playing criminal mastermind and stuffing the model with endless instructions in a single message. Fewer conditions, higher accuracy. If you want a reliable result, put what matters first, consider a step-by-step approach, and don't make the model juggle too many balls. Otherwise you won't get a masterpiece, you'll get a mediocre brush-off that technically has everything but is useless in practice.

AI lab assistant

Apply this study to your task

Describe what you're working on. AI takes the method and the measured results from this paper and adapts them to your case, or writes a ready prompt built on them.

Based on The primacy effect: with long instruction lists, LLMs drop the later ones first arXiv:2507.11538

Sign in to apply this study New accounts get 50 free credits.

Get the week’s best papers by email

One email on Sundays with the highest-rated new techniques. Unsubscribe with one click.