TL;DR
How role-play works inside an LLM: When you phrase a role, an adverb ("work accurately") shapes the model's behavior more than an adjective ("an accurate worker"). Using activation patching — a technique that traces how information flows inside the model — the researchers found that role information is encoded in the early layers, almost independently of the task itself.
The headline finding: Swapping just three words in a role description changes the model's performance fivefold (nDCG goes from 0.1 to 0.5). For example, "expert assistant that ranks accurately" is more stable than "accurate expert assistant". Positive roles ("careful", "expert") give more predictable results than negative ones ("sloppy", "confused"). The reason: the role locks in early and directly shapes the final behavior, barely mixing with the content of the request.
The pecking order of role components: Testing the template "You are a/an {adjective} assistant that {modal verb} work {adverb}", the researchers found: the adverb carries the main signal (accurately, carefully), the adjective plays a supporting part (expert, reliable), and the modal verb (can, will) barely matters. That explains why focusing on the action ("rank accurately") beats focusing on status ("an accurate ranker").
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Example in Practice
⚠️ Method limitations: The study was run on document-ranking tasks. The effect may vary in other domains, but the principle of how to phrase a role is universal.
Task: Reviewing a contract before signing. You need to catch every risk and hidden trap — the job demands maximum attention.
Prompt A (status focus):
You are an experienced lawyer. Review this contract for risks.
[contract text]Prompt B (action focus):
You are a lawyer. Read every clause carefully, hunt for hidden risks, check the wording for ambiguity. Here is the contract:
[contract text]Result: Prompt B produces a more detailed analysis with specific notes on the wording. The model focuses on the review process (read carefully, hunt, check) instead of just the "experienced lawyer" label. You get a clause-by-clause breakdown with the potential risks flagged in each one.
Why It Works
LLM weakness: Abstract roles ("seasoned expert") leave the model too much room for interpretation. The same role can play out differently depending on context.
LLM strength: Models are great at following concrete instructions about actions. The more precisely you describe HOW to work, the more stable the result.
How it works: Role information is encoded in the model's early layers and barely interacts with the content of the request (query/document). So the role acts as a behavioral settingthat applies to everything that follows. The adverb pins THAT setting down ("work accurately"), while the adjective stays abstract ("an accurate worker" — and what does that mean in practice?).
Control levers:
- Adverbs — the main lever of role strength. Swap them for different working styles: "carefully" vs "quickly", "critically" vs "supportively"
- Spelling out actions — add verbs to amplify: "check every clause", "look for contradictions"
- Positive phrasing — positive roles are more stable. Instead of "don't be superficial" → "analyze in depth"
The Formula
Instead of an abstract role built on an adjective:
❌ You are a {adjective} {role}Phrase the role through concrete actions:
✅ You are a {role}. {Verb_1} {adverb_1}, {verb_2} {adverb_2}Before-and-after examples:
| Before (status) | After (action) |
|---|---|
| You are a critical analyst | You are an analyst. Analyze critically, look for weak spots |
| You are an attentive editor | You are an editor. Read carefully, check every detail |
| You are a creative copywriter | You are a copywriter. Write vividly, surprise with your phrasing |
How to use it: Plug in your role and 2-3 concrete actions with adverbs. The adverbs should capture HOW exactly the work needs to be done.
Limitations
⚠️ Research domain: The effect was tested on document-ranking tasks (IR). In other domains (text generation, analysis, code) the difference may be less dramatic.
⚠️ Sensitivity to wording: Even positive roles can produce different results depending on the exact words. Negative roles ("inexperienced", "slow") behave unpredictably — sometimes even better than the baseline prompt with no role.
⚠️ Language specifics: The study was done in English. In other languages a literal "assistant that works accurately" construction can sound clunky. Carry over the principle instead: focus on the action via verb + adverb.
How They Studied It
The researchers built a role template with three swappable slots: {adjective} {modal verb} {adverb}. For example: "You are a talented assistant that can rank passages carefully". They generated 300 variants (10 adjectives × 3 modals × 10 adverbs), split into positive ("expert", "carefully") and negative ("confused", "poorly").
They tested on LLaMA-3.1-8B-Instruct on document-ranking tasks (MS MARCO, TREC DL 2019). Quality was measured with nDCG@10 — how well the model puts relevant documents into the top 10.
Key method — activation patching: They ran the model twice — with a positive role ("expert") and a negative one ("confused"). Then they swapped activations (the model's internal states) from the positive run into the negative run at different layers. If the swap restored quality, that layer/component encodes role information.
Surprising finding: The role is encoded in the early layers (0-13 out of 32) and barely interacts with the query and document. In other words, the role works separately from the task — it sets a general behavior pattern that then gets applied to any content.
Component hierarchy: They patched each token type separately (adjective, modal verb, adverb). The adverb gave the biggest quality recovery, the adjective a medium one, the modal verb almost zero. That shows the adverb carries the main role signal.
The effect was checked on other models (Mistral-7B, Qwen2.5-7B) and another dataset (Natural Questions) — the pattern held, although the exact interaction layers differed slightly.
Resources
How role-play shapes relevance judgment in zero-shot LLM rankers
arXiv preprint, 2025
Research code
Authors: Yumeng Wang, Jirui Qi, Catherine Chen, Panagiotis Eustratiadis, Suzan Verberne
Leiden University, University of Groningen, Brown University, University of Amsterdam