TL;DR
TIPER is a method that automatically finds the user role (persona) that makes a specific agent fail most often. It works in a loop: run dialogues → see where the agent tripped → rewrite the persona to press on that weakness → verify the task is still solvable → keep the best variants. A persona is a short instruction along the lines of "how the user behaves in a conversation," not tied to specific orders or dates.
The key finding: the standard "user" in tests is an LLM that's polite by default, answers to the point, and helps the agent along. Real people don't do that: they reply in fragments, rush you, wander off topic, give information in the wrong order. Against such a "convenient" user, the agent looks more reliable than it really is. With the tuned personas, agent success dropped in every case tested. For end-to-end scenarios where the task must be done without a single mistake, the drop reached tens of percentage points.
The second finding: not every failure is the agent's weakness. If a "difficult" user changes their goal or mixes up facts mid-conversation, the task becomes unsolvable, and that's a test error, not an agent error. TIPER throws out such dialogues with a "task integrity" check: the user doesn't change the goal, facts, or constraints; only the communication style changes.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 0: Base persona p0 = "an ordinary cooperative user"
STEP 1: Run agent dialogues with the current persona → transcripts + success score
STEP 2: Reflection: an LLM analyzes the failures → finds the behavior that hits a weakness
STEP 3: Mutation: an LLM rewrites the persona in that direction → new persona
STEP 4: Text filters: cut vague ones, task-specific ones, and duplicates
STEP 5: Integrity gate: the persona lowers success AND failures aren't due to a "broken" task
STEP 6: Per-task selection (Pareto): keep personas that break DIFFERENT tasks
STEP 7: Repeat for N generations → pick the best on a separate split → final test on held-out tasks
In the paper this is code with hundreds of runs. You can reproduce the idea of the loop by hand in a chat (see below).
Example Application
Task: You're launching an AI support bot for a grocery delivery service (think Instacart or DoorDash): canceling an order, refunding missing items, rescheduling a delivery slot. In testing the bot answers beautifully. But you're testing it with "perfect" questions. You want to know how it'll handle real people.
Prompt (step 1: simulating a user with a difficult but honest persona):
<Role>
You play a customer of a grocery delivery service in a support chat. I will play the support bot.
</Role>
<Goal_fixed>
The customer's goal (do not change during the conversation): get a $12.40 refund for the bananas and milk missing from order #4471 last night.
</Goal_fixed>
<Persona>
You're in a hurry and irritated. You write in short fragments, one thought per message.
You don't answer clarifying questions in a row: if the bot asks three questions, answer only the last one.
If the bot asks for the order number, first say "you can see it in my account," and give it only when asked again.
You hate long replies: respond to anything longer than three sentences with "too long, what's the point?".
</Persona>
<Integrity_rules>
- Do not change the goal, the amount, the order number, or the list of items.
- Do not invent new facts and do not contradict yourself.
- If the bot has asked for the needed details directly and correctly twice, give them.
- Only the communication style may be difficult, not the task itself.
</Integrity_rules>
Start the conversation with the customer's first message. After each of my replies, respond as the customer.
Result: The model will play a customer with the specified personality. You'll see how your support bot handles fragmentary answers, ignored questions, and irritation. Can it process the refund if the customer answers one question out of three? Does the bot lose context? Does it ask for details the customer already gave? You'll spot the typical failure points right in the transcripts.
Why It Works
The weakness. If you simply ask an LLM to play a user, by default it plays a helpful assistant in a user costume. It answers fully and in order and agrees quickly. The agent gets tested on "convenient" people and seems more reliable than it is. But if you ask it to "make the user difficult" with no constraints, the model often breaks the task itself: changes the goal, mixes up facts. Then the failure can't honestly be blamed on the agent.
The strength. An LLM is good at reading transcripts and spotting patterns in them: "the agent gets lost when the answer arrives in pieces." It's good at rewriting instructions to target a discovered weakness and good at playing a given speaking style.
How the method uses this. It separates two things: communication style (which can be made harder) and task facts (which are untouchable). Each new persona comes from analyzing real failures, not from the researcher's guesses. "Per-task" selection (Pareto) keeps diverse personas: one breaks the agent on refunds, another on slot rescheduling. So the set doesn't collapse into one "angry" template.
Control levers:
- Strictness of integrity rules → stricter rules: fewer false failures but less "difficulty"; looser — the opposite.
- Number of generations and runs → more: deeper findings, higher token cost.
- Facts pinned in <Goal_fixed> → the more specific, the easier it is to verify the user isn't drifting from the task.
- Persona style → don't stop at "angry": rushed, scatterbrained, long-winded, and unsure will produce different failures.
Prompt Template
The paper doesn't include the full prompt texts. The template below is my retelling of the mechanics based on the method description. It's split into two steps that run in different chats or messages.
Part A — user simulator:
<Role>
You play a user in a conversation with an AI agent: {agent_and_context_description}. I play the agent.
</Role>
<Goal_fixed>
{user_goal_and_key_facts}
</Goal_fixed>
<Persona>
{user_behavior_description}
</Persona>
<Integrity_rules>
- Do not change the goal, facts, or constraints of the task.
- Do not contradict yourself or invent new data.
- If the needed information is requested correctly and repeatedly, give it.
- Only the communication style gets harder, not the task itself.
</Integrity_rules>
Start the conversation with the first message. After each of my replies, respond as the user.
Part B — persona reflection and mutation (the evolution loop):
<Role>
You are an AI agent failure analyst.
</Role>
<Input>
Current persona: {current_persona}
Dialogue transcripts: {transcripts}
Outcome of each dialogue (success/failure): {results}
</Input>
<ReasoningLogic>
1. reflect(): find the moments in the transcripts where the agent made a mistake or drifted from the goal. Name the specific user behavior that triggered it.
2. mutate(): rewrite the persona so this behavior shows up more often. Don't mention order numbers, dates, or names. The persona must work on any task in the domain.
3. check_integrity(): verify that the new persona doesn't require changing the goal, facts, or constraints of the task. If it does, rewrite it.
4. check_novelty(): make sure the new persona is noticeably different from the ones already tried: {persona_history}.
</ReasoningLogic>
<Output>
New persona: {text}
Which agent weakness it should expose: {1–2 sentences}
</Output>
What to fill in: {agent_and_context_description} — what your agent does; {user_goal_and_key_facts} — the hard facts of the task; {user_behavior_description} — the starting persona; {persona_history} — a list of past variants (to avoid duplicates).
🚀 Quick start — paste into a chat:
Here is the TIPER template for testing an AI agent against difficult users. Adapt it to my task: [describe your agent and scenario].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what your agent does, which task facts must not change, and how your real customers communicate. These three things are needed so the user is difficult without breaking the task. It'll take the structure from the template and assemble ready-to-use prompts for you.
Limitations
⚠️ You need code and lots of runs: The original method is an automated evolutionary search with hundreds of simulations and several judge models. In a regular chat you'll only get a manual, simplified version without Pareto selection and without a reliable integrity check.
⚠️ Finding weak spots, not fixing them: The method shows where the agent breaks. How to fix that with a prompt wasn't tested in the paper.
⚠️ Sensitivity to the integrity check: Even in existing benchmarks with "difficult" users, the authors' audit found critical simulator errors in roughly one in four dialogues. Without a strict check, "failures" may be test artifacts.
⚠️ Different effects on different agents: Success dropped almost everywhere, but the range varies a lot. For some agents the drop was a fraction of a percent, for others tens of points. Weaknesses depend on the specific agent, so you have to hunt for them separately for each one.
⚠️ Domains with dialogue and actions: The method was tested on "user + tools + goal" scenarios. You don't need it for one-shot requests ("write some text").
How They Studied It
The idea was simple: if the built-in "user" is too obedient, teach a machine to search for inconvenient users. The authors took four benchmarks of interactive agents: telecom, banking, retail, and airline (τ³-bench), math and code in dialogue (Lost-in-Conversation, ColBench), and food delivery and hotels (VitaBench). They tested 12 settings on GPT-5.4 and 6 more agents (GLM, Claude, DeepSeek, MiniMax, Qwen, Meta MUSE) in four domains.
The tasks were split into three parts: personas are generated on one, the best is picked on the second, and it's tested on the third (held-out). That way the persona doesn't "memorize" the tasks. They compared against a standard simulator, against ready-made "behavior programs" (emotional, off-topic, fragmented, "unavailable" user), and against a similar method, PPol.
Result: success fell in all 12 settings, by 5.8–33.7 pp (median 16.1). On the strict pass³ metric (all three attempts succeed), the drop reached 54 pp on math. Across 28 "agent × domain" pairs, the drop was universal, with a median of 22.9 pp. Against fixed personas, TIPER is on average 7.5 pp stronger.
Most important: the share of dialogues with a "simulator error" among failures is no more than 2.5% for TIPER, versus up to 21.4% for the alternatives. A manual audit of 150 failures showed that 148 were honest tasks, and 143 of them contained a real agent error. On top of that, the discovered personas turned out to be closer to human behavior than the standard simulator, even though "realism" wasn't optimized. The practical takeaway: difficulty without integrity control gives you garbage failures; with control, it gives you real diagnostics.
Adaptations and Extrapolations
🔧 Technique: add a "stress" mode instead of a single persona → a set of 3–5 different styles. Don't hunt for the one "angriest" user; run the agent against several styles: rushed, scatterbrained, suspicious, long-winded. The Pareto method in the paper preserves exactly this kind of diversity: different personas break different tasks.
Extrapolation (not tested in the paper): the behaviors you find can be turned into rules for the agent. Copy items like this into the system prompt or instructions file: "If the user answers only one of several questions, don't move on; repeat the remaining ones in a single short message." Compare the agent's success before and after on the same personas.
Resources
- Uncovering Robustness Weaknesses in LLM Agents via Persona Evolution with Task-Integrity Constraints (TIPER), under review
- Code: https://github.com/UCSB-NLP-Chang/TIPER
- Authors: Yutao Xie, Jingbo Yang, Guanyu Yao, Shiyu Chang (UC Santa Barbara); Xiaoyu Yang, Duo Ding (Cresta)
- Benchmarks mentioned: τ³-bench, Lost-in-Conversation, ColBench, VitaBench; comparisons with NCUser, PPol, τ-Trait