SOP instead of examples: how to set up a support agent so it stops handing out compensation on the first message

Task: You're setting up an AI support assistant for a fast grocery delivery service along the lines of Instacart or Gopuff. Customers write: “You brought the wrong stuff”, “You were an hour and a half late”, “The eggs are smashed”. Right now the bot answers the very first message with “We apologize, we've credited you $5.” The customer gets annoyed, and in half the cases the problem was something else, like a wrong address.

Prompt (system):

<Role>
You are a support specialist at the grocery delivery service QuickMart.
You speak calmly, like a human, without corporate-speak or canned apologies.
</Role>

<SOP>
Work strictly step by step. Don't skip ahead.

Step 1. Acknowledge the problem in one sentence. Don't make excuses.
Step 2. Gather the facts. Ask the customer 1–2 specific questions at a time:
  what exactly happened, the order number, what time it arrived,
  whether there's a photo, what the customer wants in the end.
  Don't move to step 3 until you understand what happened.
Step 3. Check against <Policy>. Decide which options are allowed.
Step 4. Offer a resolution and explain why this one.
  Mention compensation only at this step.
</SOP>

<Policy>
- Broken or spoiled items: refund of the item price after a photo.
- Delay over 60 minutes: promo code worth up to $5.
- Wrong order: redelivery or refund.
- Compensation beyond this — only through a human agent.
</Policy>

Result: On the customer's first message, the bot acknowledges the problem and asks clarifying questions: what exactly is wrong, the order number, whether there's a photo. It doesn't offer money yet. After the customer replies, it names a resolution from the policy and briefly explains the choice. The conversation gets a couple of messages longer, but the offers are more accurate and need to be walked back less often.

The model's weakness. LLM agents grab the ready-made solution fast. Money and apologies are the shortest path to closing the conversation, so the model heads there without asking questions. The authors call this premature compensation. Give it a generic “be polite and help the customer” prompt, and that's exactly what it'll do.

The strength. The model is good at following an explicit sequence of steps. When the order of actions is spelled out, it doesn't have to guess what to do on this turn. An SOP turns “can the agent ask good questions” from a personality trait into an instruction item. Four models were tested, and for all of them the SOP improved conversation quality on all three fairness dimensions (tone, process, outcome). The share of customers who abandoned the conversation dropped several-fold for some models.

Why cases plus SOP is worse. When a general procedure and concrete examples sit side by side, the model gets two sources of truth. The authors suggest it has a harder time reconciling them, and its decisions become less stable. That's the authors' hypothesis, but the result itself is measured: the “cases + SOP” combo is consistently worse than the SOP alone, and often close to a prompt with no guidance at all.

Control knobs: - Number of steps in the SOP. For simple requests cut it to three; for complex ones add a “check whether this is a repeat complaint” step. - The “not before step 4” rule. This is direct protection against premature compensation. You can tighten it: “don't name an amount until the customer has answered at least two questions”. - Number of questions at a time. Too many questions wear the customer out, so cap it at one or two. - Policy separate from the SOP. Keep the “what can be offered” rules in their own block, and the order of actions in its own. - Don't add a “reread and improve” stage. Self-refine more often made the answer worse: it got less to the point and could drag the conversation out.

The SOP text from the paper isn't given in the available portion. Below is a skeleton built on the method's logic: a step-by-step procedure plus a separate policy. Plug in your own specifics.

<Role>
You are a support specialist at {company}.
Tone: {tone}. No canned apologies or corporate-speak.
</Role>

<SOP>
Work strictly step by step. Don't skip ahead.

Step 1. Acknowledge the problem in one sentence. Don't make excuses.
Step 2. Gather the facts. Ask 1–2 questions at a time:
  {what you need to know: what happened, order number, when, photo, what the customer expects}.
  Don't move to step 3 until you understand the gist.
Step 3. Check against <Policy>. Identify the allowed options.
Step 4. Offer a resolution and briefly explain why.
  Mention compensation and money only at this step.
</SOP>

<Policy>
{company rules: what can be offered and when, limits, what to escalate to a human}
</Policy>

What to fill in. {company} and {tone} — your company and communication style. {what you need to know} — the list of facts without which you yourself wouldn't make a decision. In <Policy>, write the real rules: amounts, deadlines, exceptions. Don't add “here's how we handled it before” examples to the prompt.

🚀 Quick start — paste into the chat:

Here's a system prompt template for a support agent with a step-by-step procedure (SOP).
Adapt it to my task: [describe your business and typical complaints].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about typical complaints, which facts are needed before a decision, what compensation is allowed and where the bot's authority ends. It needs this to fill in step 2 (which questions to ask) and the policy block (what can be offered). That's what determines whether the bot will ask questions before deciding.

⚠️ Simulation, not live customers: The customer was played by an LLM, and the conversations were graded by an LLM judge too. The authors validated the judge against humans and across several models, but real customers can be angrier and less predictable.

⚠️ The SOP text isn't shown: The available portion doesn't include the actual instructions given to the agents. You can reproduce the principle (order of steps + separate policy), but not the exact wording.

⚠️ One task type: Complaint handling was tested across five industries (airlines, hotels, restaurants, retail, ride-hailing and delivery). Transfer to technical support requests or sales wasn't tested.

⚠️ Reasoning before answering depends on the model: Chain-of-Thought and planning helped strong models, but not all equally. Self-refine hurt in this task, but that doesn't mean it hurts everywhere.

⚠️ Fine-tuning isn't available to the reader: The strong result of the small model came from SFT and DPO (fine-tuning on collected data). That's not applicable for a practitioner, so we only take the prompt-related conclusions from the paper.

⚠️ Truncated analysis: The conclusion about how the agent matches compensation to complaint severity isn't fully developed in the available portion. Don't lean on it.

• Paper title: Can LLM Agents Resolve Service Failures? Simulating Multi-Turn Complaint Handling under Information Asymmetry
• The SERVICEBENCH benchmark: 2,500 complaint scenarios across five industries. Built on real customer conversations with brand support on X (the Customer Support on Twitter dataset, Axelbrooke, 2017).
• The APCS simulation method (Asymmetric Progressive Complaint Simulation): the customer reveals details gradually, and the agent sees only part of the information.
• Evaluation based on justice theory (Smith et al., 1999): tone, process, outcome.
• Authors: Wailing Ng, Mengze Hong, Chen Jason Zhang, Shiqi Fan. The Hong Kong Polytechnic University.