Apply to my task
arXiv:2507.02858 ★ 74 Jul 3, 2025

Steering through mistakes: describe the failures instead of the ideal and beat human-level results

Key result

The paradox: the more precisely you describe the ideal result to an LLM, the worse it delivers. A list of typical failures, on the other hand, works. The method gets you answers that beat humans on clarity, precision and usefulness, with no fine-tuning and no complex techniques. The trick: you don't change the goal, you change the angle. Instead of 'what it should look like,' you specify 'what must not happen.' An LLM follows concrete prohibitions better than vague wishes. Add a block of 3–5 rules based on typical failures, and the model stops guessing what you mean by 'good.'

Requirements Elicitation Follow-Up Question Generation

Key Aspects of the Research:

The research looks at how to use GPT-4o to generate high-quality follow-up questions during interviews. The authors found that if you simply ask the model to generate a question, the result is no better than a human's. But if you give the model a clear list of "typical interviewer mistakes" and ask it to generate a question that avoids those mistakes, the quality (clarity, relevance, informativeness) becomes significantly higher than a human's.

Key Result: Telling an LLM which mistakes to avoid is a more effective strategy for getting a quality result than simply asking it to "do a good job."


The Core Idea of the Method, Explained:

The essence of the method, which you could call "steering through constraints" or "mistake-guided generation," is a shift in focus when writing the prompt. Instead of trying to describe the ideal result, the user first defines what a bad, failed result looks like, and then turns those "failures" into clear rules and constraints for the LLM.

This approach works because it's much easier for an LLM to follow specific, measurable rules (for example, "the question must not contain technical jargon," "the answer must be shorter than 50 words") than to interpret abstract wishes ("make the answer good and clear").

The practical method for the user looks like this: 1. Define the task: What do you want from the LLM? (For example, "a reply to a negative customer review"). 2. Analyze the "failures": Think about what typical mistakes people (or LLMs) make on this task. (For example: "the reply sounds robotic," "blames the customer," "offers no solution," "too long"). 3. Turn the "failures" into positive rules: Rephrase each mistake as a clear instruction or criterion. * "Sounds robotic" → "Use an empathetic, human tone." * "Blames the customer" → "Show understanding and never shift the blame." * "Offers no solution" → "Always propose a concrete next step to solve the problem." 4. Structure the prompt: * Role (Persona): Start by assigning a role ("You are an experienced customer success manager"). * Context: Clearly provide all the necessary information (the text of the review). * Task: State the main goal. * Rules/Criteria: Add a separate block with your positive rules, which will steer the generation.

This method turns the user from a mere requester into the "architect" of the answer, building "guardrails" that keep the model from going off track.


Practical Applicability Analysis:

  • Direct applicability: A user can start using this method immediately. Just add a block titled QUALITY CRITERIA: or RULES TO FOLLOW: to your prompts and list constraints there based on the "typical mistakes" for your task. It works in any chat with no technical skills at all.

  • Conceptual value: The research gives the user a fundamental understanding: an LLM isn't a "thinker" but an "executor." It works better when given clear boundaries and constraints rather than abstract goals. This helps you get past the frustration of "why doesn't it understand me?" and move to a more engineering-minded approach to writing prompts. The key idea: control the process, don't just demand the result.

  • Adaptation potential: The method is universal. The user just needs to replace "interviewer mistakes" with mistakes relevant to their own task.

    • For writing an email: Mistakes: "informal tone," "no clear call to action," "grammar errors." Rules → "Keep a professional style," "End the email with one clear question or call to action," "Check the grammar carefully."
    • For building a content plan: Mistakes: "repetitive topics," "no tie-in to current events," "too complex for the audience." Rules → "Suggest 3 different content formats (article, video, post)," "Include 1-2 topics tied to current news," "All topics must be understandable to a beginner in the field."

A Practical Example:

You are an experienced customer success manager at the online electronics store "TechWorld." Your task is to respond to negative reviews while keeping the customer's loyalty and protecting the store's reputation.

**CONTEXT:**
Here is a review a customer left on our website:

**CUSTOMER REVIEW:**
"I ordered a new smartphone from you. The courier was three hours late and didn't even apologize! The phone itself seems to work, but the box was crushed like someone had been playing soccer with it. For that kind of money, the service is just awful! I'm never buying anything from you again."

**TASK:**
Write a polite, empathetic and professional reply to this review.

**REPLY QUALITY CRITERIA (follow them without exception):**
1.  **Show empathy:** Start with a sincere apology for the bad experience. Acknowledge our mistakes.
2.  **Don't be a robot:** Avoid boilerplate phrases like "Your opinion is very important to us." Write like a real, caring person.
3.  **Don't make excuses:** Don't explain why the courier might have been late. Just acknowledge that this kind of service is unacceptable.
4.  **Offer a concrete solution:** Don't stop at apologies. Offer the customer a small bonus (for example, a promo code for their next purchase) as compensation.
5.  **State the next step:** Clearly explain how the customer can get this bonus (for example, "we've already sent a promo code to your email...").

Why This Works:

This prompt is effective because it directly applies the research's "steering through constraints" method: * Role and Context set the right frame for the model. * The REPLY QUALITY CRITERIA block is the heart of the prompt. Instead of just saying "write a good reply," we listed the specific "failures" to avoid (being a robot, making excuses, offering no solution) and turned them into positive, actionable rules. * Rule #2 ("Don't be a robot") prevents soulless, canned brush-offs. * Rule #3 ("Don't make excuses") blocks the model's defensive reflex and makes it focus on the customer. * Rules #4 and #5 ("Offer a solution" and "State the step") ensure the reply is not only empathetic but constructive, which is the ultimate goal of good customer service.


Another Practical Example

You are a creative copywriter who comes up with ideas for short ad videos (up to 30 seconds) for social media.

**CONTEXT:**
Our product is a "Smart Notebook" that syncs handwritten notes to cloud storage through a smartphone app. The target audience is students and young professionals.

**TASK:**
Come up with 3 different concepts for a short ad video. Describe each concept in 2-3 sentences (scene, action, key message).

**RULES FOR THE CONCEPTS (follow them without exception):**
1.  **Focus on the problem:** Each video must open with a relatable problem (for example, you lost an important note, spilled coffee on your notes, can't find the lecture you need).
2.  **No jargon:** Don't use the words "sync," "cloud storage," "integration." Speak the user's language: "all your notes are on your phone," "save it forever."
3.  **Show, don't tell:** Describe the visuals instead of just listing features. Instead of "the notebook has a search function," describe a scene where the hero instantly finds a handwritten formula on their phone.
4.  **A clear call to action:** Each concept must end with a clear call (for example, "Learn more at the link in the description!", "Write smarter with our notebook!").
5.  **Variety:** All three concepts must differ in mood (for example, one funny, one about productivity, one about calm and order).

Explaining the Mechanism: Why This Example Works.

This prompt works on the same principle as the previous one, adapted for a creative task. It prevents the typical mistakes in creating ad concepts: * Rule #1 ("Focus on the problem") makes the model create relevant content that "hooks" the viewer from the first seconds, instead of a boring product description. * Rule #2 ("No jargon") keeps it understandable for a broad audience and prevents a "techie" video that would scare off non-technical viewers. * Rule #3 ("Show, don't tell") is a classic principle of good storytelling. It steers the LLM toward visually rich ideas that suit the video format, rather than dry text. * Rules #4 and #5 ensure practical value and variety, making the model's output immediately ready for discussion with the team.

So the prompt doesn't just ask "come up with an ad"; it creates a "creative corridor" within which the model can generate high-quality, relevant ideas while avoiding the most common failures.

Usefulness Score: 96

02

Main Evaluation Criteria

  • A. Relevance to prompting techniques: Very high. The research directly provides prompt templates (Fig. 1), describes specific techniques (persona, one-shot examples, positive phrasing) and explains why they work (section V.A).
  • B. Improving the quality of conversational answers: Very high. The main finding of the research: guided question generation (with a list of "mistakes to avoid") beats human-written questions in quality (clarity, relevance, informativeness).
  • C. Direct practical applicability: Maximum. All the proposed methods (role assignment, structuring, listing criteria) can be used by any user in any chat interface (ChatGPT, Claude, etc.) without a single line of code.
  • D. Conceptual value: High. Reveals key features of LLM behavior: sensitivity to formatting (capitalized roles), trouble with negations, improvement from step-by-step instructions and examples. Gives the user a powerful "mental model": instead of asking it to "do well," clearly define "what bad looks like" and forbid it.
  • E. New useful practice (clustering): The work falls into several key clusters at once:
    • #1 (Phrasing techniques): Explicitly demonstrates persona-based prompting and one-shot learning.
    • #2 (Behavioral patterns): Reveals sensitivity to formatting (capitalization) and problems with handling negations.
    • #3 (Structure optimization): Shows the effectiveness of clear role separators (INTERVIEWER/INTERVIEWEE) and lists of criteria.
    • #7 (Reliability and stability): The whole mistake-guided generation method is an extremely powerful technique for raising the reliability and quality of answers.
  • Practicality checklist (+15 points): Yes, the work provides ready-made constructions, shows how to structure requests, reveals non-obvious LLM behavior and offers ways to improve the accuracy and quality of answers.
03

Numerical Usefulness Score

Arguments for a score of 96: This work is a real gem for prompt-engineering practitioners. It doesn't just offer an abstract idea; it proves its effectiveness in a controlled experiment and, most valuable of all, reveals concrete mechanics for implementing it in a prompt. The key takeaway, "steering through constraints" (or "mistake-guided generation"), is a universal and extremely powerful principle. The user gets not just a tip but a whole methodology: define the failure criteria for your task, phrase them as positive rules and put them in the prompt. This immediately raises the quality and consistency of results.

Counterarguments (why the score could be lower/higher): * Why not 100? The main domain of the research (requirements gathering in IT) is fairly specialized. An everyday user who wants to write a social media post or plan a trip will have to put in a bit of mental effort to adapt "interviewer mistakes" to their task. The methodology is universal, but the examples aren't. * Why not below 85? Despite the narrow domain, the findings and techniques described in section V.A ("Prompts Design") are completely universal. Sensitivity to capitalization, positive phrasing of instructions, using one-shot examples: this is fundamental knowledge that will improve any complex prompt, regardless of the task. The practical value of these insights outweighs the specificity of the main experiment.


Research digest

Key result

The paradox: the more precisely you describe the ideal result to an LLM, the worse it delivers. A list of typical failures, on the other hand, works. The method gets you answers that beat humans on clarity, precision and usefulness, with no fine-tuning and no complex techniques. The trick: you don't change the goal, you change the angle. Instead of 'what it should look like,' you specify 'what must not happen.' An LLM follows concrete prohibitions better than vague wishes. Add a block of 3–5 rules based on typical failures, and the model stops guessing what you mean by 'good.'

Principle

Don't write 'the reply should be warm and professional.' Write 'don't sound like a robot,' 'don't make excuses,' 'propose a concrete next step.' An LLM is an executor, not a thinker. It needs boundaries, not a vision of the ideal. A 'Quality Criteria' block in a prompt is a fence the model doesn't cross. Want a good reply to a negative review? Don't explain what good is. List what's definitely bad, and turn each failure into a clear rule.

Why it works

An LLM doesn't 'understand' what a good answer is. It guesses based on its training data. An abstract request to 'do a good job' gets you the bland average: the model grabs the most frequent pattern from everything it has seen. Concrete prohibitions ('no jargon,' 'no excuses,' 'no longer than 50 words') create measurable boundaries. A model is far better at avoiding specific mistakes than at groping toward an abstract ideal. It's like giving a chef not a recipe but a list of 'things you must never add': the result is more predictable. And yes, in the experiment this approach actually beat human interviewers on three key metrics.

When to apply

Any task where you know what failure looks like. Replies to negative reviews: easy to list what's irritating about bad replies. Email campaigns: it's clear what makes an email soulless. Ad copy: you can see where content misses. Follow-up questions in negotiations: it's obvious which questions shut a conversation down instead of opening it up. It's especially powerful when the requirements are hard to describe positively but the failures are obvious. NOT for tasks where you have no experience yourself: if you don't know what a bad result looks like, you can't make a list of mistakes.

Mini recipe

1. Define the task: What do you want from the LLM? One line: 'a reply to a negative review,' 'a follow-up interview question,' 'an ad video concept.' 2. Recall 3–5 typical failures: What does a bad result look like for this specific task? 'Robotic tone,' 'excuses instead of a solution,' 'technical jargon,' 'no next step for the user.' 3. Turn each failure into a rule: Every failure → a clear instruction. 'Robotic tone' → 'talk like a real person, avoid boilerplate phrases.' 'No solution' → 'always propose a concrete next step.' 4. Add a block to the prompt: Call it QUALITY CRITERIA: or RULES and list the instructions as a separate block after the main task. 5. Assign a role and give context: Before the task, add a role: experienced customer success manager and include all the needed information: context, review text, task parameters.

Examples

[BAD]: `Write a polite reply to a negative customer review about a late courier` [GOOD]: `You are a customer success manager at an online store. Reply to the customer's review: [REVIEW TEXT] QUALITY CRITERIA: 1. Start with a sincere apology, without boilerplate like 'your opinion matters to us' 2. Don't explain why the courier was late; just acknowledge that this kind of service is unacceptable 3. Don't stop at words; offer a concrete bonus as compensation 4. End with one clear next step for the customer on how to get that bonus` The difference isn't politeness. The difference is that the second prompt closes every hole through which the model usually slides into a template.

In plain words

The gist: by default, AI models spit out either stiff corporate-speak or useless politeness that only annoys the customer. To make a model actually help, the researchers used a steering-through-constraints method. Instead of asking the AI to "be nice," you give it hard boundaries: exactly what not to say and which direction to push its thinking. This changes the very logic of generation: the model stops churning out stock phrases and starts working like a seasoned fixer who sees right through the problem.

It's like hiring an intern and, instead of saying "get to work," handing them a list of specific screwups you'll fire them for. "Don't apologize five times in a row," "don't pass the blame to the shipping department," "don't hang up until you've offered a solution." The intern instantly understands that empty chatter won't fly and has to switch their brain on to deliver a result that fits your strict rules.

The foundation is a set of response quality criteria that target an LLM's weakest spots. The "don't be a robot" rule cuts out soulless scripts, and the ban on excuses makes the model focus on the customer instead of defending the company. Most important of all is the mandatory solution offer and a concrete next step. This turns an ordinary chat into a tool where every word works toward closing the ticket rather than looking busy.

Although the study was done in customer service, this principle is universal. You can and should plug it into any prompt, from writing code to creating content. If you just ask "write an article," you'll get garbage. If you spell out negative constraints (no filler phrases, no padding, no "in conclusion"-style wrap-ups), text quality shoots up. It's a new era of prompt engineering where prohibitions matter more than wishes.

Bottom line: stop counting on the model's "intelligence"; there isn't any, just statistics. To keep AI from messing up, you need to box it in with strict rules and criteria that stop it from sliding into the typical AI mush. 6 specific rules in a prompt work better than a page of context. Whoever learns to train models like this gets predictable results, while everyone else keeps complaining that AI writes like a bot.

AI lab assistant

Apply this study to your task

Describe what you're working on. AI takes the method and the measured results from this paper and adapts them to your case, or writes a ready prompt built on them.

Based on Steering through mistakes: describe the failures instead of the ideal and beat human-level results arXiv:2507.02858

Sign in to apply this study New accounts get 50 free credits.

Get the week’s best papers by email

One email on Sundays with the highest-rated new techniques. Unsubscribe with one click.