Key Aspects of the Study:
The researchers examined how well large language models (LLMs) can give useful feedback for improving writing (using short stories as the test case). To do this, they deliberately "broke" good stories by introducing errors (for example, swapping sentences or deleting them), then asked various LLMs to review them.
Key Result: LLMs are great at giving specific, grammatically correct suggestions, but they often fail to identify the single biggest problem in a text and hand out undeserved praise when the text has serious flaws.
The Core of the Method, Explained:
The method that follows from the study is not to simply ask the LLM to "evaluate the text" or "find mistakes", but to make the model analyze against predefined criteria. A direct, open-ended request like "Find the problem in this text" turned out to be the least effective. In that case the model latches onto the first nitpick it sees (a typo, a clunky phrase) and ignores more serious logical or structural problems.
The most effective approach the researchers found is decomposing the analysis task. Instead of one general question, you give the model a clear structure: a list of aspects to check.
The technique for users: 1. Define the goal: What do you want to improve in the text? Clarity, persuasiveness, tone, structure? 2. Write a prompt with a role and a task: "You are an experienced editor. Your task is to give feedback on the text below". 3. Provide a list of analysis criteria: Instead of "check the text", write: "Analyze the text against the following criteria as a bulleted list:". * Criterion 1 (for example, "Clarity and readability") * Criterion 2 (for example, "Structure and logic") * Criterion 3 (for example, "Tone and style") * Criterion 4 (for example, "Presence of a call to action") 4. Give a clear output-format instruction: "Give your feedback as a bulleted list".
This approach forces the model to "check" the text point by point, which sharply raises the odds of catching the main problem and makes the feedback more complete and useful.
Practical Applicability:
Direct applicability: A user can start using structured prompts right away to get feedback on any of their texts: emails, blog posts, resumes, reports. Just copy the prompt structure from the example below and adapt the criteria to your task. No technical skills required.
Conceptual value: The study gives users a key conceptual idea: an LLM is not a thoughtful expert but a "lazy" assistant that needs clearly defined subtasks. Without a clear structure it will do a superficial job. Understanding this helps you write more effective prompts not only for text analysis but for any complex task, by breaking it into components.
Adaptation potential: The method adapts easily to any field. Instead of writing categories ("coherence", "style"), you can plug in any others.
- For reviewing a business plan: "Strategy", "Financial model", "Risk analysis", "Marketing".
- For code review: "Readability", "Efficiency", "Error handling", "Security".
- For trip planning: "Logistics", "Budget", "Sights", "Downtime". The adaptation mechanism is a simple swap of the items in the bulleted list of criteria.
A Practical Example:
Say you need to improve the copy of a promotional email for the launch of a new online course.
You are an experienced email marketer. Your task is to give constructive feedback on the promotional email below to make it more effective.
Analyze the text and give your recommendations as a bulleted list. Evaluate the email against the following criteria:
- **Subject line:** How catchy is it, and does it make people want to open the email?
- **Clarity of the offer:** Is it clear from the first lines what exactly is being offered and what the benefit is?
- **Structure and readability:** Is the text easy to read? Does it have logic, subheadings, lists?
- **Call to Action:** Is it clear, visible and motivating?
- **Tone of Voice:** Does the tone fit the target audience (first-time entrepreneurs)?
If a point is fine, say so. If there are problems, suggest specific improvements.
**Here is the email to analyze:**
---
**Subject:** New information
Hi,
We've worked hard and are finally ready to present our new product. It's a course that will help you figure out business. It has lots of useful lessons from experts who will explain how everything works. You'll learn about marketing, sales, finance and much more.
Our course is the result of many years of experience. We're sure you'll find it useful. Many people have already tried our other products and were happy with them.
To learn more, visit the website.
Thanks,
The LaunchPad Team
---Why It Works:
This prompt is effective because it doesn't just ask to "improve the text"; it uses several mechanics described in the study:
- Task decomposition: Instead of one general "evaluate the email" assignment, we give the model five specific subtasks (evaluate the subject line, clarity, structure, CTA, tone). That forces the LLM to focus on each aspect in turn.
- Structured analysis: Listing the criteria (
- **Subject line:**,- **Clarity of the offer:**and so on) works as a "checklist" for the model. It can't skip a point, because the prompt requires going through all of them. This prevents the situation where the model finds one small error (say, "the word 'lots' is repeated") and stops there, missing the main problem (say, a terrible subject line and a weak call to action). - Assigning a role: The phrase "You are an experienced email marketer" activates the model's knowledge and patterns tied to that profession, which raises the quality of the recommendations.
Another Practical Example
Say a user wants feedback on their vacation plan.
You are an experienced trip planner. Your task is to analyze my 7-day trip plan for Italy and give practical advice on how to improve it.
Analyze the plan and give feedback as a bulleted list against the following criteria:
- **Logistics and realism:** How realistic is the travel between cities? Is too much packed into one day? Is time for travel and check-in accounted for?
- **Budget:** Does the plan look balanced in terms of costs? Are there obvious places to save money, or, conversely, places where cutting costs is a bad idea?
- **Completeness and variety:** How varied is the plan? Does it include not just the famous sights but also downtime, local food and authentic experiences?
- **Missed opportunities:** Is there anything interesting near the planned stops that I've missed and that would be worth adding?
**Here's my trip plan:**
---
**Day 1:** Arrive in Rome in the morning. Check into the hotel. Colosseum, Roman Forum. Dinner in the evening.
**Day 2:** Vatican (St. Peter's Basilica, museums).
**Day 3:** Morning train to Florence (2 hours). Check in. Uffizi Gallery.
**Day 4:** Morning departure for Venice (2.5 hours). Check in. Stroll around St. Mark's Square, gondola ride.
**Day 5:** Morning departure for Milan (3 hours). Check in. Visit the Duomo.
**Day 6:** Shopping in Milan.
**Day 7:** Fly home from Milan.
---Why This Example Works, Explained.
This prompt works on the same principles as the previous one, just in a different domain:
- Preventing a shallow answer: Without structure, the model might just say: "Good plan, but very packed". That's useless advice.
- Forced deep analysis: The criteria structure makes the model think like a real travel agent:
- "Logistics and realism" will make it add up the time spent on travel, hotel check-ins and lines, after which it will most likely point out that the plan is unrealistic and will turn into a "race".
- "Completeness and variety" will make it notice that the plan has only the "must-see" tourist spots and no time to soak up the atmosphere of the cities.
- "Missed opportunities" will nudge the model to suggest specific alternatives, for example, stopping in Bologna on the way from Florence to Venice, or visiting lesser-known but interesting places.
In short, decomposing the task into specific criteria turns the LLM from a generator of generic phrases into a useful assistant that spots the main problems and proposes constructive solutions.
Usefulness Score: 92
Main Evaluation Criteria
- A. Relevance to prompting techniques: Yes, the study directly compares the effectiveness of 4 different prompts (from a simple "find the problem" to a structured list) and 2 approaches (zero-shot vs two-shot).
- B. Improving the quality of conversational answers: Yes, the goal of the study is to find approaches that produce higher-quality, more accurate and more useful feedback from LLMs.
- C. Direct practical applicability: Yes, the findings can be applied immediately in any chatbot with no code or special tools. Users can directly use the proposed prompt structures.
- D. Conceptual value: Very high. The study exposes a fundamental weakness of LLMs: they're good at finding local, minor flaws but often miss the main, most significant problem. That's key knowledge for any user.
- E. New useful practice: The work falls into several clusters at once:
- Cluster 1 (Phrasing techniques): A direct comparison of different instructions and few-shot.
- Cluster 2 (Behavioral patterns): Identifies the patterns of "missing the main problem" and "praising inappropriately".
- Cluster 3 (Structure optimization): Demonstrates the effectiveness of structured prompts with categories (bulleted list).
- Cluster 7 (Reliability and stability): Proposes methods for getting more relevant and useful feedback.
- Practicality checklist: Yes, the study offers ready-made constructions, shows how to structure requests and reveals non-obvious features of LLM behavior.
Numeric Usefulness Score
Arguments for a score of 92: The study has huge practical value for anyone who uses LLMs to evaluate or improve writing. It doesn't just put forward a hypothesis; it clearly demonstrates that the way you phrase a request radically affects the quality of the result.
The key finding, "a prompt that lists categories for analysis (BL full) works significantly better than a direct order to find the problem (SpotProb)", is a ready-to-use instruction that will immediately improve LLM answers in reviewing, editing and analysis tasks. On top of that, the conceptual insight that LLMs "latch onto" minor flaws while missing the point helps users understand why the model sometimes gives shallow or irrelevant answers and how to fight that with more structured prompts.
Counterarguments (why the score could be lower): * Narrow domain: The study focuses on short fiction with artificially introduced errors. Practical effectiveness in other, more complex areas (for example, analyzing business documents, technical reports or code) hasn't been proven, although the principles look universal. * Limited few-shot findings: The study showed that two-shot examples don't always help and sometimes even hurt, because the model starts copying the style of the example answers. This finding may not be fully universal and may depend on the quality of the examples themselves.
Counterarguments (why the score could be higher):
* Fundamental findings: The exposed problem of "seeing the trees but missing the forest" (i.e., finding small errors but missing the main one) is one of the fundamental traits of the current generation of LLMs. Understanding this principle is valuable for any task, not just editing. That knowledge alone is worth a lot.
* Easy to adopt: The best method proposed (BL full) is extremely easy for any user in any field to adapt, which makes the study as useful as possible for a broad audience.